Added a comment: progress?

This commit is contained in:
anarcat 2018-11-27 06:47:26 +00:00 committed by admin
parent bb9580ba66
commit 2c1a0af295

View file

@ -0,0 +1,65 @@
[[!comment format=mdwn
username="anarcat"
avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7"
subject="progress?"
date="2018-11-27T06:47:26Z"
content="""
How's that remote going, RonnyPfannschmidt? :) I can't tell from the [homepage](https://github.com/RonnyPfannschmidt/git-annex-borg/) but from the source code, it looks like initremote is supported so far, but not much else...
From what I remember, borg supports storing arbitrary blobs with the `borg debug-put-obj` function, and retrieve one with `borg debug-get-obj`. Here's an example of how this could work:
[1145]anarcat@angela:test$ sha256sum /etc/motd
a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 /etc/motd
[1146]anarcat@angela:test$ borg init -e none repo
[1147]anarcat@angela:test$ borg debug-put-obj repo /etc/motd
object a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 put.
[1148]anarcat@angela:test$ borg debug-get-obj repo a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 tmp
object a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 fetched.
[1149]anarcat@angela:test$ sha256sum tmp
a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 tmp
This assumes the underlying blob ID in borg is a SHA256 hash, but that
seems like a fair assumption to make. Naturally, this could cause
problems with git-annex, which supports multiple hashing algorithms
thanks to the multiple [[backends]] support. But maybe this can just
work this out by refusing to store non-matchin backends.
That is, if borg actually worked that way. Unfortunately, while the
above actually works, the resulting repository is not quite right:
$ borg debug dump-repo-objs .
Dumping 000000_0000000000000000000000000000000000000000000000000000000000000000.obj
Data integrity error: Chunk a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921: Invalid encryption envelope
So borg does not like the repository at all... I'm not sure why, but
it sure looks like borg \"objects\" are not as transparent as I
hoped and that this low-level interface will not be suitable for
git-annex.
The higher level interface is \"archives\", which have (more or less) a
CRUD interface (without the U, really) through the
\"create/list/extract/prune\" interface. It's far from what we need:
items are deplicated across archives so it means it is impossible to
reliably delete a key unless we walk (and modify!) the entire archive list, which is
slow and impractical. But it *could* definitely be used to add keys to
a repository, using:
$ time borg create --stdin-name SHA256-a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 .::'{utcnow}' - < /etc/motd
1.30user 0.10system 0:01.62elapsed 86%CPU (0avgtext+0avgdata 81464maxresident)k
72inputs+1496outputs (0major+31135minor)pagefaults 0swaps
As you can see, however, that is *slow* (although arguably not slower
than `debug-put-obj` which is surprising).
But even worse, that blob is now hidden behind that archive - you'd
need to list all archives (which is also expensive) to find it.
So I hit a dead end so I'm curious to hear how you were planning to
implement this, Ronny. :) Presumably there should be a way to generate
an object compatible with `debug-put-obj`, but that interface seems
very brittle and has all sorts of warnings all around it... And on the
other hand, the archive interface is clunky and slow... I wish there
was a better way, and suspect it might be worth talking with upstream
(which I'm not anymore) to see if there's a better way to work this
problem. -- [[anarcat]]
"""]]