diff --git a/doc/todo/borg_special_remote/comment_2_cec11f364d0c1e62d05c4205b3bf2a2d._comment b/doc/todo/borg_special_remote/comment_2_cec11f364d0c1e62d05c4205b3bf2a2d._comment new file mode 100644 index 0000000000..890540f0b5 --- /dev/null +++ b/doc/todo/borg_special_remote/comment_2_cec11f364d0c1e62d05c4205b3bf2a2d._comment @@ -0,0 +1,65 @@ +[[!comment format=mdwn + username="anarcat" + avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7" + subject="progress?" + date="2018-11-27T06:47:26Z" + content=""" +How's that remote going, RonnyPfannschmidt? :) I can't tell from the [homepage](https://github.com/RonnyPfannschmidt/git-annex-borg/) but from the source code, it looks like initremote is supported so far, but not much else... + +From what I remember, borg supports storing arbitrary blobs with the `borg debug-put-obj` function, and retrieve one with `borg debug-get-obj`. Here's an example of how this could work: + + [1145]anarcat@angela:test$ sha256sum /etc/motd + a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 /etc/motd + [1146]anarcat@angela:test$ borg init -e none repo + [1147]anarcat@angela:test$ borg debug-put-obj repo /etc/motd + object a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 put. + [1148]anarcat@angela:test$ borg debug-get-obj repo a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 tmp + object a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 fetched. + [1149]anarcat@angela:test$ sha256sum tmp + a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 tmp + +This assumes the underlying blob ID in borg is a SHA256 hash, but that +seems like a fair assumption to make. Naturally, this could cause +problems with git-annex, which supports multiple hashing algorithms +thanks to the multiple [[backends]] support. But maybe this can just +work this out by refusing to store non-matchin backends. + +That is, if borg actually worked that way. Unfortunately, while the +above actually works, the resulting repository is not quite right: + + $ borg debug dump-repo-objs . + Dumping 000000_0000000000000000000000000000000000000000000000000000000000000000.obj + Data integrity error: Chunk a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921: Invalid encryption envelope + +So borg does not like the repository at all... I'm not sure why, but +it sure looks like borg \"objects\" are not as transparent as I +hoped and that this low-level interface will not be suitable for +git-annex. + +The higher level interface is \"archives\", which have (more or less) a +CRUD interface (without the U, really) through the +\"create/list/extract/prune\" interface. It's far from what we need: +items are deplicated across archives so it means it is impossible to +reliably delete a key unless we walk (and modify!) the entire archive list, which is +slow and impractical. But it *could* definitely be used to add keys to +a repository, using: + + $ time borg create --stdin-name SHA256-a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 .::'{utcnow}' - < /etc/motd + 1.30user 0.10system 0:01.62elapsed 86%CPU (0avgtext+0avgdata 81464maxresident)k + 72inputs+1496outputs (0major+31135minor)pagefaults 0swaps + +As you can see, however, that is *slow* (although arguably not slower +than `debug-put-obj` which is surprising). + +But even worse, that blob is now hidden behind that archive - you'd +need to list all archives (which is also expensive) to find it. + +So I hit a dead end so I'm curious to hear how you were planning to +implement this, Ronny. :) Presumably there should be a way to generate +an object compatible with `debug-put-obj`, but that interface seems +very brittle and has all sorts of warnings all around it... And on the +other hand, the archive interface is clunky and slow... I wish there +was a better way, and suspect it might be worth talking with upstream +(which I'm not anymore) to see if there's a better way to work this +problem. -- [[anarcat]] +"""]]