From 4d24b712b05edbdc6e1fd734c713f4e33234df05 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Fri, 2 Nov 2018 19:58:06 +0000 Subject: [PATCH 01/10] Added a comment --- ..._17_c39c3b0978777987dcd8c86906aa6532._comment | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/bugs/annex_get_-J_16_via_ssh_stalls_/comment_17_c39c3b0978777987dcd8c86906aa6532._comment diff --git a/doc/bugs/annex_get_-J_16_via_ssh_stalls_/comment_17_c39c3b0978777987dcd8c86906aa6532._comment b/doc/bugs/annex_get_-J_16_via_ssh_stalls_/comment_17_c39c3b0978777987dcd8c86906aa6532._comment new file mode 100644 index 0000000000..360a9ab395 --- /dev/null +++ b/doc/bugs/annex_get_-J_16_via_ssh_stalls_/comment_17_c39c3b0978777987dcd8c86906aa6532._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 17" + date="2018-11-02T19:58:06Z" + content=""" +> If it doesn't reply with DATA, then it's something to do with that one particular key. If it does, the problem must involve concurrency or the ssh connection somehow. + +isn't the fact that it works without -j parallelization hints that it is about concurrency and not just the key? +But for the sake of completeness, I did that \"manual\" `GET 0 sub-qa/s...` snippet and indeed was bombarded with the data. + +> Dealt with it in 0b053b96119c1a2d6c2d03ba7f98f032e89b8b66 which would need to be deployed to the server side to test. + +ok, will build/test, thanks! + +"""]] From c6cb372a9d00b557e800a306f55b2e115d4e12a0 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Fri, 2 Nov 2018 21:51:16 +0000 Subject: [PATCH 02/10] Added a comment: tests fail - can't build to test the change --- ...omment_18_bffd59a35ae162f4141b6de7efcc7422._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/bugs/annex_get_-J_16_via_ssh_stalls_/comment_18_bffd59a35ae162f4141b6de7efcc7422._comment diff --git a/doc/bugs/annex_get_-J_16_via_ssh_stalls_/comment_18_bffd59a35ae162f4141b6de7efcc7422._comment b/doc/bugs/annex_get_-J_16_via_ssh_stalls_/comment_18_bffd59a35ae162f4141b6de7efcc7422._comment new file mode 100644 index 0000000000..c79f990f1a --- /dev/null +++ b/doc/bugs/annex_get_-J_16_via_ssh_stalls_/comment_18_bffd59a35ae162f4141b6de7efcc7422._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="tests fail - can't build to test the change" + date="2018-11-02T21:51:16Z" + content=""" +the 7.20181031-13-g4d24b712b (7.20181031+git7-g39bff661b did fine) is failing some tests, get/see build logs at http://neuro.debian.net/_files/_buildlogs/git-annex/7.20181031+git13-g4d24b712b/ + +isn't there some CI running tests for git-annex or my failures are standalone build specific? +"""]] From 0dc9e68cdb5119cc5606b61092fa0acd4b940ecb Mon Sep 17 00:00:00 2001 From: andrew Date: Fri, 2 Nov 2018 22:03:15 +0000 Subject: [PATCH 03/10] Added a comment --- ...comment_3_49b00584ef4388e130cce63f4f33572f._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/forum/assistant_to_have_synced_folders/comment_3_49b00584ef4388e130cce63f4f33572f._comment diff --git a/doc/forum/assistant_to_have_synced_folders/comment_3_49b00584ef4388e130cce63f4f33572f._comment b/doc/forum/assistant_to_have_synced_folders/comment_3_49b00584ef4388e130cce63f4f33572f._comment new file mode 100644 index 0000000000..0dbd65d89d --- /dev/null +++ b/doc/forum/assistant_to_have_synced_folders/comment_3_49b00584ef4388e130cce63f4f33572f._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="andrew" + avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435" + subject="comment 3" + date="2018-11-02T22:03:15Z" + content=""" +Aaah OK. You don't necessarily need to add the repositories from assistant. If you create them from the command-line you need to manually add the full path to any repos you want the assistant to watch to a file located at `~/.config/git-annex/autostart`. + +So on your server you want that `~/.config/git-annex/autostart` file to contain the full path to the repo that is located on the server and for the `~/.config/git-annex/autostart` file on your laptop you want it to contain the full path to your laptop repo. You also need to start the assistant on both your laptop and your server by running `git-annex-assistant --autostart` on your laptop and on your server. You'll also want to verify that this autostart command is happening automatically at boot (which `git-annex` might have done for you). +"""]] From 55fca72138784c053db96735c5d9fb793d0b0cd4 Mon Sep 17 00:00:00 2001 From: admin Date: Sat, 3 Nov 2018 05:24:04 +0000 Subject: [PATCH 04/10] removed --- .../comment_4_3df3ee5c70d4d1ec6963f125c479a9c9._comment | 8 -------- 1 file changed, 8 deletions(-) delete mode 100644 doc/install/comment_4_3df3ee5c70d4d1ec6963f125c479a9c9._comment diff --git a/doc/install/comment_4_3df3ee5c70d4d1ec6963f125c479a9c9._comment b/doc/install/comment_4_3df3ee5c70d4d1ec6963f125c479a9c9._comment deleted file mode 100644 index 06b8c38f2f..0000000000 --- a/doc/install/comment_4_3df3ee5c70d4d1ec6963f125c479a9c9._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="spwhitton" - avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb" - subject="armel+armhf available in stretch-backports" - date="2018-03-04T19:03:19Z" - content=""" -@Alan: stretch-backports now includes git-annex on armel and armhf. -"""]] From 4241de01c84267ae00df605096e07864d336e4ae Mon Sep 17 00:00:00 2001 From: anarcat Date: Sat, 3 Nov 2018 19:30:29 +0000 Subject: [PATCH 05/10] p2pstdio doesn't exist in stretch and buster doesn't degrade gracefully --- ...lity_issue_between_buster_and_stretch.mdwn | 172 ++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 doc/bugs/interoperability_issue_between_buster_and_stretch.mdwn diff --git a/doc/bugs/interoperability_issue_between_buster_and_stretch.mdwn b/doc/bugs/interoperability_issue_between_buster_and_stretch.mdwn new file mode 100644 index 0000000000..1b6ba9ba06 --- /dev/null +++ b/doc/bugs/interoperability_issue_between_buster_and_stretch.mdwn @@ -0,0 +1,172 @@ +### Please describe the problem. + +If a git-annex client from Debian buster (currently 6.20180913-1) tries to talk with a server running Debian stretch (currently 6.20170101-1+deb9u2), it will fail mysteriously: + + copy bar (fd:18: hClose: resource vanished (Broken pipe)) failed + +### What steps will reproduce the problem? + +On the stretch machine, setup a test git-annex repository. + +On the buster machine, setup a test repository, add the stretch +machine as a remote, then add a file in git-annex, and +git-annex-sync-content to the stretch machine. + +### What version of git-annex are you using? On what operating system? + +Debian buster and stretch standard Debian packages. + +### Please provide any additional information below. + +On the buster machine: + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + +(test) [1265]anarcat@curie:test130$ git init test +Dépôt Git vide initialisé dans /home/anarcat/test/test/.git/ +(test) [1266]anarcat@curie:test$ cd test +(test) [1267]anarcat@curie:test$ git annex init +init ok +(recording state in git...) +(test) [1268]anarcat@curie:test$ echo foo > bar +(test) [1269]anarcat@curie:test$ git annex add bar +add bar ok +(recording state in git...) +(test) [1270]anarcat@curie:test$ git remote add sal sal:test +(test) [1271]anarcat@curie:test$ git annex sync sal +commit +[master (commit racine) 05b6d24] git-annex in anarcat@curie:~/test/test + 1 file changed, 1 insertion(+) + create mode 120000 bar +ok +pull sal +warning: pas de commit commun +remote: Counting objects: 5, done. +remote: Compressing objects: 100% (3/3), done. +remote: Total 5 (delta 0), reused 0 (delta 0) +Dépaquetage des objets: 100% (5/5), fait. +Depuis sal:test + * [nouvelle branche] git-annex -> sal/git-annex +ok +(merging sal/git-annex into git-annex...) +(recording state in git...) +push sal +Énumération des objets: 18, fait. +Décompte des objets: 100% (18/18), fait. +Compression par delta en utilisant jusqu'à 4 fils d'exécution +Compression des objets: 100% (12/12), fait. +Écriture des objets: 100% (16/16), 1.92 KiB | 654.00 KiB/s, fait. +Total 16 (delta 1), réutilisés 0 (delta 0) +To sal:test + * [new branch] git-annex -> synced/git-annex + * [new branch] master -> synced/master +ok +(test) [1272]anarcat@curie:test$ git annex sync --content sal +commit +Sur la branche master +rien à valider, la copie de travail est propre +ok +pull sal +ok +copy bar (fd:18: hClose: resource vanished (Broken pipe)) failed +git-annex: sync: 1 failed + +# End of transcript or log. +"""]] + +The stretch repo is just `git init test && git -C test annex +init`. SSH communication is setup with a passwordless SSH key in +both. Debugging shows us the problem is with the p2pstdio command: + +[[!format sh """ +$ git annex sync --content sal --debug +[2018-11-03 15:25:35.018955156] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] +[2018-11-03 15:25:35.022443588] process done ExitSuccess +[2018-11-03 15:25:35.022611687] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] +[2018-11-03 15:25:35.02578792] process done ExitSuccess +[2018-11-03 15:25:35.026234842] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..24d62bcb5e253326bde98283338a855c9bfbe983","--pretty=%H","-n1"] +[2018-11-03 15:25:35.030089742] process done ExitSuccess +[2018-11-03 15:25:35.041384604] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"] +[2018-11-03 15:25:35.042220799] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] +commit +[2018-11-03 15:25:35.045907106] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit","-a","-m","git-annex in anarcat@curie:~/test/test"] +Sur la branche master +rien à valider, la copie de travail est propre +[2018-11-03 15:25:35.077421642] process done ExitFailure 1 +ok +[2018-11-03 15:25:35.077523489] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"] +[2018-11-03 15:25:35.079008549] process done ExitSuccess +[2018-11-03 15:25:35.079088278] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"] +[2018-11-03 15:25:35.080639101] process done ExitSuccess +[2018-11-03 15:25:35.080766018] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/heads/synced/master"] +[2018-11-03 15:25:35.08225223] process done ExitSuccess +[2018-11-03 15:25:35.082343323] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/heads/synced/master","--pretty=%H","-n1"] +[2018-11-03 15:25:35.084234147] process done ExitSuccess +pull sal +[2018-11-03 15:25:35.084663095] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","fetch","sal"] +[2018-11-03 15:25:37.678014829] process done ExitSuccess +[2018-11-03 15:25:37.678131075] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"] +[2018-11-03 15:25:37.68042014] process done ExitSuccess +[2018-11-03 15:25:37.680521541] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/master"] +[2018-11-03 15:25:37.682164837] process done ExitFailure 1 +[2018-11-03 15:25:37.682259432] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/synced/master"] +[2018-11-03 15:25:37.683835717] process done ExitSuccess +[2018-11-03 15:25:37.683934116] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/synced/master..refs/remotes/sal/synced/master","--pretty=%H","-n1"] +[2018-11-03 15:25:37.686212308] process done ExitSuccess +ok +[2018-11-03 15:25:37.686343742] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] +[2018-11-03 15:25:37.688215586] process done ExitSuccess +[2018-11-03 15:25:37.688305585] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] +[2018-11-03 15:25:37.6901916] process done ExitSuccess +[2018-11-03 15:25:37.690393283] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..24d62bcb5e253326bde98283338a855c9bfbe983","--pretty=%H","-n1"] +[2018-11-03 15:25:37.692726875] process done ExitSuccess +[2018-11-03 15:25:37.692906438] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--"] +copy bar [2018-11-03 15:25:37.696621566] chat: ssh ["sal","-S",".git/annex/ssh/sal","-o","ControlMaster=auto","-o","ControlPersist=yes","-T","git-annex-shell 'p2pstdio' '/~/test' '--debug' 'f73778b3-17a4-4fee-9890-402bedf02456' --uuid 02ef3aa6-34d9-42b9-814e-f9ebc47c59e1"] +[2018-11-03 15:25:37.96931512] P2P > ERROR auth failed +(fd:18: hClose: resource vanished (Broken pipe)) failed +[2018-11-03 15:25:37.96973331] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"] +[2018-11-03 15:25:37.970341712] read: git ["--version"] +[2018-11-03 15:25:37.97296492] process done ExitSuccess +[2018-11-03 15:25:37.974463782] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"] +[2018-11-03 15:25:37.978567382] process done ExitSuccess +[2018-11-03 15:25:37.979111982] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/synced/master"] +[2018-11-03 15:25:37.982068735] process done ExitSuccess +[2018-11-03 15:25:37.982213309] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/remotes/sal/synced/master..refs/heads/synced/master","--pretty=%H","-n1"] +[2018-11-03 15:25:37.986086966] process done ExitSuccess +[2018-11-03 15:25:37.986239042] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/git-annex"] +[2018-11-03 15:25:37.988954838] process done ExitSuccess +[2018-11-03 15:25:37.989110109] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/remotes/sal/git-annex..git-annex","--pretty=%H","-n1"] +[2018-11-03 15:25:37.993956367] process done ExitSuccess +[2018-11-03 15:25:37.994340874] read: ssh ["-O","stop","-S","sal","-o","ControlMaster=auto","-o","ControlPersist=yes","localhost"] +[2018-11-03 15:25:38.00909442] process done ExitSuccess +[2018-11-03 15:25:38.009887194] process done ExitSuccess +[2018-11-03 15:25:38.010571056] process done ExitSuccess +[2018-11-03 15:25:38.011368991] process done ExitSuccess +git-annex: sync: 1 failed +"""]] + +A manual invocation confirms the [[design/p2p_protocol]] not supported +by the remote, probably because it's too old: + +[[!format sh """ +$ ssh sal git-annex-shell p2pstdio +fatal: unrecognized command 'p2pstdio' +git-annex-shell: git-shell failed +"""]] + +I understand the need for this command (which has no manpage, for what +that's worth ;), but we should be able to fallback gracefully here. In +the worst case, just treat this as a remote without git-annex-shell +and revert to rsync to transfer files... + +The workaround is to install the (working) backport in stretch, of course. + +I wonder if this could be related to +[[bugs/new_git-annex-shell_protocol_hides_remote_error_messages/]]. + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Yes! I'm using git-annex like a maniac and I'm using it to archive +Brazil right now. -- [[anarcat]] From f09f846feaee95984c5b964136a1951f7d019f91 Mon Sep 17 00:00:00 2001 From: spwhitton Date: Sat, 3 Nov 2018 19:41:32 +0000 Subject: [PATCH 06/10] Added a comment --- .../comment_1_4e7d50e097418290b185f43e05ddd86b._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/interoperability_issue_between_buster_and_stretch/comment_1_4e7d50e097418290b185f43e05ddd86b._comment diff --git a/doc/bugs/interoperability_issue_between_buster_and_stretch/comment_1_4e7d50e097418290b185f43e05ddd86b._comment b/doc/bugs/interoperability_issue_between_buster_and_stretch/comment_1_4e7d50e097418290b185f43e05ddd86b._comment new file mode 100644 index 0000000000..9341320db1 --- /dev/null +++ b/doc/bugs/interoperability_issue_between_buster_and_stretch/comment_1_4e7d50e097418290b185f43e05ddd86b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="spwhitton" + avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb" + subject="comment 1" + date="2018-11-03T19:41:32Z" + content=""" +Looking at the error message alone, I am pretty sure this is fixed in recent git-annex; I am just waiting on the Haskell transition in Debian to be able to build the package. +"""]] From 16cbeb6aff669496f417bc4557d6388e7dee3273 Mon Sep 17 00:00:00 2001 From: anarcat Date: Sat, 3 Nov 2018 19:52:48 +0000 Subject: [PATCH 07/10] Added a comment: oops --- .../comment_2_768f116e98a3cef3285ef941464c0880._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/interoperability_issue_between_buster_and_stretch/comment_2_768f116e98a3cef3285ef941464c0880._comment diff --git a/doc/bugs/interoperability_issue_between_buster_and_stretch/comment_2_768f116e98a3cef3285ef941464c0880._comment b/doc/bugs/interoperability_issue_between_buster_and_stretch/comment_2_768f116e98a3cef3285ef941464c0880._comment new file mode 100644 index 0000000000..5c21266f12 --- /dev/null +++ b/doc/bugs/interoperability_issue_between_buster_and_stretch/comment_2_768f116e98a3cef3285ef941464c0880._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="anarcat" + avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7" + subject="oops" + date="2018-11-03T19:52:48Z" + content=""" +oh. oops. sorry for the noise then... maybe i should have filed this against the BTS first? :) +"""]] From d621846ab8968e42e248467548970528ad0411e3 Mon Sep 17 00:00:00 2001 From: anarcat Date: Sat, 3 Nov 2018 20:52:49 +0000 Subject: [PATCH 08/10] Added a comment: workaround with youtube-dl --- ...ent_2_58733a5065493dbac5cb8d12a029dace._comment | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/forum/importfeed_does_not_work_with_youtube_anymore/comment_2_58733a5065493dbac5cb8d12a029dace._comment diff --git a/doc/forum/importfeed_does_not_work_with_youtube_anymore/comment_2_58733a5065493dbac5cb8d12a029dace._comment b/doc/forum/importfeed_does_not_work_with_youtube_anymore/comment_2_58733a5065493dbac5cb8d12a029dace._comment new file mode 100644 index 0000000000..9380c54c15 --- /dev/null +++ b/doc/forum/importfeed_does_not_work_with_youtube_anymore/comment_2_58733a5065493dbac5cb8d12a029dace._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="anarcat" + avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7" + subject="workaround with youtube-dl" + date="2018-11-03T20:52:49Z" + content=""" +i hacked my way around this problem with this crazy commandline: + + URL=www.youtube.com/user/WHATEVER/videos # or playlist or whatever + youtube-dl --dump-json --flat-playlist $URL | jq -r '.id' | sed 's_^_https://www.youtube.com/watch?v=_' > urls.txt + git -c annex.security.allowed-http-addresses=all annex addurl --batch < urls.txt + +It's idiotic that youtube removed support for standard stuff, but that's the name of the game these days. Maybe git-annex could delegate this to youtube-dl? --[[anarcat]] +"""]] From cb122b9d1e9dec62fcb1f3c6ede08a9c64349f4d Mon Sep 17 00:00:00 2001 From: anarcat Date: Sun, 4 Nov 2018 19:29:19 +0000 Subject: [PATCH 09/10] Added a comment: rsync can do this on its own --- ..._60b7597af2b86ce6bcb31731f6d8c54e._comment | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 doc/todo/wishlist__58___global_progress_status/comment_4_60b7597af2b86ce6bcb31731f6d8c54e._comment diff --git a/doc/todo/wishlist__58___global_progress_status/comment_4_60b7597af2b86ce6bcb31731f6d8c54e._comment b/doc/todo/wishlist__58___global_progress_status/comment_4_60b7597af2b86ce6bcb31731f6d8c54e._comment new file mode 100644 index 0000000000..106c6c6d8d --- /dev/null +++ b/doc/todo/wishlist__58___global_progress_status/comment_4_60b7597af2b86ce6bcb31731f6d8c54e._comment @@ -0,0 +1,26 @@ +[[!comment format=mdwn + username="anarcat" + avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7" + subject="rsync can do this on its own" + date="2018-11-04T19:29:19Z" + content=""" +i understand, thanks for taking the time to explain the tradeoffs! :) + +i know that rsync is not used anymore, but I figured I should document this here since it's the first thing that it made me think of when i found out about it. as it turns out, rsync *does* have its own \"global status\", which has similar tradeoffs than git-annex (namely that it doesn't work in \"incremental mode\"). from #debian-til on OFTC: + +[[!format txt \"\"\" +13:30:15 TIL: rsync --partial --no-inc-recursive -ah --info=progress2 +13:30:32 explanation? +13:30:33 * juliank apologizes for the : +13:30:45 So, --info=progress2 gives you overall progress +13:30:53 overall? +13:30:58 Oh, w/ multiple files? +13:31:00 and --no-inc-recursive makes it really overall by doing scanning up front +13:31:08 s/scanning/recursion/ +13:31:09 Oh, now that's very neat. +13:31:16 and -h gives you human-readable sizes +13:31:16 WTB that in debmirror :3 +\"\"\"]] + +So rsync allows you to switch back to the \"costly mode\" which *does* a full scan before starting. It's slower, but it allows you to get global progress info. It's a nice tradeoff and it's especially useful to be able to enable it on deman. I understand this might be complicated to implement in git-annex because there are many places where such an option would be required (and it's unclear how it would be named), but it's something that would certainly be useful for my use cases, where some repos have large files but not so many so are fairly fast to scan (e.g. i could do this on my video repo, but not the music repo). -- [[anarcat]] +"""]] From 8570c07d49233ec19824021933db6687a7fa6661 Mon Sep 17 00:00:00 2001 From: seantparsons Date: Sun, 4 Nov 2018 20:59:39 +0000 Subject: [PATCH 10/10] --- doc/todo/Retire_Esqueleto_as_a_dependency.mdwn | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 doc/todo/Retire_Esqueleto_as_a_dependency.mdwn diff --git a/doc/todo/Retire_Esqueleto_as_a_dependency.mdwn b/doc/todo/Retire_Esqueleto_as_a_dependency.mdwn new file mode 100644 index 0000000000..8871b6f688 --- /dev/null +++ b/doc/todo/Retire_Esqueleto_as_a_dependency.mdwn @@ -0,0 +1,4 @@ +Since Esqueleto is seemingly abandoned and it's causing grief with NixOS: [[https://git-annex.branchable.com/bugs/Old_dependencies_are_causing_pain_with_NixOS_18.09/]] I did the work to remove it as a dependency in this commit +[[https://github.com/seanparsons/git-annex/commit/62f291262f37327d9b2a864cd701aacb03766115]] + +Alongside that is a bunch of nix specific files that support the building and/or development of git-annex, which is pretty much anything with "nix" in the name in the patch.