From 077a99bebb5becbd4e4a4d4ef3bacdd0ab43d6a1 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Thu, 4 Jan 2024 20:23:34 +0000 Subject: [PATCH 1/6] new issue about stalling transfers --- ...ng___34__Transfer_stalled__34____63__.mdwn | 73 +++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__.mdwn diff --git a/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__.mdwn b/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__.mdwn new file mode 100644 index 0000000000..af10cc024d --- /dev/null +++ b/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__.mdwn @@ -0,0 +1,73 @@ +### Please describe the problem. + +Doing backup of data using `git annex move` from local to dropbox via `git-annex-remote-rclone`. + +Full invocation here is + +```shell +( source ~/git-annexes/10.20231227+git24-gd37dbd62b8.env; OUR_LOG_FILE=~/.cache/rclone/logs/`date --iso-8601=seconds`-movedebug; RCLONE_LOG_LEVEL=DEBUG RCLONE_LOG_FILE=${OUR_LOG_FILE}_rclone.log git annex --debug move --to dandi-dandisets-dropbox sub-484677/sub-484677_ses-20210419T161140_behavior+ecephys+ogen.nwb 2>&1 | tee ${OUR_LOG_FILE}_git-annex.log ; ) +``` +under `/mnt/backup/dandi/dandisets/000363` . On git-annex side we see + +```shell +dandi@drogon:~/.cache/rclone/logs$ grep -4 'Transfer stalled' 2024-01-04T14:28:29-05:00-movedebug_git-annex.log +[2024-01-04 14:43:24.5722525] (Annex.TransferrerPool) < op 295000000000 +75% 274.74 GiB 7 MiB/s 3h46m2024/01/04 14:44:49 DEBUG : Setting --log-level "DEBUG" from environment variable RCLONE_LOG_LEVEL="DEBUG" +2024/01/04 14:44:49 DEBUG : Setting --log-file "/home/dandi/.cache/rclone/logs/2024-01-04T14:28:29-05:00-movedebug_rclone.log" from environment variable RCLONE_LOG_FILE="/home/dandi/.cache/rclone/logs/2024-01-04T14:28:29-05:00-movedebug_rclone.log" + + Transfer stalled +[2024-01-04 14:45:40.042053663] (Utility.Process) process [2329029] chat: /home/dandi/git-annexes/10.20231227+git24-gd37dbd62b8/usr/lib/git-annex.linux/git-annex ["transferrer","-c","annex.debug=true"] + +``` + +and rclone logs for around that time point(s): + +``` +2024/01/04 14:44:49 DEBUG : rclone: Version "v1.61.1-DEV" starting with parameters ["rclone" "copy" ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb" "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/"] +2024/01/04 14:44:49 DEBUG : Creating backend with remote ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb" +2024/01/04 14:44:49 DEBUG : Using config file from "/home/dandi/.rclone.conf" +2024/01/04 14:44:49 DEBUG : fs cache: adding new entry for parent of ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb", "/mnt/backup/dandi/dandisets/000363/.git/annex/tmp" +2024/01/04 14:44:49 DEBUG : Creating backend with remote "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/" +2024/01/04 14:44:49 DEBUG : fs cache: renaming cache item "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/" to be canonical "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757" +2024/01/04 14:44:49 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Need to transfer - File not found at Destination +2024/01/04 14:44:50 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 1/20 +... +2024/01/04 14:45:36 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 19/20 +2024/01/04 14:45:38 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 20/20 +2024/01/04 14:45:39 INFO : Signal received: interrupt +2024/01/04 14:45:39 INFO : Dropbox root 'dandi-dandisets/annexstore/b4f/757': Committing uploads - please wait... +2024/01/04 14:45:39 INFO : Exiting... +``` + +
+using older git-annex I got similar crash but on earlier chunk, so it is not pertinent to last chunk + +```shell +2024/01/04 13:56:29 DEBUG : rclone: Version "v1.61.1-DEV" starting with parameters ["rclone" "copy" ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb" "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/"] +2024/01/04 13:56:29 DEBUG : Creating backend with remote ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb" +2024/01/04 13:56:29 DEBUG : Using config file from "/home/dandi/.rclone.conf" +2024/01/04 13:56:29 DEBUG : fs cache: adding new entry for parent of ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb", "/mnt/backup/dandi/dandisets/000363/.git/annex/tmp" +2024/01/04 13:56:29 DEBUG : Creating backend with remote "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/" +2024/01/04 13:56:30 DEBUG : fs cache: renaming cache item "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/" to be canonical "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757" +2024/01/04 13:56:30 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Need to transfer - File not found at Destination +2024/01/04 13:56:30 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 1/20 +... +2024/01/04 13:57:08 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 17/20 +2024/01/04 13:57:10 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 18/20 +2024/01/04 13:57:11 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 19/20 +2024/01/04 13:57:14 INFO : Signal received: interrupt +2024/01/04 13:57:14 INFO : Dropbox root 'dandi-dandisets/annexstore/b4f/757': Committing uploads - please wait... +2024/01/04 13:57:14 INFO : Exiting... +``` +
+ +So it seems that stall detection (I didn't look how it even decides on that) is flawed in case of large (chunked) files being transferred through external custom remotes which have no way to report progress as it is the case here in https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/blob/master/git-annex-remote-rclone (filed [a bug report](https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/issues/76) to see that addressed). + + +### What version of git-annex are you using? On what operating system? + +first was a year old version, then tried with bleeding edge 10.20231227+git24-gd37dbd62b8 + + +[[!meta author=yoh]] +[[!tag projects/dandi]] From d08804ef9e2de897cb4c07371b311c6dc8afb936 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Fri, 5 Jan 2024 21:00:52 +0000 Subject: [PATCH 2/6] Added a comment --- ...comment_7_92bf43545c46d302089820cfdd03141b._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/comment_7_92bf43545c46d302089820cfdd03141b._comment diff --git a/doc/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/comment_7_92bf43545c46d302089820cfdd03141b._comment b/doc/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/comment_7_92bf43545c46d302089820cfdd03141b._comment new file mode 100644 index 0000000000..e205116f56 --- /dev/null +++ b/doc/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/comment_7_92bf43545c46d302089820cfdd03141b._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 7" + date="2024-01-05T21:00:52Z" + content=""" +so there remains no way to configure assistant to add files in regular locked mode where files would \"migrate\" to become symlinks after having been added to annex? + +Our use case is the \"data acquisition\" box (for repronim's reprostim project) which we are configuring to collect videos, which are then to be transferred to another box. Size will be large, we will rely on branches and copies/moves of the files, so having them to be symlinks is kinda \"mandatory\" to keep operations fast and to not waste space. +"""]] From 25e2e827f6319da66de0ac12b012c4da6125bd13 Mon Sep 17 00:00:00 2001 From: nobodyinperson Date: Sat, 6 Jan 2024 09:22:54 +0000 Subject: [PATCH 3/6] Added a comment --- .../comment_8_a6ae8bc3b5caf0d3d49a919ab5ec5778._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/comment_8_a6ae8bc3b5caf0d3d49a919ab5ec5778._comment diff --git a/doc/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/comment_8_a6ae8bc3b5caf0d3d49a919ab5ec5778._comment b/doc/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/comment_8_a6ae8bc3b5caf0d3d49a919ab5ec5778._comment new file mode 100644 index 0000000000..b300907a88 --- /dev/null +++ b/doc/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/comment_8_a6ae8bc3b5caf0d3d49a919ab5ec5778._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="nobodyinperson" + avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5" + subject="comment 8" + date="2024-01-06T09:22:54Z" + content=""" +Not that I know of. Simplest solution for you would be probably a `while true;git annex assist`done` instead of the assistant. +"""]] From 46ab66d97ade6a602c402c29acfae85a4d506f1a Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Sat, 6 Jan 2024 16:00:05 +0000 Subject: [PATCH 4/6] reporting about --listen --- ...ebapp_--listen_port_is_not_used__63__.mdwn | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/bugs/webapp_--listen_port_is_not_used__63__.mdwn diff --git a/doc/bugs/webapp_--listen_port_is_not_used__63__.mdwn b/doc/bugs/webapp_--listen_port_is_not_used__63__.mdwn new file mode 100644 index 0000000000..56d76c9518 --- /dev/null +++ b/doc/bugs/webapp_--listen_port_is_not_used__63__.mdwn @@ -0,0 +1,21 @@ +### Please describe the problem. + +``` +reprostim@reproiner:/data/reprostim/Videos$ ps auxw | grep webapp +reprost+ 25249 0.0 0.0 9892 2100 pts/5 S+ Jan05 0:00 git annex --debug webapp --listen 0.0.0.0:8888 +reprost+ 25250 5.4 0.2 1074346616 65556 ? Ssl Jan05 60:39 /usr/bin/git-annex --debug webapp --listen 0.0.0.0:8888 +reprost+ 224039 0.0 0.0 6332 2116 pts/6 S+ 10:55 0:00 grep webapp + +reprostim@reproiner:/data/reprostim/Videos$ lsof -i :8888 + +reprostim@reproiner:/data/reprostim/Videos$ lsof -i | grep git-annex +git-annex 25250 reprostim 14u IPv4 129033 0t0 UDP *:55556 +git-annex 221230 reprostim 14u IPv4 129033 0t0 UDP *:55556 + +reprostim@reproiner:/data/reprostim/Videos$ git annex version +git-annex version: 10.20230126 + +``` + +[[!meta author=yoh]] +[[!tag projects/repronim]] From 61384f155e770f5c5c0c23a989382fc8e6778061 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Sat, 6 Jan 2024 18:31:18 +0000 Subject: [PATCH 5/6] --- ...justed_branch_subtree_regression__58___FAIL_.mdwn | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/bugs/____adjusted_branch_subtree_regression__58___FAIL_.mdwn diff --git a/doc/bugs/____adjusted_branch_subtree_regression__58___FAIL_.mdwn b/doc/bugs/____adjusted_branch_subtree_regression__58___FAIL_.mdwn new file mode 100644 index 0000000000..9fb711bb76 --- /dev/null +++ b/doc/bugs/____adjusted_branch_subtree_regression__58___FAIL_.mdwn @@ -0,0 +1,12 @@ +### Please describe the problem. + +in recent builds [eg](https://github.com/datalad/git-annex/actions/runs/7417377022/job/20183877530#step:7:1288) + +``` + adjusted branch subtree regression: FAIL (1.26s) + Test.hs:1675: + a/b/x/y missing from master after adjusted branch sync + Use -p '/adjusted branch subtree regression/' to rerun this test only. +``` + +may be more From 6e4d59c05a27032af58688fcc8c0470f0b9d9416 Mon Sep 17 00:00:00 2001 From: psxvoid Date: Sun, 7 Jan 2024 04:11:37 +0000 Subject: [PATCH 6/6] Added a comment: Using fuse annexize - works like a charm --- ..._12_041ae920e2e6fd83967dc98646452fa7._comment | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/forum/__34__du__34___equivalent_on_an_annex__63__/comment_12_041ae920e2e6fd83967dc98646452fa7._comment diff --git a/doc/forum/__34__du__34___equivalent_on_an_annex__63__/comment_12_041ae920e2e6fd83967dc98646452fa7._comment b/doc/forum/__34__du__34___equivalent_on_an_annex__63__/comment_12_041ae920e2e6fd83967dc98646452fa7._comment new file mode 100644 index 0000000000..bf3cbca594 --- /dev/null +++ b/doc/forum/__34__du__34___equivalent_on_an_annex__63__/comment_12_041ae920e2e6fd83967dc98646452fa7._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="psxvoid" + avatar="http://cdn.libravatar.org/avatar/fde068fbdeabeea31e3be7aa9c55d84b" + subject="Using fuse annexize - works like a charm" + date="2024-01-07T04:11:37Z" + content=""" +Hey @wzhd, + +Thanks a lot for this tool - annexize works like a charm even in tags views! + +It's basically solve the most important problem for me: + +I can use ncdu for organizing files in my local git annex repository that +does not contain any actual files (only file links), and then just sync +with linked repos that do store those files. +"""]]