mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2024-01-09 15:32:26 -04:00
commit 8a72aa8d99
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
6 changed files with 140 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
doc/bugs/____adjusted_branch_subtree_regression__58___FAIL_.mdwn Normal file
View file

@ -0,0 +1,12 @@
### Please describe the problem.
in recent builds [eg](https://github.com/datalad/git-annex/actions/runs/7417377022/job/20183877530#step:7:1288)
```
adjusted branch subtree regression: FAIL (1.26s)
Test.hs:1675:
a/b/x/y missing from master after adjusted branch sync
Use -p '/adjusted branch subtree regression/' to rerun this test only.
```
may be more

73
doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__.mdwn Normal file
View file

@ -0,0 +1,73 @@
### Please describe the problem.
Doing backup of data using `git annex move` from local to dropbox via `git-annex-remote-rclone`.
Full invocation here is
```shell
( source ~/git-annexes/10.20231227+git24-gd37dbd62b8.env; OUR_LOG_FILE=~/.cache/rclone/logs/`date --iso-8601=seconds`-movedebug; RCLONE_LOG_LEVEL=DEBUG RCLONE_LOG_FILE=${OUR_LOG_FILE}_rclone.log git annex --debug move --to dandi-dandisets-dropbox sub-484677/sub-484677_ses-20210419T161140_behavior+ecephys+ogen.nwb 2>&1 | tee ${OUR_LOG_FILE}_git-annex.log ; )
```
under `/mnt/backup/dandi/dandisets/000363` . On git-annex side we see
```shell
dandi@drogon:~/.cache/rclone/logs$ grep -4 'Transfer stalled' 2024-01-04T14:28:29-05:00-movedebug_git-annex.log
[2024-01-04 14:43:24.5722525] (Annex.TransferrerPool) < op 295000000000
75% 274.74 GiB 7 MiB/s 3h46m2024/01/04 14:44:49 DEBUG : Setting --log-level "DEBUG" from environment variable RCLONE_LOG_LEVEL="DEBUG"
2024/01/04 14:44:49 DEBUG : Setting --log-file "/home/dandi/.cache/rclone/logs/2024-01-04T14:28:29-05:00-movedebug_rclone.log" from environment variable RCLONE_LOG_FILE="/home/dandi/.cache/rclone/logs/2024-01-04T14:28:29-05:00-movedebug_rclone.log"
Transfer stalled
[2024-01-04 14:45:40.042053663] (Utility.Process) process [2329029] chat: /home/dandi/git-annexes/10.20231227+git24-gd37dbd62b8/usr/lib/git-annex.linux/git-annex ["transferrer","-c","annex.debug=true"]
```
and rclone logs for around that time point(s):
```
2024/01/04 14:44:49 DEBUG : rclone: Version "v1.61.1-DEV" starting with parameters ["rclone" "copy" ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb" "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/"]
2024/01/04 14:44:49 DEBUG : Creating backend with remote ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb"
2024/01/04 14:44:49 DEBUG : Using config file from "/home/dandi/.rclone.conf"
2024/01/04 14:44:49 DEBUG : fs cache: adding new entry for parent of ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb", "/mnt/backup/dandi/dandisets/000363/.git/annex/tmp"
2024/01/04 14:44:49 DEBUG : Creating backend with remote "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/"
2024/01/04 14:44:49 DEBUG : fs cache: renaming cache item "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/" to be canonical "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757"
2024/01/04 14:44:49 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Need to transfer - File not found at Destination
2024/01/04 14:44:50 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 1/20
...
2024/01/04 14:45:36 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 19/20
2024/01/04 14:45:38 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 20/20
2024/01/04 14:45:39 INFO : Signal received: interrupt
2024/01/04 14:45:39 INFO : Dropbox root 'dandi-dandisets/annexstore/b4f/757': Committing uploads - please wait...
2024/01/04 14:45:39 INFO : Exiting...
```
<details>
<summary>using older git-annex I got similar crash but on earlier chunk, so it is not pertinent to last chunk</summary>
```shell
2024/01/04 13:56:29 DEBUG : rclone: Version "v1.61.1-DEV" starting with parameters ["rclone" "copy" ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb" "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/"]
2024/01/04 13:56:29 DEBUG : Creating backend with remote ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb"
2024/01/04 13:56:29 DEBUG : Using config file from "/home/dandi/.rclone.conf"
2024/01/04 13:56:29 DEBUG : fs cache: adding new entry for parent of ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb", "/mnt/backup/dandi/dandisets/000363/.git/annex/tmp"
2024/01/04 13:56:29 DEBUG : Creating backend with remote "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/"
2024/01/04 13:56:30 DEBUG : fs cache: renaming cache item "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/" to be canonical "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757"
2024/01/04 13:56:30 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Need to transfer - File not found at Destination
2024/01/04 13:56:30 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 1/20
...
2024/01/04 13:57:08 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 17/20
2024/01/04 13:57:10 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 18/20
2024/01/04 13:57:11 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 19/20
2024/01/04 13:57:14 INFO : Signal received: interrupt
2024/01/04 13:57:14 INFO : Dropbox root 'dandi-dandisets/annexstore/b4f/757': Committing uploads - please wait...
2024/01/04 13:57:14 INFO : Exiting...
```
</details>
So it seems that stall detection (I didn't look how it even decides on that) is flawed in case of large (chunked) files being transferred through external custom remotes which have no way to report progress as it is the case here in https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/blob/master/git-annex-remote-rclone (filed [a bug report](https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/issues/76) to see that addressed).
### What version of git-annex are you using? On what operating system?
first was a year old version, then tried with bleeding edge 10.20231227+git24-gd37dbd62b8
[[!meta author=yoh]]
[[!tag projects/dandi]]

21
doc/bugs/webapp_--listen_port_is_not_used__63__.mdwn Normal file
View file

@ -0,0 +1,21 @@
### Please describe the problem.
```
reprostim@reproiner:/data/reprostim/Videos$ ps auxw | grep webapp
reprost+ 25249 0.0 0.0 9892 2100 pts/5 S+ Jan05 0:00 git annex --debug webapp --listen 0.0.0.0:8888
reprost+ 25250 5.4 0.2 1074346616 65556 ? Ssl Jan05 60:39 /usr/bin/git-annex --debug webapp --listen 0.0.0.0:8888
reprost+ 224039 0.0 0.0 6332 2116 pts/6 S+ 10:55 0:00 grep webapp
reprostim@reproiner:/data/reprostim/Videos$ lsof -i :8888
reprostim@reproiner:/data/reprostim/Videos$ lsof -i | grep git-annex
git-annex 25250 reprostim 14u IPv4 129033 0t0 UDP *:55556
git-annex 221230 reprostim 14u IPv4 129033 0t0 UDP *:55556
reprostim@reproiner:/data/reprostim/Videos$ git annex version
git-annex version: 10.20230126
```
[[!meta author=yoh]]
[[!tag projects/repronim]]

10
doc/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/comment_7_92bf43545c46d302089820cfdd03141b._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 7"
date="2024-01-05T21:00:52Z"
content="""
so there remains no way to configure assistant to add files in regular locked mode where files would \"migrate\" to become symlinks after having been added to annex?
Our use case is the \"data acquisition\" box (for repronim's reprostim project) which we are configuring to collect videos, which are then to be transferred to another box. Size will be large, we will rely on branches and copies/moves of the files, so having them to be symlinks is kinda \"mandatory\" to keep operations fast and to not waste space.
"""]]

8
doc/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/comment_8_a6ae8bc3b5caf0d3d49a919ab5ec5778._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="nobodyinperson"
avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5"
subject="comment 8"
date="2024-01-06T09:22:54Z"
content="""
Not that I know of. Simplest solution for you would be probably a `while true;git annex assist`done` instead of the assistant.
"""]]

16
doc/forum/__34__du__34___equivalent_on_an_annex__63__/comment_12_041ae920e2e6fd83967dc98646452fa7._comment Normal file
View file

@ -0,0 +1,16 @@
[[!comment format=mdwn
username="psxvoid"
avatar="http://cdn.libravatar.org/avatar/fde068fbdeabeea31e3be7aa9c55d84b"
subject="Using fuse annexize - works like a charm"
date="2024-01-07T04:11:37Z"
content="""
Hey @wzhd,
Thanks a lot for this tool - annexize works like a charm even in tags views!
It's basically solve the most important problem for me:
I can use ncdu for organizing files in my local git annex repository that
does not contain any actual files (only file links), and then just sync
with linked repos that do store those files.
"""]]