mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2025-01-27 09:35:44 -04:00
commit de33ad5ce1
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
8 changed files with 212 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

65
doc/bugs/FTBFS_for_the_past_week__58___Variable_not_in_scope.mdwn Normal file
View file

@ -0,0 +1,65 @@
### Please describe the problem.
I have in my mailbox
```
80 T Jan 26 GitHub Actions *-3.6* (3.7K/0) datalad/git-annex daily summary: 4 FAILED, 8 INCOMPLETE, 1 PASSED, 3 ABSENT
206 N T Jan 25 GitHub Actions *-3.8* (3.7K/0) datalad/git-annex daily summary: 4 FAILED, 8 INCOMPLETE, 1 PASSED, 3 ABSENT
357 T Jan 24 GitHub Actions *-4.4* (6.3K/0) datalad/git-annex daily summary: 12 FAILED, 8 INCOMPLETE, 1 PASSED, 3 ABSENT
1279 T Jan 23 GitHub Actions *-4.5* (3.7K/0) datalad/git-annex daily summary: 5 FAILED, 8 INCOMPLETE, 3 ABSENT
1715 T Jan 22 GitHub Actions *-5.0* (3.7K/0) datalad/git-annex daily summary: 5 FAILED, 8 INCOMPLETE, 3 ABSENT 2335 T Jan 21 GitHub Actions *-3.9* (3.7K/0) datalad/git-annex daily summary: 5 FAILED, 8 INCOMPLETE, 3 ABSENT
2656 T Jan 20 GitHub Actions *-4.3* (6.8K/0) datalad/git-annex daily summary: 28 PASSED, 2 ABSENT
2862 T Jan 19 GitHub Actions *-5.0* (6.8K/0) datalad/git-annex daily summary: 28 PASSED, 2 ABSENT
```
and looking at the [latest ubuntu build logs](https://github.com/datalad/git-annex/actions/runs/12970824274/job/36176536041) I see
```
I: the tail of the log
Build/LinuxMkLibs.hs:101:17: error:
Variable not in scope:
createDirectoryIfMissing :: Bool -> [Char] -> IO a3
|
101 | createDirectoryIfMissing True (top ++ libdir </> takeDirectory d)
| ^^^^^^^^^^^^^^^^^^^^^^^^
Build/LinuxMkLibs.hs:149:9: error:
Variable not in scope:
createDirectoryIfMissing :: Bool -> FilePath -> IO a2
|
149 | createDirectoryIfMissing True (top </> shimdir)
| ^^^^^^^^^^^^^^^^^^^^^^^^
Build/LinuxMkLibs.hs:150:9: error:
Variable not in scope:
createDirectoryIfMissing :: Bool -> FilePath -> IO a1
|
150 | createDirectoryIfMissing True (top </> exedir)
| ^^^^^^^^^^^^^^^^^^^^^^^^
Build/LinuxMkLibs.hs:160:19: error:
* Variable not in scope:
renameFile :: FilePath -> FilePath -> IO ()
* Perhaps you meant `readFile' (imported from Prelude)
|
160 | , renameFile exe exedest
| ^^^^^^^^^^
Build/LinuxMkLibs.hs:165:18: error:
Variable not in scope: doesFileExist :: FilePath -> IO Bool
|
165 | unlessM (doesFileExist (top </> exelink)) $
| ^^^^^^^^^^^^^
Build/LinuxMkLibs.hs:181:9: error:
Variable not in scope:
createDirectoryIfMissing :: Bool -> FilePath -> IO a0
|
181 | createDirectoryIfMissing True destdir
| ^^^^^^^^^^^^^^^^^^^^^^^^
make[3]: *** [Makefile:156: Build/Standalone] Error 1
make[3]: Leaving directory '/home/runner/work/git-annex/git-annex/git-annex-source'
make[2]: *** [Makefile:164: linuxstandalone] Error 2
```

24
doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn
View file

@ -4,13 +4,13 @@ I have a pretty big repository with around 300 000 files in the workdir of a bra
I wanted to unlock all those files from that branch on a machine, so I tried to use git-annex-adjust --unlock.
Sadly, the command do not seems to finish, ever.
Executing the command with debug from a clone(to avoid interacting with the broken index from the first), it seems to deadlock after executing between 10000 and 20000 "thawing" processes when executing the filter-process logic over the files in the worktree.
The problem seems to be reproducible with any repository with a lot of files in the worktree as far as I can tell, independant of file size.
Executing the command with the debug flag from a clone(to avoid interacting with the broken index from the first), it seems to deadlock after executing 10240 completed processes for the filter-process logic over the files in the worktree, which happens to match the annex.queuesize configuration value in use in those repositories.
The problem seems to be reproducible with any repository with more than the aforementioned count of files in the worktree as far as I can tell, independant of file size.
The deadlock described makes higher-level commands like git annex sync also block indefinitely when checkout-ing the unlocked branch for any reason.
The deadlock described makes higher-level commands like git annex sync also block indefinitely when checkout-ing the unlocked branch for any reason in these kinds of unlocked repository du to implcit call to the deadlocking git-annex smudge code.
Also, because the filtering is not completely applied, the index is pretty scrambled, its easier to clone the repo and move the annex than fix it, for me at least.
I call the behavior "deadlock" due to the absence of debug log output and low cpu usage on the process when in that state. This seems to indicate some kind of multiprocessing deadlock to me.
I call the behavior "deadlock" due to the absence of debug log output after the 10240 th process and 0% cpu usage on the remaining git and git-annex processes when the bug happens. This seems to indicate some kind of multiprocessing deadlock to me.
### What steps will reproduce the problem?
@ -27,10 +27,13 @@ Here is a minimum set of bash commands that generate the deadlock on my end:
git annex add
git commit -m "add all empty files"
# This will get stuck after around ~10000-20000 processes from Utility.Process in the debug log while the git annex thaws files into unlocked files
# The deadlock seems to happens after outputing the start of a new thawing, ctrl-c seems to be the only end state for this
git annex adjust --unlock --debug
# This will get stuck after 10240 processes from Utility.Process completed in the debug log while git annex thaws files into unlocked files
# The deadlock seems to happens after outputing the start of the last thawing in the queue, ctrl-c seems to be the only end state for this
git annex adjust --unlock --debug 2> ~/unlock-log
# Ctrl-c the command above once the debug output cease to output new lines without exiting.
# This commands output the number of processes ran for the command above, which is 10240 for me
cat ~/unlock-log | grep Perms | wc -l
### What version of git-annex are you using? On what operating system?
@ -64,14 +67,15 @@ Debian Bookworm [Compiled via "building from source on Debian"]
### Please provide any additional information below.
Excerpt of the last lines from the huge debug log:
Excerpt of the last lines from the huge debug log from the git annex adjust above:
[2025-01-16 23:30:27.913022014] (Utility.Process) process [493397] done ExitSuccess
[2025-01-16 23:30:27.91309169] (Annex.Perms) thawing content .git/annex/othertmp/BKQKGR.0/BKQKGR
Given the huge debug log produced, it may be easier to reproduce the bug to have it than copying it here. If wanted, I can generate one as required.
Given the huge debug log produced for this bug, it may be easier to reproduce the bug to have it than copying it here. If wanted, I can generate one as required with the process documented in for the bug repoduction above.
Repeatedly calling this(and ctrl-c it when it inevitably get stuck) seems to eventually unlock the files, but its not really a valid solution in my case.
Repeatedly calling this(and ctrl-c it when it inevitably get stuck) seems to eventually unlock the files ion batches of 10240, but its not really a valid solution in my case.
git annex smudge --update --debug

69
doc/bugs/rsyncurl_without___34____58____34___creates_local_folder_as_remote.mdwn Normal file
View file

@ -0,0 +1,69 @@
### Please describe the problem.
When setting up an (SSH) rsync remote, and _not_ adding the `:` at the end of the hostname, it will create a local folder instead of copying to remote.
```
[joe@laptop]$ git annex initremote myremote type=rsync rsyncurl=ssh.example.com encryption=hybrid keyid=00001111222233334444
[joe@laptop]$ git annex copy . --to myremote
copy metal-arm64.raw (to rpi50...)
ok
copy nixos-gnome-24.11.712512.3f0a8ac25fb6-x86_64-linux.iso (to myremote...)
ok
(recording state in git...)
[joe@laptop]$ ls -l
insgesamt 246792
lrwxrwxrwx. 1 joe joe 204 20. Jan 21:01 metal-arm64.raw -> .git/annex/objects/mG/21/SHA256E-s1306525696--21308f635774faf611ba35c9b04d638aeb7afb1b1c1db949ae65ff81cdafe8b7.raw/SHA256E-s1306525696--21308f635774faf611ba35c9b04d638aeb7afb1b1c1db949ae65ff81cdafe8b7.raw
lrwxrwxrwx. 1 joe joe 204 20. Jan 21:01 nixos-gnome-24.11.712512.3f0a8ac25fb6-x86_64-linux.iso -> .git/annex/objects/fX/g9/SHA256E-s2550136832--da2fe173a279d273bf5a999eafdb618db0642f4a3df95fd94a6585c45082a7f0.iso/SHA256E-s2550136832--da2fe173a279d273bf5a999eafdb618db0642f4a3df95fd94a6585c45082a7f0.iso
drwxr-xr-x. 1 joe joe 12 26. Jan 11:32 ssh.example.com # <---- for me, that was not expected behaviour
```
It might be a feature I don't understand, but because I couldn't find documentation about it, I am leaning towards non-intended behaviour. My assumption would be, that a rsync operation to a local directory is already implemented with the [directory special remote](https://git-annex.branchable.com/special_remotes/directory/).
### What steps will reproduce the problem?
Have a remote rsync server, where you don't need to specify the base directory. In my case [this is done with NixOS and this configuration which uses `rrsync`](https://wiki.nixos.org/wiki/Rsync).
The following configures the rsync remote, and later pushed files to it (so far expected behaviour):
```
git annex initremote myremote type=rsync rsyncurl=ssh.example.com: encryption=hybrid keyid=00001111222233334444
git annex copy . --to myremote
```
This however, doesn't copy to the correct remote, but creates a local folder named `ssh.example.com` in my annexed directory instead (note the missing `:` after the hostname):
```
git annex initremote myremote type=rsync rsyncurl=ssh.example.com encryption=hybrid keyid=00001111222233334444
git annex copy . --to myremote # will copy successfully, BUT
ls -l # shows the folder `ssh.example.com` in my directory with the files in it, the rsync remote is empty
```
### What version of git-annex are you using? On what operating system?
* Fedora 41
```
git-annex version: 10.20240701
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.24.1 bloomfilter-2.0.1.2 crypton-0.34 DAV-1.3.4 feed-1.3.2.1 ghc-9.6.6 http-client-0.7.17 persistent-sqlite-2.13.3.0 torrent-10000.1.3 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 10
```
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I am just now starting to _really_ use git-annex, after following it's development and every blogpost you wrote about it for almost a decade now. Thank you for a tool desperately needed!

12
doc/bugs/rsyncurl_without___34____58____34___creates_local_folder_as_remote/comment_1_b218e908bd2f897415e6d34137f8536b._comment Normal file
View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="matrss"
avatar="http://cdn.libravatar.org/avatar/cd1c0b3be1af288012e49197918395f0"
subject="comment 1"
date="2025-01-27T11:28:43Z"
content="""
I'd say this is intended behavior: I assume that the rsyncurl option is more less passed verbatim to rsync, and rsync can act on both local and remote paths. There is the possibility to use `rsync://` URLs, remote paths via SSH where the host and path are separated by a colon, and local paths.
The rsync special remote with local paths behaves a bit differently than the directory special remote, namely the rsyncurl is remembered (e.g. for autoenable) while the directory special remote does not remember the directory. There can be use-cases for both.
Besides, most of the time I think one would want to specify a remote directory with rsync, in which case the colon is necessary anyway.
"""]]

10
doc/forum/Deduplication_between_two_repos_on_the_same_drive__63__/comment_1_464adfa71d322249dfed4ba65c24995d._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="Atemu"
avatar="http://cdn.libravatar.org/avatar/6ac9c136a74bb8760c66f422d3d6dc32"
subject="comment 1"
date="2025-01-26T02:36:51Z"
content="""
It will not realise this.
Why do you have separate repos for this though? You can absolutely just use a non-plain git repo for synchronisation purposes too.
"""]]

8
doc/forum/Deduplication_between_two_repos_on_the_same_drive__63__/comment_2_e4cd3108130efbfa796e1ff5e5f55116._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="jnkl"
avatar="http://cdn.libravatar.org/avatar/2ab576f3bf2e0d96b1ee935bb7f33dbe"
subject="comment 2"
date="2025-01-26T13:09:04Z"
content="""
Sorry, I am new to git. I thought pushes are only allowed to bare repositories. Am I wrong?
"""]]

8
doc/forum/Deduplication_between_two_repos_on_the_same_drive__63__/comment_3_f2069c83af180c7026700a102a528827._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Atemu"
avatar="http://cdn.libravatar.org/avatar/6ac9c136a74bb8760c66f422d3d6dc32"
subject="comment 3"
date="2025-01-26T13:30:10Z"
content="""
git-annex synchronises branch state via the `synced/branchnamehere` branches. The actual checked out branch in the worktree will only be updated when you run a `merge` or `sync` in the worktree.
"""]]

26
doc/forum/How_to_figure_out_why_files_aren__39__t_being_dropped__63__/comment_2_ba87cf91217ba01415ff55d33550a75b._comment Normal file
View file

@ -0,0 +1,26 @@
[[!comment format=mdwn
username="Atemu"
avatar="http://cdn.libravatar.org/avatar/6ac9c136a74bb8760c66f422d3d6dc32"
subject="comment 2"
date="2025-01-26T02:54:18Z"
content="""
My issue apparently had to do with numcopies? I first passed `--numcopies 2` because I was curious but it didn't change anything. Then I passed `--numcopies 1` and it immediately dropped all the files as I'd have expected it to at `numcopies=3`. Running another sync without `--numcopies` didn't attempt to pull in the dropped files either.
This smells like a bug? If numcopies was actually violated, it should attempt to correct that again, right? (All files were available from a connected repo.)
Here are the numcopies stats from `git annex info .`:
```
numcopies stats:
numcopies +1: 1213
numcopies +0: 25310
```
Some more background: I have a bunch of drives that are offline that I have set to be trusted. One repo on my NAS is online at all times and semitrusted.
I have two offline groups: `cold` and `lukewarm`. All drives in those groups are trusted.
It's weird that it didn't work with 2 but did work with 1. This leads me to believe it could have been due to the one repo being online while the others are offline and trusted; acting more like mincopies. Was behaviour changed in this regard recently?
I'd still like to know how to debug wanted expressions too though.
"""]]