From 9104e64d3e79134ef6704e9c02702de270218e50 Mon Sep 17 00:00:00 2001 From: mih Date: Tue, 11 Jan 2022 18:59:36 +0000 Subject: [PATCH 01/17] Added a comment: Thanks! --- .../comment_2_fac648fb7550f0c3c433ce423e3ea7f6._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all/comment_2_fac648fb7550f0c3c433ce423e3ea7f6._comment diff --git a/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all/comment_2_fac648fb7550f0c3c433ce423e3ea7f6._comment b/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all/comment_2_fac648fb7550f0c3c433ce423e3ea7f6._comment new file mode 100644 index 0000000000..ed38f6ac40 --- /dev/null +++ b/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all/comment_2_fac648fb7550f0c3c433ce423e3ea7f6._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="mih" + avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd" + subject="Thanks!" + date="2022-01-11T18:59:36Z" + content=""" +That was the missing piece for me. Thank you! +"""]] From d427afb3471456cdc4c939ae81e97fdde051da60 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Tue, 11 Jan 2022 17:04:25 -0400 Subject: [PATCH 02/17] v9-locking branch still wip --- .../comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment | 2 +- doc/todo/v9_changes.mdwn | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment index a1b85d21ee..de67df2e7c 100644 --- a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment +++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment @@ -3,7 +3,7 @@ subject="""comment 8""" date="2022-01-11T18:48:15Z" content=""" -This is implemented in the `v9-locking` branch, +This is being implemented in the `v9-locking` branch, which will only get merged when v9 happens, whenever that is. See [[todo/v9_changes]]. """]] diff --git a/doc/todo/v9_changes.mdwn b/doc/todo/v9_changes.mdwn index b4aa21b4d0..87ac7b39f7 100644 --- a/doc/todo/v9_changes.mdwn +++ b/doc/todo/v9_changes.mdwn @@ -12,7 +12,7 @@ could change and if it does, these things could be included. Note that windows already uses a separate lock file. - This is implemented in git branch `v9-locking`. + This is being implemented in git branch `v9-locking`. * Possibly enable `git-annex filter-process` by default. If the tradeoffs seem worth it. From 3993421d36e05daef5337d531465b080aae3b645 Mon Sep 17 00:00:00 2001 From: anarcat Date: Wed, 12 Jan 2022 15:56:25 +0000 Subject: [PATCH 03/17] Added a comment: indeed. --- ..._2_5be34b5ffaee39013d24daf83a8dd437._comment | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/bugs/git-annex-shell_fails_to_run_true/comment_2_5be34b5ffaee39013d24daf83a8dd437._comment diff --git a/doc/bugs/git-annex-shell_fails_to_run_true/comment_2_5be34b5ffaee39013d24daf83a8dd437._comment b/doc/bugs/git-annex-shell_fails_to_run_true/comment_2_5be34b5ffaee39013d24daf83a8dd437._comment new file mode 100644 index 0000000000..a9d422e054 --- /dev/null +++ b/doc/bugs/git-annex-shell_fails_to_run_true/comment_2_5be34b5ffaee39013d24daf83a8dd437._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="anarcat" + avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7" + subject="indeed." + date="2022-01-12T15:56:25Z" + content=""" +the problem was that i had a `~/git-shell-commands/rsync` overriding the rsync command to try to restrict *that*, since i'm using a gcrypt remote... it was clunky and badly designed and, ultimately, insecure. i replaced it with `rrsync /home/anarcat/offsite/` but even that is not the best either, as the script says it \"assumes someone will not subvert the rsync protocol\" which hardly seems reassuring. + +i also had to change my remotes to follow the \"chroot\" of sorts: + + git config remote.offsite-annex.annex-rsyncurl anarcat@remote-annex:Videos.annex/ + git remote set-url offsite-git gcrypt::rsync://anarcat@remote-annex:Videos.git/ + +Running `git fetch` by hand convinced me the problem wasn't git-annex related... From there I reread the git-annex-shell and git-shell commands and noticed that bit about the obscure `git-shell-commands` directory, and found that jem. I even had an error in `journalctl -f` that confirm the problem, something i overlooked as well... + +so this indeed seems like a problem on my end. sorry for the delay and the noise! +"""]] From 658a896869b2c09bf7dde124a28b269e1e855cb3 Mon Sep 17 00:00:00 2001 From: anarcat Date: Wed, 12 Jan 2022 16:24:17 +0000 Subject: [PATCH 04/17] --- doc/bugs/git-annex-shell_fails_to_run_true.mdwn | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/bugs/git-annex-shell_fails_to_run_true.mdwn b/doc/bugs/git-annex-shell_fails_to_run_true.mdwn index bbace3f830..230a94a340 100644 --- a/doc/bugs/git-annex-shell_fails_to_run_true.mdwn +++ b/doc/bugs/git-annex-shell_fails_to_run_true.mdwn @@ -162,3 +162,5 @@ Key part above: ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) I'm a long time git-annex user (early adopter, even?) and git-annex is generally serving my uses very well, both as a backup and archival system. The encryption features, however, are a bit more obscure than my tolerance level right now. Thankfully, there is the [[special_remotes/borg/]] I'm thinking of using to replace all this complexity in the future... + +-- [[done]], problem was with a configuration in the gcrypt remote sandbox. -- [[anarcat]] From 63851bfec485021bad9427e6b27f63396da83393 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 12 Jan 2022 13:19:06 -0400 Subject: [PATCH 05/17] complication --- ..._f74642e1dfc4e533b9582e5a13a16898._comment | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_9_f74642e1dfc4e533b9582e5a13a16898._comment diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_9_f74642e1dfc4e533b9582e5a13a16898._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_9_f74642e1dfc4e533b9582e5a13a16898._comment new file mode 100644 index 0000000000..f72e43dfe8 --- /dev/null +++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_9_f74642e1dfc4e533b9582e5a13a16898._comment @@ -0,0 +1,41 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 9""" + date="2022-01-12T16:29:40Z" + content=""" +While this is mostly implemented in the branch, the upgrade to it has a +serious danger point. + +If a long-running git-annex process, eg a large drop, is running during the +upgrade, then it will keep on using the current locking method. Meanwhile, +other processes run after the upgrade will use the new locking method. So +this could cause data loss: Old git-annex locks a file to drop it, and at +the same time new git-annex-shell is used to lock the same file, to prevent +it from being dropped. + +Avoiding that seems to require a way to make sure there are no running +git-annex processes when performing a repository upgrade. (Which would be +nice in general but has somehow not been necessary until now.) + +One way to do it would be to have a shared lock file that all git-annex +processes hold while they are running. And upgrade takes an exclusive lock. +That locking would need to be implemented first and somehow be known +that any git-annex process that is running is using it, before performing +the repository upgrade. Ideally without taking years in between to wait +for all git-annex binaries to be upgraded. + +I suppose that git-annex v9 could ship with that added lock file, and not +upgrade the repository to v9 immediately. Instead, stat the lock file, and +only when its ctime is sufficiently old that it seems safe to assume any +running git-annex process would be using it, do the repository upgrade. Eg +after 3 months or so or perhaps when the ctime is older than the last +reboot. But this would not avoid problems if an older git-annex +version was also used in the same repository as the new version. + +Or git-annex upgrade could hunt for other running git-annex processes that +are using the repository and refuse to perform the v9 upgrade. But that is +hard because a processes's cwd is not necessarily inside the repository +it's using (eg a remote). It would have to look for git child processes +of git-annex processes that are using the repository, such as git cat-file. +Also network filesystems would be a problem. +"""]] From 0ba7f2ec913aa12b8f11e167a24a13c49c1888f4 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 12 Jan 2022 15:25:35 -0400 Subject: [PATCH 06/17] so close --- ..._2e61ccc41839b8da70ad26d31e40ad74._comment | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_10_2e61ccc41839b8da70ad26d31e40ad74._comment diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_10_2e61ccc41839b8da70ad26d31e40ad74._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_10_2e61ccc41839b8da70ad26d31e40ad74._comment new file mode 100644 index 0000000000..95809e9d9b --- /dev/null +++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_10_2e61ccc41839b8da70ad26d31e40ad74._comment @@ -0,0 +1,25 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 10""" + date="2022-01-12T19:14:10Z" + content=""" +This would almost work: + +Continue taking a shared lock of the content +file when locking to prevent dropping. That does not need write access, +only an exclusive lock does, so the content file can have its write bit +removed. Also lock the new lock file, with a shared lock to prevent +dropping, or an exclusive when dropping. + +The old git-annex version, when dropping, will fail to exclusively lock the +content file, either because it's not writable, or because of a shared +lock intended to prevent dropping. So a git-annex drop that was in progress +may start to fail, but it will not lose any data. + +Problem: The old git-annex version, when locking to prevent dropping +(eg git-annex move --from remote), +will take the shared lock of the content file. If the new git-annex version +is locking to drop, it will also take the shared lock of the content file, +followed by the exlusive lock of the new lock file. So the old git-annex will +not be able to prevent the new git-annex from dropping. +"""]] From 6b435719a03da66071be9e131dd9792115c5781c Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Wed, 12 Jan 2022 19:33:16 +0000 Subject: [PATCH 07/17] Added a comment --- .../comment_3_ab590b41983d84c8fe7d5dc7276689b8._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist/comment_3_ab590b41983d84c8fe7d5dc7276689b8._comment diff --git a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist/comment_3_ab590b41983d84c8fe7d5dc7276689b8._comment b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist/comment_3_ab590b41983d84c8fe7d5dc7276689b8._comment new file mode 100644 index 0000000000..2023ad7858 --- /dev/null +++ b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist/comment_3_ab590b41983d84c8fe7d5dc7276689b8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 3" + date="2022-01-12T19:33:15Z" + content=""" +Still checking but so far looks good with `8.20211231+git69-ga55fc567c-1~ndall+1` -- thank you! +"""]] From 8add0ec60e96f8ec51f471a7a7c69d5089f43a73 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 12 Jan 2022 15:54:58 -0400 Subject: [PATCH 08/17] liking this solution pretty well --- ..._e9af4784ffb84b49b6faaa21aa7c8efc._comment | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment new file mode 100644 index 0000000000..2ddaccd98e --- /dev/null +++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment @@ -0,0 +1,28 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 11""" + date="2022-01-12T19:40:35Z" + content=""" +This is going to need two repository version bumps: + +v9: Add the upgrade lock file, and all git-annex processes take a shared +lock to avoid the repository being upgraded out from under them. +Upgrade is skipped when assistant is running. + +v10: Skipped until the upgrade lock file is of a certain age. Take upgrade +lock before increasing annex.version. +In v10, stop locking content files and lock separate lock files. + +This way, an old version of git-annex cannot be used in the v9 repository, +and so the v10 upgrade only needs to worry about any git-annex processes +that were started in v8. + +The age could be eg 1 month, which assumes that no old git-annex process +like `git-annex move --to remote` is still running after that long. +Of course, that is still an assumption, but it can be pushed out as long as +it takes to feel comfortable with it. Maybe 1 year? + +`git-annex upgrade --version=10` could be available to speed up that +upgrade. The user would be responsible for making sure there are no such +old git-annex processes running, so that might need --force. +"""]] From b15ac539670cb5a92f01f267cfc3d8d1272ac8d5 Mon Sep 17 00:00:00 2001 From: "beryllium@5bc3c32eb8156390f96e363e4ba38976567425ec" Date: Thu, 13 Jan 2022 03:59:44 +0000 Subject: [PATCH 09/17] --- ...om_.git_under_Windows__44___get_fails.mdwn | 292 ++++++++++++++++++ 1 file changed, 292 insertions(+) create mode 100644 doc/bugs/Cloned_from_.git_under_Windows__44___get_fails.mdwn diff --git a/doc/bugs/Cloned_from_.git_under_Windows__44___get_fails.mdwn b/doc/bugs/Cloned_from_.git_under_Windows__44___get_fails.mdwn new file mode 100644 index 0000000000..188efeaf2e --- /dev/null +++ b/doc/bugs/Cloned_from_.git_under_Windows__44___get_fails.mdwn @@ -0,0 +1,292 @@ +### Please describe the problem. + +Under Windows (not tested in Unix), if an annex is cloned from the .git directory, git annex get fails and is different to cloning from workdir + +### What steps will reproduce the problem? + +[[!format sh """ +shaddy@COMPUTER-W10 U:\Temp +> git annex version +git-annex version: 8.20211118-g2e2d35869 +build flags: Assistant Webapp Pairing TorrentParser MagicMime Feeds Testsuite S3 WebDAV +dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-8.10.7 http-client-0.7.9 persistent-sqlite-2.13.0.3 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.1.2 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external +operating system: mingw32 x86_64 +supported repository versions: 8 +upgrade supported from repository versions: 2 3 4 5 6 7 + +shaddy@COMPUTER-W10 U:\Temp +> dir + Volume in drive U has no label. + Volume Serial Number is D684-6493 + + Directory of U:\Temp + +13/01/2022 01:42 PM . +13/01/2022 01:42 PM .. +10/04/2020 07:36 PM 14,572,000 vc_redist.x64.exe + 1 File(s) 14,572,000 bytes + 2 Dir(s) 21,238,497,280 bytes free + +shaddy@COMPUTER-W10 U:\Temp +> git init --separate-git-dir separategitmaster.git separategitmaster +Initialized empty Git repository in U:/Temp/separategitmaster.git/ + +shaddy@COMPUTER-W10 U:\Temp +> cd separategitmaster + +shaddy@COMPUTER-W10 U:\Temp\separategitmaster +> git annex init master +init master + Detected a filesystem without fifo support. + + Disabling ssh connection caching. + + Detected a crippled filesystem. + + Entering an adjusted branch where files are unlocked as this filesystem does not support locked files. + +Switched to branch 'adjusted/master(unlocked)' +ok +(recording state in git...) + +shaddy@COMPUTER-W10 U:\Temp\separategitmaster +> copy ..\vc_redist.x64.exe . + 1 file(s) copied. + +shaddy@COMPUTER-W10 U:\Temp\separategitmaster +> git annex add vc_redist.x64.exe +add vc_redist.x64.exe +ok +(recording state in git...) + +shaddy@COMPUTER-W10 U:\Temp\separategitmaster +> git commit -m vc_redist.x64.exe +[adjusted/master(unlocked) a32d7ba] vc_redist.x64.exe + 1 file changed, 1 insertion(+) + create mode 100755 vc_redist.x64.exe + +shaddy@COMPUTER-W10 U:\Temp\separategitmaster +> cd .. + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitslave +> REM note, cloning from workdir, not .git + +shaddy@COMPUTER-W10 U:\Temp +> git clone separategitmaster separategitslave +Cloning into 'separategitslave'... +done. + +shaddy@COMPUTER-W10 U:\Temp +> cd separategitslave + +shaddy@COMPUTER-W10 U:\Temp\separategitslave +> dir + Volume in drive U has no label. + Volume Serial Number is D684-6493 + + Directory of U:\Temp\separategitslave + +13/01/2022 01:45 PM . +13/01/2022 01:45 PM .. +13/01/2022 01:45 PM 108 vc_redist.x64.exe + 1 File(s) 108 bytes + 2 Dir(s) 21,209,219,072 bytes free + +shaddy@COMPUTER-W10 U:\Temp\separategitslave +> git config user.name "Shaddy Baddah" + +shaddy@COMPUTER-W10 U:\Temp\separategitslave +> git annex init slave +init slave + Detected a filesystem without fifo support. + + Disabling ssh connection caching. + + Detected a crippled filesystem. +ok +(recording state in git...) + +shaddy@COMPUTER-W10 U:\Temp\separategitslave +> git annex sync +commit +On branch adjusted/master(unlocked) +Your branch is up to date with 'origin/adjusted/master(unlocked)'. + +nothing to commit, working tree clean +ok +pull origin +ok +push origin +Enumerating objects: 9, done. +Counting objects: 100% (9/9), done. +Delta compression using up to 2 threads +Compressing objects: 100% (5/5), done. +Writing objects: 100% (6/6), 718 bytes | 359.00 KiB/s, done. +Total 6 (delta 0), reused 0 (delta 0), pack-reused 0 +To U:/Temp/separategitmaster + * [new branch] master -> synced/master + * [new branch] git-annex -> synced/git-annex +ok + +shaddy@COMPUTER-W10 U:\Temp\separategitslave +> git annex get vc_redist.x64.exe +get vc_redist.x64.exe (from origin...) +ok +(recording state in git...) + +shaddy@COMPUTER-W10 U:\Temp\separategitslave +> REM All ok. But now lets clone from the .git dir + +shaddy@COMPUTER-W10 U:\Temp\separategitslave +> cd .. + +shaddy@COMPUTER-W10 U:\Temp +> git init --separate-git-dir separatetake2gitmaster.git separatetake2gitmaster +Initialized empty Git repository in U:/Temp/separatetake2gitmaster.git/ + +shaddy@COMPUTER-W10 U:\Temp +> cd separatetake2gitmaster + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster +> git annex init master +init master + Detected a filesystem without fifo support. + + Disabling ssh connection caching. + + Detected a crippled filesystem. + + Entering an adjusted branch where files are unlocked as this filesystem does not support locked files. + +Switched to branch 'adjusted/master(unlocked)' +ok +(recording state in git...) + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster +> copy ..\vc_redist.x64.exe . + 1 file(s) copied. + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster +> git annex add vc_redist.x64.exe +add vc_redist.x64.exe +ok +(recording state in git...) + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster +> git commit -m vc_redist.x64.exe +git-annex.exe: .\vc_redist.x64.exe: DeleteFile "\\\\?\\U:\\Temp\\separatetake2gitmaster\\vc_redist.x64.exe": permission denied (The process cannot access the file because it is being used by another process.) +error: external filter 'git-annex smudge --clean -- %f' failed 1 +error: external filter 'git-annex smudge --clean -- %f' failed +[adjusted/master(unlocked) 50bb6c7] vc_redist.x64.exe + 1 file changed, 1 insertion(+) + create mode 100755 vc_redist.x64.exe + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster +> REM unfortunate Windows gotcha, but I believe it doesnt effect the scenario + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster +> dir + Volume in drive U has no label. + Volume Serial Number is D684-6493 + + Directory of U:\Temp\separatetake2gitmaster + +13/01/2022 01:47 PM . +13/01/2022 01:47 PM .. +10/04/2020 07:36 PM 14,572,000 vc_redist.x64.exe + 1 File(s) 14,572,000 bytes + 2 Dir(s) 21,149,949,952 bytes free + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster +> cd ..\ + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster +> REM cloning from the .git this time + +shaddy@COMPUTER-W10 U:\Temp +> git clone separatetake2gitmaster.git separatetake2gitslave +Cloning into 'separatetake2gitslave'... +done. + +shaddy@COMPUTER-W10 U:\Temp +> cd separatetake2gitslave + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitslave +> git annex init slave +init slave + Detected a filesystem without fifo support. + + Disabling ssh connection caching. + + Detected a crippled filesystem. +ok +(recording state in git...) + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitslave +> dir + Volume in drive U has no label. + Volume Serial Number is D684-6493 + + Directory of U:\Temp\separatetake2gitslave + +13/01/2022 01:48 PM . +13/01/2022 01:48 PM .. +13/01/2022 01:48 PM 108 vc_redist.x64.exe + 1 File(s) 108 bytes + 2 Dir(s) 21,149,868,032 bytes free + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitslave +> git annex sync +commit +On branch adjusted/master(unlocked) +Your branch is up to date with 'origin/adjusted/master(unlocked)'. + +nothing to commit, working tree clean +ok +pull origin +ok +push origin +Enumerating objects: 9, done. +Counting objects: 100% (9/9), done. +Delta compression using up to 2 threads +Compressing objects: 100% (5/5), done. +Writing objects: 100% (6/6), 719 bytes | 719.00 KiB/s, done. +Total 6 (delta 0), reused 0 (delta 0), pack-reused 0 +To U:/Temp/separatetake2gitmaster.git + * [new branch] master -> synced/master + * [new branch] git-annex -> synced/git-annex +ok + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitslave +> git annex get vc_redist.x64.exe +get vc_redist.x64.exe + Unable to access these remotes: origin + + No other repository is known to contain the file. +failed +get: 1 failed + +shaddy@COMPUTER-W10 U:\Temp\separatetake2gitslave +> REM this is a confusing situation and repeatable +# End of transcript or log. +"""]] + +### What version of git-annex are you using? On what operating system? + +shaddy@COMPUTER-W10 U:\Temp +> git annex version +git-annex version: 8.20211118-g2e2d35869 + +### Please provide any additional information below. + +[[!format sh """ +Note, in my example, I'm using a .git directory that has been "separated" from its workdir. + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Yes. Git annex is very good. I acknowledge that the way I use it can be unconventional. From 06e34313482f6f84be8837dc5021e24f451f00c6 Mon Sep 17 00:00:00 2001 From: Lukey Date: Thu, 13 Jan 2022 18:08:25 +0000 Subject: [PATCH 10/17] --- doc/tips/using_nested_git_repositories.mdwn | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 doc/tips/using_nested_git_repositories.mdwn diff --git a/doc/tips/using_nested_git_repositories.mdwn b/doc/tips/using_nested_git_repositories.mdwn new file mode 100644 index 0000000000..6ebd9e64c8 --- /dev/null +++ b/doc/tips/using_nested_git_repositories.mdwn @@ -0,0 +1,7 @@ +Using nested git repositories in git is not possible and thus this also applies to git-annex. However, here is a good workaround that I found: + +Rename the `.git` directory of the nested repo to `dotgit` (or similar), `git annex add` it and then create a symbolic link from `.git` to `dotgit`. It's important that the link is created only after the nested repo has been `git annex add`'ed. Also, the link needs to be created manually on each clone. Finally you'll need to hide the `dotgit` directory from the nested repo itself by adding `/dotgit` to `dotgit/info/exclude`. + + mv nested/.git nested/dotgit; echo "/dotgit" >>nested/dotgit/info/exclude + git annex add nested; git commit -m "add nested" + cd nested; ln -s dotgit .git # needs to be done on every clone From 1c0e04c83c090eb4caeaf7c5d8bd1cf403de655e Mon Sep 17 00:00:00 2001 From: Lukey Date: Thu, 13 Jan 2022 18:10:22 +0000 Subject: [PATCH 11/17] Added a comment --- .../comment_2_6ede677f8113fcd6215ecd68de237f06._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X/comment_2_6ede677f8113fcd6215ecd68de237f06._comment diff --git a/doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X/comment_2_6ede677f8113fcd6215ecd68de237f06._comment b/doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X/comment_2_6ede677f8113fcd6215ecd68de237f06._comment new file mode 100644 index 0000000000..260ea540e1 --- /dev/null +++ b/doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X/comment_2_6ede677f8113fcd6215ecd68de237f06._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 2" + date="2022-01-13T18:10:22Z" + content=""" +I found an even better workaround, see [[tips/using_nested_git_repositories/]] +"""]] From c6c7f7b444f67f1cff9827a75693e3d24830432d Mon Sep 17 00:00:00 2001 From: Lukey Date: Thu, 13 Jan 2022 18:19:32 +0000 Subject: [PATCH 12/17] Added a comment --- .../comment_6_4a5ddb1980b9ba0378531a8d4fb93523._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/Storing_git_repos_in_git-annex/comment_6_4a5ddb1980b9ba0378531a8d4fb93523._comment diff --git a/doc/forum/Storing_git_repos_in_git-annex/comment_6_4a5ddb1980b9ba0378531a8d4fb93523._comment b/doc/forum/Storing_git_repos_in_git-annex/comment_6_4a5ddb1980b9ba0378531a8d4fb93523._comment new file mode 100644 index 0000000000..d0a162c77a --- /dev/null +++ b/doc/forum/Storing_git_repos_in_git-annex/comment_6_4a5ddb1980b9ba0378531a8d4fb93523._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 6" + date="2022-01-13T18:19:31Z" + content=""" +The main problem is that the git developers stubbornly refuse to support nested git repos, so it's impossible for git-annex to support it. However I found a good workaround, see [[tips/using_nested_git_repositories/]]. +"""]] From e17c155f18b3f45f96c92fbe40348d09ebe33ddc Mon Sep 17 00:00:00 2001 From: ainohzoa Date: Fri, 14 Jan 2022 00:37:24 +0000 Subject: [PATCH 13/17] --- ...git_status__96___is_slow_after_update.mdwn | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update.mdwn diff --git a/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update.mdwn b/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update.mdwn new file mode 100644 index 0000000000..37ba9a94f0 --- /dev/null +++ b/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update.mdwn @@ -0,0 +1,74 @@ +Hi, if anyone can suggest some steps to take to diagnose a sudden performance problem in my git-annex repo, I'd appreciate it. + +I have been using git annex for several years on this same repo and performance has never been a problem. +I recently did a OS update of all my packages (arch linux), which updated git annex, and now performance is really bad whenever I run `git status` or `git commit` in the repo (performance is fine in git repos without git annex). For an example of how bad the performance is, I first noticed the problem when trying to make a commit, and I waited 20 minutes before cancelling with Ctrl+C. The update took me from package version "8.20210310-16" to "8.20210803-63". After noticing the problem I updated again to package version "8.20210803-81", since I was hoping there might be a fix in the most recent version, but that didn't resolve it. Those versions might be specific to arch linux, so if that's the case and anybody needs the true versions let me know and I can try to figure out which git-annex commit they are at. + +To try to get a sense of what is going on, I started by running `GIT_TRACE_PERFORMANCE=1 git commit`. There seems to be one command that git is running that has a non-trivial time difference between other commands, and its + + 18:14:31.968248 trace.c:487 performance: 247.576906378 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' + +The time difference between this command and the next is over 1 second, and it seems that `git commit` is running this type of command many times. I posted a sample of the full output below. + +When searching the web for that git command, I came across this conversation: https://git-annex.branchable.com/todo/speed_up_git_annex_sync_--content_--all/ + +Could that be related to my issue? + +What should I do next to figure out what the problem is? + +I posted the output of `git annex version` and `git annex info` below in case that is helpful. + +Thanks! + +---------- + +`git annex version`: + + git-annex version: 8.20210803-g9cae7c5bb + build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV + dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-9.0.2 http-client-0.7.9 persistent-sqlite-2.13.0.3 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.1.2 + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* + remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external + operating system: linux x86_64 + supported repository versions: 8 + upgrade supported from repository versions: 0 1 2 3 4 5 6 7 + local repository version: 8 + + +`git annex info` (omitting repository info): + + available local disk space: 146.18 gigabytes (+1 megabyte reserved) + local annex keys: 2120 + local annex size: 2.59 gigabytes + annexed files in working tree: 1945 + size of annexed files in working tree: 2.55 gigabytes + bloom filter size: 32 mebibytes (0.4% full) + backend usage: + SHA256E: 1945 + + +sample of the output of `GIT_TRACE_PERFORMANCE=1 git commit`: + + 18:14:31.949865 trace.c:487 performance: 0.031945775 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' --buffer + 18:14:31.950372 trace.c:487 performance: 0.031692879 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch=%(objectname) %(objecttype) %(objectsize)' --buffer + 18:14:31.963858 trace.c:487 performance: 0.007070777 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs hash-object --stdin-paths --no-filters + 18:14:31.967006 trace.c:487 performance: 247.578879398 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch + 18:14:31.968248 trace.c:487 performance: 247.576906378 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' + 18:14:33.217226 trace.c:487 performance: 0.000427304 s: git command: git config --null --list + 18:14:33.244772 read-cache.c:2398 performance: 0.011644939 s: read cache .git/index + 18:14:33.245219 read-cache.c:2398 performance: 0.001136355 s: read cache .git/index + 18:14:33.263160 trace.c:487 performance: 0.001024382 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs show-ref --hash refs/annex/last-index + 18:14:33.274091 read-cache.c:2398 performance: 0.001037856 s: read cache .git/index + 18:14:33.284793 unpack-trees.c:1802 performance: 0.000024725 s: traverse_trees + 18:14:33.284872 unpack-trees.c:418 performance: 0.000001050 s: check_updates + 18:14:33.284917 unpack-trees.c:1892 performance: 0.000349269 s: unpack_trees + 18:14:33.289148 diff-lib.c:629 performance: 0.004585392 s: diff-index + 18:14:33.289220 trace.c:487 performance: 0.017392572 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs -c filter.annex.smudge= -c filter.annex.clean= -c diff.external= diff d62ecb841c32ff0d62df4f8ca7cb0567616cf415 --staged --raw -z --no-abbrev -G/annex/objects/ --no-renames --ignore-submodules=all --no-ext-diff + 18:14:33.294517 trace.c:487 performance: 0.025457813 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' --buffer + 18:14:33.297215 trace.c:487 performance: 0.024423869 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch=%(objectname) %(objecttype) %(objectsize)' --buffer + 18:14:33.303299 trace.c:487 performance: 0.000746262 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs hash-object --stdin-paths --no-filters + 18:14:33.305448 trace.c:487 performance: 0.076031969 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch + 18:14:33.308580 trace.c:487 performance: 0.075975445 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' + 18:14:34.537129 trace.c:487 performance: 0.000424618 s: git command: git config --null --list + 18:14:34.570720 read-cache.c:2398 performance: 0.008743858 s: read cache .git/index + 18:14:34.573279 read-cache.c:2398 performance: 0.001075431 s: read cache .git/index + From 34e3823602b02ec6915b7d0e8196452da7f9c0e8 Mon Sep 17 00:00:00 2001 From: "https://christian.amsuess.com/chrysn" Date: Fri, 14 Jan 2022 13:02:37 +0000 Subject: [PATCH 14/17] Added a comment: nested git repositories are git submodules --- ...omment_1_cd7f673de1c39d2f37d9958357d5e72d._comment | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 doc/tips/using_nested_git_repositories/comment_1_cd7f673de1c39d2f37d9958357d5e72d._comment diff --git a/doc/tips/using_nested_git_repositories/comment_1_cd7f673de1c39d2f37d9958357d5e72d._comment b/doc/tips/using_nested_git_repositories/comment_1_cd7f673de1c39d2f37d9958357d5e72d._comment new file mode 100644 index 0000000000..92c56b1916 --- /dev/null +++ b/doc/tips/using_nested_git_repositories/comment_1_cd7f673de1c39d2f37d9958357d5e72d._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="https://christian.amsuess.com/chrysn" + nickname="chrysn" + avatar="http://christian.amsuess.com/avatar/c6c0d57d63ac88f3541522c4b21198c3c7169a665a2f2d733b4f78670322ffdc" + subject="nested git repositories are git submodules" + date="2022-01-14T13:02:36Z" + content=""" +Using nested git repositories is well possible; if they are checked in they are called submodules, otherwise they just sit there unadded. + +Apart from some [[odd quirx you never run into in normal operation|bugs/creating dot-git-as-symlink workaround drops worktree configuration from submodules]], submodules work fine also with git-annex. +"""]] From b08fb6fec462a0041d59ce9504423301db1515e1 Mon Sep 17 00:00:00 2001 From: ainohzoa Date: Fri, 14 Jan 2022 21:10:13 +0000 Subject: [PATCH 15/17] Added a comment: Now it's working normally again --- .../comment_1_68429f9e44ce24250c608da8ffffe643._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update/comment_1_68429f9e44ce24250c608da8ffffe643._comment diff --git a/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update/comment_1_68429f9e44ce24250c608da8ffffe643._comment b/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update/comment_1_68429f9e44ce24250c608da8ffffe643._comment new file mode 100644 index 0000000000..fc65820127 --- /dev/null +++ b/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update/comment_1_68429f9e44ce24250c608da8ffffe643._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="ainohzoa" + avatar="http://cdn.libravatar.org/avatar/0d6a2dbd95f6c4f410cc41d32beaebe9" + subject="Now it's working normally again" + date="2022-01-14T21:10:13Z" + content=""" +After running a `git status` today (the first time running it was slow as usual) the problem seems to have gone away. Perhaps it just needed to rebuild some kind of cache or something. I believe that I had waited for a `git status` command to completely finish before, so maybe it takes several tries to resolve this performance problem? Or possibly I'm misremembering and I never actually waited for the `git status` command to fully finish. I'm baffled by this, but happy that it's back to normal. I'll post an update if the problem comes back. +"""]] From 1723d4e88a8b87c5ad779c9b7d54bf7fe1c3e27e Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 19 Jan 2022 11:49:37 -0400 Subject: [PATCH 16/17] close --- ..._58___pidlock__58___getFileStatus__58___does_not_exist.mdwn | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn index 1bfd4b6859..758406251a 100644 --- a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn +++ b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn @@ -18,4 +18,5 @@ conda linux nodep (standalone) build 8.20211012-geb95ed486 and then with another [[!meta author=yoh]] [[!tag projects/datalad]] -[[!tag moreinfo]] +> [[done]] provisionally, followup if further testing shows otherwise +> --[[Joey]] From 2cb79146995325a73dfd3ec540c6d9262c341c7f Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 19 Jan 2022 11:56:00 -0400 Subject: [PATCH 17/17] commeent --- ..._63f30b652c0cbdb0acf6745891f9f09e._comment | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 doc/tips/using_nested_git_repositories/comment_2_63f30b652c0cbdb0acf6745891f9f09e._comment diff --git a/doc/tips/using_nested_git_repositories/comment_2_63f30b652c0cbdb0acf6745891f9f09e._comment b/doc/tips/using_nested_git_repositories/comment_2_63f30b652c0cbdb0acf6745891f9f09e._comment new file mode 100644 index 0000000000..bc6b5fbdd7 --- /dev/null +++ b/doc/tips/using_nested_git_repositories/comment_2_63f30b652c0cbdb0acf6745891f9f09e._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2022-01-19T15:51:17Z" + content=""" +I agree, submodules are the usual way to nest git repositories, and will +more or less just work with git-annex. + +I think that the author of this tip is wanting to version control the +contents of `.git` itself. Eg, to version control `.git/config` and +`.git/hooks/`. + +One problem with this approach is that when the outer repository has +"dotgit/annex/objects/` files added to it, running `git-annex drop` inside +the nested git repository will drop the content, but the outer repository +will still contain a copy too. You would have to use `git-annex unused` +to eventually clean up those copies. And it stores 2 copies of every +annexed file to use it this way. +"""]]