diff --git a/doc/internals/lockdown/comment_3_caf6d5318703d188a2135737093d8323._comment b/doc/internals/lockdown/comment_3_caf6d5318703d188a2135737093d8323._comment new file mode 100644 index 0000000000..7ddbca9a40 --- /dev/null +++ b/doc/internals/lockdown/comment_3_caf6d5318703d188a2135737093d8323._comment @@ -0,0 +1,421 @@ +[[!comment format=mdwn + username="nick.guenther@e418ed3c763dff37995c2ed5da4232a7c6cee0a9" + nickname="nick.guenther" + avatar="http://cdn.libravatar.org/avatar/9e85c6ca61c3f877fef4f91c2bf6e278" + subject="How to disable lockdown in bare repos?" + date="2022-05-15T21:30:50Z" + content=""" +I've set up a project server for my team with annexes in most repos. I'm using [gitolite](https://gitolite.com) with its [git-annex-shell](https://github.com/sitaramc/gitolite/blob/master/src/commands/git-annex-shell) plugin. It's been going well for a year, and my team finds git-annex very useful for managing our large projects, so we have a large debt to you for that :) + +## Problem + +But when my users delete repos, the repos aren't fully deleted because any `annex/objects/*/*/SHA256-*/SHA256-*` file is locked down. + + +### Gitolite + +For example: + +
test-git-annex-write + +``` + +test-gitea-annex-write() { +REPO=$1; shift + + +(set -e; cd $(mktemp -d) + git init + echo '# testing' > README.md && git add README.md && git commit -m \"Initial commit\" + git annex init + dd if=/dev/urandom of=large.bin bs=1M count=16 && git annex add large.bin && git commit -m \"Annex a file\" + + git remote add origin \"$REPO\" + git config annex.jobs 1 + git annex sync --content origin + git annex sync --content origin # it only uploads the branch, but doesn't upload content, if I only do this once +) +} +``` + +
+ +
Create+Upload: test-gitea-annex-write git@data:datasets/test-jank.git + +``` +$ test-gitea-annex-write git@data:datasets/test-jank.git +Initialized empty Git repository in /tmp/tmp.ak87a0yp1e/.git/ +[master (root-commit) 0a7be36] Initial commit + 1 file changed, 1 insertion(+) + create mode 100644 README.md +init ok +(recording state in git...) +16+0 records in +16+0 records out +16777216 bytes (17 MB, 16 MiB) copied, 0.0608327 s, 276 MB/s +add large.bin +ok +(recording state in git...) +[master 4a55ea5] Annex a file + 1 file changed, 1 insertion(+) + create mode 120000 large.bin + + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true + + Unable to parse git config from origin +FATAL: autocreate denied + +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. + + + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + annex.sshcaching is not set to true + + Unable to parse git config from origin +FATAL: autocreate denied + +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. +On branch master +nothing to commit, working tree clean +commit ok +pull origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +FATAL: autocreate denied + +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. +ok +push origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +hint: Using 'master' as the name for the initial branch. This default branch name +hint: is subject to change. To configure the initial branch name to use in all +hint: of your new repositories, which will suppress this warning, call: +hint: +hint: git config --global init.defaultBranch +hint: +hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and +hint: 'development'. The just-created branch can be renamed via this command: +hint: +hint: git branch -m +Initialized empty Git repository in /srv/git/repositories/datasets/test-jank.git/ +ok + + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +On branch master +nothing to commit, working tree clean +commit ok +pull origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +ok +copy large.bin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +(to origin...) ok +pull origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +ok +(recording state in git...) +push origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +ok +``` + +
+ +
Delete the repo + +``` +$ ssh git@data D unlock datasets/test-jank +'datasets/test-jank' is now unlocked +$ ssh git@data D rm datasets/test-jank +rm: cannot remove 'datasets/test-jank.git/annex/objects/968/4c0/SHA256E-s16777216--9d8ccb3ebe399a8f6801cde009e03a867151ea4e4bc609848abbd29dd335688f.bin/SHA256E-s16777216--9d8ccb3ebe399a8f6801cde009e03a867151ea4e4bc609848abbd29dd335688f.bin': Permission denied +'datasets/test-jank' is now gone! +``` + +
+ +Notice the \"Permission denied\" error -- but gitolite *thinks* its work is done: + +``` +$ ssh git@data info | grep test-jank +$ +``` + +but if I try to recreate the same repo, it fails: + +
test-gitea-annex-write git@data:datasets/test-jank.git + +``` +$ test-gitea-annex-write git@data:datasets/test-jank.git +Initialized empty Git repository in /tmp/tmp.IBSNFKRgRg/.git/ +[master (root-commit) e11344b] Initial commit + 1 file changed, 1 insertion(+) + create mode 100644 README.md +init ok +(recording state in git...) +16+0 records in +16+0 records out +16777216 bytes (17 MB, 16 MiB) copied, 0.0631523 s, 266 MB/s +add large.bin +ok +(recording state in git...) +[master 0cedc5c] Annex a file + 1 file changed, 1 insertion(+) + create mode 120000 large.bin + + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true + + Unable to parse git config from origin +FATAL: R any datasets/test-jank nguenther DENIED by fallthru +(or you mis-spelled the reponame) +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. + + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true + + Unable to parse git config from origin +FATAL: R any datasets/test-jank nguenther DENIED by fallthru +(or you mis-spelled the reponame) +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. +On branch master +nothing to commit, working tree clean +commit ok +pull origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +FATAL: R any datasets/test-jank nguenther DENIED by fallthru +(or you mis-spelled the reponame) +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. +ok +push origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +FATAL: W any datasets/test-jank nguenther DENIED by fallthru +(or you mis-spelled the reponame) +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. +FATAL: W any datasets/test-jank nguenther DENIED by fallthru +(or you mis-spelled the reponame) +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. + + Pushing to origin failed. +failed +sync: 1 failed +``` + +
+ +Because of course if I log in on the server I can see: + +``` +git@data:~/repositories/datasets$ tree test-jank.git/ +test-jank.git/ +└── annex + └── objects + └── 968 + └── 4c0 + └── SHA256E-s16777216--9d8ccb3ebe399a8f6801cde009e03a867151ea4e4bc609848abbd29dd335688f.bin + └── SHA256E-s16777216--9d8ccb3ebe399a8f6801cde009e03a867151ea4e4bc609848abbd29dd335688f.bin + +5 directories, 1 file +``` + + +### Gitea + +I've been [porting `git-annex-shell`](https://github.com/neuropoly/gitea/pull/1/) into [Gitea](https://gitea.io/) as well to get a more familiar UI for my team, and I have discovered the exact same problem there: if `test-git-annex-write` to my test instance, then delete that repo, Gitea dutifully reports + +> The repository has been deleted. + +but if I then try to recreate it balks with + +> Files already exist for this repository. Either adopt them or delete them. + +## Solutions + +There doesn't seem to be much benefit to lockdown in a bare repo: there's no checkout that might corrupt the content. Plus in my case there's gitolite/gitea in the way which is an extra layer of protection against direct modification. So could **lockdown be turned off**? + +I'd like it best if you detected when you're run in a bare repo and skipped the freezing and thawing steps. But I'd also just be able to work with a config setting (`git config --global annex.lockdown false`?). + +#### Workaround #1 + +In the meantime, so far I have found one workaround: I can misuse [`annex.freezecontent-command`](https://git-annex.branchable.com/todo/lockdown_hooks/): + +``` +ssh root@data +su -l git +git config --global annex.freezecontent-command \"chmod -R +w %path\" +``` + +
Example + +``` +$ test-gitea-annex-write git@data:datasets/test-jank3 +Initialized empty Git repository in /tmp/tmp.I9wno7oZXg/.git/ +[master (root-commit) 4f47840] Initial commit + 1 file changed, 1 insertion(+) + create mode 100644 README.md +init ok +(recording state in git...) +16+0 records in +16+0 records out +16388608 bytes (16.2 MB, 16.0 MiB) copied, 0.0313028 s, 268 MB/s +add large.bin +ok +(recording state in git...) +[master 27e4c8c] Annex a file + 1 file changed, 1 insertion(+) + create mode 120000 large.bin + + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true + + Unable to parse git config from origin +FATAL: autocreate denied + +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. + + + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + annex.sshcaching is not set to true + + Unable to parse git config from origin +FATAL: autocreate denied + +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. +On branch master +nothing to commit, working tree clean +commit ok +pull origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +FATAL: autocreate denied + +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. +ok +push origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +hint: Using 'master' as the name for the initial branch. This default branch name +hint: is subject to change. To configure the initial branch name to use in all +hint: of your new repositories, which will suppress this warning, call: +hint: +hint: git config --global init.defaultBranch +hint: +hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and +hint: 'development'. The just-created branch can be renamed via this command: +hint: +hint: git branch -m +Initialized empty Git repository in /srv/git/repositories/datasets/test-jank3.git/ +ok + + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +On branch master +nothing to commit, working tree clean +commit ok +pull origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +ok +copy large.bin + + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + annex.sshcaching is not set to true +(to origin...) ok +pull origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +ok +(recording state in git...) +push origin + You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time. + + annex.sshcaching is not set to true +ok +$ ssh git@data D unlock datasets/test-jank3 +'datasets/test-jank3' is now unlocked +$ ssh git@data D rm datasets/test-jank3 +'datasets/test-jank3' is now gone! +``` + +(notice it doesn't give any error this time) + +
+ +But this only works with a relatively new git-annex; I haven't looked up when this went in, but I know 8.20210223, from barely a year ago, doesn't have this feature, while 10.20220322 does. And also it's very much a workaround: it immediately undoes the work git-annex does, which will cause unnecessary disk I/O. + +Here's an upgrade to this workaround, that limits the effect to bare repos (though the only repos ever created by the remote `git` user should be bare): + +``` +git config --global annex.freezecontent-command 'sh -c '\"'\"'[ \"$(git config core.bare)\" = \"true\" ] && chmod -R +w %path'\"'\" +``` + + +#### Workaround #2 + +The advice above says to + +> (The only bad consequence of this is that `rm -rf .git` doesn't work unless you first run `chmod -R +w .git`) + +so another solution would be to patch gitolite/gitea's `rm` subroutines to be git-annex aware, i.e. to run `chmod -R +w` before doing anything else. + +That looks more feasible for me to do in gitea, where git-annex support is turning out to need a whole bunch of patches scattered across the codebase, but it's a lot less appealing to do in gitolite where git-annex support is currently contained in one very elegant file. +"""]]