From 4d3178b8ea96947335cec5ec16799506ea50eb25 Mon Sep 17 00:00:00 2001 From: jkniiv Date: Thu, 9 Jun 2022 02:32:19 +0000 Subject: [PATCH 1/3] Added a comment --- ...ment_2_d51bb87dc5ed0f4ede808c5a9594240d._comment | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 doc/bugs/initremote_type__61__git_is_not_working_for_unkn_reason/comment_2_d51bb87dc5ed0f4ede808c5a9594240d._comment diff --git a/doc/bugs/initremote_type__61__git_is_not_working_for_unkn_reason/comment_2_d51bb87dc5ed0f4ede808c5a9594240d._comment b/doc/bugs/initremote_type__61__git_is_not_working_for_unkn_reason/comment_2_d51bb87dc5ed0f4ede808c5a9594240d._comment new file mode 100644 index 0000000000..085d00ea50 --- /dev/null +++ b/doc/bugs/initremote_type__61__git_is_not_working_for_unkn_reason/comment_2_d51bb87dc5ed0f4ede808c5a9594240d._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="jkniiv" + avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d" + subject="comment 2" + date="2022-06-09T02:32:18Z" + content=""" +Wouldn't it be possible to support (absolute) file:// urls, eg. something similar to +`file:///home/jkniiv/test-VEfBrTZ/remote2`? In my mind they feel like a reasonable approximation +of ssh:// urls and could be useful for getting a feel for git special remotes before setting +up a bare git-repo/annex on an ssh-server. I know they are not the same thing implementation wise +but I feel that being able to try this feature out on a least-effort basis would be useful +from a pedagogical standpoint. +"""]] From 2d4126e742d914c9602b334c3fb071273edbfb03 Mon Sep 17 00:00:00 2001 From: mih Date: Thu, 9 Jun 2022 11:43:47 +0000 Subject: [PATCH 2/3] --- ...cannot_probe_for_untrusted_remote_key.mdwn | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 doc/bugs/10.20220525_cannot_probe_for_untrusted_remote_key.mdwn diff --git a/doc/bugs/10.20220525_cannot_probe_for_untrusted_remote_key.mdwn b/doc/bugs/10.20220525_cannot_probe_for_untrusted_remote_key.mdwn new file mode 100644 index 0000000000..2ecf1db6d0 --- /dev/null +++ b/doc/bugs/10.20220525_cannot_probe_for_untrusted_remote_key.mdwn @@ -0,0 +1,25 @@ +### Please describe the problem. +10.20220525 forcefully untrusts `exporttree=yes`, causing `fsck` to unconditionally fail when probing for a key that only exists on such a remote. + +The full story is at https://github.com/datalad/datalad-next/issues/72 + +The summary is that an application needs to know whether a particular key is on such a remote, and it can only be on that remote (there is no other). Because the availability info from `whereis` cannot be trusted (same rational as the one causing the change in git-annex implementation) it uses `annex fsck --fast --key -f ...` which now fails. + +Parsing the JSON output of the failing `fsck` call seems to be the only way left to accomplish this goal. + +``` +% git -C myclone/.git/dl-repoannex/origin/repoannex -c diff.ignoreSubmodules=none annex fsck -f origin --fast --key XDLRA--refs -c annex.dotfiles=true --json + Only these untrusted locations may have copies of XDLRA--refs + d8735795-663e-4702-8f7e-8c684264a9df -- [origin] + Back it up to trusted locations with git-annex copy. +{"dead":[],"command":"fsck","note":"(Avoid this check by running: git annex dead --key )","success":false,"input":[],"untrusted":[{"here":false,"uuid":"d8735795-663e-4702-8f7e-8c684264a9df","description":"[origin]"}],"key":"XDLRA--refs","error-messages":[],"file":null} +fsck: 1 failed +``` + +That however, is substantially more complex than looking at the exit code -- given that the entire call is about a single key. + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +99.9% are just splendid. Wondering here whether this particular consequence of the change was intentional and avoidable? + + From 1a2bff370d98c55ae5d9d39d6587fd255f08f87e Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Thu, 9 Jun 2022 17:17:56 +0000 Subject: [PATCH 3/3] initial report on difficulty with 0 length file --- ...file_size_0_from_external_spec_remote.mdwn | 88 +++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 doc/bugs/error_out_on_file_size_0_from_external_spec_remote.mdwn diff --git a/doc/bugs/error_out_on_file_size_0_from_external_spec_remote.mdwn b/doc/bugs/error_out_on_file_size_0_from_external_spec_remote.mdwn new file mode 100644 index 0000000000..b2358f32a3 --- /dev/null +++ b/doc/bugs/error_out_on_file_size_0_from_external_spec_remote.mdwn @@ -0,0 +1,88 @@ +### Please describe the problem. + +Here is a reproducer using the place where I encountered it -- using git-annex-remote-rclone (tiny bash script; needs fixing though for recent rclone -- take from [my fork where I am tuning testing etc](https://github.com/yarikoptic/git-annex-remote-rclone/blob/enh-tests/git-annex-remote-rclone)) [https://github.com/DanielDent/git-annex-remote-rclone/](https://github.com/DanielDent/git-annex-remote-rclone/) + +
+this is a reproducer which creates repo and commits to annex of size 0 + +```bash +#!/bin/bash + +cd "$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)" + +set -eux + +# provide versioning information to possibly ease troubleshooting +git annex version +rclone --version + +export HOME=$PWD +echo -e '[local]\ntype = local\nnounc =' > ~/.rclone.conf +# to pacify git/git-annex +git config --global user.name Me +git config --global user.email me@example.com +git config --global init.defaultBranch master + +# Prepare rclone remote local store +mkdir rclone-local +export RCLONE_PREFIX=$PWD/rclone-local + +git-annex version +mkdir testrepo +cd testrepo +git init . +git-annex init +git-annex initremote GA-rclone-CI type=external externaltype=rclone target=local prefix=$RCLONE_PREFIX chunk=100MiB encryption=shared mac=HMACSHA512 + +# Rudimentary test, spaces in the filename must be ok, 0 length files should be ok + +touch "test 0" + +#echo 1 > "test 1" + +git-annex add * +git-annex copy * --to GA-rclone-CI +git-annex drop * +git-annex get --debug * + +``` +
+ +where it ends with + +``` ++ git-annex get --debug 'test 0' +[2022-06-09 13:16:20.668338168] (Utility.Process) process [1022440] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--error-unmatch","--","test 0"] +[2022-06-09 13:16:20.668784836] (Utility.Process) process [1022441] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"] +[2022-06-09 13:16:20.669111286] (Utility.Process) process [1022442] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"] +[2022-06-09 13:16:20.66965775] (Utility.Process) process [1022443] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"] +[2022-06-09 13:16:20.670318546] (Utility.Process) process [1022443] done ExitSuccess +[2022-06-09 13:16:20.67074703] (Utility.Process) process [1022444] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"] +[2022-06-09 13:16:20.671504449] (Utility.Process) process [1022444] done ExitSuccess +[2022-06-09 13:16:20.672125442] (Utility.Process) process [1022445] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..0da4aa78e4397feb9906400d1e08aaebccac1be1","--pretty=%H","-n1"] +[2022-06-09 13:16:20.673019668] (Utility.Process) process [1022445] done ExitSuccess +[2022-06-09 13:16:20.673440353] (Utility.Process) process [1022446] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"] +get test 0 [2022-06-09 13:16:20.674594171] (Utility.Process) process [1022448] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"] +(from GA-rclone-CI...) +[2022-06-09 13:16:20.682205613] (Annex.Perms) freezing content directory .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 + +git-annex: .git/annex/tmp/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855: rename: does not exist (No such file or directory) +failed +[2022-06-09 13:16:20.682296739] (Utility.Process) process [1022446] done ExitSuccess +[2022-06-09 13:16:20.682333234] (Utility.Process) process [1022442] done ExitSuccess +[2022-06-09 13:16:20.682384256] (Utility.Process) process [1022441] done ExitSuccess +[2022-06-09 13:16:20.682408227] (Utility.Process) process [1022440] done ExitSuccess +[2022-06-09 13:16:20.682701178] (Utility.Process) process [1022448] done ExitSuccess +get: 1 failed + +``` + +- you can see that there is not even an attempt to talk to the remote. git-annex crashes before then with that " rename: does not exist (No such file or directory)" + +so I do not think it is an issue of the remote (which might also have some size 0 issues from looking at some conditions) + + +### What version of git-annex are you using? On what operating system? + + +tried both 8.20211123 and 10.20220504 from debian