From b1ac11856cb539f7a8cd09c15cca92f7528c877f Mon Sep 17 00:00:00 2001 From: "nick.guenther@e418ed3c763dff37995c2ed5da4232a7c6cee0a9" Date: Wed, 6 Jul 2022 05:04:08 +0000 Subject: [PATCH 01/10] Added a comment --- ..._6a5932706a9fee097c3fdf2cd6808f1d._comment | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/devblog/day_641__an_alternative_smudge_filter/comment_5_6a5932706a9fee097c3fdf2cd6808f1d._comment diff --git a/doc/devblog/day_641__an_alternative_smudge_filter/comment_5_6a5932706a9fee097c3fdf2cd6808f1d._comment b/doc/devblog/day_641__an_alternative_smudge_filter/comment_5_6a5932706a9fee097c3fdf2cd6808f1d._comment new file mode 100644 index 0000000000..2c8343980c --- /dev/null +++ b/doc/devblog/day_641__an_alternative_smudge_filter/comment_5_6a5932706a9fee097c3fdf2cd6808f1d._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="nick.guenther@e418ed3c763dff37995c2ed5da4232a7c6cee0a9" + nickname="nick.guenther" + avatar="http://cdn.libravatar.org/avatar/9e85c6ca61c3f877fef4f91c2bf6e278" + subject="comment 5" + date="2022-07-06T05:04:07Z" + content=""" +Until this is ready, at the moment, I got around the slowness [by setting](https://github.com/neuropoly/intranet.neuro.polymtl.ca/blob/d5c320c/geek-tips/git-annex.md#new-repo) + +``` +cat > .gitattributes < Date: Wed, 6 Jul 2022 06:24:06 +0000 Subject: [PATCH 02/10] --- .../testremote_is_not_honoring_--backend.mdwn | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 doc/bugs/testremote_is_not_honoring_--backend.mdwn diff --git a/doc/bugs/testremote_is_not_honoring_--backend.mdwn b/doc/bugs/testremote_is_not_honoring_--backend.mdwn new file mode 100644 index 0000000000..4c4fa5894c --- /dev/null +++ b/doc/bugs/testremote_is_not_honoring_--backend.mdwn @@ -0,0 +1,35 @@ +### Please describe the problem. + +I am running `testremote` on a windows CI system to test a special remote implementation for dataverse.org. I run into this error: + +``` +git-annex: MoveFileEx "C:\\DLTMP\\ran2133" Just ".git\\annex\\objects\\f76\\373\\SHA256E-s1048576--813fea02438e9569e6222f802958fcd89bee742d06ffe9aabe27fd940ef01196.this-is-a-test-key\\SHA256E-s1048576--813fea02438e9569e6222f802958fcd89bee742d06ffe9aabe27fd940ef01196.this-is-a-test-key": does not exist (The system cannot find the path specified.) +``` + +I suspect this could be a path-length issue (the system reports a max length of 285, and the relative path given above is already 230 chars. + +I thought to run `git annex testremote --backend=MD5E` instead, to shorten the key length, but this options is not honored (enough), the error showing a SHA256 key remains the same. + +`testremote` man page says "Also the git-annex-common-options(1) can be used." and `--backend` is explicitly listed in the help output, hence I assumed this should work. + + +### What steps will reproduce the problem? + +It happens when running the https://github.com/datalad/datalad-dataverse tests on a windows appveyor worker. Running on a crippled FS is not enough to trigger the initial `testremote` error, it only happens on windows proper. However, I assume that `--backend` not having the effect that I assumed it should have, is not platform specific. + +Here is a demo test log: https://ci.appveyor.com/project/mih/datalad-dataverse/builds/44079592/job/b38woai0ekmq7bn5#L856 + +The corresponding datalad issue is https://github.com/datalad/datalad-dataverse/issues/127 + + +### What version of git-annex are you using? On what operating system? + +CI used + +- annex: 8.20211117-gc3af94eff +- git: 2.37.0.windows.1 + + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +All the time! Sorry to mostly show up when there is an issue! From a223f402f385de78c50b9c655b2e0c491078f540 Mon Sep 17 00:00:00 2001 From: mih Date: Wed, 6 Jul 2022 12:18:36 +0000 Subject: [PATCH 03/10] Added a comment: Recent version tested --- .../comment_1_2b119f8bce39266630dd7c0f4d8add98._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/testremote_is_not_honoring_--backend/comment_1_2b119f8bce39266630dd7c0f4d8add98._comment diff --git a/doc/bugs/testremote_is_not_honoring_--backend/comment_1_2b119f8bce39266630dd7c0f4d8add98._comment b/doc/bugs/testremote_is_not_honoring_--backend/comment_1_2b119f8bce39266630dd7c0f4d8add98._comment new file mode 100644 index 0000000000..c14740668a --- /dev/null +++ b/doc/bugs/testremote_is_not_honoring_--backend/comment_1_2b119f8bce39266630dd7c0f4d8add98._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="mih" + avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd" + subject="Recent version tested" + date="2022-07-06T12:18:36Z" + content=""" +The behavior is the same for the more recent git-annex 10.20220624-g17e4081d4 +"""]] From 96e91cd6a8ec482bf068e4904097e1ac332e392e Mon Sep 17 00:00:00 2001 From: drunken_sapo Date: Thu, 7 Jul 2022 00:57:32 +0000 Subject: [PATCH 04/10] --- ...le__62___fails___40__Action_blocked_by_GIT__41__.mdwn | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/forum/git_annex_get___60__file__62___fails___40__Action_blocked_by_GIT__41__.mdwn diff --git a/doc/forum/git_annex_get___60__file__62___fails___40__Action_blocked_by_GIT__41__.mdwn b/doc/forum/git_annex_get___60__file__62___fails___40__Action_blocked_by_GIT__41__.mdwn new file mode 100644 index 0000000000..0010a5f98b --- /dev/null +++ b/doc/forum/git_annex_get___60__file__62___fails___40__Action_blocked_by_GIT__41__.mdwn @@ -0,0 +1,9 @@ +Hi, + After a long time, and having a fresh machine, I cloned a repo I've been using without problems. When I try to get the content of some files, I get the following error: + +> git annex get im/fixed_regions/melanomas/melanomas_mp/ejemplo18/mean_bounds.png +get im/fixed_regions/melanomas/melanomas_mp/ejemplo18/mean_bounds.png (from origin...) +git-annex-shell: Action blocked by GIT_ANNEX_SHELL_LIMITED + +I've checked that the file is present in the origin remote. I'm sure it is a pretty silly thing, but it's been a while since I used git-annex. +Thanks for the help and kudos for this great tool! From d272ef197a9879fc8a57e6fe920da31fa6571d37 Mon Sep 17 00:00:00 2001 From: drunken_sapo Date: Thu, 7 Jul 2022 00:58:29 +0000 Subject: [PATCH 05/10] --- ..._60__file__62___fails___40__Action_blocked_by_GIT__41__.mdwn | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/forum/git_annex_get___60__file__62___fails___40__Action_blocked_by_GIT__41__.mdwn b/doc/forum/git_annex_get___60__file__62___fails___40__Action_blocked_by_GIT__41__.mdwn index 0010a5f98b..8e38629922 100644 --- a/doc/forum/git_annex_get___60__file__62___fails___40__Action_blocked_by_GIT__41__.mdwn +++ b/doc/forum/git_annex_get___60__file__62___fails___40__Action_blocked_by_GIT__41__.mdwn @@ -2,7 +2,9 @@ Hi, After a long time, and having a fresh machine, I cloned a repo I've been using without problems. When I try to get the content of some files, I get the following error: > git annex get im/fixed_regions/melanomas/melanomas_mp/ejemplo18/mean_bounds.png + get im/fixed_regions/melanomas/melanomas_mp/ejemplo18/mean_bounds.png (from origin...) + git-annex-shell: Action blocked by GIT_ANNEX_SHELL_LIMITED I've checked that the file is present in the origin remote. I'm sure it is a pretty silly thing, but it's been a while since I used git-annex. From ccdb234a0e47bb6c8f0222cd4869f07b905d83fa Mon Sep 17 00:00:00 2001 From: pat Date: Thu, 7 Jul 2022 07:09:46 +0000 Subject: [PATCH 06/10] --- ..._status_is_slow_twice__44___then_fast.mdwn | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/forum/git_status_is_slow_twice__44___then_fast.mdwn diff --git a/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn b/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn new file mode 100644 index 0000000000..e09198371d --- /dev/null +++ b/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn @@ -0,0 +1,21 @@ +I don't know if this is a git thing, or a git annex thing, but it's weird. + +After I add a bunch of files to the annex (and there are many more local files not yet added), `git status` is slow. That's not really a surprise. It shows the `refresh index` calculation as usual. + +I run it a second time, and I get the exact same result: + +``` +% time git status +Refresh index: 100% (2997/2997), done. + + +git status 38.64s user 42.08s system 80% cpu 1:39.95 total +``` + +I run it a third time and it's _instant_. + +This happens reliably. Two slow runs, and then a fast run. + +I don't love the initial slow run, but it makes sense, so I can live with it. The second slow run doesn't make sense to me though. Is there any way to avoid it? I guess I would expect the first slow run to refresh the index and speed up subsequent runs. + +- @pat From 0cb20421336f59c9e6596d151c31419de383113d Mon Sep 17 00:00:00 2001 From: pat Date: Thu, 7 Jul 2022 07:52:26 +0000 Subject: [PATCH 07/10] --- ...t_status_is_slow_twice__44___then_fast.mdwn | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn b/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn index e09198371d..3151056ace 100644 --- a/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn +++ b/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn @@ -18,4 +18,22 @@ This happens reliably. Two slow runs, and then a fast run. I don't love the initial slow run, but it makes sense, so I can live with it. The second slow run doesn't make sense to me though. Is there any way to avoid it? I guess I would expect the first slow run to refresh the index and speed up subsequent runs. +It seems like a similar symptom to [git keeps refreshing index](https://git-annex.branchable.com/bugs/git_keeps_refreshing_index/). + +I am using git and git-annex installed via homebrew on macOS 10.15.7 + +``` +git version 2.37.0 + +git-annex version: 10.20220624 +build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Feeds Testsuite S3 WebDAV +dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.30 DAV-1.3.4 feed-1.3.2.1 ghc-8.10.7 http-client-0.7.11 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external +operating system: darwin x86_64 +supported repository versions: 8 9 10 +upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 +local repository version: 8 +``` + - @pat From 661a89881159a4874ac932affdd6fc7e4af4cb67 Mon Sep 17 00:00:00 2001 From: pat Date: Thu, 7 Jul 2022 07:57:27 +0000 Subject: [PATCH 08/10] --- doc/forum/git_status_is_slow_twice__44___then_fast.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn b/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn index 3151056ace..de7ef1026f 100644 --- a/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn +++ b/doc/forum/git_status_is_slow_twice__44___then_fast.mdwn @@ -20,7 +20,7 @@ I don't love the initial slow run, but it makes sense, so I can live with it. Th It seems like a similar symptom to [git keeps refreshing index](https://git-annex.branchable.com/bugs/git_keeps_refreshing_index/). -I am using git and git-annex installed via homebrew on macOS 10.15.7 +I am using git and git-annex installed via homebrew on macOS 10.15.7. I have `annex.addunlocked` set to true. ``` git version 2.37.0 From 22453fca7cbf1cdb3b050995a44171bd8b7adc76 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Thu, 7 Jul 2022 19:50:15 +0000 Subject: [PATCH 09/10] request for anonymous access to S3 --- .../allow_for_annonymous_AWS_S3_access.mdwn | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 doc/todo/allow_for_annonymous_AWS_S3_access.mdwn diff --git a/doc/todo/allow_for_annonymous_AWS_S3_access.mdwn b/doc/todo/allow_for_annonymous_AWS_S3_access.mdwn new file mode 100644 index 0000000000..123bc70f6c --- /dev/null +++ b/doc/todo/allow_for_annonymous_AWS_S3_access.mdwn @@ -0,0 +1,18 @@ +Should be possible for any public bucket which allows for annonymous access. ATM git-annex demands AWS credentials for importtree on such public bucket + +```shell +(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ git annex initremote s3-origin type=S3 importtree=yes encryption=none autoenable=true bucket=dandiarchive fileprefix=zarr-checksums/2ac71edb-738c-40ac-bd8c-8ca985adaa12/ initremote s3-origin (checking bucket...) + Set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use S3 + +git-annex: No S3 credentials configured +failed +initremote: 1 failed + +(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ git annex version +git-annex version: 10.20220525-gf1fdc90 +``` + +in my searches information on "anonymous access to S3" is scarce, but in DataLad we rely on old-ish version of `boto` library which does it for us, more info/pointers [eg here on SO](https://stackoverflow.com/questions/34865927/can-i-use-boto3-anonymously) + +[[!meta author=yoh]] +[[!tag projects/dandi]] From e7f499ba364983f0439e6dd1f74a89684deca8ce Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Thu, 7 Jul 2022 19:56:06 +0000 Subject: [PATCH 10/10] initial report on inability to connect to S3 --- ...enticate_into_S3_for_initremote__63__.mdwn | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 doc/bugs/fails_to_authenticate_into_S3_for_initremote__63__.mdwn diff --git a/doc/bugs/fails_to_authenticate_into_S3_for_initremote__63__.mdwn b/doc/bugs/fails_to_authenticate_into_S3_for_initremote__63__.mdwn new file mode 100644 index 0000000000..cb103f7436 --- /dev/null +++ b/doc/bugs/fails_to_authenticate_into_S3_for_initremote__63__.mdwn @@ -0,0 +1,28 @@ +### Please describe the problem. + +``` +(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ AWS_ACCESS_KEY_ID=$(awk '/^access_key/{print $3}' ~/.s3cfg-dandi-backup) AWS_SECRET_ACCESS_KEY=$(awk '/^secret_key/{print $3}' ~/.s3cfg-dandi-backup) git annex initremote s3-origin type=S3 importtree=yes encryption=none autoenable=true bucket=dandiarchive fileprefix=zarr-checksums/2ac71edb-738c-40ac-bd8c-8ca985adaa12/ +initremote s3-origin (checking bucket...) (creating bucket in US...) +git-annex: S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "InvalidRequest", s3ErrorMessage = "The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.", s3ErrorResource = Nothing, s3ErrorHostId = Just "????=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing} +failed +initremote: 1 failed + +(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ git annex version +git-annex version: 10.20220525-gf1fdc90 +build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV +dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-8.10.7 http-client-0.7.9 persistent-sqlite-2.13.0.3 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.1.2 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external +operating system: linux x86_64 +supported repository versions: 8 9 10 +upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 +local repository version: 8 + +``` + +what exactly does it want from me for AWS4-HMAC-SHA256 ? here is how those secrets look like (after slight obfuscation): + +``` +$ echo AWS_ACCESS_KEY_ID=$(awk '/^access_key/{print $3}' ~/.s3cfg-dandi-backup) AWS_SECRET_ACCESS_KEY=$(awk '/^secret_key/{print $3}' ~/.s3cfg-dandi-backup) | tr 'a-z0-9' '?-?' +AWS_ACCESS_KEY_ID=AKIA?GIMZPVVE??Y?NKH AWS_SECRET_ACCESS_KEY=/??U??TV?LH?L???KZJ??RF?G???Y+?S??????LM +```