mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2022-07-08 10:33:54 -04:00
commit 9c6e45c0b2
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
7 changed files with 159 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
doc/bugs/fails_to_authenticate_into_S3_for_initremote__63__.mdwn Normal file
View file

@ -0,0 +1,28 @@
### Please describe the problem.
```
(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ AWS_ACCESS_KEY_ID=$(awk '/^access_key/{print $3}' ~/.s3cfg-dandi-backup) AWS_SECRET_ACCESS_KEY=$(awk '/^secret_key/{print $3}' ~/.s3cfg-dandi-backup) git annex initremote s3-origin type=S3 importtree=yes encryption=none autoenable=true bucket=dandiarchive fileprefix=zarr-checksums/2ac71edb-738c-40ac-bd8c-8ca985adaa12/
initremote s3-origin (checking bucket...) (creating bucket in US...)
git-annex: S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "InvalidRequest", s3ErrorMessage = "The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.", s3ErrorResource = Nothing, s3ErrorHostId = Just "????=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}
failed
initremote: 1 failed
(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ git annex version
git-annex version: 10.20220525-gf1fdc90
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-8.10.7 http-client-0.7.9 persistent-sqlite-2.13.0.3 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.1.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 8
```
what exactly does it want from me for AWS4-HMAC-SHA256 ? here is how those secrets look like (after slight obfuscation):
```
$ echo AWS_ACCESS_KEY_ID=$(awk '/^access_key/{print $3}' ~/.s3cfg-dandi-backup) AWS_SECRET_ACCESS_KEY=$(awk '/^secret_key/{print $3}' ~/.s3cfg-dandi-backup) | tr 'a-z0-9' '?-?'
AWS_ACCESS_KEY_ID=AKIA?GIMZPVVE??Y?NKH AWS_SECRET_ACCESS_KEY=/??U??TV?LH?L???KZJ??RF?G???Y+?S??????LM
```

35
doc/bugs/testremote_is_not_honoring_--backend.mdwn Normal file
View file

@ -0,0 +1,35 @@
### Please describe the problem.
I am running `testremote` on a windows CI system to test a special remote implementation for dataverse.org. I run into this error:
```
git-annex: MoveFileEx "C:\\DLTMP\\ran2133" Just ".git\\annex\\objects\\f76\\373\\SHA256E-s1048576--813fea02438e9569e6222f802958fcd89bee742d06ffe9aabe27fd940ef01196.this-is-a-test-key\\SHA256E-s1048576--813fea02438e9569e6222f802958fcd89bee742d06ffe9aabe27fd940ef01196.this-is-a-test-key": does not exist (The system cannot find the path specified.)
```
I suspect this could be a path-length issue (the system reports a max length of 285, and the relative path given above is already 230 chars.
I thought to run `git annex testremote --backend=MD5E` instead, to shorten the key length, but this options is not honored (enough), the error showing a SHA256 key remains the same.
`testremote` man page says "Also the git-annex-common-options(1) can be used." and `--backend` is explicitly listed in the help output, hence I assumed this should work.
### What steps will reproduce the problem?
It happens when running the https://github.com/datalad/datalad-dataverse tests on a windows appveyor worker. Running on a crippled FS is not enough to trigger the initial `testremote` error, it only happens on windows proper. However, I assume that `--backend` not having the effect that I assumed it should have, is not platform specific.
Here is a demo test log: https://ci.appveyor.com/project/mih/datalad-dataverse/builds/44079592/job/b38woai0ekmq7bn5#L856
The corresponding datalad issue is https://github.com/datalad/datalad-dataverse/issues/127
### What version of git-annex are you using? On what operating system?
CI used
- annex: 8.20211117-gc3af94eff
- git: 2.37.0.windows.1
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
All the time! Sorry to mostly show up when there is an issue!

8
doc/bugs/testremote_is_not_honoring_--backend/comment_1_2b119f8bce39266630dd7c0f4d8add98._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="mih"
avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd"
subject="Recent version tested"
date="2022-07-06T12:18:36Z"
content="""
The behavior is the same for the more recent git-annex 10.20220624-g17e4081d4
"""]]

20
doc/devblog/day_641__an_alternative_smudge_filter/comment_5_6a5932706a9fee097c3fdf2cd6808f1d._comment Normal file
View file

@ -0,0 +1,20 @@
[[!comment format=mdwn
username="nick.guenther@e418ed3c763dff37995c2ed5da4232a7c6cee0a9"
nickname="nick.guenther"
avatar="http://cdn.libravatar.org/avatar/9e85c6ca61c3f877fef4f91c2bf6e278"
subject="comment 5"
date="2022-07-06T05:04:07Z"
content="""
Until this is ready, at the moment, I got around the slowness [by setting](https://github.com/neuropoly/intranet.neuro.polymtl.ca/blob/d5c320c/geek-tips/git-annex.md#new-repo)
```
cat > .gitattributes <<EOF
*.mkv filter=anex annex.largefiles=anything
*.mp3 filter=anex annex.largefiles=anything
EOF
```
This way the file size is ignored and instead git makes the decision for me based on file extension. This is much faster than running every file through `git annex smudge`. I'
I might end up sticking with this even if you use the faster filter API.
"""]]

11
doc/forum/git_annex_get___60__file__62___fails___40__Action_blocked_by_GIT__41__.mdwn Normal file
View file

@ -0,0 +1,11 @@
Hi,
After a long time, and having a fresh machine, I cloned a repo I've been using without problems. When I try to get the content of some files, I get the following error:
> git annex get im/fixed_regions/melanomas/melanomas_mp/ejemplo18/mean_bounds.png
get im/fixed_regions/melanomas/melanomas_mp/ejemplo18/mean_bounds.png (from origin...)
git-annex-shell: Action blocked by GIT_ANNEX_SHELL_LIMITED
I've checked that the file is present in the origin remote. I'm sure it is a pretty silly thing, but it's been a while since I used git-annex.
Thanks for the help and kudos for this great tool!

39
doc/forum/git_status_is_slow_twice__44___then_fast.mdwn Normal file
View file

@ -0,0 +1,39 @@
I don't know if this is a git thing, or a git annex thing, but it's weird.
After I add a bunch of files to the annex (and there are many more local files not yet added), `git status` is slow. That's not really a surprise. It shows the `refresh index` calculation as usual.
I run it a second time, and I get the exact same result:
```
% time git status
Refresh index: 100% (2997/2997), done.
<snip>
git status 38.64s user 42.08s system 80% cpu 1:39.95 total
```
I run it a third time and it's _instant_.
This happens reliably. Two slow runs, and then a fast run.
I don't love the initial slow run, but it makes sense, so I can live with it. The second slow run doesn't make sense to me though. Is there any way to avoid it? I guess I would expect the first slow run to refresh the index and speed up subsequent runs.
It seems like a similar symptom to [git keeps refreshing index](https://git-annex.branchable.com/bugs/git_keeps_refreshing_index/).
I am using git and git-annex installed via homebrew on macOS 10.15.7. I have `annex.addunlocked` set to true.
```
git version 2.37.0
git-annex version: 10.20220624
build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.30 DAV-1.3.4 feed-1.3.2.1 ghc-8.10.7 http-client-0.7.11 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: darwin x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 8
```
- @pat

18
doc/todo/allow_for_annonymous_AWS_S3_access.mdwn Normal file
View file

@ -0,0 +1,18 @@
Should be possible for any public bucket which allows for annonymous access. ATM git-annex demands AWS credentials for importtree on such public bucket
```shell
(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ git annex initremote s3-origin type=S3 importtree=yes encryption=none autoenable=true bucket=dandiarchive fileprefix=zarr-checksums/2ac71edb-738c-40ac-bd8c-8ca985adaa12/ initremote s3-origin (checking bucket...)
Set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use S3
git-annex: No S3 credentials configured
failed
initremote: 1 failed
(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ git annex version
git-annex version: 10.20220525-gf1fdc90
```
in my searches information on "anonymous access to S3" is scarce, but in DataLad we rely on old-ish version of `boto` library which does it for us, more info/pointers [eg here on SO](https://stackoverflow.com/questions/34865927/can-i-use-boto3-anonymously)
[[!meta author=yoh]]
[[!tag projects/dandi]]