Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2022-07-08 10:33:54 -04:00
commit 9c6e45c0b2
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
7 changed files with 159 additions and 0 deletions

View file

@ -0,0 +1,28 @@
### Please describe the problem.
```
(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ AWS_ACCESS_KEY_ID=$(awk '/^access_key/{print $3}' ~/.s3cfg-dandi-backup) AWS_SECRET_ACCESS_KEY=$(awk '/^secret_key/{print $3}' ~/.s3cfg-dandi-backup) git annex initremote s3-origin type=S3 importtree=yes encryption=none autoenable=true bucket=dandiarchive fileprefix=zarr-checksums/2ac71edb-738c-40ac-bd8c-8ca985adaa12/
initremote s3-origin (checking bucket...) (creating bucket in US...)
git-annex: S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "InvalidRequest", s3ErrorMessage = "The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.", s3ErrorResource = Nothing, s3ErrorHostId = Just "????=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}
failed
initremote: 1 failed
(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ git annex version
git-annex version: 10.20220525-gf1fdc90
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-8.10.7 http-client-0.7.9 persistent-sqlite-2.13.0.3 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.1.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 8
```
what exactly does it want from me for AWS4-HMAC-SHA256 ? here is how those secrets look like (after slight obfuscation):
```
$ echo AWS_ACCESS_KEY_ID=$(awk '/^access_key/{print $3}' ~/.s3cfg-dandi-backup) AWS_SECRET_ACCESS_KEY=$(awk '/^secret_key/{print $3}' ~/.s3cfg-dandi-backup) | tr 'a-z0-9' '?-?'
AWS_ACCESS_KEY_ID=AKIA?GIMZPVVE??Y?NKH AWS_SECRET_ACCESS_KEY=/??U??TV?LH?L???KZJ??RF?G???Y+?S??????LM
```

View file

@ -0,0 +1,35 @@
### Please describe the problem.
I am running `testremote` on a windows CI system to test a special remote implementation for dataverse.org. I run into this error:
```
git-annex: MoveFileEx "C:\\DLTMP\\ran2133" Just ".git\\annex\\objects\\f76\\373\\SHA256E-s1048576--813fea02438e9569e6222f802958fcd89bee742d06ffe9aabe27fd940ef01196.this-is-a-test-key\\SHA256E-s1048576--813fea02438e9569e6222f802958fcd89bee742d06ffe9aabe27fd940ef01196.this-is-a-test-key": does not exist (The system cannot find the path specified.)
```
I suspect this could be a path-length issue (the system reports a max length of 285, and the relative path given above is already 230 chars.
I thought to run `git annex testremote --backend=MD5E` instead, to shorten the key length, but this options is not honored (enough), the error showing a SHA256 key remains the same.
`testremote` man page says "Also the git-annex-common-options(1) can be used." and `--backend` is explicitly listed in the help output, hence I assumed this should work.
### What steps will reproduce the problem?
It happens when running the https://github.com/datalad/datalad-dataverse tests on a windows appveyor worker. Running on a crippled FS is not enough to trigger the initial `testremote` error, it only happens on windows proper. However, I assume that `--backend` not having the effect that I assumed it should have, is not platform specific.
Here is a demo test log: https://ci.appveyor.com/project/mih/datalad-dataverse/builds/44079592/job/b38woai0ekmq7bn5#L856
The corresponding datalad issue is https://github.com/datalad/datalad-dataverse/issues/127
### What version of git-annex are you using? On what operating system?
CI used
- annex: 8.20211117-gc3af94eff
- git: 2.37.0.windows.1
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
All the time! Sorry to mostly show up when there is an issue!

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="mih"
avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd"
subject="Recent version tested"
date="2022-07-06T12:18:36Z"
content="""
The behavior is the same for the more recent git-annex 10.20220624-g17e4081d4
"""]]