Merge branch 'master' of ssh://git-annex.branchable.com
This commit is contained in:
commit
c99526dd41
12 changed files with 272 additions and 0 deletions
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="comment 1"
|
||||
date="2019-09-02T15:49:57Z"
|
||||
content="""
|
||||
Post the trace (with `git annex --debug --verbose`) and your config files (`git config --list`)? (Redact any sensitive info). Maybe, `annex.security.allowed-ip-addresses` excludes the IP address of the URL, so the built-in web remote never gets a chance to claim the URL? I recall running into something like you describe, and the issue was with some config setting I had.
|
||||
"""]]
|
|
@ -0,0 +1,50 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="another trace of concurrent copy failure"
|
||||
date="2019-09-05T06:09:13Z"
|
||||
content="""
|
||||
Some more concurrent copy failures when non-concurrent works:
|
||||
|
||||
[[!format sh \"\"\"
|
||||
(master_env_v156_py36) 23:35 [viral-ngs-benchmarks] $ git annex copy -J16 --to viral-ngs-benchmarks-s3 --not --in=viral-ngs-benchmark\
|
||||
s-s3 --and --not --in=mygs --and --not --in=dnanexus
|
||||
copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades-3.13.0/files/calls/assemble_denovo/assemb\
|
||||
le/0/outputs/subsampBam/LASV_NGA_2016_0014.ll2.subsamp.bam (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok
|
||||
copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades-3.13.0/files/calls/assemble_denovo/refine\
|
||||
_2x_and_plot/0/outputs/aligned_bam_idx/LASV_NGA_2016_0014.ll2.all.bai (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s\
|
||||
3...) ok
|
||||
copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades-3.13.0/files/calls/assemble_denovo/refine\
|
||||
_2x_and_plot/0/outputs/aligned_only_reads_bam/LASV_NGA_2016_0014.ll2.mapped.bam (checking viral-ngs-benchmarks-s3...) (to viral-ngs-be\
|
||||
nchmarks-s3...) ok
|
||||
...
|
||||
|
||||
copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades-3.13.0/cromwell_submit_output.txt (checki\
|
||||
ng viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...)
|
||||
S3Error {s3StatusCode = Status {statusCode = 403, statusMessage = \"Forbidden\"}, s3ErrorCode = \"SignatureDoesNotMatch\", s3ErrorMessag\
|
||||
e = \"The request signature we calculated does not match the signature you provided. Check your key and signing method.\", s3ErrorResour\
|
||||
ce = Nothing, s3ErrorHostId = Just \"7uk2qBwWawhwYhROH7SzY+i3Y/49OQ9OiIT81eRkoEvZowo56xVkATE9qPU4Zs78x4C7gPsu744=\", s3ErrorAccessKeyId \
|
||||
= Just \"AKIAXXXXXXXXXXXXXXX\", s3ErrorStringToSign = Just \"PUT\n\nin\nThu, 05 Sep 2019 03:35:47 GMT\nx-amz-storage-class:STANDARD\n/vi\
|
||||
ral-ngs-benchmarks/MD5E-s628-S16777216-C1--065d8c01e5f7bd3b997c6d24109005c7.txt\", s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothin\
|
||||
g, s3ErrorEndpoint = Nothing}
|
||||
ok
|
||||
|
||||
...
|
||||
copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades/files/calls/assemble_denovo/refine_2x_and\
|
||||
_plot/0/outputs/aligned_only_reads_bam/LASV_NGA_2016_0014.ll2.mapped.bam (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmark\
|
||||
s-s3...) ok
|
||||
copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades/files/calls/assemble_denovo/assemble/0/st\
|
||||
dout/stdout (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok
|
||||
double free or corruption (out)
|
||||
error: git-annex died of signal 6
|
||||
(master_env_v156_py36) 23:35 [viral-ngs-benchmarks] $ git annex copy --to viral-ngs-benchmarks-s3 --not --in=viral-ngs-benchmarks-s3\
|
||||
--and --not --in=mygs --and --not --in=dnanexus
|
||||
...
|
||||
(recording state in git...)
|
||||
(master_env_v156_py36) 01:17 [viral-ngs-benchmarks] $ echo $?
|
||||
0
|
||||
|
||||
|
||||
|
||||
\"\"\"]]
|
||||
"""]]
|
|
@ -0,0 +1,67 @@
|
|||
### Please describe the problem.
|
||||
|
||||
It is probably not a regression (since happens with now elderly git-annex in debian sid), it could be something either changed on debian sid or on datalad side (originally ran into it while trying to build 0.11.7 package for debian sid) to make test directory name even more "tricky" than before.
|
||||
|
||||
Summary: `git annex init` fails even though `git init` (and other commands, not shown here) works just fine:
|
||||
|
||||
|
||||
[[!format sh """
|
||||
root@smaug:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1# pwd
|
||||
/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1
|
||||
|
||||
# git init was ran before already
|
||||
root@smaug:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1# git init
|
||||
Reinitialized existing Git repository in /tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1/.git/
|
||||
|
||||
root@smaug:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1# git annex init
|
||||
init fatal: Unable to create '/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&c `| 1/.git/annex/index.lock': No such file or directory
|
||||
fatal: Unable to create '/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&c `| 1/.git/annex/index.lock': No such file or directory
|
||||
|
||||
git-annex: failed to read sha from git write-tree
|
||||
CallStack (from HasCallStack):
|
||||
error, called at ./Git/Sha.hs:18:15 in main:Git.Sha
|
||||
failed
|
||||
git-annex: init: 1 failed
|
||||
|
||||
root@smaug:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1# git annex version
|
||||
git-annex version: 7.20190129
|
||||
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 5 7
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6
|
||||
|
||||
"""]]
|
||||
|
||||
so above version is the one in Debian proper (apparently quite dated!). I have tried with 7.20190819+git2-g908476a9b-1~ndall+1 and 7.20190819+git60-gcdb679818-1~ndall+1
|
||||
|
||||
You can see that `git-annex` got unicode portion of the directory name stripped (reports ``` "';a&b&c `| 1``` instead of ``` "';a&b&cΔЙקم๗あ `| 1```).
|
||||
|
||||
My wild guess is that full path (instead of a relative path) is used somewhere, encoded/decoded with losses, and then kaboom comes.
|
||||
|
||||
<details>
|
||||
<summary>locale is just C</summary>
|
||||
|
||||
[[!format sh """
|
||||
root@smaug:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1# locale
|
||||
LANG=C
|
||||
LANGUAGE=
|
||||
LC_CTYPE="C"
|
||||
LC_NUMERIC="C"
|
||||
LC_TIME="C"
|
||||
LC_COLLATE="C"
|
||||
LC_MONETARY="C"
|
||||
LC_MESSAGES="C"
|
||||
LC_PAPER="C"
|
||||
LC_NAME="C"
|
||||
LC_ADDRESS="C"
|
||||
LC_TELEPHONE="C"
|
||||
LC_MEASUREMENT="C"
|
||||
LC_IDENTIFICATION="C"
|
||||
LC_ALL=C
|
||||
"""]]
|
||||
</details>
|
||||
|
||||
[[!meta author=yoh]]
|
|
@ -0,0 +1,9 @@
|
|||
### Please describe the problem.
|
||||
|
||||
`git-annex-copy --json --json-error-messages` writes errors messages to stdout, but not to stderr. This can be confusing in that the stderr output is empty, even though there were errors. It would be more intuitive to (also) output error messages as json records to stderr.
|
||||
|
||||
Also, the error messages that *are* written to stderr are not always in json format, e.g. the `git-annex: thread blocked indefinitely in an STM transaction` messages in [[bugs/git-annex-copy_fails_with___34__thread_blocked_indefinitely_in_an_STM_transaction__34__]]. Adding `--debug --verbose` also causes some non-json output.
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
git-annex has been an extremely flexible and versatile tool. So far I've been able to adapt to most usage scenarios I've encountered.
|
|
@ -0,0 +1,56 @@
|
|||
### Please describe the problem.
|
||||
`git-annex-copy` fails after a short time with the message "thread blocked indefinitely in an STM transaction". Re-running it brings the same problem.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
See below
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
[[!format sh """
|
||||
(master_env_v156_py36) 14:22 [viral-ngs-benchmarks] $ git annex version
|
||||
git-annex version: 7.20190819-ge4cecf8
|
||||
build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sql\
|
||||
ite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_2\
|
||||
24 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE\
|
||||
2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224\
|
||||
BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 5 7
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6
|
||||
local repository version: 5
|
||||
(master_env_v156_py36) 14:24 [viral-ngs-benchmarks] $ uname -a
|
||||
Linux ip-172-31-81-103.ec2.internal 4.14.138-114.102.amzn2.x86_64 #1 SMP Thu Aug 15 15:29:58 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
|
||||
(master_env_v156_py36) 14:24 [viral-ngs-benchmarks] $
|
||||
|
||||
"""]]
|
||||
|
||||
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
# If you can, paste a complete transcript of the problem occurring here.
|
||||
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
|
||||
|
||||
(master_env_v156_py36) 14:18 [viral-ngs-benchmarks] $ git annex --debug --verbose copy -J1 --not --in dnanexus --and --not --in mygs \
|
||||
--to viral-ngs-benchmarks-s3 --json --json-error-messages > tmp/copy.out.txt 2> tmp/copy.err.txt
|
||||
C-c C-c
|
||||
(master_env_v156_py36) 14:22 [viral-ngs-benchmarks] $ git annex copy -J1 --not --in dnanexus --and --not --in mygs --to viral-ngs-ben\
|
||||
chmarks-s3 --json --json-error-messages > tmp/copy.out.txt 2> tmp/copy.err.txt
|
||||
(master_env_v156_py36) 14:22 [viral-ngs-benchmarks] $ echo $?
|
||||
1
|
||||
(master_env_v156_py36) 14:22 [viral-ngs-benchmarks] $ cat tmp/copy.err.txt
|
||||
git-annex: thread blocked indefinitely in an STM transaction
|
||||
git-annex: thread blocked indefinitely in an STM transaction
|
||||
(
|
||||
|
||||
# End of transcript or log.
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="the issue persists in master"
|
||||
date="2019-09-02T14:17:14Z"
|
||||
content="""
|
||||
FWIW, the issue persists with current version in master (which comes after comment from 3 days ago where wider exception catching was told to be implemented):
|
||||
|
||||
[[!format sh \"\"\"
|
||||
[bids@rolando git-annex] > ~/git-annexes/datalad/tools/bisect-git-annex.scripts/bisect-nfs-lock.sh
|
||||
Initialized empty Git repository in /inbox/BIDS/.git/tmp/repo/.git/
|
||||
init
|
||||
git-annex: waitToSetLock: resource exhausted (No locks available)
|
||||
failed
|
||||
git-annex: init: 1 failed
|
||||
[bids@rolando git-annex] > echo $?
|
||||
1
|
||||
[bids@rolando git-annex] > git annex version
|
||||
git-annex version: 7.20190826-gf951a6221
|
||||
build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 5 7
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6
|
||||
|
||||
\"\"\"]]
|
||||
"""]]
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="buffering chunks in memory when uploading to remote"
|
||||
date="2019-09-04T19:50:42Z"
|
||||
content="""
|
||||
\"git-annex has to buffer chunks in memory before they are sent to a remote\" -- would it be hard to remove this restriction? I want to have a non-chunked S3 remote, so that files in it can be accessed through a URL. But if some files are 50GB, then git-annex running on a 16GB machine would fail when talking to this repo?
|
||||
"""]]
|
|
@ -0,0 +1,3 @@
|
|||
Hi,
|
||||
|
||||
Is it possible to use hashdirlower layout with S3 special remote, instead of storing all the files in the root of the bucket?
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="TroisSinges"
|
||||
avatar="http://cdn.libravatar.org/avatar/f4c62f9ebe09016b8852b04f1dd6612f"
|
||||
subject="comment 1"
|
||||
date="2019-09-02T22:19:04Z"
|
||||
content="""
|
||||
Hmmm, this is strange, because according to <http://git-annex.branchable.com/special_remotes/S3/#comment-e9d48ba37aa5b01c236ab894057660b7>, files should be \"stored in a hashed directory structure with the names of their key used\". This isn't the case for me, using S3 special remote for Wasabi.
|
||||
"""]]
|
|
@ -0,0 +1,16 @@
|
|||
[[!comment format=mdwn
|
||||
username="johnmario.itec19@69a7b742534851b36216e0f951f1a00dbb9067cd"
|
||||
nickname="johnmario.itec19"
|
||||
avatar="http://cdn.libravatar.org/avatar/2f07ffce1656bdcd6aa19aaab7517975"
|
||||
subject="commenting on git-annex-add"
|
||||
date="2019-09-02T06:21:27Z"
|
||||
content="""
|
||||
Yes you can do that. Simplest way is to git add the files you want to directly be in the git repo (e.g. the source code) and git annex add the large files.
|
||||
|
||||
You can then check in any changes to the source code files (or anything else you added with git add) to github as normal.
|
||||
|
||||
You can manage the storage and versioning of the large files using git annex commands. Git annex supports using AWS S3 and/or glacier for backing up the files. It can also back them up to a server you control over ssh or to an external drive (or any combination of the above). http://git-annex.branchable.com/special_remotes/
|
||||
|
||||
With the latest version of git annex, you can also set up automatically filters that decide which types/sizes of files to check in directly to git vs which ones to store as links in the annex. https://git-annex.branchable.com/tips/largefiles/
|
||||
For more tech related assistance or support <a href=\"https://uaedatarecovery.com/data-recovery-dubai/\">Data Recovery Dubai</a>
|
||||
"""]]
|
|
@ -0,0 +1,15 @@
|
|||
[[!comment format=mdwn
|
||||
username="ginquistador@86f226616ead98d2733e249429918f241f928064"
|
||||
nickname="ginquistador"
|
||||
avatar="http://cdn.libravatar.org/avatar/f0ef7d68c0ff5d4948a9b0d282987195"
|
||||
subject="Disappointed with `git add`"
|
||||
date="2019-09-03T07:30:28Z"
|
||||
content="""
|
||||
I first have to say, I have been following and using git annex for ages (5+ years at least), and is my trusted source for all my data. However, for the first time in all these years, I'm seeing a decision that I do not agree with or understand.
|
||||
|
||||
Specifically, using `git add .` to add a file to git annex as the default pattern just seems a fundamentally wrong design to me (at least for my usage pattern). I want to be able to use git normally, and have git-annex only get involved when I explicitly request it to, and not for all files. AFAIK, git-lfs does do it right. I understand [annex.largefiles: configuring mixed content repositories](http://git-annex.branchable.com/tips/largefiles/) can be configured to get the behavior I want. However, the default behavior should add it to vanilla git, and any other desired behavior can be obtained by the user via annex attributes, or extra command line flags to `git annex add`
|
||||
|
||||
Knowing Joey, I assume there's a strong rationale as always, and would love to hear it, but I would still like to STRONGLY REQUEST changing the default behavior.
|
||||
|
||||
|
||||
"""]]
|
3
doc/todo/S3_export_redirecting_to_key-value_store.mdwn
Normal file
3
doc/todo/S3_export_redirecting_to_key-value_store.mdwn
Normal file
|
@ -0,0 +1,3 @@
|
|||
S3 lets you [redirect](https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html) requests for an object to another object, or to a URL. This could be used to export a git branch, in the manner of [[`git-annex-export`|git-annex-export]], but with annexed objects redirecting to a key-value S3 remote in the same bucket.
|
||||
|
||||
Related: [[todo/simpler__44___trusted_export_remotes]] ; [[forum/Using_hashdirlower_layout_for_S3_special_remote]].
|
Loading…
Reference in a new issue