From 330a8a27e0024d00d2620f66790d17950371c0b6 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 3 Jul 2019 21:38:06 +0000 Subject: [PATCH] added bug report for failing parallel copy --- doc/bugs/parallel_copy_fails.mdwn | 79 +++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 doc/bugs/parallel_copy_fails.mdwn diff --git a/doc/bugs/parallel_copy_fails.mdwn b/doc/bugs/parallel_copy_fails.mdwn new file mode 100644 index 0000000000..f12f3556f7 --- /dev/null +++ b/doc/bugs/parallel_copy_fails.mdwn @@ -0,0 +1,79 @@ +### Please describe the problem. + +I keep encountering situations where git-annex fails on parallel operations. Retrying the operation several times sometimes resolves the issue -- it would be better if git-annex could be configured to do the retrying. I have set `annex.retry` and `annex.retry-delay`, but these don't seem to be enough. Maybe, these config vars could apply to all idempotent operations? + +I have also seen a parallel copy operation, copying to an S3 remote, hang indefinitely. Killing and retrying fixed the problem; it would be better if git-annex could have a timeout for operations, and kill and retry them automatically. This especially matters when building scripts on top of git-annex. + +### What steps will reproduce the problem? + +I have included the log below. I think repeatedly trying to copy from a local repo to an S3 bucket (non-export-tree) should eventually reproduce the problem. + +### What version of git-annex are you using? On what operating system? + +[[!format sh """ + +(master_env_v142_py36) 17:29 [v1.23.0-rc29] $ git annex version +git-annex version: 7.20190626-g85db42d +build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite +dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.4.2 http-client-0.5.14 persistent-sql\ +ite-2.9.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_2\ +24 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE\ +2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256\ + BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external +operating system: linux x86_64 +supported repository versions: 5 7 +upgrade supported from repository versions: 0 1 2 3 4 5 6 +local repository version: 5 +(master_env_v142_py36) 17:32 [v1.23.0-rc29] $ uname -a +Linux ip-172-31-81-187.ec2.internal 4.14.128-112.105.amzn2.x86_64 #1 SMP Wed Jun 19 16:53:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux +(master_env_v142_py36) 17:32 [v1.23.0-rc29] $ + +# End of transcript or log. +"""]] + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + +2019-07-03 17:30:22,365 - workflow_utils:289:_run - INFO - running command: git annex copy . --to=viral-ngs-benchmarks-s3 cwd=/dev/shm\ +/crogrun_190703_162305_36571/viral-ngs-benchmarks kw={'cwd': '/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks/benchmarks/rab\ +ies200/analysis-FK7B7p00761bfq2516zQb58F/benchmark_variants/v1.23.0-rc29'} +copy assemble_denovo.wdl (checking viral-ngs-benchmarks-s3...) ok +copy input-spec.json (checking viral-ngs-benchmarks-s3...) ok +copy analysis_labels.json (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok +copy inputs-git-links.json (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok +copy inputs-local.json (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok +copy inputs-orig.json (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok +copy imports.zip (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok +(recording state in git...) +error: git-annex died of signal 11 +2019-07-03 17:30:23,429 - workflow_utils:303:_run - INFO - RETRY 2 sleep 4 command (cwd=/dev/shm/crogrun_190703_162305_36571/viral-ngs\ +-benchmarks, kw={'cwd': '/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks/benchmarks/rabies200/analysis-FK7B7p00761bfq2516zQb\ +58F/benchmark_variants/v1.23.0-rc29'}) in 1.063608169555664s: git annex copy . --to=viral-ngs-benchmarks-s3 exception Command 'git ann\ +ex copy . --to=viral-ngs-benchmarks-s3' returned non-zero exit status 139. +2019-07-03 17:30:27,434 - workflow_utils:312:_run - INFO - command (cwd=/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks, kw=\ +{'cwd': '/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks/benchmarks/rabies200/analysis-FK7B7p00761bfq2516zQb58F/benchmark_va\ +riants/v1.23.0-rc29'}) FAILED in 5.06814980506897s: git annex copy . --to=viral-ngs-benchmarks-s3 +copy assemble_denovo.wdl (checking viral-ngs-benchmarks-s3...) ok +copy analysis_labels.json (checking viral-ngs-benchmarks-s3...) ok +copy inputs-local.json (checking viral-ngs-benchmarks-s3...) ok +copy inputs-orig.json (checking viral-ngs-benchmarks-s3...) ok +copy input-spec.json (checking viral-ngs-benchmarks-s3...) ok +copy imports.zip (checking viral-ngs-benchmarks-s3...) ok +copy inputs-git-links.json (checking viral-ngs-benchmarks-s3...) ok +2019-07-03 17:30:27,952 - workflow_utils:312:_run - INFO - command (cwd=/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks, kw=\ +{'cwd': '/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks/benchmarks/rabies200/analysis-FK7B7p00761bfq2516zQb58F/benchmark_va\ +riants/v1.23.0-rc29'}) SUCCEEDED in 5.586166620254517s: git annex copy . --to=viral-ngs-benchmarks-s3 +2 + + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +