mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Browse source
This commit is contained in:
david@8424c56b088b73b1f9bd7d0497ca9b5752776099 2019-08-03 22:42:50 +00:00 • committed by admin
parent fba7da498b
commit 9108927b86
1 changed files with 61 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

61
doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails.mdwn Normal file
View file

@ -0,0 +1,61 @@
### Please describe the problem.
I'm trying to copy a directory full of photos (~9GB) to an S3 special remote. It seems to fail when running multiple jobs. The machine I am using has two cores (4 logical due to hyperthreading) and I initially used -J8 and that failed quickly. I haven't retried that yet to see if it always fails quickly but I immediately retried with -J4 and that seemed to run okay for a little over an hour and a half before failing again. This is the second large directory I've tried to copy to this remote and had a similar problem the last time (many retries seemed to allow me to get through).
### What steps will reproduce the problem?
### What version of git-annex are you using? On what operating system?
git-annex was installed via homebrew on macos 10.14.5
[[!format sh """
15:23 $ git annex version
git-annex version: 7.20190322
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.19 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.2.2 http-client-0.5.14 persistent-sqlite-2.6.4 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLA
KE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
operating system: darwin x86_64
supported repository versions: 5 7
upgrade supported from repository versions: 0 1 2 3 4 5 6
"""]]
### Please provide any additional information below.
[[!format sh """
22:56 $ git annex copy -J8 photos/ --to s3
copy photos/2001/01/IMAG0003.JPG (checking s3...) (to s3...) ok
copy photos/2001/01/IMAG0006.JPG (checking s3...) (to s3...)
88% 319.84 KiB 68 KiB/s 0s
copy photos/2001/01/IMAG0001-1.JPG (checking s3...) (to s3...)
63% 255.88 KiB 25 KiB/s 6s
copy photos/2001/01/IMAG0002.JPG (checking s3...) (to s3...)
63% 223.89 KiB 17 KiB/s 7s
copy photos/2001/01/IMAG0004.JPG (checking s3...) (to s3...)
copy photos/2001/01/IMAG0005.JPG (checking s3...) (to s3...)
80% 287.86 KiB 159 KiB/s 0s
copy photos/2001/01/IMAG0001.JPG (checking s3...) (to s3...)
74% 255.88 KiB 49 KiB/s 1s
copy photos/2001/01/IMAG0002-1.JPG (checking s3...) (to s3...)
ParseError {errorContexts = [], errorMessage = "Failed reading: takeWhile1", errorPosition = 4:2 (285)}
copy photos/2001/01/IMAG0007.JPG (checking s3...) (to s3...)
10% 31.98 KiB 102 KiB/s 2s
git-annex(21112,0x70000dd5b000) malloc: Incorrect checksum for freed object 0x7fe0b3e02bf8: probably modified after being freed.
Corrupt value: 0x6574636f2f6e6f69
git-annex(21112,0x70000dd5b000) malloc: *** set a breakpoint in malloc_error_break to debug
error: git-annex died of signal 6
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I'm a new git-annex user and besides these occasional issues with concurrent with git-annex-copy I found it really easy to get going with git-annex and to manage my imported files. In particular I really appreciate the extensive documentation!