8faeb25076
Untested and not even compiled yet. Testing should include checks that file content streams through without buffering in memory. Note that CL.consume causes all the etags to be buffered in memory. This is probably nearly unavoidable, since a request has to be constructed that contains the list of etags in its body. (While it might be possible to stream generation of the body, that would entail making a http request that dribbles out parts of the body as the multipart uploads complete, which is not likely to work well.. To limit this being a problem, it's best for partsize to be set to some suitably large value, like 1gb. Then a full terabyte file will need only 1024 etags to be stored, which will probably use around 1 mb of memory.
62 lines
2 KiB
Markdown
62 lines
2 KiB
Markdown
What steps will reproduce the problem?
|
|
|
|
> Try to copy/move a file greater than 5G to S3.
|
|
|
|
git annex copy large_file.tgz --to cloud
|
|
|
|
What is the expected output? What do you see instead?
|
|
|
|
> Looks like git-annex may not be using the Multipart Upload API: http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html
|
|
|
|
> Expected transfer to succeed, instead this error is output:
|
|
|
|
copy large-file.tgz (gpg) (checking cloud...) (to cloud...) Reading passphrase from file descriptor 12
|
|
|
|
|
|
Your proposed upload exceeds the maximum allowed size
|
|
failed
|
|
git-annex: copy: 1 failed
|
|
|
|
What version of git-annex are you using? On what operating system?
|
|
|
|
> OSX 10.8.2
|
|
|
|
Please provide any additional information below.
|
|
|
|
annex [master●] % git annex status
|
|
supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
|
|
supported remote types: git S3 bup directory rsync web webdav glacier hook
|
|
repository mode: indirect
|
|
trusted repositories: 0
|
|
semitrusted repositories: 3
|
|
00000000-0000-0000-0000-000000000001 -- web
|
|
BE1D8EC7-C64B-47DE-AD4E-2A50437532B4 -- cloud
|
|
E84568BA-6A4B-4AA1-B622-605B9248EDB1 -- here (eric laptop)
|
|
untrusted repositories: 0
|
|
dead repositories: 0
|
|
transfers in progress: none
|
|
available local disk space: 169 gigabytes (+1 megabyte reserved)
|
|
temporary directory size: 218 megabytes (clean up with git-annex unused)
|
|
local annex keys: 24
|
|
local annex size: 8 gigabytes
|
|
known annex keys: 25
|
|
known annex size: 8 gigabytes
|
|
bloom filter size: 16 mebibytes (0% full)
|
|
backend usage:
|
|
SHA256E: 49
|
|
annex [master●] % git annex version
|
|
git-annex version: 3.20130114
|
|
local repository version: 3
|
|
default repository version: 3
|
|
supported repository versions: 3
|
|
upgrade supported from repository versions: 0 1 2
|
|
|
|
[[!tag confirmed]]
|
|
|
|
> [[fixed|done]] This is now supported, when git-annex is built with a new
|
|
> enough version of the aws library. You need to configure the remote to
|
|
> use an appropriate value for multipart, eg:
|
|
>
|
|
> git annex enableremote cloud multipart=1GiB
|
|
>
|
|
> --[[Joey]]
|