mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

WIP multipart S3 upload

Browse source 
I'm a little stuck on getting the list of etags of the parts.
This seems to require taking the md5 of each part locally,
which doesn't get along well with lazily streaming in the part from the
file. It would need to read the file twice, or lose laziness and buffer a
whole part -- but parts might be quite large.

This seems to be a problem with the API provided; S3 is supposed to return
an etag, but that is not exposed. I have filed a bug:
https://github.com/aristidb/aws/issues/141
This commit is contained in:
Joey Hess 2014-10-28 14:17:30 -04:00
parent e535ff8fa4
commit 6e89d070bc
3 changed files with 49 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

5
doc/special_remotes/S3.mdwn
View file

@ -21,6 +21,11 @@ the S3 remote.
* `chunk` - Enables [[chunking]] when storing large files.
`chunk=1MiB` is a good starting point for chunking.
* `partsize` - Specifies the largest object to attempt to store in the
bucket. Multipart uploads will be used when storing larger objects.
This is not enabled by default, but can be enabled or changed at any
time. Setting `partsize=1GiB` is reasonable for S3.
* `keyid` - Specifies the gpg key to use for [[encryption]].
* `embedcreds` - Optional. Set to "yes" embed the login credentials inside