work around minimum part size problem

When uploading the last part of a file, which was 640229 bytes, S3 rejected
that part: "Your proposed upload is smaller than the minimum allowed size"

I don't know what the minimum is, but the fix is just to include the last
part into the previous part. Since this can result in a part that's
double-sized, use half-sized parts normally.
This commit is contained in:
Joey Hess 2014-11-04 16:06:13 -04:00
parent ad2125e24a
commit a42022d8ff
2 changed files with 19 additions and 8 deletions

View file

@ -181,9 +181,16 @@ store r h = fileStorer $ \k f p -> do
}
uploadid <- S3.imurUploadId <$> sendS3Handle h startreq
-- The actual part size will be a even multiple of the
-- 32k chunk size that hGetUntilMetered uses.
let partsz' = (partsz `div` toInteger defaultChunkSize) * toInteger defaultChunkSize
{- The actual part size will be a even multiple of the
- 32k chunk size that hGetUntilMetered uses.
-
- Also, half-size parts are used. This is so that
- the final part of a file can be rolled into the
- last full-size part, which avoids a problem when the
- final part could otherwise be too small for S3 to accept
- it.
-}
let partsz' = (partsz `div` toInteger defaultChunkSize `div` 2) * toInteger defaultChunkSize
-- Send parts of the file, taking care to stream each part
-- w/o buffering in memory, since the parts can be large.
@ -195,7 +202,7 @@ store r h = fileStorer $ \k f p -> do
else do
-- Calculate size of part that will
-- be read.
let sz = if fsz - pos < partsz'
let sz = if fsz - pos < partsz' * 2
then fsz - pos
else partsz'
let p' = offsetMeterUpdate p (toBytesProcessed pos)

View file

@ -21,10 +21,14 @@ the S3 remote.
* `chunk` - Enables [[chunking]] when storing large files.
`chunk=1MiB` is a good starting point for chunking.
* `partsize` - Specifies the largest object to attempt to store in the
bucket. Multipart uploads will be used when storing larger objects.
This is not enabled by default, but can be enabled or changed at any
time. Setting `partsize=1GiB` is reasonable for S3.
* `partsize` - Amazon S3 only accepts uploads up to a certian file size,
and storing larger files requires a multipart upload process.
Setting `partsize=1GiB` is recommended for Amazon S3; this will
cause multipart uploads to be done using parts up to 1GiB in size.
This is not enabled by default, since other S3 implementations may
not support multipart uploads, but can be enabled or changed at any
time.
* `keyid` - Specifies the gpg key to use for [[encryption]].