work around minimum part size problem
When uploading the last part of a file, which was 640229 bytes, S3 rejected that part: "Your proposed upload is smaller than the minimum allowed size" I don't know what the minimum is, but the fix is just to include the last part into the previous part. Since this can result in a part that's double-sized, use half-sized parts normally.
This commit is contained in:
parent
ad2125e24a
commit
a42022d8ff
2 changed files with 19 additions and 8 deletions
15
Remote/S3.hs
15
Remote/S3.hs
|
@ -181,9 +181,16 @@ store r h = fileStorer $ \k f p -> do
|
|||
}
|
||||
uploadid <- S3.imurUploadId <$> sendS3Handle h startreq
|
||||
|
||||
-- The actual part size will be a even multiple of the
|
||||
-- 32k chunk size that hGetUntilMetered uses.
|
||||
let partsz' = (partsz `div` toInteger defaultChunkSize) * toInteger defaultChunkSize
|
||||
{- The actual part size will be a even multiple of the
|
||||
- 32k chunk size that hGetUntilMetered uses.
|
||||
-
|
||||
- Also, half-size parts are used. This is so that
|
||||
- the final part of a file can be rolled into the
|
||||
- last full-size part, which avoids a problem when the
|
||||
- final part could otherwise be too small for S3 to accept
|
||||
- it.
|
||||
-}
|
||||
let partsz' = (partsz `div` toInteger defaultChunkSize `div` 2) * toInteger defaultChunkSize
|
||||
|
||||
-- Send parts of the file, taking care to stream each part
|
||||
-- w/o buffering in memory, since the parts can be large.
|
||||
|
@ -195,7 +202,7 @@ store r h = fileStorer $ \k f p -> do
|
|||
else do
|
||||
-- Calculate size of part that will
|
||||
-- be read.
|
||||
let sz = if fsz - pos < partsz'
|
||||
let sz = if fsz - pos < partsz' * 2
|
||||
then fsz - pos
|
||||
else partsz'
|
||||
let p' = offsetMeterUpdate p (toBytesProcessed pos)
|
||||
|
|
|
@ -21,10 +21,14 @@ the S3 remote.
|
|||
* `chunk` - Enables [[chunking]] when storing large files.
|
||||
`chunk=1MiB` is a good starting point for chunking.
|
||||
|
||||
* `partsize` - Specifies the largest object to attempt to store in the
|
||||
bucket. Multipart uploads will be used when storing larger objects.
|
||||
This is not enabled by default, but can be enabled or changed at any
|
||||
time. Setting `partsize=1GiB` is reasonable for S3.
|
||||
* `partsize` - Amazon S3 only accepts uploads up to a certian file size,
|
||||
and storing larger files requires a multipart upload process.
|
||||
Setting `partsize=1GiB` is recommended for Amazon S3; this will
|
||||
cause multipart uploads to be done using parts up to 1GiB in size.
|
||||
|
||||
This is not enabled by default, since other S3 implementations may
|
||||
not support multipart uploads, but can be enabled or changed at any
|
||||
time.
|
||||
|
||||
* `keyid` - Specifies the gpg key to use for [[encryption]].
|
||||
|
||||
|
|
Loading…
Reference in a new issue