git-annex/doc/bugs/S3_memory_leaks.mdwn

S3 has memory leaks

Sending a file to S3 causes a slow memory increase toward the file size.

> This is fixed, now that it uses aws. --[[Joey]]

Copying the file back from S3 causes a slow memory increase toward the
file size.

> [[fixed|done]] too! --[[Joey]]

The author of hS3 is aware of the problem, and working on it. I think I
have identified the root cause of the buffering; it's done by hS3 so it can
resend the data if S3 sends it a 307 redirect. --[[Joey]]

At least the send leak should be fixed by the patch in the s3-memory-leak
branch in git. That needs a patch to hS3, which I have sent to its author.
--[[Joey]] 
[[!tag confirmed]]
fix S3 upload buffering problem Provide file size to new version of hS3. 2011-04-21 14:31:54 +00:00			`S3 has memory leaks`
update 2011-04-19 20:03:27 +00:00
update on S3 memory leaks The remaining leaks are in hS3. The leak with encryption was worked around by the use of the temp file. (And was probably originally caused by gpgCipherHandle sparking a thread which kept a reference to the start of the byte string.) 2011-04-21 15:06:29 +00:00			`Sending a file to S3 causes a slow memory increase toward the file size.`
heard from hS3 author 2011-04-19 21:56:56 +00:00
remove dead code 2014-08-09 18:30:28 +00:00			`> This is fixed, now that it uses aws. --[[Joey]]`

update on S3 memory leaks The remaining leaks are in hS3. The leak with encryption was worked around by the use of the temp file. (And was probably originally caused by gpgCipherHandle sparking a thread which kept a reference to the start of the byte string.) 2011-04-21 15:06:29 +00:00			`Copying the file back from S3 causes a slow memory increase toward the`
			`file size.`
heard from hS3 author 2011-04-19 21:56:56 +00:00
S3: finish converting to aws library Implemented the Retriever. Unfortunately, it is a fileRetriever and not a byteRetriever. It should be possible to convert this to a byteRetiever, but I got stuck: The conduit sink needs to process individual chunks, but a byteRetriever needs to pass a single L.ByteString to its callback for processing. I looked into using unsafeInerlaveIO to build up the bytestring lazily, but the sink is already operating under conduit's inversion of control, and does not run directly in IO anyway. On the plus side, no more memory leak.. 2014-08-09 19:58:01 +00:00			`> [[fixed\|done]] too! --[[Joey]]`

root caused 2011-04-21 16:13:17 +00:00			`The author of hS3 is aware of the problem, and working on it. I think I`
			`have identified the root cause of the buffering; it's done by hS3 so it can`
			`resend the data if S3 sends it a 307 redirect. --[[Joey]]`
update; branch 2012-10-08 03:10:28 +00:00
			`At least the send leak should be fixed by the patch in the s3-memory-leak`
			`branch in git. That needs a patch to hS3, which I have sent to its author.`
			`--[[Joey]]`
bug triage; add new 'confirmed' tag 2014-05-24 20:29:41 +00:00			`[[!tag confirmed]]`