Joey Hess
0f78f197eb
casts; now fully working.. but still leaking
...
Still seems to buffer the whole partsize in memory, but I'm pretty sure my
code is not what's doing it. See https://github.com/aristidb/aws/issues/142
2014-11-03 21:12:15 -04:00
Joey Hess
f0551578d6
this should avoid leaking memory
2014-11-03 20:49:30 -04:00
Joey Hess
bd09046291
switch from hGetSome to hGet
...
This should be essentially no-op change for hGetContentsMetered, since it
always gets the entire contents. So the only difference is that each chunk
of the lazy bytestring will always be the full chunk size. So, I'm pretty
sure this is safe. Also, the only current users of hGetContentsMetered are
reading files, so the stream won't block for long in the middle.
The improvement is that hGetUntilMetered will always get some multiple of
the defaultChunkSize. This will allow the S3 multipart code to pick a fixed
size and know that hGetUntilMetered will really get that size.
2014-11-03 20:36:11 -04:00
Joey Hess
4230b56b79
logic error
2014-11-03 20:15:33 -04:00
Joey Hess
62de9a39bf
WIP 3
2014-11-03 20:04:42 -04:00
Joey Hess
d16382e99f
WIP 2
2014-11-03 19:50:33 -04:00
Joey Hess
5360417436
WIP try sending using RequestBodyStreamChunked
...
May not work; if it does this is gonna be the simplest way to get good
memory size and progress reporting.
2014-11-03 19:18:46 -04:00
Joey Hess
628304136b
Merge branch 'master' into s3-aws-multipart
2014-11-03 18:37:32 -04:00
Joey Hess
0602b26314
hGetUntilMetered
2014-11-03 18:37:05 -04:00
Joey Hess
8f61bfad51
link to memory leak bug
2014-11-03 17:55:05 -04:00
Joey Hess
068bcdb3f3
comment
2014-11-03 17:31:12 -04:00
Joey Hess
711b18a6eb
improve info display for multipart
2014-11-03 17:24:53 -04:00
Joey Hess
2c53f331bd
fix build
2014-11-03 17:23:46 -04:00
Joey Hess
6a965cf8d7
adjust version check
...
I assume 0.10.6 will have the fix for the bug I reported, which got fixed
in master already..
2014-11-03 16:23:00 -04:00
Joey Hess
5c3d9d6caa
show multipart configuration in git annex info s3remote
2014-11-03 16:07:41 -04:00
Joey Hess
a3ec6ed73b
Merge branch 'master' into s3-aws-multipart
2014-11-03 16:05:03 -04:00
Joey Hess
8faeb25076
finish multipart support using unreleased update to aws lib to yield etags
...
Untested and not even compiled yet.
Testing should include checks that file content streams through without
buffering in memory.
Note that CL.consume causes all the etags to be buffered in memory.
This is probably nearly unavoidable, since a request has to be constructed
that contains the list of etags in its body. (While it might be possible to
stream generation of the body, that would entail making a http request that
dribbles out parts of the body as the multipart uploads complete, which is
not likely to work well..
To limit this being a problem, it's best for partsize to be set to some
suitably large value, like 1gb. Then a full terabyte file will need only
1024 etags to be stored, which will probably use around 1 mb of memory.
2014-11-03 16:04:55 -04:00
https://www.google.com/accounts/o8/id?id=AItOawlUGPPMvAP5Hu0NyNqeRMPC4pANeJNHZ0o
3dcc73e8ba
2014-11-03 18:17:56 +00:00
Joey Hess
bf57edbdd6
comment
2014-11-03 13:17:55 -04:00
Joey Hess
46180eab4d
todo item
2014-11-03 13:06:26 -04:00
Joey Hess
4f80dcc02a
comment
2014-11-03 13:05:21 -04:00
Joey Hess
e57745b8d6
followup
2014-11-03 13:02:31 -04:00
Joey Hess
2819a466c7
close non-bug
2014-11-03 12:50:44 -04:00
Joey Hess
8a66df7848
comment
2014-11-03 12:48:57 -04:00
https://gregy.mojeid.cz/#RQxWK6xlii
ec2c4bf67c
2014-11-03 10:21:31 +00:00
https://gregy.mojeid.cz/#RQxWK6xlii
8887b6df78
2014-11-03 10:10:59 +00:00
https://gregy.mojeid.cz/#RQxWK6xlii
fb1a3d2867
2014-11-03 10:08:00 +00:00
https://gregy.mojeid.cz/#RQxWK6xlii
6ed2654ada
2014-11-03 10:04:36 +00:00
https://www.google.com/accounts/o8/id?id=AItOawlVB7wf1TFFrNPvM62fg78X-simVV-R1Cg
e83416496d
removed
2014-11-03 02:21:58 +00:00
https://www.google.com/accounts/o8/id?id=AItOawlVB7wf1TFFrNPvM62fg78X-simVV-R1Cg
0e4b6a65bd
Added a comment
2014-11-03 02:21:32 +00:00
https://www.google.com/accounts/o8/id?id=AItOawlVB7wf1TFFrNPvM62fg78X-simVV-R1Cg
06903f6c2c
Added a comment
2014-11-03 02:21:17 +00:00
https://www.google.com/accounts/o8/id?id=AItOawk9SYh6N-JUMkYkW4aOk55zC3Vr9KonDV4
3fd6a3171b
2014-11-02 19:19:28 +00:00
https://www.google.com/accounts/o8/id?id=AItOawlUGPPMvAP5Hu0NyNqeRMPC4pANeJNHZ0o
1b6b8a0633
Added a comment: why worry about modifications ?
2014-11-02 17:46:57 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo
3630a12420
2014-11-02 13:03:49 +00:00
https://id.koumbit.net/lavamind
8e7dd00353
2014-11-02 00:25:53 +00:00
http://svario.it/gioele
e99809528b
new bug "git annex status does not report added files"
2014-11-01 16:53:07 +00:00
http://svario.it/gioele
7447ef29fb
new wish "--dry-run option for all commands"
2014-11-01 16:40:56 +00:00
http://svario.it/gioele
624ae956de
new forum post "Why is git annex status slow?"
2014-11-01 14:12:35 +00:00
https://renaud.casenave.fr/
a029e55878
2014-11-01 01:57:53 +00:00
Joey Hess
bfac20c744
comment
2014-10-31 17:23:02 -04:00
Joey Hess
f1ef94fd05
comment
2014-10-31 16:42:49 -04:00
Joey Hess
1970563dd9
comment and associated todo
2014-10-31 16:39:19 -04:00
Joey Hess
746b1cb7bc
sssheeesh
2014-10-31 16:31:00 -04:00
Joey Hess
b67500d806
comment
2014-10-31 16:24:52 -04:00
Joey Hess
d1fe814a81
comment
2014-10-31 16:19:23 -04:00
Joey Hess
0f6aaf8012
Windows: Fix crash when user.name is not set in git config.
2014-10-31 16:14:12 -04:00
Joey Hess
8c622ec100
commnt
2014-10-31 15:48:52 -04:00
Joey Hess
86b326782a
getUserEntryForID does not work on windows
...
Throw an error instead, mentioning the relevant env vars.
2014-10-31 15:46:44 -04:00
Joey Hess
8f072c8db7
add example multiline data constructor
2014-10-31 12:41:08 -04:00
Joey Hess
a4810a4757
better organization and a few wording tweaks
2014-10-31 11:27:05 -04:00
