mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

comment

Browse source
This commit is contained in:
Joey Hess 2016-09-05 13:08:35 -04:00
parent 47b0cb943b
commit 30aa817f80
No known key found for this signature in database
GPG key ID: C910D9222512E3C7
1 changed files with 26 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

26
doc/forum/Large_Uploads_to_S3__63__/comment_1_c1fd2ed0b74ce58f818ab53158e581f3._comment Normal file
View file

@ -0,0 +1,26 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2016-09-05T16:52:24Z"
content="""
Googling for that message suggests it's pretty common for large file
uploads amoung different AWS implementations for different programming
languages. The error message is coming from AWS not git-annex.
If this is happening with a single file transfer, I'm pretty sure git-annex
is not keeping the S3 connection idle. (If one file transfer succeeded, and
then a later one in the same git-annex run failed, that might indicate that
the S3 connection was being reused and timed out in between.)
Based on things like <https://github.com/aws/aws-cli/issues/401>,
this seems to be down to a network connection problem, especially on
residential internet connections, such as the link
getting saturated by something else and so the transfer stalling out.
I think that finding a chunk size that works is your best bet. That
will let uploads be resumed more or less where they left off.
It might make sense for git-annex to retry an upload that fails this way,
but imagine if it were a non-chunked 1 gb file and it failed part way
through every time. That would waste a lot of bandwidth.
"""]]