mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

followup

Browse source
This commit is contained in:
Joey Hess 2019-09-06 14:20:39 -04:00
parent f70da1b359
commit 796a49a8c7
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
1 changed files with 24 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

24
doc/forum/Using_hashdirlower_layout_for_S3_special_remote/comment_5_79519a4c0fc25d41226a7048cd3a8306._comment Normal file
View file

@ -0,0 +1,24 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2019-09-06T17:53:00Z"
content="""
The stuff @Ilya found about prefix similaries causing bottlenecks in S3
infra is interesting. git-annex keys have a common prefix, and have
the hash at the end. So it could have a performance impact.
But that info also seems out of date when it talks about 6-8 characters
prefix length. And the rate limit has also been raised significantly, to
3000-5000 ops/sec per prefix. See
<https://stackoverflow.com/questions/52443839/s3-what-exactly-is-a-prefix-and-what-ratelimits-apply>
From that, it seems S3 does actually treat '/' as a prefix delimiter.
(Based on a single not very clear email from Amazon support and not
documented anywhere else..) So a single level of hash "directories"
could increase the rate limit accordingly.
If a git-annex exceeded those rate limits, it would start getting 503
responses from S3, so it wouldn't slow down but would instead fail whatever
operation it was doing. I can't recall anyone complaining of 503's from
S3.
"""]]