mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

use 20% less memory when listing unversioned S3 bucket

Browse source
This commit is contained in:
Joey Hess 2024-11-15 13:24:13 -04:00
parent 43a4adda6e
commit eb714c107b
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 35 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

16
doc/bugs/importtree_from_S3_slows_to_halt_even_with_prefix/comment_7_fe6e9bc5460f9bcd24eb3034a2f45fbc._comment Normal file
View file

@ -0,0 +1,16 @@
[[!comment format=mdwn
username="joey"
subject="""comment 7"""
date="2024-11-15T17:16:51Z"
content="""
Trying the same command but with versioning=yes, I have verified that
* it does not have the same loop forever behavior
* it does use a lot of memory quite quickly
Going back to the unversioned command, I was able to reduce the memory use
by 20% by processing each result, rather than building up a list of results
and processing at the end. It will be harder to do that in the versioning
case, but I expect it will improve it at least that much, and probably
more, since it will be able to GC all the delete markers.
"""]]