mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

blew a day investigating this and still don't understand it fully

Browse source
This commit is contained in:
Joey Hess 2020-10-13 15:33:59 -04:00
parent cf0ff9f53d
commit d41795c8ff
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 29 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

13
doc/todo/memory_use_increase/comment_6_8329ffed8b4e84440ae5326d40ece517._comment Normal file
View file

@ -0,0 +1,13 @@
[[!comment format=mdwn
username="joey"
subject="""comment 6"""
date="2020-10-13T18:59:08Z"
content="""
In the alternatepipenullsplit branch, I made what I think is the relevant
part of LsFiles not use a lazy bytestring, but instead read chunks
directly to strict bytestrings. It did not change the PINNED amount or reduce
overall memory use.
So the pinning must be happening somewhere other than where I had assumed
it was happening.
"""]]

16
doc/todo/memory_use_increase/comment_7_68f580f2df32ee9027f70b4407451937._comment Normal file
View file

@ -0,0 +1,16 @@
[[!comment format=mdwn
username="joey"
subject="""comment 7"""
date="2020-10-13T19:15:41Z"
content="""
The seekHelper version in comment 3 significantly changes the
shape of the profile. PINNED is still there, but only a couple mb,
and it's a constant amount of memory, not an upward slope.
So I think that identifies the main culprit for the mysterious PINNED.
runSegmentPaths is supposed to stream, but it seems it somehow is hanging
on to values longer than it should.
Although the max memory use of that version is still around 90 mb
according to `time`. The profile is flat at 3500 kb.
"""]]