mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

update

Browse source
This commit is contained in:
Joey Hess 2020-10-19 14:48:39 -04:00
parent 72644d919a
commit 5009c1ce68
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
4 changed files with 66 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

29
doc/profiling/comment_8_951c1c047ef414268295626218474677._comment Normal file
View file

@ -0,0 +1,29 @@
[[!comment format=mdwn
username="joey"
subject="""comment 8"""
date="2020-10-19T18:27:56Z"
content="""
Update after some recent optimisations involving seekFilteredKeys.
Mon Oct 19 14:36 2020 Time and Allocation Profiling Report (Final)
git-annex +RTS -p -RTS find
total time = 1.32 secs (1316 ticks @ 1000 us, 1 processor)
total alloc = 583,359,800 bytes (excludes profiling overheads)
COST CENTRE MODULE SRC %time %alloc
keyFile Annex.Locations Annex/Locations.hs:(602,1)-(612,30) 13.2 32.7
readObjectContent Git.CatFile Git/CatFile.hs:(122,1)-(131,42) 10.9 2.3
copyAndFreeze Data.ByteArray.Methods Data/ByteArray/Methods.hs:(237,1)-(240,21) 5.2 0.8
withPtr Basement.Block.Base Basement/Block/Base.hs:(402,1)-(411,31) 5.2 4.1
parseLinkTarget Annex.Link Annex/Link.hs:(263,1)-(271,25) 4.5 12.3
fileKey Annex.Locations Annex/Locations.hs:(618,1)-(628,41) 4.5 8.1
decimal Data.Attoparsec.ByteString.Char8 Data/Attoparsec/ByteString/Char8.hs:(447,1)-(448,49) 4.3 4.5
doesPathExist Utility.RawFilePath Utility/RawFilePath.hs:32:1-25 3.9 0.6
ifM Utility.Monad Utility/Monad.hs:(54,1)-(56,44) 3.0 3.2
catObjectStream Git.CatFile Git/CatFile.hs:(322,1)-(330,45) 2.6 2.5
</> System.FilePath.Posix.ByteString System/FilePath/Posix/../Internal.hs:841:1-15 2.4 3.4
respParser Git.CatFile Git/CatFile.hs:(175,1)-(182,39) 2.4 1.3
"""]]

18
doc/profiling/comment_9_7e1a06690b52d241b2d57be0864d6f26._comment Normal file
View file

@ -0,0 +1,18 @@
[[!comment format=mdwn
username="joey"
subject="""comment 9"""
date="2020-10-19T18:35:02Z"
content="""
Also, /usr/bin/time git-annex find:
1.70user 0.27system 0:01.55elapsed 126%CPU (0avgtext+0avgdata 97352maxresident)k
0inputs+0outputs (0major+9303minor)pagefaults 0swaps
The maxresident seems high, but a stack profile does not show a memory
leak, or such a large amount of memory use at all. Currently, I
think that memory is being preallocated by the ghc runtime,
or something like that. (See [[todo/memory_use_increase]].)
ghc 8.8.4
Should keep an eye on this with newer ghc versions.
"""]]

2
doc/todo/memory_use_increase.mdwn
View file

@ -30,3 +30,5 @@ Version 8.20200720 (cat-file --buffer)
Smoking gun. And probably reasonable. But, why exactly does that optimisation
change the memory use in this way?
[[done]]

17
doc/todo/memory_use_increase/comment_10_362b878949e691e45fac0ff1aa240688._comment Normal file
View file

@ -0,0 +1,17 @@
[[!comment format=mdwn
username="joey"
subject="""comment 10"""
date="2020-10-19T17:33:49Z"
content="""
[[/profiling]] has a history of `+RTS -p` profiles in the same repo.
Comparing against the 10 month old one there, current git-annex find
runs in same time, and actually allocates slightly less memory, 583357880
bytes down from 608475328. That's memory churn, not max memory usage,
so doesn't rule out a memory leak. But if there is one, it's memory that
was allocated before, so it would need to be a laziness bug I think.
And the profiles are not showing another such leak.
My feeling is, what's left now is all due to a change to haskell runtime's
memory management, or a library. So not worth keeping this open for since I
can't do anything about it except for keep an eye on it.
"""]]