mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

unused: Improved memory use significantly when there are a lot of differences between branches.

Browse source 
Argh, didn't need an accumulator here!

I think I use accumulators a lot more than I need to when recusively
processing lists..

This commit was sponsored by Jeff Goeke-Smith on Patreon.
This commit is contained in:
Joey Hess 2017-01-31 19:42:00 -04:00
parent 062286135c
commit ed60f60e9b
No known key found for this signature in database
GPG key ID: C910D9222512E3C7
4 changed files with 19 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

2
CHANGELOG
View file

@ -11,6 +11,8 @@ git-annex (6.20170102) UNRELEASED; urgency=medium
* Some optimisations to string splitting code. * Some optimisations to string splitting code.
* unused: When large files are checked right into git, avoid buffering * unused: When large files are checked right into git, avoid buffering
their contents in memory. their contents in memory.
* unused: Improved memory use significantly when there are a lot
of differences between branches.
-- Joey Hess <id@joeyh.name> Fri, 06 Jan 2017 15:22:06 -0400 -- Joey Hess <id@joeyh.name> Fri, 06 Jan 2017 15:22:06 -0400

8
Git/DiffTree.hs
View file

@ -101,11 +101,11 @@ getdiff command params repo = do
{- Parses --raw output used by diff-tree and git-log. -} {- Parses --raw output used by diff-tree and git-log. -}
parseDiffRaw :: [String] -> [DiffTreeItem] parseDiffRaw :: [String] -> [DiffTreeItem]
parseDiffRaw l = go l [] parseDiffRaw l = go l
where where
go [] c = c go [] = []
go (info:f:rest) c = go rest (mk info f : c) go (info:f:rest) = mk info f : go rest
go (s:[]) _ = error $ "diff-tree parse error near \"" ++ s ++ "\"" go (s:[]) = error $ "diff-tree parse error near \"" ++ s ++ "\""
mk info f = DiffTreeItem mk info f = DiffTreeItem
{ srcmode = readmode srcm { srcmode = readmode srcm

3
doc/todo/more_efficient_memory_usage_with_git-annex_unused.mdwn
View file

@ -1,3 +1,6 @@
While running *git-annex unused* on an annex with tens of thousands of items, *git-annex*'s memory usage ballooned to over 3 gigs and my PC froze. I cannot run *git-annex unused* on this annex because of this issue. While running *git-annex unused* on an annex with tens of thousands of items, *git-annex*'s memory usage ballooned to over 3 gigs and my PC froze. I cannot run *git-annex unused* on this annex because of this issue.
If it's possible, more efficient memory management would prevent this from happening. If it's possible, more efficient memory management would prevent this from happening.
> [[done]] -- assuming the memory leak I saw was the same one you saw...
> --[[Joey]]

10
doc/todo/more_efficient_memory_usage_with_git-annex_unused/comment_3_3c712e871ea3cd12916497f2d8152004._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2017-01-31T23:31:03Z"
content="""
Fixed the rest of the streaming problem.
(Also found/fixed an unrelated memory blow up in git annex unused that
only happened when a large file got checked right into git.)
"""]]