mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc/bugs/git_annex_add_memory_leak.mdwn

40 lines
1.6 KiB
Text
Raw Normal View History

add, and immediately close bug. useful documentation though
2012-02-14 05:01:38 +00:00
For the record, `git annex add` has had a series of memory leaks.
Mostly these are minor -- until you need to check in a few
million files in a single operation.
If this happens to you, git-annex will run out of memory and stop.
(Generally well before your system runs out of memory, since it has some
built-in ulimits.) You can recover by just re-running the `git annex add`
-- it will automatically pick up where it left off.
A history of the leaks:
* Originally, `git annex add` remembered all the files
it had added, and fed them to git at the end. Of course
that made its memory use grow, so it was fixed to periodically
fix memory leak when staging the journal The list of files had to be retained until the end so it could be deleted. Also, a list of update-index lines was generated and only then fed into it. Now everything streams in constant space.
2012-02-14 18:35:52 +00:00
flush its buffer. Fixed in version 0.20110417.
add, and immediately close bug. useful documentation though
2012-02-14 05:01:38 +00:00
* Something called a "lazy state monad" caused "thunks" to build
up and memory to leak. Also affected other git annex commands
than `add`. Adding files using a SHA* backend hit the worst.
Fixed in versions afer 3.20120123.
Fixed a memory leak due to excessive strictness when committing journal files. When hashing the files, the entire list of shas was read strictly. That was entirely unnecessary, since there's a cleanup action run after they're consumed.
2012-02-14 15:20:30 +00:00
* Committing journal files turned out to have another memory leak.
After adding a lot of files ran out of memory, this left the journal
fix memory leak when staging the journal The list of files had to be retained until the end so it could be deleted. Also, a list of update-index lines was generated and only then fed into it. Now everything streams in constant space.
2012-02-14 18:35:52 +00:00
behind and could affect other git-annex commands. Fixed in versions afer
Fixed a memory leak due to excessive strictness when committing journal files. When hashing the files, the entire list of shas was read strictly. That was entirely unnecessary, since there's a cleanup action run after they're consumed.
2012-02-14 15:20:30 +00:00
3.20120123.
fix failure count memory leak This is the last memory leak that prevents git-annex from running in constant space, as far as I can see. I can now run git annex find dummied up to repeatedly find the same file over and over, on millions olf files, and memory stays entirely constant.
2012-02-15 18:35:49 +00:00
* The count of the number of failed commands was updated lazily, which
caused a slow leak when running on a lot of files. Fixed in versions afer
3.20120123.
fix memory leak when staging the journal The list of files had to be retained until the end so it could be deleted. Also, a list of update-index lines was generated and only then fed into it. Now everything streams in constant space.
2012-02-14 18:35:52 +00:00
add, and immediately close bug. useful documentation though
2012-02-14 05:01:38 +00:00
* (Note that `git ls-files --others`, which is used to find files to add,
also uses surpsisingly large amounts
of memory when you have a lot of files. It buffers
the entire list, so it can compare it with the files in the index,
before outputting anything.
This is Not Our Problem, but I'm sure the git developers
would appreciate a patch that fixes it.)
fix failure count memory leak This is the last memory leak that prevents git-annex from running in constant space, as far as I can see. I can now run git annex find dummied up to repeatedly find the same file over and over, on millions olf files, and memory stays entirely constant.
2012-02-15 18:35:49 +00:00
[[done]]
Reference in a new issue Copy permalink