mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2014-02-24 21:30:06 -04:00
commit fdc7200b25
2 changed files with 58 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

19
doc/bugs/Auto-repair_greatly_slows_down_the_machine.mdwn Normal file
View file

@ -0,0 +1,19 @@
### Please describe the problem.
The assistant regulary ends up trying to perform repair (I don't know why, it happens fairly often, once a week or so). When it does so, it ends up creating a huge (2.4G) .git/objects directory, and a git prune-packed process uses so much I/O the machine really slows down.
### What steps will reproduce the problem?
I don't have any reliable way to reproduce it. The repository ends up being attempted to be repaired around once a week. This week the repair (and the slowdown) also happened on a second computer.
### What version of git-annex are you using? On what operating system?
git-annex version: 5.20140221-gbdfc8e1 (using the standalone 64bit builds)
This is on an up-to-date Arch Linux. It also happened on Fedora 20.
### Please provide any additional information below.
The daemon.log is fairly long, but not particulary interesting: [[https://ssl.zerodogg.org/~zerodogg/private/tmp/daemon.log-2014-02-25.1]]
The «resource vanished (Broken pipe)» at the end is the result of me killing the prune-packed in order to be able to use the machine again.

39
doc/bugs/pages_of_packfile_errors/comment_3_73b9f574e8ce36d5e0d0f6c6a89006b7._comment Normal file
View file

@ -0,0 +1,39 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.172"
subject="comment 3"
date="2014-02-24T23:39:46Z"
content="""
Well, if there's a bug here, it might be that this particular problem has caused the repair process to loop repeatedly trying to unpack a pack file.
I don't see how that could happen, looking at the code it will try to unpack each pack file only once.
If you run `git annex repair --debug`, you can see the git commands it runs, and so see if it's somehow looping. When I do this with some corrupt pack files (actually, I swapped one pack file for another one), I see, for example:
<pre>
[2014-02-24 19:11:42 JEST] feed: git [\"--git-dir=/home/joey/tmp/git/.git\",\"--work-tree=/home/joey/tmp/git\",\"unpack-objects\",\"-r\"]
error: packfile /home/joey/tmp/git/.git/objects/pack/pack-857c07e35d98e8f063fdae6846d1f6f7453e1312.pack claims to have 862 objects while index indicates 1431 objects
warning: packfile /home/joey/tmp/git/.git/objects/pack/pack-857c07e35d98e8f063fdae6846d1f6f7453e1312.pack cannot be accessed
error: packfile /home/joey/tmp/git/.git/objects/pack/pack-857c07e35d98e8f063fdae6846d1f6f7453e1312.pack claims to have 862 objects while index indicates 1431 objects
warning: packfile /home/joey/tmp/git/.git/objects/pack/pack-857c07e35d98e8f063fdae6846d1f6f7453e1312.pack cannot be accessed
error: packfile /home/joey/tmp/git/.git/objects/pack/pack-857c07e35d98e8f063fdae6846d1f6f7453e1312.pack claims to have 862 objects while index indicates 1431 objects
...
</pre>
Which shows that git-annex only ran `git unpack-objects -r` once, and yet it printed out the same error repeatedly.
One possibility is a problem using `-r`, which makes it keep going on errors. Which seemed like a good idea at the time to unpack as much as possible from a damaged file. It might be that `git unpack-objects` is itself getting stuck in some kind of loop with the -r.
In my case, it did not get stuck; it eventually quit and it moved on to the next pack file, after 900-some repitions of the error message:
<pre>
[2014-02-24 19:16:47 JEST] feed: git [\"--git-dir=/home/joey/tmp/git/.git\",\"--work-tree=/home/joey/tmp/git\",\"unpack-objects\",\"-r\"]
error: packfile /home/joey/tmp/git/.git/objects/pack/pack-857c07e35d98e8f063fdae6846d1f6f7453e1312.pack claims to have 862 objects while index indicates 1431 objects
warning: packfile /home/joey/tmp/git/.git/objects/pack/pack-857c07e35d98e8f063fdae6846d1f6f7453e1312.pack cannot be accessed
</pre>
Intesting that it's again complaining about the same pack file, despite having moved from one pack file on to the next one. I think what's going on here is while unpacking pack files A..Y (which may all be fine), it's checking pack file Z, which is corrupt, to see if the objects exist in it, and complaining each time.
So, I can improve this a lot by moving *all* the pack files out of the way before trying to unpack any of them. In my test case, that completely eliminated the errors, and probably also sped it up a bit.
If I were you, I'd either try stopping your running git-annex and run `git annex repair --debug` and analize the log like I did above, or get the next daily build which has that change, and see if it helps in your case.
"""]]