git write-tree was being run once per file git-annex acts on when eg,
getting files, which is slow when the remote repository has a large
tree.
onLocal calls quiesce after each action, and quiesce closes the keys db
since [[!commit ba7ecbc6a9c]]. Which has a relevant comment about
performance. I have not addressed that, the keys db still gets closed and
reopened after each file.
Turns out that, since git write-tree was run by each call to
reconcileStaged, the .git/annex/keysdb.cache value was never the
same as the git index's inode. Because git write-tree updates the index's
mtime even when no changes have been made.
And so, when the database got closed and reopened, reconcileStaged would
see a changed index, and run git write-tree again. Over and over.
I considered writing the index's new inodecache after write-tree to the
keysdb.cache, but that would be vulnerable to a race, if the index was
changed just after write-tree.
The fix was to stop using keysb.cache at all. When the database is closed
and later reopened by the same process, avoid re-doing reconcileStaged.
Now that .git/annex/keysdb.cache is no longer used. It could be removed,
but the time overhead of removing it would be more than the space overhead
of keeping it. Defferred removal to the v11 upgrade.
Sponsored-by: unqueued
4fd71c125e