30 lines
1.4 KiB
Text
30 lines
1.4 KiB
Text
|
Users sometimes expect `git-annex import --from remote` to be faster than
|
||
|
|
it is, when importing hundreds of thousands of files, particularly
|
||
|
|
from a directory special remote.
|
||
|
|
|
||
|
|
I think generally, they're expecting something that is not achievable.
|
||
|
|
It is always going to be slower than using git in a repository with that
|
||
|
|
many files, because git operates at a lower level of abstraction (the
|
||
|
|
filesystem), so has more optimisations available to it. (Also git has its
|
||
|
|
own scalability limits with many files.)
|
||
|
|
|
||
|
|
Still, it would be good to find some ways to speed it up.
|
||
|
|
|
||
|
|
Hmm... What if it generated a git tree, where each file in the tree is
|
||
|
|
a sha1 hash of the ContentIdentifier. The tree can just be recorded locally
|
||
|
|
somewhere. It's ok if it gets garbage collected; it's only an optimisation.
|
||
|
|
On the next sync, diff from the old to the new tree. It only needs to
|
||
|
|
import the changed files!
|
||
|
|
|
||
|
|
(That is assuming that ContentIdentifiers don't tend to sha1 collide.
|
||
|
|
If there was a collision it would fail to import the new file. But it seems
|
||
|
|
reasonable, because git loses data on sha1 collisions anyway, and ContentIdentifiers
|
||
|
|
are no more likely to collide than the content of files, and probably less
|
||
|
|
likely overall..)
|
||
|
|
|
||
|
|
Another idea would to be use something faster than sqlite to record the cid
|
||
|
|
to key mappings. Looking up those mappings is the main thing that makes
|
||
|
|
import slow when only a few files have changed and a large number have not.
|
||
|
|
|
||
|
|
--[[Joey]]
|