mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

update

Browse source
This commit is contained in:
Joey Hess 2023-05-30 15:49:52 -04:00
parent aaeae746f0
commit 9ca81ed02a
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
1 changed files with 8 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

9
doc/todo/speed_up_import_tree.mdwn
View file

@ -10,7 +10,9 @@ own scalability limits with many files.)
Still, it would be good to find some ways to speed it up. Still, it would be good to find some ways to speed it up.
Hmm... What if it generated a git tree, where each file in the tree is ---
What if it generated a git tree, where each file in the tree is
a sha1 hash of the ContentIdentifier. The tree can just be recorded locally a sha1 hash of the ContentIdentifier. The tree can just be recorded locally
somewhere. It's ok if it gets garbage collected; it's only an optimisation. somewhere. It's ok if it gets garbage collected; it's only an optimisation.
On the next sync, diff from the old to the new tree. It only needs to On the next sync, diff from the old to the new tree. It only needs to
@ -22,6 +24,11 @@ reasonable, because git loses data on sha1 collisions anyway, and ContentIdentif
are no more likely to collide than the content of files, and probably less are no more likely to collide than the content of files, and probably less
likely overall..) likely overall..)
How fast can a git tree of say, 10000 files be generated? Is it faster than
querying sqlite 10000 times?
----
Another idea would to be use something faster than sqlite to record the cid Another idea would to be use something faster than sqlite to record the cid
to key mappings. Looking up those mappings is the main thing that makes to key mappings. Looking up those mappings is the main thing that makes
import slow when only a few files have changed and a large number have not. import slow when only a few files have changed and a large number have not.