mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

use importChanges optimisation

Browse source 
Large speed up to importing trees from special remotes that contain a lot
of files, by only processing changed files.

Benchmarks:

Importing from a special remote that has 10000 files, that have all been
imported before, and 1 new file sped up from 26.06 to 2.59 seconds.

An import with no change and 10000 unchanged files sped up from 24.3 to
1.99 seconds.

Going up to 20000 files, an import with no changes sped up from
125.95 to 3.84 seconds.

Sponsored-by: k0ld on Patreon
This commit is contained in:
Joey Hess 2023-06-01 13:46:16 -04:00
parent 029b08f54b
commit 40017089f2
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
5 changed files with 174 additions and 86 deletions
Download patch file Download diff file Expand all files Collapse all files

6
Command/Sync.hs
View file

@ -586,9 +586,9 @@ pullThirdPartyPopulated o remote
| otherwise = void $ includeCommandAction $ starting "list" ai si $
Command.Import.listContents' remote ImportTree (CheckGitIgnore False) go
where
go (Just importable) = importKeys remote ImportTree False True importable >>= \case
ImportFinished importablekeys -> do
(_imported, updatestate) <- recordImportTree remote ImportTree importablekeys
go (Just importable) = importChanges remote ImportTree False True importable >>= \case
ImportFinished imported -> do
(_t, updatestate) <- recordImportTree remote ImportTree imported
next $ do
updatestate
return True