mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

avoid uncessary keys db writes; doubled speed!

Browse source 
When running eg git-annex get, for each file it has to read from and
write to the keys database. But it's reading exclusively from one table,
and writing to a different table. So, it is not necessary to flush the
write to the database before reading. This avoids writing the database
once per file, instead it will buffer 1000 changes before writing.

Benchmarking getting 1000 small files from a local origin,
git-annex get now takes 13.62s, down from 22.41s!
git-annex drop now takes 9.07s, down from 18.63s!
Wowowowowowowow!

(It would perhaps have been better if there were separate databases for
the two tables. At least it would have avoided this complexity. Ah well,
this is better than splitting the table in a annex.version upgrade.)

Sponsored-by: Dartmouth College's Datalad project
This commit is contained in:
Joey Hess 2022-10-12 15:21:19 -04:00
parent ba7ecbc6a9
commit 6fbd337e34
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
7 changed files with 117 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

4
Upgrade/V7.hs
View file

@ -16,6 +16,7 @@ import Types.Upgrade
import Annex.CatFile
import qualified Database.Keys
import qualified Database.Keys.SQL
import Database.Keys.Tables
import qualified Git.LsFiles as LsFiles
import qualified Git
import Git.FilePath
@ -114,8 +115,9 @@ populateKeysDb = unlessM isBareRepo $ do
Nothing -> noop
Just k -> do
topf <- inRepo $ toTopFilePath $ toRawFilePath f
Database.Keys.runWriter $ \h -> liftIO $ do
Database.Keys.runWriter AssociatedTable $ \h -> liftIO $
Database.Keys.SQL.addAssociatedFile k topf h
Database.Keys.runWriter ContentTable $ \h -> liftIO $
Database.Keys.SQL.addInodeCaches k [ic] h
liftIO $ void cleanup
Database.Keys.closeDb