mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

speed up very first tree import by 25%

Browse source 
Reading from the cidsdb is responsible for about 25% of the runtime of
an import. Since the cidmap is used to store the same information in
ram, the cidsdb is not written to during an import any longer. And so,
if it started off empty (and updateFromLog wasn't needed), those reads
can just be skipped.

This is kind of a cheesy optimisation, since after any import from any
special remote, the database will no longer be empty, so it's a single
use optimisation. But it's probably not uncommon to start by importing a
lot of files, and it can save a lot of time then.

Sponsored-by: Brock Spratlen on Patreon
This commit is contained in:
Joey Hess 2023-06-02 13:30:30 -04:00
parent b43fb4923f
commit fe1b2dfb4b
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
4 changed files with 38 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

20
Annex/Import.hs
View file

@ -626,9 +626,9 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
-- avoid two threads both importing the same content identifier.
importing <- liftIO $ newTVarIO S.empty
withciddb $ \db -> do
CIDDb.needsUpdateFromLog db
>>= maybe noop (CIDDb.updateFromLog db)
(prepclock (run cidmap importing db))
db' <- CIDDb.needsUpdateFromLog db
>>= maybe (pure db) (CIDDb.updateFromLog db)
(prepclock (run cidmap importing db'))
where
-- When not importing content, reuse the same vector
-- clock for all state that's recorded. This can save
@ -925,10 +925,16 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
getTopFilePath subdir P.</> fromImportLocation loc
getcidkey cidmap db cid = liftIO $
CIDDb.getContentIdentifierKeys db rs cid >>= \case
[] -> atomically $
maybeToList . M.lookup cid <$> readTVar cidmap
l -> return l
-- Avoiding querying the database when it's empty speeds up
-- the initial import.
if CIDDb.databaseIsEmpty db
then getcidkeymap cidmap cid
else CIDDb.getContentIdentifierKeys db rs cid >>= \case
[] -> getcidkeymap cidmap cid
l -> return l
getcidkeymap cidmap cid =
atomically $ maybeToList . M.lookup cid <$> readTVar cidmap
recordcidkey cidmap cid k = do
liftIO $ atomically $ modifyTVar' cidmap $