mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

avoid redundant export uploads

Browse source 
export, sync --content: Avoid unnecessarily trying to upload files to an
exporttree remote that already contains the files.

When the export was origianly made in one repo and now git-annex is
running in a different repo, the export database is not yet populated with
information about the exportLocation of files. So, it was trying to upload
the files to the export, even when it already contained them.

sync --content would first download the content from the export, and then
re-upload the content back.

And this also led to "not available" failures for each file that was not
locally present yet.

Fix: Just use checkPresentExport before uploading; if it succeeds update
the database.

This is a surprising oversight, it's possible it fixes a reversion because
I would have thought I'd have noticed this problem when originally
developing exporttree remotes.

This commit was sponsored by Jochen Bartl on Patreon.
This commit is contained in:
Joey Hess 2018-11-14 11:47:40 -04:00
parent d65df7ab21
commit c3fa1f2b08
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 15 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
CHANGELOG
View file

@ -12,6 +12,8 @@ git-annex (7.20181106) UNRELEASED; urgency=medium
already actually downloaded. already actually downloaded.
* When an export conflict prevents accessing a special remote, * When an export conflict prevents accessing a special remote,
be clearer about what the problem is and how to resolve it. be clearer about what the problem is and how to resolve it.
* export, sync --content: Avoid unnecessarily trying to upload files
to an exporttree remote that already contains the files.
-- Joey Hess <id@joeyh.name> Tue, 06 Nov 2018 12:44:27 -0400 -- Joey Hess <id@joeyh.name> Tue, 06 Nov 2018 12:44:27 -0400

21
Command/Export.hs
View file

@ -204,15 +204,19 @@ fillExport r ea db new = do
startExport :: Remote -> ExportActions Annex -> ExportHandle -> MVar Bool -> Git.LsTree.TreeItem -> CommandStart startExport :: Remote -> ExportActions Annex -> ExportHandle -> MVar Bool -> Git.LsTree.TreeItem -> CommandStart
startExport r ea db cvar ti = do startExport r ea db cvar ti = do
ek <- exportKey (Git.LsTree.sha ti) ek <- exportKey (Git.LsTree.sha ti)
stopUnless (notpresent ek) $ do stopUnless (notrecordedpresent ek) $ do
showStart ("export " ++ name r) f showStart ("export " ++ name r) f
liftIO $ modifyMVar_ cvar (pure . const True) ifM (either (const False) id <$> tryNonAsync (checkPresentExport ea (asKey ek) loc))
next $ performExport r ea db ek af (Git.LsTree.sha ti) loc ( next $ next $ cleanupExport r db ek loc False
, do
liftIO $ modifyMVar_ cvar (pure . const True)
next $ performExport r ea db ek af (Git.LsTree.sha ti) loc
)
where where
loc = mkExportLocation f loc = mkExportLocation f
f = getTopFilePath (Git.LsTree.file ti) f = getTopFilePath (Git.LsTree.file ti)
af = AssociatedFile (Just f) af = AssociatedFile (Just f)
notpresent ek = (||) notrecordedpresent ek = (||)
<$> liftIO (notElem loc <$> getExportedLocation db (asKey ek)) <$> liftIO (notElem loc <$> getExportedLocation db (asKey ek))
-- If content was removed from the remote, the export db -- If content was removed from the remote, the export db
-- will still list it, so also check location tracking. -- will still list it, so also check location tracking.
@ -245,13 +249,14 @@ performExport r ea db ek af contentsha loc = do
liftIO $ hClose h liftIO $ hClose h
storer tmp sha1k loc m storer tmp sha1k loc m
if sent if sent
then next $ cleanupExport r db ek loc then next $ cleanupExport r db ek loc True
else stop else stop
cleanupExport :: Remote -> ExportHandle -> ExportKey -> ExportLocation -> CommandCleanup cleanupExport :: Remote -> ExportHandle -> ExportKey -> ExportLocation -> Bool -> CommandCleanup
cleanupExport r db ek loc = do cleanupExport r db ek loc sent = do
liftIO $ addExportedLocation db (asKey ek) loc liftIO $ addExportedLocation db (asKey ek) loc
logChange (asKey ek) (uuid r) InfoPresent when sent $
logChange (asKey ek) (uuid r) InfoPresent
return True return True
startUnexport :: Remote -> ExportActions Annex -> ExportHandle -> TopFilePath -> [Git.Sha] -> CommandStart startUnexport :: Remote -> ExportActions Annex -> ExportHandle -> TopFilePath -> [Git.Sha] -> CommandStart