mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

avoid redundant export uploads

Browse source 
export, sync --content: Avoid unnecessarily trying to upload files to an
exporttree remote that already contains the files.

When the export was origianly made in one repo and now git-annex is
running in a different repo, the export database is not yet populated with
information about the exportLocation of files. So, it was trying to upload
the files to the export, even when it already contained them.

sync --content would first download the content from the export, and then
re-upload the content back.

And this also led to "not available" failures for each file that was not
locally present yet.

Fix: Just use checkPresentExport before uploading; if it succeeds update
the database.

This is a surprising oversight, it's possible it fixes a reversion because
I would have thought I'd have noticed this problem when originally
developing exporttree remotes.

This commit was sponsored by Jochen Bartl on Patreon.
This commit is contained in:
Joey Hess 2018-11-14 11:47:40 -04:00
parent d65df7ab21
commit c3fa1f2b08
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 15 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
CHANGELOG
View file

@ -12,6 +12,8 @@ git-annex (7.20181106) UNRELEASED; urgency=medium
already actually downloaded.
* When an export conflict prevents accessing a special remote,
be clearer about what the problem is and how to resolve it.
* export, sync --content: Avoid unnecessarily trying to upload files
to an exporttree remote that already contains the files.
-- Joey Hess <id@joeyh.name> Tue, 06 Nov 2018 12:44:27 -0400

21
Command/Export.hs
View file

@ -204,15 +204,19 @@ fillExport r ea db new = do
startExport :: Remote -> ExportActions Annex -> ExportHandle -> MVar Bool -> Git.LsTree.TreeItem -> CommandStart
startExport r ea db cvar ti = do
ek <- exportKey (Git.LsTree.sha ti)
stopUnless (notpresent ek) $ do
stopUnless (notrecordedpresent ek) $ do
showStart ("export " ++ name r) f
liftIO $ modifyMVar_ cvar (pure . const True)
next $ performExport r ea db ek af (Git.LsTree.sha ti) loc
ifM (either (const False) id <$> tryNonAsync (checkPresentExport ea (asKey ek) loc))
( next $ next $ cleanupExport r db ek loc False
, do
liftIO $ modifyMVar_ cvar (pure . const True)
next $ performExport r ea db ek af (Git.LsTree.sha ti) loc
)
where
loc = mkExportLocation f
f = getTopFilePath (Git.LsTree.file ti)
af = AssociatedFile (Just f)
notpresent ek = (||)
notrecordedpresent ek = (||)
<$> liftIO (notElem loc <$> getExportedLocation db (asKey ek))
-- If content was removed from the remote, the export db
-- will still list it, so also check location tracking.
@ -245,13 +249,14 @@ performExport r ea db ek af contentsha loc = do
liftIO $ hClose h
storer tmp sha1k loc m
if sent
then next $ cleanupExport r db ek loc
then next $ cleanupExport r db ek loc True
else stop
cleanupExport :: Remote -> ExportHandle -> ExportKey -> ExportLocation -> CommandCleanup
cleanupExport r db ek loc = do
cleanupExport :: Remote -> ExportHandle -> ExportKey -> ExportLocation -> Bool -> CommandCleanup
cleanupExport r db ek loc sent = do
liftIO $ addExportedLocation db (asKey ek) loc
logChange (asKey ek) (uuid r) InfoPresent
when sent $
logChange (asKey ek) (uuid r) InfoPresent
return True
startUnexport :: Remote -> ExportActions Annex -> ExportHandle -> TopFilePath -> [Git.Sha] -> CommandStart