cache remotes via annex-speculate-present

Added remote.name.annex-speculate-present config that can be used to
make cache remotes.

Implemented it in Remote.keyPossibilities, which is used by the
get/move/copy/mirror commands, and nothing else. This way, things like
whereis will not show content that's speculatively present.

The assistant and sync --content were not using Remote.keyPossibilities,
and were changed to use it.

The efficiency hit should be small; Remote.keyPossibilities is only
used before transferring a file, which is the expensive operation.
And, it's only doing one lookup of the remoteList and a very cheap
filter over it.

Note that, git-annex still updates the location log when copying content
to a remote with annex-speculate-present set. In this case, the location
tracking will indicate that content is present in the remote. This may
not be wanted for caches, or may not be a real problem for them. TBD.

This commit was supported by the NSF-funded DataLad project.
This commit is contained in:
Joey Hess 2018-08-01 14:22:52 -04:00
parent 2884637cab
commit fd5a392006
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
7 changed files with 114 additions and 4 deletions

View file

@ -1,6 +1,6 @@
{- git-annex remotes
-
- Copyright 2011 Joey Hess <id@joeyh.name>
- Copyright 2011-2018 Joey Hess <id@joeyh.name>
-
- Licensed under the GNU GPL version 3 or higher.
-}
@ -278,13 +278,21 @@ keyLocations key = trustExclude DeadTrusted =<< loggedLocations key
{- Cost ordered lists of remotes that the location log indicates
- may have a key.
-
- Also includes remotes with remoteAnnexSpeculatePresent set.
-}
keyPossibilities :: Key -> Annex [Remote]
keyPossibilities key = do
u <- getUUID
-- uuids of all remotes that are recorded to have the key
locations <- filter (/= u) <$> keyLocations key
fst <$> remoteLocations locations []
speclocations <- map uuid
. filter (remoteAnnexSpeculatePresent . gitconfig)
<$> remoteList
-- there are unlikely to be many speclocations, so building a Set
-- is not worth the expense
let locations' = speclocations ++ filter (`notElem` speclocations) locations
fst <$> remoteLocations locations' []
{- Given a list of locations of a key, and a list of all
- trusted repositories, generates a cost-ordered list of