external special remotes mostly implemented (untested)

This has not been tested at all. It compiles!

The only known missing things are support for encryption, and for get/set
of special remote configuration, and of key state. (The latter needs
separate work to add a new per-key log file to store that state.)

Only thing I don't much like is that initremote needs to be passed both
type=external and externaltype=foo. It would be better to have just
type=foo

Most of this is quite straightforward code, that largely wrote itself given
the types. The only tricky parts were:

* Need to lock the remote when using it to eg make a request, because
  in theory git-annex could have multiple threads that each try to use
  a remote at the same time. I don't think that git-annex ever does
  that currently, but better safe than sorry.

* Rather than starting up every external special remote program when
  git-annex starts, they are started only on demand, when first used.
  This will avoid slowdown, especially when running fast git-annex query
  commands. Once started, they keep running until git-annex stops, currently,
  which may not be ideal, but it's hard to know a better time to stop them.

* Bit of a chicken and egg problem with caching the cost of the remote,
  because setting annex-cost in the git config needs the remote to already
  be set up. Managed to finesse that.

This commit was sponsored by Lukas Anzinger.
This commit is contained in:
Joey Hess 2013-12-26 18:23:13 -04:00
parent 409a85b264
commit 6c565ec905
10 changed files with 509 additions and 205 deletions

View file

@ -12,7 +12,6 @@ import qualified Git
import qualified Git.Config
import qualified Git.Command
import qualified Annex
import qualified Types.Remote as Remote
import Config.Cost
type UnqualifiedConfigKey = String
@ -55,14 +54,16 @@ annexConfig key = ConfigKey $ "annex." ++ key
- by remote.<name>.annex-cost, or if remote.<name>.annex-cost-command
- is set and prints a number, that is used. -}
remoteCost :: RemoteGitConfig -> Cost -> Annex Cost
remoteCost c def = case remoteAnnexCostCommand c of
Just cmd | not (null cmd) -> liftIO $
(fromMaybe def . readish) <$>
readProcess "sh" ["-c", cmd]
_ -> return $ fromMaybe def $ remoteAnnexCost c
remoteCost c def = fromMaybe def <$> remoteCost' c
setRemoteCost :: Remote -> Cost -> Annex ()
setRemoteCost r c = setConfig (remoteConfig (Remote.repo r) "cost") (show c)
remoteCost' :: RemoteGitConfig -> Annex (Maybe Cost)
remoteCost' c = case remoteAnnexCostCommand c of
Just cmd | not (null cmd) -> liftIO $
readish <$> readProcess "sh" ["-c", cmd]
_ -> return $ remoteAnnexCost c
setRemoteCost :: Git.Repo -> Cost -> Annex ()
setRemoteCost r c = setConfig (remoteConfig r "cost") (show c)
getNumCopies :: Maybe Int -> Annex Int
getNumCopies (Just v) = return v