2012-01-10 17:11:16 +00:00
|
|
|
{- git-annex remote list
|
|
|
|
-
|
2020-01-14 16:35:08 +00:00
|
|
|
- Copyright 2011-2020 Joey Hess <id@joeyh.name>
|
2012-01-10 17:11:16 +00:00
|
|
|
-
|
2019-03-13 19:48:14 +00:00
|
|
|
- Licensed under the GNU AGPL version 3 or higher.
|
2012-01-10 17:11:16 +00:00
|
|
|
-}
|
|
|
|
|
add RemoteStateHandle
This solves the problem of sameas remotes trampling over per-remote
state. Used for:
* per-remote state, of course
* per-remote metadata, also of course
* per-remote content identifiers, because two remote implementations
could in theory generate the same content identifier for two different
peices of content
While chunk logs are per-remote data, they don't use this, because the
number and size of chunks stored is a common property across sameas
remotes.
External special remote had a complication, where it was theoretically
possible for a remote to send SETSTATE or GETSTATE during INITREMOTE or
EXPORTSUPPORTED. Since the uuid of the remote is typically generate in
Remote.setup, it would only be possible to pass a Maybe
RemoteStateHandle into it, and it would otherwise have to construct its
own. Rather than go that route, I decided to send an ERROR in this case.
It seems unlikely that any existing external special remote will be
affected. They would have to make up a git-annex key, and set state for
some reason during INITREMOTE. I can imagine such a hack, but it doesn't
seem worth complicating the code in such an ugly way to support it.
Unfortunately, both TestRemote and Annex.Import needed the Remote
to have a new field added that holds its RemoteStateHandle.
2019-10-14 16:33:27 +00:00
|
|
|
{-# LANGUAGE CPP #-}
|
|
|
|
|
2012-01-10 17:11:16 +00:00
|
|
|
module Remote.List where
|
|
|
|
|
|
|
|
import qualified Data.Map as M
|
|
|
|
|
2016-01-20 20:36:33 +00:00
|
|
|
import Annex.Common
|
2012-01-10 17:11:16 +00:00
|
|
|
import qualified Annex
|
|
|
|
import Logs.Remote
|
|
|
|
import Types.Remote
|
add RemoteStateHandle
This solves the problem of sameas remotes trampling over per-remote
state. Used for:
* per-remote state, of course
* per-remote metadata, also of course
* per-remote content identifiers, because two remote implementations
could in theory generate the same content identifier for two different
peices of content
While chunk logs are per-remote data, they don't use this, because the
number and size of chunks stored is a common property across sameas
remotes.
External special remote had a complication, where it was theoretically
possible for a remote to send SETSTATE or GETSTATE during INITREMOTE or
EXPORTSUPPORTED. Since the uuid of the remote is typically generate in
Remote.setup, it would only be possible to pass a Maybe
RemoteStateHandle into it, and it would otherwise have to construct its
own. Rather than go that route, I decided to send an ERROR in this case.
It seems unlikely that any existing external special remote will be
affected. They would have to make up a git-annex key, and set state for
some reason during INITREMOTE. I can imagine such a hack, but it doesn't
seem worth complicating the code in such an ugly way to support it.
Unfortunately, both TestRemote and Annex.Import needed the Remote
to have a new field added that holds its RemoteStateHandle.
2019-10-14 16:33:27 +00:00
|
|
|
import Types.RemoteState
|
2012-01-10 17:11:16 +00:00
|
|
|
import Annex.UUID
|
2012-03-04 20:00:24 +00:00
|
|
|
import Remote.Helper.Hooks
|
2014-01-02 17:12:32 +00:00
|
|
|
import Remote.Helper.ReadOnly
|
2019-02-20 19:55:01 +00:00
|
|
|
import Remote.Helper.ExportImport
|
2020-01-14 16:35:08 +00:00
|
|
|
import Annex.SpecialRemote.Config
|
2012-07-22 17:48:50 +00:00
|
|
|
import qualified Git
|
2012-08-05 18:49:47 +00:00
|
|
|
import qualified Git.Config
|
2012-01-10 17:11:16 +00:00
|
|
|
|
|
|
|
import qualified Remote.Git
|
2013-09-07 22:38:00 +00:00
|
|
|
import qualified Remote.GCrypt
|
2016-12-06 16:19:47 +00:00
|
|
|
import qualified Remote.P2P
|
2012-04-14 18:22:33 +00:00
|
|
|
#ifdef WITH_S3
|
2012-02-14 07:10:01 +00:00
|
|
|
import qualified Remote.S3
|
2012-04-14 18:22:33 +00:00
|
|
|
#endif
|
2012-01-10 17:11:16 +00:00
|
|
|
import qualified Remote.Bup
|
|
|
|
import qualified Remote.Directory
|
|
|
|
import qualified Remote.Rsync
|
|
|
|
import qualified Remote.Web
|
2014-12-17 03:22:46 +00:00
|
|
|
import qualified Remote.BitTorrent
|
2012-11-15 00:25:00 +00:00
|
|
|
#ifdef WITH_WEBDAV
|
|
|
|
import qualified Remote.WebDAV
|
|
|
|
#endif
|
2018-03-27 16:41:57 +00:00
|
|
|
import qualified Remote.Adb
|
2014-01-08 20:14:37 +00:00
|
|
|
import qualified Remote.Tahoe
|
2012-11-20 20:43:58 +00:00
|
|
|
import qualified Remote.Glacier
|
2014-05-15 18:44:00 +00:00
|
|
|
import qualified Remote.Ddar
|
2019-08-01 19:11:45 +00:00
|
|
|
import qualified Remote.GitLFS
|
2012-01-10 17:11:16 +00:00
|
|
|
import qualified Remote.Hook
|
external special remotes mostly implemented (untested)
This has not been tested at all. It compiles!
The only known missing things are support for encryption, and for get/set
of special remote configuration, and of key state. (The latter needs
separate work to add a new per-key log file to store that state.)
Only thing I don't much like is that initremote needs to be passed both
type=external and externaltype=foo. It would be better to have just
type=foo
Most of this is quite straightforward code, that largely wrote itself given
the types. The only tricky parts were:
* Need to lock the remote when using it to eg make a request, because
in theory git-annex could have multiple threads that each try to use
a remote at the same time. I don't think that git-annex ever does
that currently, but better safe than sorry.
* Rather than starting up every external special remote program when
git-annex starts, they are started only on demand, when first used.
This will avoid slowdown, especially when running fast git-annex query
commands. Once started, they keep running until git-annex stops, currently,
which may not be ideal, but it's hard to know a better time to stop them.
* Bit of a chicken and egg problem with caching the cost of the remote,
because setting annex-cost in the git config needs the remote to already
be set up. Managed to finesse that.
This commit was sponsored by Lukas Anzinger.
2013-12-26 22:23:13 +00:00
|
|
|
import qualified Remote.External
|
2012-01-10 17:11:16 +00:00
|
|
|
|
|
|
|
remoteTypes :: [RemoteType]
|
2019-03-04 20:02:56 +00:00
|
|
|
remoteTypes = map adjustExportImportRemoteType
|
2012-01-10 17:11:16 +00:00
|
|
|
[ Remote.Git.remote
|
2013-09-07 22:38:00 +00:00
|
|
|
, Remote.GCrypt.remote
|
2016-12-06 16:19:47 +00:00
|
|
|
, Remote.P2P.remote
|
2012-04-14 18:22:33 +00:00
|
|
|
#ifdef WITH_S3
|
2012-02-14 07:10:01 +00:00
|
|
|
, Remote.S3.remote
|
2012-04-14 18:22:33 +00:00
|
|
|
#endif
|
2012-01-10 17:11:16 +00:00
|
|
|
, Remote.Bup.remote
|
|
|
|
, Remote.Directory.remote
|
|
|
|
, Remote.Rsync.remote
|
|
|
|
, Remote.Web.remote
|
2014-12-17 03:22:46 +00:00
|
|
|
, Remote.BitTorrent.remote
|
2012-11-15 00:25:00 +00:00
|
|
|
#ifdef WITH_WEBDAV
|
|
|
|
, Remote.WebDAV.remote
|
2014-01-08 20:14:37 +00:00
|
|
|
#endif
|
2018-03-27 16:41:57 +00:00
|
|
|
, Remote.Adb.remote
|
2014-01-08 20:14:37 +00:00
|
|
|
, Remote.Tahoe.remote
|
2012-11-20 20:43:58 +00:00
|
|
|
, Remote.Glacier.remote
|
2014-05-15 18:44:00 +00:00
|
|
|
, Remote.Ddar.remote
|
2019-08-01 19:11:45 +00:00
|
|
|
, Remote.GitLFS.remote
|
2012-01-10 17:11:16 +00:00
|
|
|
, Remote.Hook.remote
|
external special remotes mostly implemented (untested)
This has not been tested at all. It compiles!
The only known missing things are support for encryption, and for get/set
of special remote configuration, and of key state. (The latter needs
separate work to add a new per-key log file to store that state.)
Only thing I don't much like is that initremote needs to be passed both
type=external and externaltype=foo. It would be better to have just
type=foo
Most of this is quite straightforward code, that largely wrote itself given
the types. The only tricky parts were:
* Need to lock the remote when using it to eg make a request, because
in theory git-annex could have multiple threads that each try to use
a remote at the same time. I don't think that git-annex ever does
that currently, but better safe than sorry.
* Rather than starting up every external special remote program when
git-annex starts, they are started only on demand, when first used.
This will avoid slowdown, especially when running fast git-annex query
commands. Once started, they keep running until git-annex stops, currently,
which may not be ideal, but it's hard to know a better time to stop them.
* Bit of a chicken and egg problem with caching the cost of the remote,
because setting annex-cost in the git config needs the remote to already
be set up. Managed to finesse that.
This commit was sponsored by Lukas Anzinger.
2013-12-26 22:23:13 +00:00
|
|
|
, Remote.External.remote
|
2012-01-10 17:11:16 +00:00
|
|
|
]
|
|
|
|
|
|
|
|
{- Builds a list of all available Remotes.
|
|
|
|
- Since doing so can be expensive, the list is cached. -}
|
|
|
|
remoteList :: Annex [Remote]
|
|
|
|
remoteList = do
|
|
|
|
rs <- Annex.getState Annex.remotes
|
|
|
|
if null rs
|
2015-08-05 17:49:54 +00:00
|
|
|
then remoteList' False
|
2012-01-10 17:11:16 +00:00
|
|
|
else return rs
|
2015-08-05 17:49:54 +00:00
|
|
|
|
|
|
|
remoteList' :: Bool -> Annex [Remote]
|
|
|
|
remoteList' autoinit = do
|
|
|
|
m <- readRemoteLog
|
|
|
|
rs <- concat <$> mapM (process m) remoteTypes
|
|
|
|
Annex.changeState $ \s -> s { Annex.remotes = rs }
|
|
|
|
return rs
|
2012-11-11 04:51:07 +00:00
|
|
|
where
|
2015-08-05 17:49:54 +00:00
|
|
|
process m t = enumerate t autoinit
|
|
|
|
>>= mapM (remoteGen m t)
|
|
|
|
>>= return . catMaybes
|
2012-07-22 17:48:50 +00:00
|
|
|
|
2012-08-05 18:49:47 +00:00
|
|
|
{- Forces the remoteList to be re-generated, re-reading the git config. -}
|
|
|
|
remoteListRefresh :: Annex [Remote]
|
|
|
|
remoteListRefresh = do
|
|
|
|
newg <- inRepo Git.Config.reRead
|
|
|
|
Annex.changeState $ \s -> s
|
|
|
|
{ Annex.remotes = []
|
2018-10-29 20:01:08 +00:00
|
|
|
, Annex.gitremotes = Nothing
|
2012-08-05 18:49:47 +00:00
|
|
|
, Annex.repo = newg
|
|
|
|
}
|
|
|
|
remoteList
|
|
|
|
|
2012-07-22 17:48:50 +00:00
|
|
|
{- Generates a Remote. -}
|
2013-09-26 03:19:01 +00:00
|
|
|
remoteGen :: M.Map UUID RemoteConfig -> RemoteType -> Git.Repo -> Annex (Maybe Remote)
|
2019-03-04 20:02:56 +00:00
|
|
|
remoteGen m t g = do
|
|
|
|
u <- getRepoUUID g
|
|
|
|
gc <- Annex.getRemoteGitConfig g
|
2019-10-11 16:32:11 +00:00
|
|
|
let cu = fromMaybe u $ remoteAnnexConfigUUID gc
|
add RemoteStateHandle
This solves the problem of sameas remotes trampling over per-remote
state. Used for:
* per-remote state, of course
* per-remote metadata, also of course
* per-remote content identifiers, because two remote implementations
could in theory generate the same content identifier for two different
peices of content
While chunk logs are per-remote data, they don't use this, because the
number and size of chunks stored is a common property across sameas
remotes.
External special remote had a complication, where it was theoretically
possible for a remote to send SETSTATE or GETSTATE during INITREMOTE or
EXPORTSUPPORTED. Since the uuid of the remote is typically generate in
Remote.setup, it would only be possible to pass a Maybe
RemoteStateHandle into it, and it would otherwise have to construct its
own. Rather than go that route, I decided to send an ERROR in this case.
It seems unlikely that any existing external special remote will be
affected. They would have to make up a git-annex key, and set state for
some reason during INITREMOTE. I can imagine such a hack, but it doesn't
seem worth complicating the code in such an ugly way to support it.
Unfortunately, both TestRemote and Annex.Import needed the Remote
to have a new field added that holds its RemoteStateHandle.
2019-10-14 16:33:27 +00:00
|
|
|
let rs = RemoteStateHandle cu
|
2019-10-11 16:32:11 +00:00
|
|
|
let c = fromMaybe M.empty $ M.lookup cu m
|
add LISTCONFIGS to external special remote protocol
Special remote programs that use GETCONFIG/SETCONFIG are recommended
to implement it.
The description is not yet used, but will be useful later when adding a way
to make initremote list all accepted configs.
configParser now takes a RemoteConfig parameter. Normally, that's not
needed, because configParser returns a parter, it does not parse it
itself. But, it's needed to look at externaltype and work out what
external remote program to run for LISTCONFIGS.
Note that, while externalUUID is changed to a Maybe UUID, checkExportSupported
used to use NoUUID. The code that now checks for Nothing used to behave
in some undefined way if the external program made requests that
triggered it.
Also, note that in externalSetup, once it generates external,
it parses the RemoteConfig strictly. That generates a
ParsedRemoteConfig, which is thrown away. The reason it's ok to throw
that away, is that, if the strict parse succeeded, the result must be
the same as the earlier, lenient parse.
initremote of an external special remote now runs the program three
times. First for LISTCONFIGS, then EXPORTSUPPORTED, and again
LISTCONFIGS+INITREMOTE. It would not be hard to eliminate at least
one of those, and it should be possible to only run the program once.
2020-01-17 19:30:14 +00:00
|
|
|
pc <- either (const mempty) id . parseRemoteConfig c <$> configParser t c
|
2020-01-14 16:35:08 +00:00
|
|
|
generate t g u pc gc rs >>= \case
|
2019-03-04 20:02:56 +00:00
|
|
|
Nothing -> return Nothing
|
add RemoteStateHandle
This solves the problem of sameas remotes trampling over per-remote
state. Used for:
* per-remote state, of course
* per-remote metadata, also of course
* per-remote content identifiers, because two remote implementations
could in theory generate the same content identifier for two different
peices of content
While chunk logs are per-remote data, they don't use this, because the
number and size of chunks stored is a common property across sameas
remotes.
External special remote had a complication, where it was theoretically
possible for a remote to send SETSTATE or GETSTATE during INITREMOTE or
EXPORTSUPPORTED. Since the uuid of the remote is typically generate in
Remote.setup, it would only be possible to pass a Maybe
RemoteStateHandle into it, and it would otherwise have to construct its
own. Rather than go that route, I decided to send an ERROR in this case.
It seems unlikely that any existing external special remote will be
affected. They would have to make up a git-annex key, and set state for
some reason during INITREMOTE. I can imagine such a hack, but it doesn't
seem worth complicating the code in such an ugly way to support it.
Unfortunately, both TestRemote and Annex.Import needed the Remote
to have a new field added that holds its RemoteStateHandle.
2019-10-14 16:33:27 +00:00
|
|
|
Just r -> Just <$> adjustExportImport (adjustReadOnly (addHooks r)) rs
|
2012-01-10 17:11:16 +00:00
|
|
|
|
2012-08-05 18:49:47 +00:00
|
|
|
{- Updates a local git Remote, re-reading its git config. -}
|
2013-09-12 19:54:35 +00:00
|
|
|
updateRemote :: Remote -> Annex (Maybe Remote)
|
2012-08-05 18:49:47 +00:00
|
|
|
updateRemote remote = do
|
|
|
|
m <- readRemoteLog
|
2018-06-04 18:31:55 +00:00
|
|
|
remote' <- updaterepo =<< getRepo remote
|
2012-08-05 18:49:47 +00:00
|
|
|
remoteGen m (remotetype remote) remote'
|
2012-11-11 04:51:07 +00:00
|
|
|
where
|
|
|
|
updaterepo r
|
|
|
|
| Git.repoIsLocal r || Git.repoIsLocalUnknown r =
|
2015-08-05 17:49:54 +00:00
|
|
|
Remote.Git.configRead False r
|
2012-11-11 04:51:07 +00:00
|
|
|
| otherwise = return r
|
2012-08-05 18:49:47 +00:00
|
|
|
|
2013-09-09 13:58:17 +00:00
|
|
|
{- Checks if a remote is syncable using git. -}
|
2020-02-19 17:58:26 +00:00
|
|
|
gitSyncableRemoteType :: RemoteType -> Bool
|
|
|
|
gitSyncableRemoteType t = t `elem`
|
2019-11-18 20:13:21 +00:00
|
|
|
[ Remote.Git.remote
|
|
|
|
, Remote.GCrypt.remote
|
|
|
|
, Remote.P2P.remote
|
|
|
|
, Remote.GitLFS.remote
|
|
|
|
]
|