2012-01-10 17:11:16 +00:00
|
|
|
{- git-annex remote list
|
|
|
|
-
|
2020-01-14 16:35:08 +00:00
|
|
|
- Copyright 2011-2020 Joey Hess <id@joeyh.name>
|
2012-01-10 17:11:16 +00:00
|
|
|
-
|
2019-03-13 19:48:14 +00:00
|
|
|
- Licensed under the GNU AGPL version 3 or higher.
|
2012-01-10 17:11:16 +00:00
|
|
|
-}
|
|
|
|
|
|
|
|
module Remote.List where
|
|
|
|
|
|
|
|
import qualified Data.Map as M
|
|
|
|
|
2016-01-20 20:36:33 +00:00
|
|
|
import Annex.Common
|
2012-01-10 17:11:16 +00:00
|
|
|
import qualified Annex
|
|
|
|
import Logs.Remote
|
|
|
|
import Types.Remote
|
add RemoteStateHandle
This solves the problem of sameas remotes trampling over per-remote
state. Used for:
* per-remote state, of course
* per-remote metadata, also of course
* per-remote content identifiers, because two remote implementations
could in theory generate the same content identifier for two different
peices of content
While chunk logs are per-remote data, they don't use this, because the
number and size of chunks stored is a common property across sameas
remotes.
External special remote had a complication, where it was theoretically
possible for a remote to send SETSTATE or GETSTATE during INITREMOTE or
EXPORTSUPPORTED. Since the uuid of the remote is typically generate in
Remote.setup, it would only be possible to pass a Maybe
RemoteStateHandle into it, and it would otherwise have to construct its
own. Rather than go that route, I decided to send an ERROR in this case.
It seems unlikely that any existing external special remote will be
affected. They would have to make up a git-annex key, and set state for
some reason during INITREMOTE. I can imagine such a hack, but it doesn't
seem worth complicating the code in such an ugly way to support it.
Unfortunately, both TestRemote and Annex.Import needed the Remote
to have a new field added that holds its RemoteStateHandle.
2019-10-14 16:33:27 +00:00
|
|
|
import Types.RemoteState
|
2012-01-10 17:11:16 +00:00
|
|
|
import Annex.UUID
|
2012-03-04 20:00:24 +00:00
|
|
|
import Remote.Helper.Hooks
|
2014-01-02 17:12:32 +00:00
|
|
|
import Remote.Helper.ReadOnly
|
2019-02-20 19:55:01 +00:00
|
|
|
import Remote.Helper.ExportImport
|
2012-07-22 17:48:50 +00:00
|
|
|
import qualified Git
|
2012-01-10 17:11:16 +00:00
|
|
|
|
|
|
|
import qualified Remote.Git
|
2013-09-07 22:38:00 +00:00
|
|
|
import qualified Remote.GCrypt
|
2016-12-06 16:19:47 +00:00
|
|
|
import qualified Remote.P2P
|
2012-02-14 07:10:01 +00:00
|
|
|
import qualified Remote.S3
|
2012-01-10 17:11:16 +00:00
|
|
|
import qualified Remote.Bup
|
|
|
|
import qualified Remote.Directory
|
|
|
|
import qualified Remote.Rsync
|
|
|
|
import qualified Remote.Web
|
2014-12-17 03:22:46 +00:00
|
|
|
import qualified Remote.BitTorrent
|
2012-11-15 00:25:00 +00:00
|
|
|
import qualified Remote.WebDAV
|
2018-03-27 16:41:57 +00:00
|
|
|
import qualified Remote.Adb
|
2014-01-08 20:14:37 +00:00
|
|
|
import qualified Remote.Tahoe
|
2012-11-20 20:43:58 +00:00
|
|
|
import qualified Remote.Glacier
|
2014-05-15 18:44:00 +00:00
|
|
|
import qualified Remote.Ddar
|
2019-08-01 19:11:45 +00:00
|
|
|
import qualified Remote.GitLFS
|
2020-09-02 14:41:27 +00:00
|
|
|
import qualified Remote.HttpAlso
|
2020-12-18 20:03:51 +00:00
|
|
|
import qualified Remote.Borg
|
2024-04-17 19:19:42 +00:00
|
|
|
import qualified Remote.Rclone
|
2012-01-10 17:11:16 +00:00
|
|
|
import qualified Remote.Hook
|
external special remotes mostly implemented (untested)
This has not been tested at all. It compiles!
The only known missing things are support for encryption, and for get/set
of special remote configuration, and of key state. (The latter needs
separate work to add a new per-key log file to store that state.)
Only thing I don't much like is that initremote needs to be passed both
type=external and externaltype=foo. It would be better to have just
type=foo
Most of this is quite straightforward code, that largely wrote itself given
the types. The only tricky parts were:
* Need to lock the remote when using it to eg make a request, because
in theory git-annex could have multiple threads that each try to use
a remote at the same time. I don't think that git-annex ever does
that currently, but better safe than sorry.
* Rather than starting up every external special remote program when
git-annex starts, they are started only on demand, when first used.
This will avoid slowdown, especially when running fast git-annex query
commands. Once started, they keep running until git-annex stops, currently,
which may not be ideal, but it's hard to know a better time to stop them.
* Bit of a chicken and egg problem with caching the cost of the remote,
because setting annex-cost in the git config needs the remote to already
be set up. Managed to finesse that.
This commit was sponsored by Lukas Anzinger.
2013-12-26 22:23:13 +00:00
|
|
|
import qualified Remote.External
|
2012-01-10 17:11:16 +00:00
|
|
|
|
|
|
|
remoteTypes :: [RemoteType]
|
2019-03-04 20:02:56 +00:00
|
|
|
remoteTypes = map adjustExportImportRemoteType
|
2012-01-10 17:11:16 +00:00
|
|
|
[ Remote.Git.remote
|
2013-09-07 22:38:00 +00:00
|
|
|
, Remote.GCrypt.remote
|
2016-12-06 16:19:47 +00:00
|
|
|
, Remote.P2P.remote
|
2012-02-14 07:10:01 +00:00
|
|
|
, Remote.S3.remote
|
2012-01-10 17:11:16 +00:00
|
|
|
, Remote.Bup.remote
|
|
|
|
, Remote.Directory.remote
|
|
|
|
, Remote.Rsync.remote
|
|
|
|
, Remote.Web.remote
|
2014-12-17 03:22:46 +00:00
|
|
|
, Remote.BitTorrent.remote
|
2012-11-15 00:25:00 +00:00
|
|
|
, Remote.WebDAV.remote
|
2018-03-27 16:41:57 +00:00
|
|
|
, Remote.Adb.remote
|
2014-01-08 20:14:37 +00:00
|
|
|
, Remote.Tahoe.remote
|
2012-11-20 20:43:58 +00:00
|
|
|
, Remote.Glacier.remote
|
2014-05-15 18:44:00 +00:00
|
|
|
, Remote.Ddar.remote
|
2019-08-01 19:11:45 +00:00
|
|
|
, Remote.GitLFS.remote
|
2020-09-02 14:41:27 +00:00
|
|
|
, Remote.HttpAlso.remote
|
2020-12-18 20:03:51 +00:00
|
|
|
, Remote.Borg.remote
|
2024-04-17 19:19:42 +00:00
|
|
|
, Remote.Rclone.remote
|
2012-01-10 17:11:16 +00:00
|
|
|
, Remote.Hook.remote
|
external special remotes mostly implemented (untested)
This has not been tested at all. It compiles!
The only known missing things are support for encryption, and for get/set
of special remote configuration, and of key state. (The latter needs
separate work to add a new per-key log file to store that state.)
Only thing I don't much like is that initremote needs to be passed both
type=external and externaltype=foo. It would be better to have just
type=foo
Most of this is quite straightforward code, that largely wrote itself given
the types. The only tricky parts were:
* Need to lock the remote when using it to eg make a request, because
in theory git-annex could have multiple threads that each try to use
a remote at the same time. I don't think that git-annex ever does
that currently, but better safe than sorry.
* Rather than starting up every external special remote program when
git-annex starts, they are started only on demand, when first used.
This will avoid slowdown, especially when running fast git-annex query
commands. Once started, they keep running until git-annex stops, currently,
which may not be ideal, but it's hard to know a better time to stop them.
* Bit of a chicken and egg problem with caching the cost of the remote,
because setting annex-cost in the git config needs the remote to already
be set up. Managed to finesse that.
This commit was sponsored by Lukas Anzinger.
2013-12-26 22:23:13 +00:00
|
|
|
, Remote.External.remote
|
2012-01-10 17:11:16 +00:00
|
|
|
]
|
|
|
|
|
|
|
|
{- Builds a list of all available Remotes.
|
|
|
|
- Since doing so can be expensive, the list is cached. -}
|
|
|
|
remoteList :: Annex [Remote]
|
|
|
|
remoteList = do
|
|
|
|
rs <- Annex.getState Annex.remotes
|
|
|
|
if null rs
|
2015-08-05 17:49:54 +00:00
|
|
|
then remoteList' False
|
2012-01-10 17:11:16 +00:00
|
|
|
else return rs
|
2015-08-05 17:49:54 +00:00
|
|
|
|
|
|
|
remoteList' :: Bool -> Annex [Remote]
|
|
|
|
remoteList' autoinit = do
|
2020-09-22 17:52:26 +00:00
|
|
|
m <- remoteConfigMap
|
2015-08-05 17:49:54 +00:00
|
|
|
rs <- concat <$> mapM (process m) remoteTypes
|
|
|
|
Annex.changeState $ \s -> s { Annex.remotes = rs }
|
|
|
|
return rs
|
2012-11-11 04:51:07 +00:00
|
|
|
where
|
2015-08-05 17:49:54 +00:00
|
|
|
process m t = enumerate t autoinit
|
|
|
|
>>= mapM (remoteGen m t)
|
|
|
|
>>= return . catMaybes
|
2012-07-22 17:48:50 +00:00
|
|
|
|
|
|
|
{- Generates a Remote. -}
|
2013-09-26 03:19:01 +00:00
|
|
|
remoteGen :: M.Map UUID RemoteConfig -> RemoteType -> Git.Repo -> Annex (Maybe Remote)
|
2019-03-04 20:02:56 +00:00
|
|
|
remoteGen m t g = do
|
|
|
|
u <- getRepoUUID g
|
|
|
|
gc <- Annex.getRemoteGitConfig g
|
2019-10-11 16:32:11 +00:00
|
|
|
let cu = fromMaybe u $ remoteAnnexConfigUUID gc
|
add RemoteStateHandle
This solves the problem of sameas remotes trampling over per-remote
state. Used for:
* per-remote state, of course
* per-remote metadata, also of course
* per-remote content identifiers, because two remote implementations
could in theory generate the same content identifier for two different
peices of content
While chunk logs are per-remote data, they don't use this, because the
number and size of chunks stored is a common property across sameas
remotes.
External special remote had a complication, where it was theoretically
possible for a remote to send SETSTATE or GETSTATE during INITREMOTE or
EXPORTSUPPORTED. Since the uuid of the remote is typically generate in
Remote.setup, it would only be possible to pass a Maybe
RemoteStateHandle into it, and it would otherwise have to construct its
own. Rather than go that route, I decided to send an ERROR in this case.
It seems unlikely that any existing external special remote will be
affected. They would have to make up a git-annex key, and set state for
some reason during INITREMOTE. I can imagine such a hack, but it doesn't
seem worth complicating the code in such an ugly way to support it.
Unfortunately, both TestRemote and Annex.Import needed the Remote
to have a new field added that holds its RemoteStateHandle.
2019-10-14 16:33:27 +00:00
|
|
|
let rs = RemoteStateHandle cu
|
2019-10-11 16:32:11 +00:00
|
|
|
let c = fromMaybe M.empty $ M.lookup cu m
|
fix encryption of content to gcrypt and git-lfs
Fix serious regression in gcrypt and encrypted git-lfs remotes.
Since version 7.20200202.7, git-annex incorrectly stored content
on those remotes without encrypting it.
Problem was, Remote.Git enumerates all git remotes, including git-lfs
and gcrypt. It then dispatches to those. So, Remote.List used the
RemoteConfigParser from Remote.Git, instead of from git-lfs or gcrypt,
and that parser does not know about encryption fields, so did not
include them in the ParsedRemoteConfig. (Also didn't include other
fields specific to those remotes, perhaps chunking etc also didn't
get through.)
To fix, had to move RemoteConfig parsing down into the generate methods
of each remote, rather than doing it in Remote.List.
And a consequence of that was that ParsedRemoteConfig had to change to
include the RemoteConfig that got parsed, so that testremote can
generate a new remote based on an existing remote.
(I would have rather fixed this just inside Remote.Git, but that was not
practical, at least not w/o re-doing work that Remote.List already did.
Big ugly mostly mechanical patch seemed preferable to making git-annex
slower.)
2020-02-26 21:20:56 +00:00
|
|
|
generate t g u c gc rs >>= \case
|
2019-03-04 20:02:56 +00:00
|
|
|
Nothing -> return Nothing
|
add RemoteStateHandle
This solves the problem of sameas remotes trampling over per-remote
state. Used for:
* per-remote state, of course
* per-remote metadata, also of course
* per-remote content identifiers, because two remote implementations
could in theory generate the same content identifier for two different
peices of content
While chunk logs are per-remote data, they don't use this, because the
number and size of chunks stored is a common property across sameas
remotes.
External special remote had a complication, where it was theoretically
possible for a remote to send SETSTATE or GETSTATE during INITREMOTE or
EXPORTSUPPORTED. Since the uuid of the remote is typically generate in
Remote.setup, it would only be possible to pass a Maybe
RemoteStateHandle into it, and it would otherwise have to construct its
own. Rather than go that route, I decided to send an ERROR in this case.
It seems unlikely that any existing external special remote will be
affected. They would have to make up a git-annex key, and set state for
some reason during INITREMOTE. I can imagine such a hack, but it doesn't
seem worth complicating the code in such an ugly way to support it.
Unfortunately, both TestRemote and Annex.Import needed the Remote
to have a new field added that holds its RemoteStateHandle.
2019-10-14 16:33:27 +00:00
|
|
|
Just r -> Just <$> adjustExportImport (adjustReadOnly (addHooks r)) rs
|
2012-01-10 17:11:16 +00:00
|
|
|
|
2012-08-05 18:49:47 +00:00
|
|
|
{- Updates a local git Remote, re-reading its git config. -}
|
2013-09-12 19:54:35 +00:00
|
|
|
updateRemote :: Remote -> Annex (Maybe Remote)
|
2012-08-05 18:49:47 +00:00
|
|
|
updateRemote remote = do
|
2020-09-22 17:52:26 +00:00
|
|
|
m <- remoteConfigMap
|
2018-06-04 18:31:55 +00:00
|
|
|
remote' <- updaterepo =<< getRepo remote
|
2012-08-05 18:49:47 +00:00
|
|
|
remoteGen m (remotetype remote) remote'
|
2012-11-11 04:51:07 +00:00
|
|
|
where
|
|
|
|
updaterepo r
|
|
|
|
| Git.repoIsLocal r || Git.repoIsLocalUnknown r =
|
2015-08-05 17:49:54 +00:00
|
|
|
Remote.Git.configRead False r
|
2012-11-11 04:51:07 +00:00
|
|
|
| otherwise = return r
|
2012-08-05 18:49:47 +00:00
|
|
|
|
2013-09-09 13:58:17 +00:00
|
|
|
{- Checks if a remote is syncable using git. -}
|
2020-02-19 17:58:26 +00:00
|
|
|
gitSyncableRemoteType :: RemoteType -> Bool
|
|
|
|
gitSyncableRemoteType t = t `elem`
|
2019-11-18 20:13:21 +00:00
|
|
|
[ Remote.Git.remote
|
|
|
|
, Remote.GCrypt.remote
|
|
|
|
, Remote.P2P.remote
|
|
|
|
, Remote.GitLFS.remote
|
|
|
|
]
|