2011-03-27 19:56:43 +00:00
{- Standard git remotes.
-
2024-06-07 16:03:48 +00:00
- Copyright 2011 - 2024 Joey Hess < id @ joeyh . name >
2011-03-27 19:56:43 +00:00
-
2018-03-09 05:03:11 +00:00
- Licensed under the GNU AGPL version 3 or higher .
2011-03-27 19:56:43 +00:00
- }
2013-05-11 20:03:00 +00:00
{- # LANGUAGE CPP # -}
2019-11-27 20:54:11 +00:00
{- # LANGUAGE OverloadedStrings # -}
2013-05-11 20:03:00 +00:00
2012-07-22 17:48:50 +00:00
module Remote.Git (
remote ,
configRead ,
2020-04-17 21:09:29 +00:00
onLocalRepo ,
2012-07-22 17:48:50 +00:00
) where
2011-03-27 19:56:43 +00:00
2016-01-20 20:36:33 +00:00
import Annex.Common
2013-05-14 17:53:29 +00:00
import Annex.Ssh
2011-06-02 01:56:04 +00:00
import Types.Remote
2013-01-01 17:52:47 +00:00
import Types.GitConfig
2011-06-30 17:16:57 +00:00
import qualified Git
2011-12-13 19:05:07 +00:00
import qualified Git.Config
import qualified Git.Construct
2012-10-12 17:45:14 +00:00
import qualified Git.Command
2013-09-05 20:02:39 +00:00
import qualified Git.GCrypt
2014-05-22 17:42:17 +00:00
import qualified Git.Types as Git
2011-03-27 19:56:43 +00:00
import qualified Annex
2014-03-22 14:42:38 +00:00
import Annex.Transfer
2021-04-14 18:06:43 +00:00
import Annex.CopyFile
2021-08-16 19:56:24 +00:00
import Annex.Verify
2022-01-07 17:17:43 +00:00
import Annex.Content ( verificationOfContentFailed )
2011-10-15 21:47:03 +00:00
import Annex.UUID
2011-10-04 04:40:47 +00:00
import qualified Annex.Content
2011-12-12 21:38:46 +00:00
import qualified Annex.BranchState
2012-02-25 22:02:49 +00:00
import qualified Annex.Branch
2013-09-28 18:35:21 +00:00
import qualified Annex.Url as Url
2019-10-10 17:08:17 +00:00
import qualified Annex.SpecialRemote.Config as SpecialRemote
2013-05-12 23:19:28 +00:00
import Utility.Tmp
2011-03-28 01:43:25 +00:00
import Config
2013-03-13 20:16:01 +00:00
import Config.Cost
2020-01-14 16:35:08 +00:00
import Annex.SpecialRemote.Config
2017-08-17 16:26:14 +00:00
import Config.DynamicConfig
2014-01-26 20:36:31 +00:00
import Annex.Init
2014-03-13 23:06:26 +00:00
import Types.CleanupActions
2014-01-26 20:32:55 +00:00
import qualified CmdLine.GitAnnexShell.Fields as Fields
2012-12-12 23:20:38 +00:00
import Logs.Location
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
import Logs.Proxy
2024-06-25 14:06:28 +00:00
import Logs.Cluster.Basic
2013-03-28 21:03:04 +00:00
import Utility.Metered
2013-10-14 19:05:10 +00:00
import Utility.Env
2013-10-11 20:03:18 +00:00
import Utility.Batch
2013-09-07 22:38:00 +00:00
import Remote.Helper.Git
2013-09-24 17:37:41 +00:00
import Remote.Helper.Messages
2019-02-20 19:55:01 +00:00
import Remote.Helper.ExportImport
2013-09-24 17:37:41 +00:00
import qualified Remote.Helper.Ssh as Ssh
2013-09-07 22:38:00 +00:00
import qualified Remote.GCrypt
2019-08-01 19:11:45 +00:00
import qualified Remote.GitLFS
2016-12-06 16:19:47 +00:00
import qualified Remote.P2P
2018-03-08 20:21:16 +00:00
import qualified Remote.Helper.P2P as P2PHelper
import P2P.Address
2015-02-28 21:23:13 +00:00
import Annex.Path
2014-05-22 17:42:17 +00:00
import Creds
2015-10-09 17:07:03 +00:00
import Types.NumCopies
2020-01-10 18:10:20 +00:00
import Types.ProposedAccepted
2017-09-30 02:36:08 +00:00
import Annex.Action
2018-03-12 21:56:39 +00:00
import Messages.Progress
2011-03-27 19:56:43 +00:00
2012-09-20 17:35:53 +00:00
import Control.Concurrent
2013-09-07 22:38:00 +00:00
import qualified Data.Map as M
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
import qualified Data.Set as S
import qualified Data.ByteString as B
2023-03-03 16:58:39 +00:00
import qualified Utility.RawFilePath as R
2014-08-10 18:52:58 +00:00
import Network.URI
2012-09-20 17:35:53 +00:00
2011-12-31 08:11:39 +00:00
remote :: RemoteType
2017-09-07 17:45:31 +00:00
remote = RemoteType
{ typename = " git "
, enumerate = list
, generate = gen
2020-01-14 17:18:15 +00:00
, configParser = mkRemoteConfigParser
2020-01-20 19:20:04 +00:00
[ optionalStringParser locationField
( FieldDesc " url of git remote to remember with special remote " )
]
2017-09-07 17:45:31 +00:00
, setup = gitSetup
, exportSupported = exportUnsupported
2019-02-20 19:55:01 +00:00
, importSupported = importUnsupported
add thirdPartyPopulated interface
This is to support, eg a borg repo as a special remote, which is
populated not by running git-annex commands, but by using borg. Then
git-annex sync lists the content of the remote, learns which files are
annex objects, and treats those as present in the remote.
So, most of the import machinery is reused, to a new purpose. While
normally importtree maintains a remote tracking branch, this does not,
because the files stored in the remote are annex object files, not
user-visible filenames. But, internally, a git tree is still generated,
of the files on the remote that are annex objects. This tree is used
by retrieveExportWithContentIdentifier, etc. As with other import/export
remotes, that the tree is recorded in the export log, and gets grafted
into the git-annex branch.
importKey changed to be able to return Nothing, to indicate when an
ImportLocation is not an annex object and so should be skipped from
being included in the tree.
It did not seem to make sense to have git-annex import do this, since
from the user's perspective, it's not like other imports. So only
git-annex sync does it.
Note that, git-annex sync does not yet download objects from such
remotes that are preferred content. importKeys is run with
content downloading disabled, to avoid getting the content of all
objects. Perhaps what's needed is for seekSyncContent to be run with these
remotes, but I don't know if it will just work (in particular, it needs
to avoid trying to transfer objects to them), so I skipped that for now.
(Untested and unused as of yet.)
This commit was sponsored by Jochen Bartl on Patreon.
2020-12-18 18:52:57 +00:00
, thirdPartyPopulated = False
2017-09-07 17:45:31 +00:00
}
2011-03-29 03:51:07 +00:00
2020-01-14 16:35:08 +00:00
locationField :: RemoteConfigField
locationField = Accepted " location "
2015-08-05 17:49:54 +00:00
list :: Bool -> Annex [ Git . Repo ]
list autoinit = do
2011-12-14 19:30:14 +00:00
c <- fromRepo Git . config
2018-01-09 19:36:56 +00:00
rs <- mapM ( tweakurl c ) =<< Annex . getGitRemotes
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
rs' <- mapM ( configRead autoinit ) ( filter ( not . isGitRemoteAnnex ) rs )
2024-06-19 01:31:32 +00:00
proxies <- doQuietAction getProxies
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
if proxies == mempty
then return rs'
else do
proxied <- listProxied proxies rs'
return ( proxied ++ rs' )
2012-10-29 01:27:15 +00:00
where
2020-02-19 17:45:11 +00:00
annexurl r = remoteConfig r " annexurl "
2012-10-29 01:27:15 +00:00
tweakurl c r = do
let n = fromJust $ Git . remoteName r
2020-02-19 17:45:11 +00:00
case M . lookup ( annexurl r ) c of
2012-10-29 01:27:15 +00:00
Nothing -> return r
Just url -> inRepo $ \ g ->
Git . Construct . remoteNamed n $
2023-03-23 19:19:04 +00:00
Git . Construct . fromRemoteLocation ( Git . fromConfigValue url ) False g
2012-07-22 17:48:50 +00:00
2024-05-07 19:01:22 +00:00
isGitRemoteAnnex :: Git . Repo -> Bool
isGitRemoteAnnex r = " annex:: " ` isPrefixOf ` Git . repoLocation r
2022-06-09 17:16:50 +00:00
{- Git remotes are normally set up using standard git commands, not
2014-05-22 17:42:17 +00:00
- git - annex initremote and enableremote .
-
2022-06-09 17:16:50 +00:00
- For initremote , probe the location to find the uuid .
- and set up a git remote .
2014-05-22 17:42:17 +00:00
-
- enableremote simply sets up a git remote using the stored location .
- No attempt is made to make the remote be accessible via ssh key setup ,
- etc .
- }
2017-02-07 18:35:58 +00:00
gitSetup :: SetupStage -> Maybe UUID -> Maybe CredPair -> RemoteConfig -> RemoteGitConfig -> Annex ( RemoteConfig , UUID )
gitSetup Init mu _ c _ = do
2022-06-09 17:16:50 +00:00
let location = maybe ( giveup " Specify location=url " ) fromProposedAccepted $
M . lookup locationField c
2023-03-23 19:19:04 +00:00
r <- inRepo $ Git . Construct . fromRemoteLocation location False
2022-06-09 17:16:50 +00:00
r' <- tryGitConfigRead False r False
let u = getUncachedUUID r'
if u == NoUUID
2022-06-09 17:40:05 +00:00
then giveup " git repository does not have an annex uuid "
2022-06-09 17:16:50 +00:00
else if isNothing mu || mu == Just u
then enableRemote ( Just u ) c
else giveup " git repository does not have specified uuid "
2021-03-17 13:41:12 +00:00
gitSetup ( Enable _ ) mu _ c _ = enableRemote mu c
gitSetup ( AutoEnable _ ) mu _ c _ = enableRemote mu c
enableRemote :: Maybe UUID -> RemoteConfig -> Annex ( RemoteConfig , UUID )
enableRemote ( Just u ) c = do
2023-04-18 18:00:02 +00:00
rs <- Annex . getGitRemotes
unless ( any ( \ r -> Git . remoteName r == Just cname ) rs ) $
inRepo $ Git . Command . run
[ Param " remote "
, Param " add "
, Param cname
, Param clocation
]
2014-05-22 17:42:17 +00:00
return ( c , u )
2023-04-18 18:00:02 +00:00
where
cname = fromMaybe ( giveup " no name " ) ( SpecialRemote . lookupName c )
clocation = maybe ( giveup " no location " ) fromProposedAccepted ( M . lookup locationField c )
2022-06-09 17:16:50 +00:00
enableRemote Nothing _ = giveup " unable to enable git remote with no specified uuid "
2014-05-22 17:42:17 +00:00
2012-07-22 17:48:50 +00:00
{- It's assumed to be cheap to read the config of non - URL remotes, so this is
2018-01-10 18:21:18 +00:00
- done each time git - annex is run in a way that uses remotes , unless
- annex - checkuuid is false .
2012-07-22 17:48:50 +00:00
-
- Conversely , the config of an URL remote is only read when there is no
- cached UUID value . - }
2015-08-05 17:49:54 +00:00
configRead :: Bool -> Git . Repo -> Annex Git . Repo
configRead autoinit r = do
2014-05-16 20:08:20 +00:00
gc <- Annex . getRemoteGitConfig r
avoid beware of the leopard situation
* Display a warning message when a remote uses a protocol, such as
git://, that git-annex does not support. Silently skipping such a
remote was confusing behavior.
It sets annex-ignore, so the warning is only displayed once.
* Also display a warning message when a remote, without a known uuid,
is located in a directory that does not currently exist, to avoid
silently skipping such a remote.
This is a bit more debatable, since git-annex get will say,
try making repository available. And since it does not set annex-ignore,
the warning will be displayed repeatedly. It's also an extreme edge case,
I don't think I've ever seen it happen in real life.
2020-05-04 17:01:11 +00:00
hasuuid <- ( /= NoUUID ) <$> getRepoUUID r
2017-08-17 16:26:14 +00:00
annexignore <- liftIO $ getDynamicConfig ( remoteAnnexIgnore gc )
avoid beware of the leopard situation
* Display a warning message when a remote uses a protocol, such as
git://, that git-annex does not support. Silently skipping such a
remote was confusing behavior.
It sets annex-ignore, so the warning is only displayed once.
* Also display a warning message when a remote, without a known uuid,
is located in a directory that does not currently exist, to avoid
silently skipping such a remote.
This is a bit more debatable, since git-annex get will say,
try making repository available. And since it does not set annex-ignore,
the warning will be displayed repeatedly. It's also an extreme edge case,
I don't think I've ever seen it happen in real life.
2020-05-04 17:01:11 +00:00
case ( repoCheap r , annexignore , hasuuid ) of
2013-01-01 17:52:47 +00:00
( _ , True , _ ) -> return r
2018-01-10 18:21:18 +00:00
( True , _ , _ )
avoid beware of the leopard situation
* Display a warning message when a remote uses a protocol, such as
git://, that git-annex does not support. Silently skipping such a
remote was confusing behavior.
It sets annex-ignore, so the warning is only displayed once.
* Also display a warning message when a remote, without a known uuid,
is located in a directory that does not currently exist, to avoid
silently skipping such a remote.
This is a bit more debatable, since git-annex get will say,
try making repository available. And since it does not set annex-ignore,
the warning will be displayed repeatedly. It's also an extreme edge case,
I don't think I've ever seen it happen in real life.
2020-05-04 17:01:11 +00:00
| remoteAnnexCheckUUID gc -> tryGitConfigRead autoinit r hasuuid
2018-01-10 18:21:18 +00:00
| otherwise -> return r
avoid beware of the leopard situation
* Display a warning message when a remote uses a protocol, such as
git://, that git-annex does not support. Silently skipping such a
remote was confusing behavior.
It sets annex-ignore, so the warning is only displayed once.
* Also display a warning message when a remote, without a known uuid,
is located in a directory that does not currently exist, to avoid
silently skipping such a remote.
This is a bit more debatable, since git-annex get will say,
try making repository available. And since it does not set annex-ignore,
the warning will be displayed repeatedly. It's also an extreme edge case,
I don't think I've ever seen it happen in real life.
2020-05-04 17:01:11 +00:00
( False , _ , False ) -> configSpecialGitRemotes r >>= \ case
Nothing -> tryGitConfigRead autoinit r False
2019-11-18 20:09:09 +00:00
Just r' -> return r'
2012-07-22 17:48:50 +00:00
_ -> return r
2011-03-28 01:43:25 +00:00
2024-06-27 19:21:03 +00:00
fix encryption of content to gcrypt and git-lfs
Fix serious regression in gcrypt and encrypted git-lfs remotes.
Since version 7.20200202.7, git-annex incorrectly stored content
on those remotes without encrypting it.
Problem was, Remote.Git enumerates all git remotes, including git-lfs
and gcrypt. It then dispatches to those. So, Remote.List used the
RemoteConfigParser from Remote.Git, instead of from git-lfs or gcrypt,
and that parser does not know about encryption fields, so did not
include them in the ParsedRemoteConfig. (Also didn't include other
fields specific to those remotes, perhaps chunking etc also didn't
get through.)
To fix, had to move RemoteConfig parsing down into the generate methods
of each remote, rather than doing it in Remote.List.
And a consequence of that was that ParsedRemoteConfig had to change to
include the RemoteConfig that got parsed, so that testremote can
generate a new remote based on an existing remote.
(I would have rather fixed this just inside Remote.Git, but that was not
practical, at least not w/o re-doing work that Remote.List already did.
Big ugly mostly mechanical patch seemed preferable to making git-annex
slower.)
2020-02-26 21:20:56 +00:00
gen :: Git . Repo -> UUID -> RemoteConfig -> RemoteGitConfig -> RemoteStateHandle -> Annex ( Maybe Remote )
gen r u rc gc rs
2019-08-05 17:24:21 +00:00
-- Remote.GitLFS may be used with a repo that is also encrypted
-- with gcrypt so is checked first.
fix encryption of content to gcrypt and git-lfs
Fix serious regression in gcrypt and encrypted git-lfs remotes.
Since version 7.20200202.7, git-annex incorrectly stored content
on those remotes without encrypting it.
Problem was, Remote.Git enumerates all git remotes, including git-lfs
and gcrypt. It then dispatches to those. So, Remote.List used the
RemoteConfigParser from Remote.Git, instead of from git-lfs or gcrypt,
and that parser does not know about encryption fields, so did not
include them in the ParsedRemoteConfig. (Also didn't include other
fields specific to those remotes, perhaps chunking etc also didn't
get through.)
To fix, had to move RemoteConfig parsing down into the generate methods
of each remote, rather than doing it in Remote.List.
And a consequence of that was that ParsedRemoteConfig had to change to
include the RemoteConfig that got parsed, so that testremote can
generate a new remote based on an existing remote.
(I would have rather fixed this just inside Remote.Git, but that was not
practical, at least not w/o re-doing work that Remote.List already did.
Big ugly mostly mechanical patch seemed preferable to making git-annex
slower.)
2020-02-26 21:20:56 +00:00
| remoteAnnexGitLFS gc = Remote . GitLFS . gen r u rc gc rs
| Git . GCrypt . isEncrypted r = Remote . GCrypt . chainGen r u rc gc rs
2016-12-06 16:19:47 +00:00
| otherwise = case repoP2PAddress r of
2018-01-10 18:21:18 +00:00
Nothing -> do
use P2P protocol for checkpresent, retrieve, and store
Note that, due to not using rsync to transfer files to ssh remotes
any longer, permissions and other file metadata of annexed files
will no longer be preserved when copying them to ssh remotes.
Other remotes never supported preserving that information, so
this is not considered a regression. Added NEWS item about this.
Another significant side effect of this is that, even when rsync is run to
retrieve a file, its progress display will no longer be shown, and
instead the native git-annex progress display will appear. It would be
possible to use the rsync process display when rsync is used (old
git-annex-shell and also retrieval from a local repository), but it
would have complicated the code unncessarily, and been inconsistent
behavior.
(I'd been thinking for a while about eliminating the rsync progress
display, since it's got some annoying verbosities, including display of
the key and the "(xfr#1, to-chk=0/1)" bit and was already somewhat
inconsistent.)
retrieveKeyFileCheap still uses rsync, since that ensures that it gets
the actual file content from the remote. Using the P2P protocol would
use the local content, as long as the local and remote size are the
same.
This commit was sponsored by John Pellman on Patreon.
2018-03-09 16:57:32 +00:00
st <- mkState r u gc
fix encryption of content to gcrypt and git-lfs
Fix serious regression in gcrypt and encrypted git-lfs remotes.
Since version 7.20200202.7, git-annex incorrectly stored content
on those remotes without encrypting it.
Problem was, Remote.Git enumerates all git remotes, including git-lfs
and gcrypt. It then dispatches to those. So, Remote.List used the
RemoteConfigParser from Remote.Git, instead of from git-lfs or gcrypt,
and that parser does not know about encryption fields, so did not
include them in the ParsedRemoteConfig. (Also didn't include other
fields specific to those remotes, perhaps chunking etc also didn't
get through.)
To fix, had to move RemoteConfig parsing down into the generate methods
of each remote, rather than doing it in Remote.List.
And a consequence of that was that ParsedRemoteConfig had to change to
include the RemoteConfig that got parsed, so that testremote can
generate a new remote based on an existing remote.
(I would have rather fixed this just inside Remote.Git, but that was not
practical, at least not w/o re-doing work that Remote.List already did.
Big ugly mostly mechanical patch seemed preferable to making git-annex
slower.)
2020-02-26 21:20:56 +00:00
c <- parsedRemoteConfig remote rc
2024-06-27 19:21:03 +00:00
go st c <$> remoteCost gc c ( defaultRepoCost r )
fix encryption of content to gcrypt and git-lfs
Fix serious regression in gcrypt and encrypted git-lfs remotes.
Since version 7.20200202.7, git-annex incorrectly stored content
on those remotes without encrypting it.
Problem was, Remote.Git enumerates all git remotes, including git-lfs
and gcrypt. It then dispatches to those. So, Remote.List used the
RemoteConfigParser from Remote.Git, instead of from git-lfs or gcrypt,
and that parser does not know about encryption fields, so did not
include them in the ParsedRemoteConfig. (Also didn't include other
fields specific to those remotes, perhaps chunking etc also didn't
get through.)
To fix, had to move RemoteConfig parsing down into the generate methods
of each remote, rather than doing it in Remote.List.
And a consequence of that was that ParsedRemoteConfig had to change to
include the RemoteConfig that got parsed, so that testremote can
generate a new remote based on an existing remote.
(I would have rather fixed this just inside Remote.Git, but that was not
practical, at least not w/o re-doing work that Remote.List already did.
Big ugly mostly mechanical patch seemed preferable to making git-annex
slower.)
2020-02-26 21:20:56 +00:00
Just addr -> Remote . P2P . chainGen addr r u rc gc rs
2012-10-29 01:27:15 +00:00
where
fix encryption of content to gcrypt and git-lfs
Fix serious regression in gcrypt and encrypted git-lfs remotes.
Since version 7.20200202.7, git-annex incorrectly stored content
on those remotes without encrypting it.
Problem was, Remote.Git enumerates all git remotes, including git-lfs
and gcrypt. It then dispatches to those. So, Remote.List used the
RemoteConfigParser from Remote.Git, instead of from git-lfs or gcrypt,
and that parser does not know about encryption fields, so did not
include them in the ParsedRemoteConfig. (Also didn't include other
fields specific to those remotes, perhaps chunking etc also didn't
get through.)
To fix, had to move RemoteConfig parsing down into the generate methods
of each remote, rather than doing it in Remote.List.
And a consequence of that was that ParsedRemoteConfig had to change to
include the RemoteConfig that got parsed, so that testremote can
generate a new remote based on an existing remote.
(I would have rather fixed this just inside Remote.Git, but that was not
practical, at least not w/o re-doing work that Remote.List already did.
Big ugly mostly mechanical patch seemed preferable to making git-annex
slower.)
2020-02-26 21:20:56 +00:00
go st c cst = Just new
2013-01-01 17:52:47 +00:00
where
new = Remote
{ uuid = u
, cost = cst
, name = Git . repoDescribe r
use P2P protocol for checkpresent, retrieve, and store
Note that, due to not using rsync to transfer files to ssh remotes
any longer, permissions and other file metadata of annexed files
will no longer be preserved when copying them to ssh remotes.
Other remotes never supported preserving that information, so
this is not considered a regression. Added NEWS item about this.
Another significant side effect of this is that, even when rsync is run to
retrieve a file, its progress display will no longer be shown, and
instead the native git-annex progress display will appear. It would be
possible to use the rsync process display when rsync is used (old
git-annex-shell and also retrieval from a local repository), but it
would have complicated the code unncessarily, and been inconsistent
behavior.
(I'd been thinking for a while about eliminating the rsync progress
display, since it's got some annoying verbosities, including display of
the key and the "(xfr#1, to-chk=0/1)" bit and was already somewhat
inconsistent.)
retrieveKeyFileCheap still uses rsync, since that ensures that it gets
the actual file content from the remote. Using the P2P protocol would
use the local content, as long as the local and remote size are the
same.
This commit was sponsored by John Pellman on Patreon.
2018-03-09 16:57:32 +00:00
, storeKey = copyToRemote new st
, retrieveKeyFile = copyFromRemote new st
2021-02-10 17:29:12 +00:00
, retrieveKeyFileCheap = copyFromRemoteCheap st r
2018-06-21 15:35:27 +00:00
, retrievalSecurityPolicy = RetrievalAllKeysSecure
use P2P protocol for checkpresent, retrieve, and store
Note that, due to not using rsync to transfer files to ssh remotes
any longer, permissions and other file metadata of annexed files
will no longer be preserved when copying them to ssh remotes.
Other remotes never supported preserving that information, so
this is not considered a regression. Added NEWS item about this.
Another significant side effect of this is that, even when rsync is run to
retrieve a file, its progress display will no longer be shown, and
instead the native git-annex progress display will appear. It would be
possible to use the rsync process display when rsync is used (old
git-annex-shell and also retrieval from a local repository), but it
would have complicated the code unncessarily, and been inconsistent
behavior.
(I'd been thinking for a while about eliminating the rsync progress
display, since it's got some annoying verbosities, including display of
the key and the "(xfr#1, to-chk=0/1)" bit and was already somewhat
inconsistent.)
retrieveKeyFileCheap still uses rsync, since that ensures that it gets
the actual file content from the remote. Using the P2P protocol would
use the local content, as long as the local and remote size are the
same.
This commit was sponsored by John Pellman on Patreon.
2018-03-09 16:57:32 +00:00
, removeKey = dropKey new st
, lockContent = Just ( lockKey new st )
, checkPresent = inAnnex new st
2014-08-06 17:45:19 +00:00
, checkPresentCheap = repoCheap r
2017-09-01 17:02:07 +00:00
, exportActions = exportUnsupported
2019-02-20 19:55:01 +00:00
, importActions = importUnsupported
2013-01-01 17:52:47 +00:00
, whereisKey = Nothing
2013-10-11 20:03:18 +00:00
, remoteFsck = if Git . repoIsUrl r
then Nothing
else Just $ fsckOnRemote r
2013-10-27 19:38:59 +00:00
, repairRepo = if Git . repoIsUrl r
then Nothing
else Just $ repairRemote r
2013-11-02 20:37:28 +00:00
, config = c
2013-09-07 22:38:00 +00:00
, localpath = localpathCalc r
2018-06-04 20:48:26 +00:00
, getRepo = getRepoFromState st
removal of the rest of remoteGitConfig
In keyUrls, the GitConfig is used only by annexLocations
to support configured Differences. Since such configurations affect all
clones of a repository, the local repo's GitConfig must have the same
information as the remote's GitConfig would have. So, used getGitConfig
to get the local GitConfig, which is cached and so available cheaply.
That actually fixed a bug noone had ever noticed: keyUrls is
used for remotes accessed over http. The full git config of such a
remote is normally not available, so the remoteGitConfig that keyUrls
used would not have the necessary information in it.
In copyFromRemoteCheap', it uses gitAnnexLocation,
which does need the GitConfig of the remote repo itself in order to
check if it's crippled, supports symlinks, etc. So, made the
State include that GitConfig, cached. The use of gitAnnexLocation is
within a (not $ Git.repoIsUrl repo) guard, so it's local, and so
its git config will always be read and available.
(Note that gitAnnexLocation in turn calls annexLocations, so the
Differences config it uses in this case comes from the remote repo's
GitConfig and not from the local repo's GitConfig. As explained above
this is ok since they must have the same value.)
Not very happy with this mess of different GitConfigs not type-safe and
some read only sometimes etc. Very hairy. Think I got it this change
right. Test suite passes..
This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
, gitconfig = gc
2013-01-01 17:52:47 +00:00
, readonly = Git . repoIsHttp r
2018-08-30 15:12:18 +00:00
, appendonly = False
2020-12-28 19:08:53 +00:00
, untrustworthy = False
2023-08-16 18:31:31 +00:00
, availability = repoAvail r
2013-01-01 17:52:47 +00:00
, remotetype = remote
fix encryption of content to gcrypt and git-lfs
Fix serious regression in gcrypt and encrypted git-lfs remotes.
Since version 7.20200202.7, git-annex incorrectly stored content
on those remotes without encrypting it.
Problem was, Remote.Git enumerates all git remotes, including git-lfs
and gcrypt. It then dispatches to those. So, Remote.List used the
RemoteConfigParser from Remote.Git, instead of from git-lfs or gcrypt,
and that parser does not know about encryption fields, so did not
include them in the ParsedRemoteConfig. (Also didn't include other
fields specific to those remotes, perhaps chunking etc also didn't
get through.)
To fix, had to move RemoteConfig parsing down into the generate methods
of each remote, rather than doing it in Remote.List.
And a consequence of that was that ParsedRemoteConfig had to change to
include the RemoteConfig that got parsed, so that testremote can
generate a new remote based on an existing remote.
(I would have rather fixed this just inside Remote.Git, but that was not
practical, at least not w/o re-doing work that Remote.List already did.
Big ugly mostly mechanical patch seemed preferable to making git-annex
slower.)
2020-02-26 21:20:56 +00:00
, mkUnavailable = unavailable r u rc gc rs
2015-01-13 22:11:03 +00:00
, getInfo = gitRepoInfo new
2014-12-08 17:40:15 +00:00
, claimUrl = Nothing
2014-12-11 19:32:42 +00:00
, checkUrl = Nothing
add RemoteStateHandle
This solves the problem of sameas remotes trampling over per-remote
state. Used for:
* per-remote state, of course
* per-remote metadata, also of course
* per-remote content identifiers, because two remote implementations
could in theory generate the same content identifier for two different
peices of content
While chunk logs are per-remote data, they don't use this, because the
number and size of chunks stored is a common property across sameas
remotes.
External special remote had a complication, where it was theoretically
possible for a remote to send SETSTATE or GETSTATE during INITREMOTE or
EXPORTSUPPORTED. Since the uuid of the remote is typically generate in
Remote.setup, it would only be possible to pass a Maybe
RemoteStateHandle into it, and it would otherwise have to construct its
own. Rather than go that route, I decided to send an ERROR in this case.
It seems unlikely that any existing external special remote will be
affected. They would have to make up a git-annex key, and set state for
some reason during INITREMOTE. I can imagine such a hack, but it doesn't
seem worth complicating the code in such an ugly way to support it.
Unfortunately, both TestRemote and Annex.Import needed the Remote
to have a new field added that holds its RemoteStateHandle.
2019-10-14 16:33:27 +00:00
, remoteStateHandle = rs
2013-01-01 17:52:47 +00:00
}
2011-03-27 19:56:43 +00:00
2024-06-27 19:21:03 +00:00
defaultRepoCost :: Git . Repo -> Cost
defaultRepoCost r
| repoCheap r = cheapRemoteCost
| otherwise = expensiveRemoteCost
fix encryption of content to gcrypt and git-lfs
Fix serious regression in gcrypt and encrypted git-lfs remotes.
Since version 7.20200202.7, git-annex incorrectly stored content
on those remotes without encrypting it.
Problem was, Remote.Git enumerates all git remotes, including git-lfs
and gcrypt. It then dispatches to those. So, Remote.List used the
RemoteConfigParser from Remote.Git, instead of from git-lfs or gcrypt,
and that parser does not know about encryption fields, so did not
include them in the ParsedRemoteConfig. (Also didn't include other
fields specific to those remotes, perhaps chunking etc also didn't
get through.)
To fix, had to move RemoteConfig parsing down into the generate methods
of each remote, rather than doing it in Remote.List.
And a consequence of that was that ParsedRemoteConfig had to change to
include the RemoteConfig that got parsed, so that testremote can
generate a new remote based on an existing remote.
(I would have rather fixed this just inside Remote.Git, but that was not
practical, at least not w/o re-doing work that Remote.List already did.
Big ugly mostly mechanical patch seemed preferable to making git-annex
slower.)
2020-02-26 21:20:56 +00:00
unavailable :: Git . Repo -> UUID -> RemoteConfig -> RemoteGitConfig -> RemoteStateHandle -> Annex ( Maybe Remote )
add RemoteStateHandle
This solves the problem of sameas remotes trampling over per-remote
state. Used for:
* per-remote state, of course
* per-remote metadata, also of course
* per-remote content identifiers, because two remote implementations
could in theory generate the same content identifier for two different
peices of content
While chunk logs are per-remote data, they don't use this, because the
number and size of chunks stored is a common property across sameas
remotes.
External special remote had a complication, where it was theoretically
possible for a remote to send SETSTATE or GETSTATE during INITREMOTE or
EXPORTSUPPORTED. Since the uuid of the remote is typically generate in
Remote.setup, it would only be possible to pass a Maybe
RemoteStateHandle into it, and it would otherwise have to construct its
own. Rather than go that route, I decided to send an ERROR in this case.
It seems unlikely that any existing external special remote will be
affected. They would have to make up a git-annex key, and set state for
some reason during INITREMOTE. I can imagine such a hack, but it doesn't
seem worth complicating the code in such an ugly way to support it.
Unfortunately, both TestRemote and Annex.Import needed the Remote
to have a new field added that holds its RemoteStateHandle.
2019-10-14 16:33:27 +00:00
unavailable r = gen r'
2014-08-10 18:52:58 +00:00
where
r' = case Git . location r of
Git . Local { Git . gitdir = d } ->
r { Git . location = Git . LocalUnknown d }
Git . Url url -> case uriAuthority url of
Just auth ->
let auth' = auth { uriRegName = " !dne! " }
in r { Git . location = Git . Url ( url { uriAuthority = Just auth' } ) }
Nothing -> r { Git . location = Git . Unknown }
_ -> r -- already unavailable
2011-03-28 01:43:25 +00:00
{- Tries to read the config for a specified remote, updates state, and
- returns the updated repo . - }
avoid beware of the leopard situation
* Display a warning message when a remote uses a protocol, such as
git://, that git-annex does not support. Silently skipping such a
remote was confusing behavior.
It sets annex-ignore, so the warning is only displayed once.
* Also display a warning message when a remote, without a known uuid,
is located in a directory that does not currently exist, to avoid
silently skipping such a remote.
This is a bit more debatable, since git-annex get will say,
try making repository available. And since it does not set annex-ignore,
the warning will be displayed repeatedly. It's also an extreme edge case,
I don't think I've ever seen it happen in real life.
2020-05-04 17:01:11 +00:00
tryGitConfigRead :: Bool -> Git . Repo -> Bool -> Annex Git . Repo
tryGitConfigRead autoinit r hasuuid
2013-04-24 00:06:02 +00:00
| haveconfig r = return r -- already read
2019-11-18 20:09:09 +00:00
| Git . repoIsSsh r = storeUpdatedRemote $ do
2017-02-15 19:08:46 +00:00
v <- Ssh . onRemote NoConsumeStdin r
2021-03-24 18:19:32 +00:00
( pipedconfig Git . Config . ConfigList autoinit ( Git . repoDescribe r )
, return ( Left " configlist failed " )
)
2017-02-15 19:08:46 +00:00
" configlist " [] configlistfields
2013-04-24 00:06:02 +00:00
case v of
Right r'
| haveconfig r' -> return r'
| otherwise -> configlist_failed
Left _ -> configlist_failed
2019-11-18 20:09:09 +00:00
| Git . repoIsHttp r = storeUpdatedRemote geturlconfig
2020-02-19 17:45:11 +00:00
| Git . GCrypt . isEncrypted r = handlegcrypt =<< getConfigMaybe ( remoteAnnexConfig r " uuid " )
avoid beware of the leopard situation
* Display a warning message when a remote uses a protocol, such as
git://, that git-annex does not support. Silently skipping such a
remote was confusing behavior.
It sets annex-ignore, so the warning is only displayed once.
* Also display a warning message when a remote, without a known uuid,
is located in a directory that does not currently exist, to avoid
silently skipping such a remote.
This is a bit more debatable, since git-annex get will say,
try making repository available. And since it does not set annex-ignore,
the warning will be displayed repeatedly. It's also an extreme edge case,
I don't think I've ever seen it happen in real life.
2020-05-04 17:01:11 +00:00
| Git . repoIsUrl r = do
set_ignore " uses a protocol not supported by git-annex " False
return r
| otherwise = storeUpdatedRemote $
2021-07-26 15:39:57 +00:00
readlocalannexconfig
avoid beware of the leopard situation
* Display a warning message when a remote uses a protocol, such as
git://, that git-annex does not support. Silently skipping such a
remote was confusing behavior.
It sets annex-ignore, so the warning is only displayed once.
* Also display a warning message when a remote, without a known uuid,
is located in a directory that does not currently exist, to avoid
silently skipping such a remote.
This is a bit more debatable, since git-annex get will say,
try making repository available. And since it does not set annex-ignore,
the warning will be displayed repeatedly. It's also an extreme edge case,
I don't think I've ever seen it happen in real life.
2020-05-04 17:01:11 +00:00
` catchNonAsync ` const failedreadlocalconfig
2012-10-29 01:27:15 +00:00
where
2013-04-24 00:06:02 +00:00
haveconfig = not . M . null . Git . config
2020-04-13 17:05:41 +00:00
pipedconfig st mustincludeuuuid configloc cmd params = do
v <- liftIO $ Git . Config . fromPipe r cmd params st
2013-09-24 21:51:12 +00:00
case v of
2020-01-22 17:20:06 +00:00
Right ( r' , val , _err ) -> do
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
unless ( isUUIDConfigured r' || val == mempty || not mustincludeuuuid ) $ do
filter out control characters in warning messages
Converted warning and similar to use StringContainingQuotedPath. Most
warnings are static strings, some do refer to filepaths that need to be
quoted, and others don't need quoting.
Note that, since quote filters out control characters of even
UnquotedString, this makes all warnings safe, even when an attacker
sneaks in a control character in some other way.
When json is being output, no quoting is done, since json gets its own
quoting.
This does, as a side effect, make warning messages in json output not
be indented. The indentation is only needed to offset warning messages
underneath the display of the file they apply to, so that's ok.
Sponsored-by: Brett Eisenberg on Patreon
2023-04-10 18:47:32 +00:00
warning $ UnquotedString $ " Failed to get annex.uuid configuration of repository " ++ Git . repoDescribe r
warning $ UnquotedString $ " Instead, got: " ++ show val
warning " This is unexpected; please check the network transport! "
2013-09-24 21:51:12 +00:00
return $ Right r'
2020-01-22 17:20:06 +00:00
Left l -> do
filter out control characters in warning messages
Converted warning and similar to use StringContainingQuotedPath. Most
warnings are static strings, some do refer to filepaths that need to be
quoted, and others don't need quoting.
Note that, since quote filters out control characters of even
UnquotedString, this makes all warnings safe, even when an attacker
sneaks in a control character in some other way.
When json is being output, no quoting is done, since json gets its own
quoting.
This does, as a side effect, make warning messages in json output not
be indented. The indentation is only needed to offset warning messages
underneath the display of the file they apply to, so that's ok.
Sponsored-by: Brett Eisenberg on Patreon
2023-04-10 18:47:32 +00:00
warning $ UnquotedString $ " Unable to parse git config from " ++ configloc
2021-03-24 18:19:32 +00:00
return $ Left ( show l )
2011-08-17 00:48:11 +00:00
2020-01-22 20:13:48 +00:00
geturlconfig = Url . withUrlOptionsPromptingCreds $ \ uo -> do
2021-03-24 18:19:32 +00:00
let url = Git . repoLocation r ++ " /config "
2019-11-12 14:07:27 +00:00
v <- withTmpFile " git-annex.tmp " $ \ tmpfile h -> do
liftIO $ hClose h
2021-08-18 18:49:01 +00:00
Url . download' nullMeterUpdate Nothing url tmpfile uo >>= \ case
2021-03-24 18:19:32 +00:00
Right () -> pipedconfig Git . Config . ConfigNullList
False url " git "
[ Param " config "
, Param " --null "
, Param " --list "
, Param " --file "
, File tmpfile
]
Left err -> return ( Left err )
2013-05-25 05:47:19 +00:00
case v of
2021-03-24 18:19:32 +00:00
Right r' -> do
2014-01-26 17:03:25 +00:00
-- Cache when http remote is not bare for
-- optimisation.
2023-02-14 18:11:23 +00:00
unless ( fromMaybe False $ Git . Config . isBare r' ) $
2016-05-24 19:48:22 +00:00
setremote setRemoteBare False
2014-01-26 17:03:25 +00:00
return r'
2021-03-24 18:19:32 +00:00
Left err -> do
2017-03-29 16:43:47 +00:00
set_ignore " not usable by git-annex " False
filter out control characters in warning messages
Converted warning and similar to use StringContainingQuotedPath. Most
warnings are static strings, some do refer to filepaths that need to be
quoted, and others don't need quoting.
Note that, since quote filters out control characters of even
UnquotedString, this makes all warnings safe, even when an attacker
sneaks in a control character in some other way.
When json is being output, no quoting is done, since json gets its own
quoting.
This does, as a side effect, make warning messages in json output not
be indented. The indentation is only needed to offset warning messages
underneath the display of the file they apply to, so that's ok.
Sponsored-by: Brett Eisenberg on Patreon
2023-04-10 18:47:32 +00:00
warning $ UnquotedString $ url ++ " " ++ err
2017-03-29 16:43:47 +00:00
return r
2011-08-17 00:48:11 +00:00
2013-04-24 00:06:02 +00:00
{- Is this remote just not available, or does
- it not have git - annex - shell ?
- Find out by trying to fetch from the remote . - }
configlist_failed = case Git . remoteName r of
Nothing -> return r
Just n -> do
2014-05-16 16:58:50 +00:00
whenM ( inRepo $ Git . Command . runBool [ Param " fetch " , Param " --quiet " , Param n ] ) $ do
set_ignore " does not have git-annex installed " True
2013-04-24 00:06:02 +00:00
return r
2013-05-25 05:47:19 +00:00
2014-05-16 16:58:50 +00:00
set_ignore msg longmessage = do
2014-01-26 17:03:25 +00:00
case Git . remoteName r of
Nothing -> noop
2014-05-16 16:58:50 +00:00
Just n -> do
filter out control characters in warning messages
Converted warning and similar to use StringContainingQuotedPath. Most
warnings are static strings, some do refer to filepaths that need to be
quoted, and others don't need quoting.
Note that, since quote filters out control characters of even
UnquotedString, this makes all warnings safe, even when an attacker
sneaks in a control character in some other way.
When json is being output, no quoting is done, since json gets its own
quoting.
This does, as a side effect, make warning messages in json output not
be indented. The indentation is only needed to offset warning messages
underneath the display of the file they apply to, so that's ok.
Sponsored-by: Brett Eisenberg on Patreon
2023-04-10 18:47:32 +00:00
warning $ UnquotedString $ " Remote " ++ n ++ " " ++ msg ++ " ; setting annex-ignore "
2014-05-16 16:58:50 +00:00
when longmessage $
filter out control characters in warning messages
Converted warning and similar to use StringContainingQuotedPath. Most
warnings are static strings, some do refer to filepaths that need to be
quoted, and others don't need quoting.
Note that, since quote filters out control characters of even
UnquotedString, this makes all warnings safe, even when an attacker
sneaks in a control character in some other way.
When json is being output, no quoting is done, since json gets its own
quoting.
This does, as a side effect, make warning messages in json output not
be indented. The indentation is only needed to offset warning messages
underneath the display of the file they apply to, so that's ok.
Sponsored-by: Brett Eisenberg on Patreon
2023-04-10 18:47:32 +00:00
warning $ UnquotedString $ " This could be a problem with the git-annex installation on the remote. Please make sure that git-annex-shell is available in PATH when you ssh into the remote. Once you have fixed the git-annex installation, run: git annex enableremote " ++ n
2016-05-24 19:48:22 +00:00
setremote setRemoteIgnore True
2014-01-26 17:03:25 +00:00
2016-05-24 19:48:22 +00:00
setremote setter v = case Git . remoteName r of
2013-05-25 05:47:19 +00:00
Nothing -> noop
2016-05-27 15:15:52 +00:00
Just _ -> setter r v
2016-05-24 19:48:22 +00:00
2013-09-08 19:19:14 +00:00
handlegcrypt Nothing = return r
handlegcrypt ( Just _cacheduuid ) = do
-- Generate UUID from the gcrypt-id
g <- gitRepo
case Git . GCrypt . remoteRepoId g ( Git . remoteName r ) of
Nothing -> return r
2019-11-18 20:09:09 +00:00
Just v -> storeUpdatedRemote $ liftIO $ setUUID r $
2023-10-26 17:12:57 +00:00
genUUIDInNameSpace gCryptNameSpace ( encodeBS v )
2013-04-24 00:06:02 +00:00
2014-07-15 18:45:27 +00:00
{- The local repo may not yet be initialized, so try to initialize
- it if allowed . However , if that fails , still return the read
- git config . - }
2014-07-15 18:27:43 +00:00
readlocalannexconfig = do
2017-09-30 02:36:08 +00:00
let check = do
2014-07-15 18:27:43 +00:00
Annex . BranchState . disableUpdate
remove dead nodes when loading the cluster log
This is to avoid inserting a cluster uuid into the location log when
only dead nodes in the cluster contain the content of a key.
One reason why this is necessary is Remote.keyLocations, which excludes
dead repositories from the list. But there are probably many more.
Implementing this was challenging, because Logs.Location importing
Logs.Cluster which imports Logs.Trust which imports Remote.List resulted
in an import cycle through several other modules.
Resorted to making Logs.Location not import Logs.Cluster, and instead
it assumes that Annex.clusters gets populated when necessary before it's
called.
That's done in Annex.Startup, which is run by the git-annex command
(but not other commands) at early startup in initialized repos. Or,
is run after initialization.
Note that is Remote.Git, it is unable to import Annex.Startup, because
Remote.Git importing Logs.Cluster leads the the same import cycle.
So ensureInitialized is not passed annexStartup in there.
Other commands, like git-annex-shell currently don't run annexStartup
either.
So there are cases where Logs.Location will not see clusters. So it won't add
any cluster UUIDs when loading the log. That's ok, the only reason to do
that is to make display of where objects are located include clusters,
and to make commands like git-annex get --from treat keys as being located
in a cluster. git-annex-shell certainly does not do anything like that,
and I'm pretty sure Remote.Git (and callers to Remote.Git.onLocalRepo)
don't either.
2024-06-16 18:35:07 +00:00
catchNonAsync ( autoInitialize noop ( pure [] ) ) $ \ e ->
filter out control characters in warning messages
Converted warning and similar to use StringContainingQuotedPath. Most
warnings are static strings, some do refer to filepaths that need to be
quoted, and others don't need quoting.
Note that, since quote filters out control characters of even
UnquotedString, this makes all warnings safe, even when an attacker
sneaks in a control character in some other way.
When json is being output, no quoting is done, since json gets its own
quoting.
This does, as a side effect, make warning messages in json output not
be indented. The indentation is only needed to offset warning messages
underneath the display of the file they apply to, so that's ok.
Sponsored-by: Brett Eisenberg on Patreon
2023-04-10 18:47:32 +00:00
warning $ UnquotedString $ " Remote " ++ Git . repoDescribe r ++
2023-02-14 16:34:07 +00:00
" : " ++ show e
2014-07-15 18:27:43 +00:00
Annex . getState Annex . repo
2023-09-07 18:56:26 +00:00
let r' = r { Git . repoPathSpecifiedExplicitly = True }
2023-09-07 18:36:16 +00:00
s <- newLocal r'
avoid flushing keys db queue after each Annex action
The flush was only done Annex.run' to make sure that the queue was flushed
before git-annex exits. But, doing it there means that as soon as one
change gets queued, it gets flushed soon after, which contributes to
excessive writes to the database, slowing git-annex down.
(This does not yet speed git-annex up, but it is a stepping stone to
doing so.)
Database queues do not autoflush when garbage collected, so have to
be flushed explicitly. I don't think it's possible to make them
autoflush (except perhaps if git-annex sqitched to using ResourceT..).
The comment in Database.Keys.closeDb used to be accurate, since the
automatic flushing did mean that all writes reached the database even
when closeDb was not called. But now, closeDb or flushDb needs to be
called before stopping using an Annex state. So, removed that comment.
In Remote.Git, change to using quiesce everywhere that it used to use
stopCoProcesses. This means that uses on onLocal in there are just as
slow as before. I considered only calling closeDb on the local git remotes
when git-annex exits. But, the reason that Remote.Git calls stopCoProcesses
in each onLocal is so as not to leave git processes running that have files
open on the remote repo, when it's on removable media. So, it seemed to make
sense to also closeDb after each one, since sqlite may also keep files
open. Although that has not seemed to cause problems with removable
media so far. It was also just easier to quiesce in each onLocal than
once at the end. This does likely leave performance on the floor, so
could be revisited.
In Annex.Content.saveState, there was no reason to close the db,
flushing it is enough.
The rest of the changes are from auditing for Annex.new, and making
sure that quiesce is called, after any action that might possibly need
it.
After that audit, I'm pretty sure that the change to Annex.run' is
safe. The only concern might be that this does let more changes get
queued for write to the db, and if git-annex is interrupted, those will be
lost. But interrupting git-annex can obviously already prevent it from
writing the most recent change to the db, so it must recover from such
lost data... right?
Sponsored-by: Dartmouth College's Datalad project
2022-10-12 17:50:46 +00:00
liftIO $ Annex . eval s $ check
` finally ` quiesce True
2015-08-05 17:49:54 +00:00
avoid beware of the leopard situation
* Display a warning message when a remote uses a protocol, such as
git://, that git-annex does not support. Silently skipping such a
remote was confusing behavior.
It sets annex-ignore, so the warning is only displayed once.
* Also display a warning message when a remote, without a known uuid,
is located in a directory that does not currently exist, to avoid
silently skipping such a remote.
This is a bit more debatable, since git-annex get will say,
try making repository available. And since it does not set annex-ignore,
the warning will be displayed repeatedly. It's also an extreme edge case,
I don't think I've ever seen it happen in real life.
2020-05-04 17:01:11 +00:00
failedreadlocalconfig = do
unless hasuuid $ case Git . remoteName r of
Nothing -> noop
Just n -> do
filter out control characters in warning messages
Converted warning and similar to use StringContainingQuotedPath. Most
warnings are static strings, some do refer to filepaths that need to be
quoted, and others don't need quoting.
Note that, since quote filters out control characters of even
UnquotedString, this makes all warnings safe, even when an attacker
sneaks in a control character in some other way.
When json is being output, no quoting is done, since json gets its own
quoting.
This does, as a side effect, make warning messages in json output not
be indented. The indentation is only needed to offset warning messages
underneath the display of the file they apply to, so that's ok.
Sponsored-by: Brett Eisenberg on Patreon
2023-04-10 18:47:32 +00:00
warning $ UnquotedString $ " Remote " ++ n ++ " cannot currently be accessed. "
avoid beware of the leopard situation
* Display a warning message when a remote uses a protocol, such as
git://, that git-annex does not support. Silently skipping such a
remote was confusing behavior.
It sets annex-ignore, so the warning is only displayed once.
* Also display a warning message when a remote, without a known uuid,
is located in a directory that does not currently exist, to avoid
silently skipping such a remote.
This is a bit more debatable, since git-annex get will say,
try making repository available. And since it does not set annex-ignore,
the warning will be displayed repeatedly. It's also an extreme edge case,
I don't think I've ever seen it happen in real life.
2020-05-04 17:01:11 +00:00
return r
2015-08-05 17:49:54 +00:00
configlistfields = if autoinit
then [ ( Fields . autoInit , " 1 " ) ]
else []
2014-07-15 18:27:43 +00:00
2019-11-18 20:09:09 +00:00
{- Handles special remotes that can be enabled by the presence of
- regular git remotes .
-
- When a remote repo is found to be such a special remote , its
- UUID is cached in the git config , and the repo returned with
- the UUID set .
- }
configSpecialGitRemotes :: Git . Repo -> Annex ( Maybe Git . Repo )
configSpecialGitRemotes r = Remote . GitLFS . configKnownUrl r >>= \ case
Nothing -> return Nothing
Just r' -> Just <$> storeUpdatedRemote ( return r' )
storeUpdatedRemote :: Annex Git . Repo -> Annex Git . Repo
storeUpdatedRemote = observe $ \ r' -> do
l <- Annex . getGitRemotes
let rs = exchange l r'
Annex . changeState $ \ s -> s { Annex . gitremotes = Just rs }
where
exchange [] _ = []
exchange ( old : ls ) new
| Git . remoteName old == Git . remoteName new =
new : exchange ls new
| otherwise =
old : exchange ls new
2014-08-06 17:45:19 +00:00
{- Checks if a given remote has the content for a key in its annex. -}
use P2P protocol for checkpresent, retrieve, and store
Note that, due to not using rsync to transfer files to ssh remotes
any longer, permissions and other file metadata of annexed files
will no longer be preserved when copying them to ssh remotes.
Other remotes never supported preserving that information, so
this is not considered a regression. Added NEWS item about this.
Another significant side effect of this is that, even when rsync is run to
retrieve a file, its progress display will no longer be shown, and
instead the native git-annex progress display will appear. It would be
possible to use the rsync process display when rsync is used (old
git-annex-shell and also retrieval from a local repository), but it
would have complicated the code unncessarily, and been inconsistent
behavior.
(I'd been thinking for a while about eliminating the rsync progress
display, since it's got some annoying verbosities, including display of
the key and the "(xfr#1, to-chk=0/1)" bit and was already somewhat
inconsistent.)
retrieveKeyFileCheap still uses rsync, since that ensures that it gets
the actual file content from the remote. Using the P2P protocol would
use the local content, as long as the local and remote size are the
same.
This commit was sponsored by John Pellman on Patreon.
2018-03-09 16:57:32 +00:00
inAnnex :: Remote -> State -> Key -> Annex Bool
2018-06-04 18:31:55 +00:00
inAnnex rmt st key = do
repo <- getRepo rmt
inAnnex' repo rmt st key
inAnnex' :: Git . Repo -> Remote -> State -> Key -> Annex Bool
2020-04-17 21:09:29 +00:00
inAnnex' repo rmt st @ ( State connpool duc _ _ _ ) key
2018-06-04 18:31:55 +00:00
| Git . repoIsHttp repo = checkhttp
| Git . repoIsUrl repo = checkremote
2011-11-09 22:33:15 +00:00
| otherwise = checklocal
2012-10-29 01:27:15 +00:00
where
2014-02-25 01:29:37 +00:00
checkhttp = do
removal of the rest of remoteGitConfig
In keyUrls, the GitConfig is used only by annexLocations
to support configured Differences. Since such configurations affect all
clones of a repository, the local repo's GitConfig must have the same
information as the remote's GitConfig would have. So, used getGitConfig
to get the local GitConfig, which is cached and so available cheaply.
That actually fixed a bug noone had ever noticed: keyUrls is
used for remotes accessed over http. The full git config of such a
remote is normally not available, so the remoteGitConfig that keyUrls
used would not have the necessary information in it.
In copyFromRemoteCheap', it uses gitAnnexLocation,
which does need the GitConfig of the remote repo itself in order to
check if it's crippled, supports symlinks, etc. So, made the
State include that GitConfig, cached. The use of gitAnnexLocation is
within a (not $ Git.repoIsUrl repo) guard, so it's local, and so
its git config will always be read and available.
(Note that gitAnnexLocation in turn calls annexLocations, so the
Differences config it uses in this case comes from the remote repo's
GitConfig and not from the local repo's GitConfig. As explained above
this is ok since they must have the same value.)
Not very happy with this mess of different GitConfigs not type-safe and
some read only sometimes etc. Very hairy. Think I got it this change
right. Test suite passes..
This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
gc <- Annex . getGitConfig
2022-08-08 15:57:24 +00:00
Url . withUrlOptionsPromptingCreds $ \ uo ->
anyM ( \ u -> Url . checkBoth u ( fromKey keySize key ) uo )
( keyUrls gc repo rmt key )
remove git-annex-shell compat code
* Removed support for accessing git remotes that use versions of
git-annex older than 6.20180312.
* git-annex-shell: Removed several commands that were only needed to
support git-annex versions older than 6.20180312.
(lockcontent, recvkey, sendkey, transferinfo, commit)
The P2P protocol was added in that version, and used ever since, so
this code was only needed for interop with older versions.
"git-annex-shell commit" is used by newer git-annex versions, though
unnecessarily so, because the p2pstdio command makes a single commit at
shutdown. Luckily, it was run with stderr and stdout sent to /dev/null,
and non-zero exit status or other exceptions are caught and ignored. So,
that was able to be removed from git-annex-shell too.
git-annex-shell inannex, recvkey, sendkey, and dropkey are still used by
gcrypt special remotes accessed over ssh, so those had to be kept.
It would probably be possible to convert that to using the P2P protocol,
but it would be another multi-year transition.
Some git-annex-shell fields were able to be removed. I hoped to remove
all of them, and the very concept of them, but unfortunately autoinit
is used by git-annex sync, and gcrypt uses remoteuuid.
The main win here is really in Remote.Git, removing piles of hairy fallback
code.
Sponsored-by: Luke Shumaker
2021-10-11 19:35:54 +00:00
checkremote = P2PHelper . checkpresent ( Ssh . runProto rmt connpool ( cantCheck rmt ) ) key
2018-01-10 18:21:18 +00:00
checklocal = ifM duc
2018-06-04 18:31:55 +00:00
( guardUsable repo ( cantCheck repo ) $
maybe ( cantCheck repo ) return
2020-04-17 21:09:29 +00:00
=<< onLocalFast st ( Annex . Content . inAnnexSafe key )
2018-06-04 18:31:55 +00:00
, cantCheck repo
2018-01-10 18:21:18 +00:00
)
2011-08-17 01:04:23 +00:00
removal of the rest of remoteGitConfig
In keyUrls, the GitConfig is used only by annexLocations
to support configured Differences. Since such configurations affect all
clones of a repository, the local repo's GitConfig must have the same
information as the remote's GitConfig would have. So, used getGitConfig
to get the local GitConfig, which is cached and so available cheaply.
That actually fixed a bug noone had ever noticed: keyUrls is
used for remotes accessed over http. The full git config of such a
remote is normally not available, so the remoteGitConfig that keyUrls
used would not have the necessary information in it.
In copyFromRemoteCheap', it uses gitAnnexLocation,
which does need the GitConfig of the remote repo itself in order to
check if it's crippled, supports symlinks, etc. So, made the
State include that GitConfig, cached. The use of gitAnnexLocation is
within a (not $ Git.repoIsUrl repo) guard, so it's local, and so
its git config will always be read and available.
(Note that gitAnnexLocation in turn calls annexLocations, so the
Differences config it uses in this case comes from the remote repo's
GitConfig and not from the local repo's GitConfig. As explained above
this is ok since they must have the same value.)
Not very happy with this mess of different GitConfigs not type-safe and
some read only sometimes etc. Very hairy. Think I got it this change
right. Test suite passes..
This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
keyUrls :: GitConfig -> Git . Repo -> Remote -> Key -> [ String ]
keyUrls gc repo r key = map tourl locs'
2012-10-29 01:27:15 +00:00
where
2018-06-04 18:31:55 +00:00
tourl l = Git . repoLocation repo ++ " / " ++ l
2014-01-26 17:03:25 +00:00
-- If the remote is known to not be bare, try the hash locations
-- used for non-bare repos first, as an optimisation.
locs
2021-07-16 18:16:05 +00:00
| remoteAnnexBare remoteconfig == Just False = annexLocationsNonBare gc key
| otherwise = annexLocationsBare gc key
2013-08-02 16:27:32 +00:00
# ifndef mingw32_HOST_OS
2019-12-11 18:12:22 +00:00
locs' = map fromRawFilePath locs
2013-07-07 17:35:06 +00:00
# else
2019-12-11 18:12:22 +00:00
locs' = map ( replace " \ \ " " / " . fromRawFilePath ) locs
2013-07-07 17:35:06 +00:00
# endif
2015-01-28 20:51:40 +00:00
remoteconfig = gitconfig r
2011-08-17 01:04:23 +00:00
2020-05-14 18:08:09 +00:00
dropKey :: Remote -> State -> Key -> Annex ()
2018-06-04 18:31:55 +00:00
dropKey r st key = do
repo <- getRepo r
2020-05-14 18:08:09 +00:00
dropKey' repo r st key
2018-06-04 18:31:55 +00:00
2020-05-14 18:08:09 +00:00
dropKey' :: Git . Repo -> Remote -> State -> Key -> Annex ()
2020-04-17 21:09:29 +00:00
dropKey' repo r st @ ( State connpool duc _ _ _ ) key
2018-06-04 18:31:55 +00:00
| not $ Git . repoIsUrl repo = ifM duc
2020-05-14 18:08:09 +00:00
( guardUsable repo ( giveup " cannot access remote " ) $
2020-04-17 21:09:29 +00:00
commitOnCleanup repo r st $ onLocalFast st $ do
2014-08-08 23:18:08 +00:00
whenM ( Annex . Content . inAnnex key ) $ do
2020-07-25 15:54:34 +00:00
let cleanup = logStatus key InfoMissing
Annex . Content . lockContentForRemoval key cleanup $ \ lock -> do
2016-12-09 16:56:38 +00:00
Annex . Content . removeAnnex lock
2020-07-25 15:54:34 +00:00
cleanup
2020-05-14 18:08:09 +00:00
, giveup " remote does not have expected annex.uuid value "
2018-01-10 18:21:18 +00:00
)
2020-05-14 18:08:09 +00:00
| Git . repoIsHttp repo = giveup " dropping from http remote not supported "
2024-06-23 13:28:18 +00:00
| otherwise = P2PHelper . remove ( uuid r )
2024-06-28 18:07:23 +00:00
( Ssh . runProto r connpool ( return ( Right False , Nothing ) ) ) key
2011-03-27 19:56:43 +00:00
use P2P protocol for checkpresent, retrieve, and store
Note that, due to not using rsync to transfer files to ssh remotes
any longer, permissions and other file metadata of annexed files
will no longer be preserved when copying them to ssh remotes.
Other remotes never supported preserving that information, so
this is not considered a regression. Added NEWS item about this.
Another significant side effect of this is that, even when rsync is run to
retrieve a file, its progress display will no longer be shown, and
instead the native git-annex progress display will appear. It would be
possible to use the rsync process display when rsync is used (old
git-annex-shell and also retrieval from a local repository), but it
would have complicated the code unncessarily, and been inconsistent
behavior.
(I'd been thinking for a while about eliminating the rsync progress
display, since it's got some annoying verbosities, including display of
the key and the "(xfr#1, to-chk=0/1)" bit and was already somewhat
inconsistent.)
retrieveKeyFileCheap still uses rsync, since that ensures that it gets
the actual file content from the remote. Using the P2P protocol would
use the local content, as long as the local and remote size are the
same.
This commit was sponsored by John Pellman on Patreon.
2018-03-09 16:57:32 +00:00
lockKey :: Remote -> State -> Key -> ( VerifiedCopy -> Annex r ) -> Annex r
2018-06-04 18:31:55 +00:00
lockKey r st key callback = do
repo <- getRepo r
lockKey' repo r st key callback
lockKey' :: Git . Repo -> Remote -> State -> Key -> ( VerifiedCopy -> Annex r ) -> Annex r
2020-04-17 21:09:29 +00:00
lockKey' repo r st @ ( State connpool duc _ _ _ ) key callback
2018-06-04 18:31:55 +00:00
| not $ Git . repoIsUrl repo = ifM duc
( guardUsable repo failedlock $ do
2015-10-09 17:35:28 +00:00
inorigrepo <- Annex . makeRunner
-- Lock content from perspective of remote,
-- and then run the callback in the original
-- annex monad, not the remote's.
2020-04-17 21:09:29 +00:00
onLocalFast st $
add content retention files
This allows lockContentShared to lock content for eg, 10 minutes and
if the process then gets terminated before it can unlock, the content
will remain locked for that amount of time.
The Windows implementation is not yet tested.
In P2P.Annex, a duration of 10 minutes is used. This way, when p2pstdio
or remotedaemon is serving the P2P protocol, and is asked to
LOCKCONTENT, and that process gets killed, the content will not be
subject to deletion. This is not a perfect solution to
doc/todo/P2P_locking_connection_drop_safety.mdwn yet, but it gets most
of the way there, without needing any P2P protocol changes.
This is only done in v10 and higher repositories (or on Windows). It
might be possible to backport it to v8 or earlier, but it would
complicate locking even further, and without a separate lock file, might
be hard. I think that by the time this fix reaches a given user, they
will probably have been running git-annex 10.x long enough that their v8
repositories will have upgraded to v10 after the 1 year wait. And it's
not as if git-annex hasn't already been subject to this problem (though
I have not heard of any data loss caused by it) for 6 years already, so
waiting another fraction of a year on top of however long it takes this
fix to reach users is unlikely to be a problem.
2024-07-03 18:44:38 +00:00
Annex . Content . lockContentShared key Nothing $
make sure that lockContentShared is always paired with an inAnnex check
lockContentShared had a screwy caveat that it didn't verify that the content
was present when locking it, but in the most common case, eg indirect mode,
it failed to lock when the content is not present.
That led to a few callers forgetting to check inAnnex when using it,
but the potential data loss was unlikely to be noticed because it only
affected direct mode I think.
Fix data loss bug when the local repository uses direct mode, and a
locally modified file is dropped from a remote repsitory. The bug
caused the modified file to be counted as a copy of the original file.
(This is not a severe bug because in such a situation, dropping
from the remote and then modifying the file is allowed and has the same
end result.)
And, in content locking over tor, when the remote repository is
in direct mode, it neglected to check that the content was actually
present when locking it. This could cause git annex drop to remove
the only copy of a file when it thought the tor remote had a copy.
So, make lockContentShared do its own inAnnex check. This could perhaps
be optimised for direct mode, to avoid the check then, since locking
the content necessarily verifies it exists there, but I have not bothered
with that.
This commit was sponsored by Jeff Goeke-Smith on Patreon.
2018-03-07 18:13:02 +00:00
liftIO . inorigrepo . callback
2018-01-10 18:21:18 +00:00
, failedlock
)
2018-06-04 18:31:55 +00:00
| Git . repoIsSsh repo = do
2015-10-09 21:21:02 +00:00
showLocking r
2024-06-12 14:10:11 +00:00
let withconn = Ssh . withP2PShellConnection r connpool failedlock
2018-03-09 17:48:10 +00:00
P2PHelper . lock withconn Ssh . runProtoConn ( uuid r ) key callback
2018-03-09 17:42:55 +00:00
| otherwise = failedlock
where
2016-11-16 01:29:54 +00:00
failedlock = giveup " can't lock content "
2015-10-09 17:07:03 +00:00
2011-03-27 19:56:43 +00:00
{- Tries to copy a key's content from a remote's annex to a file. -}
2021-08-17 16:41:36 +00:00
copyFromRemote :: Remote -> State -> Key -> AssociatedFile -> FilePath -> MeterUpdate -> VerifyConfig -> Annex Verification
remove git-annex-shell compat code
* Removed support for accessing git remotes that use versions of
git-annex older than 6.20180312.
* git-annex-shell: Removed several commands that were only needed to
support git-annex versions older than 6.20180312.
(lockcontent, recvkey, sendkey, transferinfo, commit)
The P2P protocol was added in that version, and used ever since, so
this code was only needed for interop with older versions.
"git-annex-shell commit" is used by newer git-annex versions, though
unnecessarily so, because the p2pstdio command makes a single commit at
shutdown. Luckily, it was run with stderr and stdout sent to /dev/null,
and non-zero exit status or other exceptions are caught and ignored. So,
that was able to be removed from git-annex-shell too.
git-annex-shell inannex, recvkey, sendkey, and dropkey are still used by
gcrypt special remotes accessed over ssh, so those had to be kept.
It would probably be possible to convert that to using the P2P protocol,
but it would be another multi-year transition.
Some git-annex-shell fields were able to be removed. I hoped to remove
all of them, and the very concept of them, but unfortunately autoinit
is used by git-annex sync, and gcrypt uses remoteuuid.
The main win here is really in Remote.Git, removing piles of hairy fallback
code.
Sponsored-by: Luke Shumaker
2021-10-11 19:35:54 +00:00
copyFromRemote r st key file dest meterupdate vc = do
2018-06-04 18:31:55 +00:00
repo <- getRepo r
remove git-annex-shell compat code
* Removed support for accessing git remotes that use versions of
git-annex older than 6.20180312.
* git-annex-shell: Removed several commands that were only needed to
support git-annex versions older than 6.20180312.
(lockcontent, recvkey, sendkey, transferinfo, commit)
The P2P protocol was added in that version, and used ever since, so
this code was only needed for interop with older versions.
"git-annex-shell commit" is used by newer git-annex versions, though
unnecessarily so, because the p2pstdio command makes a single commit at
shutdown. Luckily, it was run with stderr and stdout sent to /dev/null,
and non-zero exit status or other exceptions are caught and ignored. So,
that was able to be removed from git-annex-shell too.
git-annex-shell inannex, recvkey, sendkey, and dropkey are still used by
gcrypt special remotes accessed over ssh, so those had to be kept.
It would probably be possible to convert that to using the P2P protocol,
but it would be another multi-year transition.
Some git-annex-shell fields were able to be removed. I hoped to remove
all of them, and the very concept of them, but unfortunately autoinit
is used by git-annex sync, and gcrypt uses remoteuuid.
The main win here is really in Remote.Git, removing piles of hairy fallback
code.
Sponsored-by: Luke Shumaker
2021-10-11 19:35:54 +00:00
copyFromRemote'' repo r st key file dest meterupdate vc
2018-06-04 18:31:55 +00:00
remove git-annex-shell compat code
* Removed support for accessing git remotes that use versions of
git-annex older than 6.20180312.
* git-annex-shell: Removed several commands that were only needed to
support git-annex versions older than 6.20180312.
(lockcontent, recvkey, sendkey, transferinfo, commit)
The P2P protocol was added in that version, and used ever since, so
this code was only needed for interop with older versions.
"git-annex-shell commit" is used by newer git-annex versions, though
unnecessarily so, because the p2pstdio command makes a single commit at
shutdown. Luckily, it was run with stderr and stdout sent to /dev/null,
and non-zero exit status or other exceptions are caught and ignored. So,
that was able to be removed from git-annex-shell too.
git-annex-shell inannex, recvkey, sendkey, and dropkey are still used by
gcrypt special remotes accessed over ssh, so those had to be kept.
It would probably be possible to convert that to using the P2P protocol,
but it would be another multi-year transition.
Some git-annex-shell fields were able to be removed. I hoped to remove
all of them, and the very concept of them, but unfortunately autoinit
is used by git-annex sync, and gcrypt uses remoteuuid.
The main win here is really in Remote.Git, removing piles of hairy fallback
code.
Sponsored-by: Luke Shumaker
2021-10-11 19:35:54 +00:00
copyFromRemote'' :: Git . Repo -> Remote -> State -> Key -> AssociatedFile -> FilePath -> MeterUpdate -> VerifyConfig -> Annex Verification
copyFromRemote'' repo r st @ ( State connpool _ _ _ _ ) key file dest meterupdate vc
2022-05-09 17:18:47 +00:00
| Git . repoIsHttp repo = verifyKeyContentIncrementally vc key $ \ iv -> do
removal of the rest of remoteGitConfig
In keyUrls, the GitConfig is used only by annexLocations
to support configured Differences. Since such configurations affect all
clones of a repository, the local repo's GitConfig must have the same
information as the remote's GitConfig would have. So, used getGitConfig
to get the local GitConfig, which is cached and so available cheaply.
That actually fixed a bug noone had ever noticed: keyUrls is
used for remotes accessed over http. The full git config of such a
remote is normally not available, so the remoteGitConfig that keyUrls
used would not have the necessary information in it.
In copyFromRemoteCheap', it uses gitAnnexLocation,
which does need the GitConfig of the remote repo itself in order to
check if it's crippled, supports symlinks, etc. So, made the
State include that GitConfig, cached. The use of gitAnnexLocation is
within a (not $ Git.repoIsUrl repo) guard, so it's local, and so
its git config will always be read and available.
(Note that gitAnnexLocation in turn calls annexLocations, so the
Differences config it uses in this case comes from the remote repo's
GitConfig and not from the local repo's GitConfig. As explained above
this is ok since they must have the same value.)
Not very happy with this mess of different GitConfigs not type-safe and
some read only sometimes etc. Very hairy. Think I got it this change
right. Test suite passes..
This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
gc <- Annex . getGitConfig
2020-05-13 21:05:56 +00:00
ok <- Url . withUrlOptionsPromptingCreds $
2021-09-01 19:28:22 +00:00
Annex . Content . downloadUrl False key meterupdate iv ( keyUrls gc repo r key ) dest
2020-05-13 21:05:56 +00:00
unless ok $
giveup " failed to download content "
| not $ Git . repoIsUrl repo = guardUsable repo ( giveup " cannot access remote " ) $ do
2012-07-01 20:59:54 +00:00
u <- getUUID
2015-09-14 16:13:38 +00:00
hardlink <- wantHardLink
2024-01-19 19:14:26 +00:00
let bwlimit = remoteAnnexBwLimitDownload ( gitconfig r )
<|> remoteAnnexBwLimit ( gitconfig r )
2012-07-01 20:59:54 +00:00
-- run copy from perspective of remote
2024-07-01 14:42:27 +00:00
onLocalFast st $ Annex . Content . prepSendAnnex' key Nothing >>= \ case
disk free checking for unsized keys
Improve disk free space checking when transferring unsized keys to
local git remotes. Since the size of the object file is known, can
check that instead.
Getting unsized keys from local git remotes does not check the actual
object size. It would be harder to handle that direction because the size
check is run locally, before anything involving the remote is done. So it
doesn't know the size of the file on the remote.
Also, transferring unsized keys to other remotes, including ssh remotes and
p2p remotes don't do disk size checking for unsized keys. This would need a
change in protocol.
(It does seem like it would be possible to implement the same thing for
directory special remotes though.)
In some sense, it might be better to not ever do disk free checking for
unsized keys, than to do it only sometimes. A user might notice this
direction working and consider it a bug that the other direction does not.
On the other hand, disk reserve checking is not implemented for most
special remotes at all, and yet it is implemented for a few, which is also
inconsistent, but best effort. And so doing this best effort seems to make
some sense. Fundamentally, if the user wants the size to always be checked,
they should not use unsized keys.
Sponsored-by: Brock Spratlen on Patreon
2024-01-16 18:29:10 +00:00
Just ( object , _sz , check ) -> do
2021-06-25 17:04:17 +00:00
let checksuccess = check >>= \ case
Just err -> giveup err
Nothing -> return True
2021-04-14 18:06:43 +00:00
copier <- mkFileCopier hardlink st
2020-05-13 21:05:56 +00:00
( ok , v ) <- runTransfer ( Transfer Download u ( fromKey id key ) )
2023-03-27 19:10:46 +00:00
Nothing file Nothing stdRetry $ \ p ->
bwlimit
Added annex.bwlimit and remote.name.annex-bwlimit config that works for git
remotes and many but not all special remotes.
This nearly works, at least for a git remote on the same disk. With it set
to 100kb/1s, the meter displays an actual bandwidth of 128 kb/s, with
occasional spikes to 160 kb/s. So it needs to delay just a bit longer...
I'm unsure why.
However, at the beginning a lot of data flows before it determines the
right bandwidth limit. A granularity of less than 1s would probably improve
that.
And, I don't know yet if it makes sense to have it be 100ks/1s rather than
100kb/s. Is there a situation where the user would want a larger
granularity? Does granulatity need to be configurable at all? I only used that
format for the config really in order to reuse an existing parser.
This can't support for external special remotes, or for ones that
themselves shell out to an external command. (Well, it could, but it
would involve pausing and resuming the child process tree, which seems
very hard to implement and very strange besides.) There could also be some
built-in special remotes that it still doesn't work for, due to them not
having a progress meter whose displays blocks the bandwidth using thread.
But I don't think there are actually any that run a separate thread for
downloads than the thread that displays the progress meter.
Sponsored-by: Graham Spencer on Patreon
2021-09-21 20:58:02 +00:00
metered ( Just ( combineMeterUpdate p meterupdate ) ) key bwlimit $ \ _ p' ->
2021-08-17 16:41:36 +00:00
copier object dest key p' checksuccess vc
2020-05-13 21:05:56 +00:00
if ok
then return v
else giveup " failed to retrieve content from remote "
Nothing -> giveup " content is not present in remote "
remove git-annex-shell compat code
* Removed support for accessing git remotes that use versions of
git-annex older than 6.20180312.
* git-annex-shell: Removed several commands that were only needed to
support git-annex versions older than 6.20180312.
(lockcontent, recvkey, sendkey, transferinfo, commit)
The P2P protocol was added in that version, and used ever since, so
this code was only needed for interop with older versions.
"git-annex-shell commit" is used by newer git-annex versions, though
unnecessarily so, because the p2pstdio command makes a single commit at
shutdown. Luckily, it was run with stderr and stdout sent to /dev/null,
and non-zero exit status or other exceptions are caught and ignored. So,
that was able to be removed from git-annex-shell too.
git-annex-shell inannex, recvkey, sendkey, and dropkey are still used by
gcrypt special remotes accessed over ssh, so those had to be kept.
It would probably be possible to convert that to using the P2P protocol,
but it would be another multi-year transition.
Some git-annex-shell fields were able to be removed. I hoped to remove
all of them, and the very concept of them, but unfortunately autoinit
is used by git-annex sync, and gcrypt uses remoteuuid.
The main win here is really in Remote.Git, removing piles of hairy fallback
code.
Sponsored-by: Luke Shumaker
2021-10-11 19:35:54 +00:00
| Git . repoIsSsh repo =
P2PHelper . retrieve
bwlimit
Added annex.bwlimit and remote.name.annex-bwlimit config that works for git
remotes and many but not all special remotes.
This nearly works, at least for a git remote on the same disk. With it set
to 100kb/1s, the meter displays an actual bandwidth of 128 kb/s, with
occasional spikes to 160 kb/s. So it needs to delay just a bit longer...
I'm unsure why.
However, at the beginning a lot of data flows before it determines the
right bandwidth limit. A granularity of less than 1s would probably improve
that.
And, I don't know yet if it makes sense to have it be 100ks/1s rather than
100kb/s. Is there a situation where the user would want a larger
granularity? Does granulatity need to be configurable at all? I only used that
format for the config really in order to reuse an existing parser.
This can't support for external special remotes, or for ones that
themselves shell out to an external command. (Well, it could, but it
would involve pausing and resuming the child process tree, which seems
very hard to implement and very strange besides.) There could also be some
built-in special remotes that it still doesn't work for, due to them not
having a progress meter whose displays blocks the bandwidth using thread.
But I don't think there are actually any that run a separate thread for
downloads than the thread that displays the progress meter.
Sponsored-by: Graham Spencer on Patreon
2021-09-21 20:58:02 +00:00
( gitconfig r )
remove git-annex-shell compat code
* Removed support for accessing git remotes that use versions of
git-annex older than 6.20180312.
* git-annex-shell: Removed several commands that were only needed to
support git-annex versions older than 6.20180312.
(lockcontent, recvkey, sendkey, transferinfo, commit)
The P2P protocol was added in that version, and used ever since, so
this code was only needed for interop with older versions.
"git-annex-shell commit" is used by newer git-annex versions, though
unnecessarily so, because the p2pstdio command makes a single commit at
shutdown. Luckily, it was run with stderr and stdout sent to /dev/null,
and non-zero exit status or other exceptions are caught and ignored. So,
that was able to be removed from git-annex-shell too.
git-annex-shell inannex, recvkey, sendkey, and dropkey are still used by
gcrypt special remotes accessed over ssh, so those had to be kept.
It would probably be possible to convert that to using the P2P protocol,
but it would be another multi-year transition.
Some git-annex-shell fields were able to be removed. I hoped to remove
all of them, and the very concept of them, but unfortunately autoinit
is used by git-annex sync, and gcrypt uses remoteuuid.
The main win here is really in Remote.Git, removing piles of hairy fallback
code.
Sponsored-by: Luke Shumaker
2021-10-11 19:35:54 +00:00
( Ssh . runProto r connpool ( return ( False , UnVerified ) ) )
2021-08-17 16:41:36 +00:00
key file dest meterupdate vc
2020-05-13 21:05:56 +00:00
| otherwise = giveup " copying from non-ssh, non-http remote not supported "
2011-08-17 01:04:23 +00:00
2021-02-10 17:29:12 +00:00
copyFromRemoteCheap :: State -> Git . Repo -> Maybe ( Key -> AssociatedFile -> FilePath -> Annex () )
2013-08-02 16:27:32 +00:00
# ifndef mingw32_HOST_OS
2021-02-10 17:29:12 +00:00
copyFromRemoteCheap st repo
2020-05-13 21:05:56 +00:00
| not $ Git . repoIsUrl repo = Just $ \ key _af file -> guardUsable repo ( giveup " cannot access remote " ) $ do
removal of the rest of remoteGitConfig
In keyUrls, the GitConfig is used only by annexLocations
to support configured Differences. Since such configurations affect all
clones of a repository, the local repo's GitConfig must have the same
information as the remote's GitConfig would have. So, used getGitConfig
to get the local GitConfig, which is cached and so available cheaply.
That actually fixed a bug noone had ever noticed: keyUrls is
used for remotes accessed over http. The full git config of such a
remote is normally not available, so the remoteGitConfig that keyUrls
used would not have the necessary information in it.
In copyFromRemoteCheap', it uses gitAnnexLocation,
which does need the GitConfig of the remote repo itself in order to
check if it's crippled, supports symlinks, etc. So, made the
State include that GitConfig, cached. The use of gitAnnexLocation is
within a (not $ Git.repoIsUrl repo) guard, so it's local, and so
its git config will always be read and available.
(Note that gitAnnexLocation in turn calls annexLocations, so the
Differences config it uses in this case comes from the remote repo's
GitConfig and not from the local repo's GitConfig. As explained above
this is ok since they must have the same value.)
Not very happy with this mess of different GitConfigs not type-safe and
some read only sometimes etc. Very hairy. Think I got it this change
right. Test suite passes..
This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
gc <- getGitConfigFromState st
loc <- liftIO $ gitAnnexLocation key repo gc
2019-12-11 18:12:22 +00:00
liftIO $ ifM ( R . doesPathExist loc )
2015-04-18 17:36:12 +00:00
( do
2020-10-30 19:55:59 +00:00
absloc <- absPath loc
R . createSymbolicLink absloc ( toRawFilePath file )
2020-05-13 21:05:56 +00:00
, giveup " remote does not contain key "
2015-04-18 17:36:12 +00:00
)
2020-05-13 21:05:56 +00:00
| otherwise = Nothing
2013-08-04 17:12:18 +00:00
# else
2021-02-15 17:35:01 +00:00
copyFromRemoteCheap _ _ = Nothing
2013-08-04 17:12:18 +00:00
# endif
2012-01-20 17:23:11 +00:00
2011-03-27 19:56:43 +00:00
{- Tries to copy a key's content to a remote's annex. -}
2024-07-01 14:42:27 +00:00
copyToRemote :: Remote -> State -> Key -> AssociatedFile -> Maybe FilePath -> MeterUpdate -> Annex ()
copyToRemote r st key af o meterupdate = do
2018-06-04 18:31:55 +00:00
repo <- getRepo r
2024-07-01 14:42:27 +00:00
copyToRemote' repo r st key af o meterupdate
2018-06-04 18:31:55 +00:00
2024-07-01 14:42:27 +00:00
copyToRemote' :: Git . Repo -> Remote -> State -> Key -> AssociatedFile -> Maybe FilePath -> MeterUpdate -> Annex ()
copyToRemote' repo r st @ ( State connpool duc _ _ _ ) key af o meterupdate
2018-06-04 18:31:55 +00:00
| not $ Git . repoIsUrl repo = ifM duc
2020-05-13 18:03:00 +00:00
( guardUsable repo ( giveup " cannot access remote " ) $ commitOnCleanup repo r st $
2024-07-01 14:42:27 +00:00
copylocal =<< Annex . Content . prepSendAnnex' key o
2020-05-13 18:03:00 +00:00
, giveup " remote does not have expected annex.uuid value "
2018-01-10 18:21:18 +00:00
)
remove git-annex-shell compat code
* Removed support for accessing git remotes that use versions of
git-annex older than 6.20180312.
* git-annex-shell: Removed several commands that were only needed to
support git-annex versions older than 6.20180312.
(lockcontent, recvkey, sendkey, transferinfo, commit)
The P2P protocol was added in that version, and used ever since, so
this code was only needed for interop with older versions.
"git-annex-shell commit" is used by newer git-annex versions, though
unnecessarily so, because the p2pstdio command makes a single commit at
shutdown. Luckily, it was run with stderr and stdout sent to /dev/null,
and non-zero exit status or other exceptions are caught and ignored. So,
that was able to be removed from git-annex-shell too.
git-annex-shell inannex, recvkey, sendkey, and dropkey are still used by
gcrypt special remotes accessed over ssh, so those had to be kept.
It would probably be possible to convert that to using the P2P protocol,
but it would be another multi-year transition.
Some git-annex-shell fields were able to be removed. I hoped to remove
all of them, and the very concept of them, but unfortunately autoinit
is used by git-annex sync, and gcrypt uses remoteuuid.
The main win here is really in Remote.Git, removing piles of hairy fallback
code.
Sponsored-by: Luke Shumaker
2021-10-11 19:35:54 +00:00
| Git . repoIsSsh repo =
2024-06-23 12:21:11 +00:00
P2PHelper . store ( uuid r ) ( gitconfig r )
2024-06-18 16:07:01 +00:00
( Ssh . runProto r connpool ( return Nothing ) )
2024-07-01 14:42:27 +00:00
key af o meterupdate
use P2P protocol for checkpresent, retrieve, and store
Note that, due to not using rsync to transfer files to ssh remotes
any longer, permissions and other file metadata of annexed files
will no longer be preserved when copying them to ssh remotes.
Other remotes never supported preserving that information, so
this is not considered a regression. Added NEWS item about this.
Another significant side effect of this is that, even when rsync is run to
retrieve a file, its progress display will no longer be shown, and
instead the native git-annex progress display will appear. It would be
possible to use the rsync process display when rsync is used (old
git-annex-shell and also retrieval from a local repository), but it
would have complicated the code unncessarily, and been inconsistent
behavior.
(I'd been thinking for a while about eliminating the rsync progress
display, since it's got some annoying verbosities, including display of
the key and the "(xfr#1, to-chk=0/1)" bit and was already somewhat
inconsistent.)
retrieveKeyFileCheap still uses rsync, since that ensures that it gets
the actual file content from the remote. Using the P2P protocol would
use the local content, as long as the local and remote size are the
same.
This commit was sponsored by John Pellman on Patreon.
2018-03-09 16:57:32 +00:00
2020-05-13 18:03:00 +00:00
| otherwise = giveup " copying to non-ssh repo not supported "
2012-12-08 21:03:39 +00:00
where
2020-05-13 18:03:00 +00:00
copylocal Nothing = giveup " content not available "
disk free checking for unsized keys
Improve disk free space checking when transferring unsized keys to
local git remotes. Since the size of the object file is known, can
check that instead.
Getting unsized keys from local git remotes does not check the actual
object size. It would be harder to handle that direction because the size
check is run locally, before anything involving the remote is done. So it
doesn't know the size of the file on the remote.
Also, transferring unsized keys to other remotes, including ssh remotes and
p2p remotes don't do disk size checking for unsized keys. This would need a
change in protocol.
(It does seem like it would be possible to implement the same thing for
directory special remotes though.)
In some sense, it might be better to not ever do disk free checking for
unsized keys, than to do it only sometimes. A user might notice this
direction working and consider it a bug that the other direction does not.
On the other hand, disk reserve checking is not implemented for most
special remotes at all, and yet it is implemented for a few, which is also
inconsistent, but best effort. And so doing this best effort seems to make
some sense. Fundamentally, if the user wants the size to always be checked,
they should not use unsized keys.
Sponsored-by: Brock Spratlen on Patreon
2024-01-16 18:29:10 +00:00
copylocal ( Just ( object , sz , check ) ) = do
2021-06-25 17:04:17 +00:00
-- The check action is going to be run in
deal with unlocked files when calling rsyncParamsRemote
In copyFromRemote, it used to check isDirect, but that was not needed;
the remote is sending the file, so it doesn't matter if the local,
receiving repository is in direct mode or not. And, since the content is not
present, yet, it's certianly not unlocked. Note that, the remote may indeed
be sending an unlocked file, but sendkey uses sendAnnex, which will detect
if the file is modified before or during transfer, and will exit nonzero,
aborting the upload. So, the receiver doesn't need any checks.
In copyToRemote, it forces recvkey to verify content whenever it's being
sent from a v6 repository. recvkey is almost always going to verify content
anyway, unless annex.verify is not set. So, this doesn't make it any more
expensive, except for in that unusual configuration. The alternative would
be to change the recvkey interface, so that the sender checks afterwards if
what it was sending changed, and the receiver then throws out the bad
transfer. That would be less expensive for the reciever, as it would not
need to do a checksum verification. But, it would mean another network
round trip, and since rsync closes the connection, it would need to open
another ssh connection to do this. Even with connction caching, that would
add latency to uploads. It would also complicate the interface, especially
because an older git-annex-shell would not have the new interface
available. For these reasons, I prefer punting on that at this time, and
instead someone might set annex.verify=false and be unhappy that it still
verifies..
(One other gotcha not dealt with is that a v5 repo could be upgraded to v6
while an upload is in progress, and a file unlocked and modified.)
(Also, I double-checked Remote.GCrypt's calls to rsyncParamsRemote, and
they're fine. When a file is being uploaded to gcrypt, or any other special
repository, it is mediated by sendAnnex, so changes will be detected at
that level and the special remote implementation doesn't need to worry
about them.)
2015-12-26 18:05:07 +00:00
-- the remote's Annex, but it needs access to the local
2013-03-12 20:41:54 +00:00
-- Annex monad's state.
2021-06-25 17:04:17 +00:00
checkio <- Annex . withCurrentState check
2012-07-01 20:59:54 +00:00
u <- getUUID
2015-09-14 16:13:38 +00:00
hardlink <- wantHardLink
2024-01-19 19:14:26 +00:00
let bwlimit = remoteAnnexBwLimitUpload ( gitconfig r )
<|> remoteAnnexBwLimit ( gitconfig r )
2011-03-27 19:56:43 +00:00
-- run copy from perspective of remote
2020-05-13 18:03:00 +00:00
res <- onLocalFast st $ ifM ( Annex . Content . inAnnex key )
2013-03-10 21:54:27 +00:00
( return True
2024-07-01 14:42:27 +00:00
, runTransfer ( Transfer Download u ( fromKey id key ) ) Nothing af Nothing stdRetry $ \ p -> do
2021-08-17 16:41:36 +00:00
let verify = RemoteVerify r
2021-04-14 18:06:43 +00:00
copier <- mkFileCopier hardlink st
2018-06-21 17:34:11 +00:00
let rsp = RetrievalAllKeysSecure
2021-06-25 17:04:17 +00:00
let checksuccess = liftIO checkio >>= \ case
Just err -> giveup err
Nothing -> return True
2024-07-01 14:42:27 +00:00
logStatusAfter key $ Annex . Content . getViaTmp rsp verify key af ( Just sz ) $ \ dest ->
bwlimit
Added annex.bwlimit and remote.name.annex-bwlimit config that works for git
remotes and many but not all special remotes.
This nearly works, at least for a git remote on the same disk. With it set
to 100kb/1s, the meter displays an actual bandwidth of 128 kb/s, with
occasional spikes to 160 kb/s. So it needs to delay just a bit longer...
I'm unsure why.
However, at the beginning a lot of data flows before it determines the
right bandwidth limit. A granularity of less than 1s would probably improve
that.
And, I don't know yet if it makes sense to have it be 100ks/1s rather than
100kb/s. Is there a situation where the user would want a larger
granularity? Does granulatity need to be configurable at all? I only used that
format for the config really in order to reuse an existing parser.
This can't support for external special remotes, or for ones that
themselves shell out to an external command. (Well, it could, but it
would involve pausing and resuming the child process tree, which seems
very hard to implement and very strange besides.) There could also be some
built-in special remotes that it still doesn't work for, due to them not
having a progress meter whose displays blocks the bandwidth using thread.
But I don't think there are actually any that run a separate thread for
downloads than the thread that displays the progress meter.
Sponsored-by: Graham Spencer on Patreon
2021-09-21 20:58:02 +00:00
metered ( Just ( combineMeterUpdate meterupdate p ) ) key bwlimit $ \ _ p' ->
2021-06-25 17:04:17 +00:00
copier object ( fromRawFilePath dest ) key p' checksuccess verify
2012-09-18 17:59:03 +00:00
)
2020-05-13 18:03:00 +00:00
unless res $
giveup " failed to send content to remote "
2011-03-27 19:56:43 +00:00
2013-10-11 20:03:18 +00:00
fsckOnRemote :: Git . Repo -> [ CommandParam ] -> Annex ( IO Bool )
fsckOnRemote r params
2021-10-11 18:55:57 +00:00
| Git . repoIsUrl r = return $ return False
2013-10-14 16:23:38 +00:00
| otherwise = return $ do
2015-02-28 21:23:13 +00:00
program <- programPath
2013-10-14 19:05:10 +00:00
r' <- Git . Config . read r
2014-06-10 23:20:14 +00:00
environ <- getEnvironment
let environ' = addEntries
2019-12-09 17:49:05 +00:00
[ ( " GIT_WORK_TREE " , fromRawFilePath $ Git . repoPath r' )
, ( " GIT_DIR " , fromRawFilePath $ Git . localGitDir r' )
2014-06-10 23:20:14 +00:00
] environ
2015-02-09 18:16:42 +00:00
batchCommandEnv program ( Param " fsck " : params ) ( Just environ' )
2013-10-11 20:03:18 +00:00
2013-10-27 19:38:59 +00:00
{- The passed repair action is run in the Annex monad of the remote. -}
repairRemote :: Git . Repo -> Annex Bool -> Annex ( IO Bool )
2014-03-06 21:12:50 +00:00
repairRemote r a = return $ do
2013-09-07 22:38:00 +00:00
s <- Annex . new r
Annex . eval s $ do
Annex . BranchState . disableUpdate
remove dead nodes when loading the cluster log
This is to avoid inserting a cluster uuid into the location log when
only dead nodes in the cluster contain the content of a key.
One reason why this is necessary is Remote.keyLocations, which excludes
dead repositories from the list. But there are probably many more.
Implementing this was challenging, because Logs.Location importing
Logs.Cluster which imports Logs.Trust which imports Remote.List resulted
in an import cycle through several other modules.
Resorted to making Logs.Location not import Logs.Cluster, and instead
it assumes that Annex.clusters gets populated when necessary before it's
called.
That's done in Annex.Startup, which is run by the git-annex command
(but not other commands) at early startup in initialized repos. Or,
is run after initialization.
Note that is Remote.Git, it is unable to import Annex.Startup, because
Remote.Git importing Logs.Cluster leads the the same import cycle.
So ensureInitialized is not passed annexStartup in there.
Other commands, like git-annex-shell currently don't run annexStartup
either.
So there are cases where Logs.Location will not see clusters. So it won't add
any cluster UUIDs when loading the log. That's ok, the only reason to do
that is to make display of where objects are located include clusters,
and to make commands like git-annex get --from treat keys as being located
in a cluster. git-annex-shell certainly does not do anything like that,
and I'm pretty sure Remote.Git (and callers to Remote.Git.onLocalRepo)
don't either.
2024-06-16 18:35:07 +00:00
ensureInitialized noop ( pure [] )
avoid flushing keys db queue after each Annex action
The flush was only done Annex.run' to make sure that the queue was flushed
before git-annex exits. But, doing it there means that as soon as one
change gets queued, it gets flushed soon after, which contributes to
excessive writes to the database, slowing git-annex down.
(This does not yet speed git-annex up, but it is a stepping stone to
doing so.)
Database queues do not autoflush when garbage collected, so have to
be flushed explicitly. I don't think it's possible to make them
autoflush (except perhaps if git-annex sqitched to using ResourceT..).
The comment in Database.Keys.closeDb used to be accurate, since the
automatic flushing did mean that all writes reached the database even
when closeDb was not called. But now, closeDb or flushDb needs to be
called before stopping using an Annex state. So, removed that comment.
In Remote.Git, change to using quiesce everywhere that it used to use
stopCoProcesses. This means that uses on onLocal in there are just as
slow as before. I considered only calling closeDb on the local git remotes
when git-annex exits. But, the reason that Remote.Git calls stopCoProcesses
in each onLocal is so as not to leave git processes running that have files
open on the remote repo, when it's on removable media. So, it seemed to make
sense to also closeDb after each one, since sqlite may also keep files
open. Although that has not seemed to cause problems with removable
media so far. It was also just easier to quiesce in each onLocal than
once at the end. This does likely leave performance on the floor, so
could be revisited.
In Annex.Content.saveState, there was no reason to close the db,
flushing it is enough.
The rest of the changes are from auditing for Annex.new, and making
sure that quiesce is called, after any action that might possibly need
it.
After that audit, I'm pretty sure that the change to Annex.run' is
safe. The only concern might be that this does let more changes get
queued for write to the db, and if git-annex is interrupted, those will be
lost. But interrupting git-annex can obviously already prevent it from
writing the most recent change to the db, so it must recover from such
lost data... right?
Sponsored-by: Dartmouth College's Datalad project
2022-10-12 17:50:46 +00:00
a ` finally ` quiesce True
2011-03-27 19:56:43 +00:00
2022-06-29 16:40:12 +00:00
data LocalRemoteAnnex = LocalRemoteAnnex Git . Repo ( MVar [ ( Annex . AnnexState , Annex . AnnexRead ) ] )
2020-04-17 21:09:29 +00:00
{- This can safely be called on a Repo that is not local, but of course
- onLocal will not work if used with the result . - }
mkLocalRemoteAnnex :: Git . Repo -> Annex ( LocalRemoteAnnex )
2022-06-29 16:40:12 +00:00
mkLocalRemoteAnnex repo = LocalRemoteAnnex repo <$> liftIO ( newMVar [] )
2020-04-17 21:09:29 +00:00
2014-03-06 21:12:50 +00:00
{- Runs an action from the perspective of a local remote.
-
- The AnnexState is cached for speed and to avoid resource leaks .
avoid flushing keys db queue after each Annex action
The flush was only done Annex.run' to make sure that the queue was flushed
before git-annex exits. But, doing it there means that as soon as one
change gets queued, it gets flushed soon after, which contributes to
excessive writes to the database, slowing git-annex down.
(This does not yet speed git-annex up, but it is a stepping stone to
doing so.)
Database queues do not autoflush when garbage collected, so have to
be flushed explicitly. I don't think it's possible to make them
autoflush (except perhaps if git-annex sqitched to using ResourceT..).
The comment in Database.Keys.closeDb used to be accurate, since the
automatic flushing did mean that all writes reached the database even
when closeDb was not called. But now, closeDb or flushDb needs to be
called before stopping using an Annex state. So, removed that comment.
In Remote.Git, change to using quiesce everywhere that it used to use
stopCoProcesses. This means that uses on onLocal in there are just as
slow as before. I considered only calling closeDb on the local git remotes
when git-annex exits. But, the reason that Remote.Git calls stopCoProcesses
in each onLocal is so as not to leave git processes running that have files
open on the remote repo, when it's on removable media. So, it seemed to make
sense to also closeDb after each one, since sqlite may also keep files
open. Although that has not seemed to cause problems with removable
media so far. It was also just easier to quiesce in each onLocal than
once at the end. This does likely leave performance on the floor, so
could be revisited.
In Annex.Content.saveState, there was no reason to close the db,
flushing it is enough.
The rest of the changes are from auditing for Annex.new, and making
sure that quiesce is called, after any action that might possibly need
it.
After that audit, I'm pretty sure that the change to Annex.run' is
safe. The only concern might be that this does let more changes get
queued for write to the db, and if git-annex is interrupted, those will be
lost. But interrupting git-annex can obviously already prevent it from
writing the most recent change to the db, so it must recover from such
lost data... right?
Sponsored-by: Dartmouth College's Datalad project
2022-10-12 17:50:46 +00:00
- However , it is quiesced after each call to avoid git processes
- hanging around on removable media .
consolidate calls to ensureInitialized
tryGitConfigRead may run ensureInitialized first, but when checkuuid = false,
that is skipped. So, make sure it's run before all onLocal actions.
ensureInitialized is inexpensive, so the extra call by tryGitConfigRead
is not a big deal. But since it was easy to do, I made it only be run
once by all calls to onLocal.
A few calls to onLocal didn't call ensureInitialized before. Notably,
the checkPresent action didn't, and does now.
That means that there's a guarantee that any necessary repo upgrades
will be run before the checkPresent action runs in the repo. Which is
important especially for the direct mode conversion, because without
that upgrade, the checkPresent action would need to support direct mode
still. Now I can remove the last bits of direct mode support in
Annex.Content without worrying that it will break accessing remotes
that have not been upgraded.
This does necessarily mean that checkPresent needs to write to the disk
when performing such a repo upgrade. The other remote actions already
did, so retrieval from a readonly remote that needed to be upgraded would
fail. Having checkPresent also fail doesn't seem like a large reversion,
especially since it already failed in the default case when checkuuid = true.
2019-08-27 16:05:20 +00:00
-
- The remote will be automatically initialized / upgraded first ,
- when possible .
2014-03-06 21:12:50 +00:00
- }
2020-04-17 21:09:29 +00:00
onLocal :: State -> Annex a -> Annex a
onLocal ( State _ _ _ _ lra ) = onLocal' lra
onLocalRepo :: Git . Repo -> Annex a -> Annex a
onLocalRepo repo a = do
lra <- mkLocalRemoteAnnex repo
onLocal' lra a
2021-07-26 15:39:57 +00:00
newLocal :: Git . Repo -> Annex ( Annex . AnnexState , Annex . AnnexRead )
newLocal repo = do
( st , rd ) <- liftIO $ Annex . new repo
debugenabled <- Annex . getRead Annex . debugenabled
debugselector <- Annex . getRead Annex . debugselector
2023-04-19 16:53:58 +00:00
force <- Annex . getRead Annex . force
2021-07-26 15:39:57 +00:00
return ( st , rd
{ Annex . debugenabled = debugenabled
, Annex . debugselector = debugselector
2023-04-19 16:53:58 +00:00
, Annex . force = force
2021-07-26 15:39:57 +00:00
} )
2020-04-17 21:09:29 +00:00
onLocal' :: LocalRemoteAnnex -> Annex a -> Annex a
2021-04-02 19:26:21 +00:00
onLocal' ( LocalRemoteAnnex repo mv ) a = liftIO ( takeMVar mv ) >>= \ case
2022-06-29 16:40:12 +00:00
[] -> do
liftIO $ putMVar mv []
2021-07-26 15:39:57 +00:00
v <- newLocal repo
remove dead nodes when loading the cluster log
This is to avoid inserting a cluster uuid into the location log when
only dead nodes in the cluster contain the content of a key.
One reason why this is necessary is Remote.keyLocations, which excludes
dead repositories from the list. But there are probably many more.
Implementing this was challenging, because Logs.Location importing
Logs.Cluster which imports Logs.Trust which imports Remote.List resulted
in an import cycle through several other modules.
Resorted to making Logs.Location not import Logs.Cluster, and instead
it assumes that Annex.clusters gets populated when necessary before it's
called.
That's done in Annex.Startup, which is run by the git-annex command
(but not other commands) at early startup in initialized repos. Or,
is run after initialization.
Note that is Remote.Git, it is unable to import Annex.Startup, because
Remote.Git importing Logs.Cluster leads the the same import cycle.
So ensureInitialized is not passed annexStartup in there.
Other commands, like git-annex-shell currently don't run annexStartup
either.
So there are cases where Logs.Location will not see clusters. So it won't add
any cluster UUIDs when loading the log. That's ok, the only reason to do
that is to make display of where objects are located include clusters,
and to make commands like git-annex get --from treat keys as being located
in a cluster. git-annex-shell certainly does not do anything like that,
and I'm pretty sure Remote.Git (and callers to Remote.Git.onLocalRepo)
don't either.
2024-06-16 18:35:07 +00:00
go ( v , ensureInitialized noop ( pure [] ) >> a )
2022-06-29 16:40:12 +00:00
( v : rest ) -> do
liftIO $ putMVar mv rest
go ( v , a )
2014-03-06 21:12:50 +00:00
where
2021-04-02 19:26:21 +00:00
go ( ( st , rd ) , a' ) = do
2015-04-04 00:08:38 +00:00
curro <- Annex . getState Annex . output
2021-04-02 19:26:21 +00:00
let act = Annex . run ( st { Annex . output = curro } , rd ) $
avoid flushing keys db queue after each Annex action
The flush was only done Annex.run' to make sure that the queue was flushed
before git-annex exits. But, doing it there means that as soon as one
change gets queued, it gets flushed soon after, which contributes to
excessive writes to the database, slowing git-annex down.
(This does not yet speed git-annex up, but it is a stepping stone to
doing so.)
Database queues do not autoflush when garbage collected, so have to
be flushed explicitly. I don't think it's possible to make them
autoflush (except perhaps if git-annex sqitched to using ResourceT..).
The comment in Database.Keys.closeDb used to be accurate, since the
automatic flushing did mean that all writes reached the database even
when closeDb was not called. But now, closeDb or flushDb needs to be
called before stopping using an Annex state. So, removed that comment.
In Remote.Git, change to using quiesce everywhere that it used to use
stopCoProcesses. This means that uses on onLocal in there are just as
slow as before. I considered only calling closeDb on the local git remotes
when git-annex exits. But, the reason that Remote.Git calls stopCoProcesses
in each onLocal is so as not to leave git processes running that have files
open on the remote repo, when it's on removable media. So, it seemed to make
sense to also closeDb after each one, since sqlite may also keep files
open. Although that has not seemed to cause problems with removable
media so far. It was also just easier to quiesce in each onLocal than
once at the end. This does likely leave performance on the floor, so
could be revisited.
In Annex.Content.saveState, there was no reason to close the db,
flushing it is enough.
The rest of the changes are from auditing for Annex.new, and making
sure that quiesce is called, after any action that might possibly need
it.
After that audit, I'm pretty sure that the change to Annex.run' is
safe. The only concern might be that this does let more changes get
queued for write to the db, and if git-annex is interrupted, those will be
lost. But interrupting git-annex can obviously already prevent it from
writing the most recent change to the db, so it must recover from such
lost data... right?
Sponsored-by: Dartmouth College's Datalad project
2022-10-12 17:50:46 +00:00
a' ` finally ` quiesce True
2021-04-02 19:26:21 +00:00
( ret , ( st' , _rd ) ) <- liftIO $ act ` onException ` cache ( st , rd )
liftIO $ cache ( st' , rd )
2014-03-06 21:12:50 +00:00
return ret
2022-06-29 16:40:12 +00:00
cache v = do
l <- takeMVar mv
putMVar mv ( v : l )
2014-03-06 21:12:50 +00:00
2017-02-17 19:21:39 +00:00
{- Faster variant of onLocal.
-
- The repository's git - annex branch is not updated , as an optimisation .
- No caller of onLocalFast can query data from the branch and be ensured
- it gets the most current value . Caller of onLocalFast can make changes
- to the branch , however .
- }
2020-04-17 21:09:29 +00:00
onLocalFast :: State -> Annex a -> Annex a
onLocalFast st a = onLocal st $ Annex . BranchState . disableUpdate >> a
2017-02-17 19:21:39 +00:00
2020-04-17 21:09:29 +00:00
commitOnCleanup :: Git . Repo -> Remote -> State -> Annex a -> Annex a
commitOnCleanup repo r st a = go ` after ` a
2012-10-29 01:27:15 +00:00
where
2020-12-11 19:28:58 +00:00
go = Annex . addCleanupAction ( RemoteCleanup $ uuid r ) cleanup
2012-10-29 01:27:15 +00:00
cleanup
2020-04-17 21:09:29 +00:00
| not $ Git . repoIsUrl repo = onLocalFast st $
2012-10-29 01:27:15 +00:00
doQuietSideAction $
2018-08-02 18:06:06 +00:00
Annex . Branch . commit =<< Annex . Branch . commitMessage
remove git-annex-shell compat code
* Removed support for accessing git remotes that use versions of
git-annex older than 6.20180312.
* git-annex-shell: Removed several commands that were only needed to
support git-annex versions older than 6.20180312.
(lockcontent, recvkey, sendkey, transferinfo, commit)
The P2P protocol was added in that version, and used ever since, so
this code was only needed for interop with older versions.
"git-annex-shell commit" is used by newer git-annex versions, though
unnecessarily so, because the p2pstdio command makes a single commit at
shutdown. Luckily, it was run with stderr and stdout sent to /dev/null,
and non-zero exit status or other exceptions are caught and ignored. So,
that was able to be removed from git-annex-shell too.
git-annex-shell inannex, recvkey, sendkey, and dropkey are still used by
gcrypt special remotes accessed over ssh, so those had to be kept.
It would probably be possible to convert that to using the P2P protocol,
but it would be another multi-year transition.
Some git-annex-shell fields were able to be removed. I hoped to remove
all of them, and the very concept of them, but unfortunately autoinit
is used by git-annex sync, and gcrypt uses remoteuuid.
The main win here is really in Remote.Git, removing piles of hairy fallback
code.
Sponsored-by: Luke Shumaker
2021-10-11 19:35:54 +00:00
| otherwise = noop
2015-09-14 16:13:38 +00:00
wantHardLink :: Annex Bool
2016-01-13 18:19:31 +00:00
wantHardLink = ( annexHardLink <$> Annex . getGitConfig )
-- Not unlocked files that are hard linked in the work tree,
-- because they can be modified at any time.
<&&> ( not <$> annexThin <$> Annex . getGitConfig )
2015-09-14 16:13:38 +00:00
2021-08-16 19:56:24 +00:00
type FileCopier = FilePath -> FilePath -> Key -> MeterUpdate -> Annex Bool -> VerifyConfig -> Annex ( Bool , Verification )
2021-08-16 19:25:06 +00:00
-- If either the remote or local repository wants to use hard links,
-- the copier will do so (falling back to copying if a hard link cannot be
-- made).
--
-- When a hard link is created, returns Verified; the repo being linked
-- from is implicitly trusted, so no expensive verification needs to be
-- done. Also returns Verified if the key's content is verified while
-- copying it.
2021-04-14 18:06:43 +00:00
mkFileCopier :: Bool -> State -> Annex FileCopier
mkFileCopier remotewanthardlink ( State _ _ copycowtried _ _ ) = do
2015-09-14 16:13:38 +00:00
localwanthardlink <- wantHardLink
2023-03-01 19:55:58 +00:00
let linker = \ src dest -> R . createLink ( toRawFilePath src ) ( toRawFilePath dest ) >> return True
2019-08-28 15:53:10 +00:00
if remotewanthardlink || localwanthardlink
2021-02-10 20:05:24 +00:00
then return $ \ src dest k p check verifyconfig ->
other 80% of avoding verification when hard linking to objects in shared repo
In c6632ee5c8e66c26ef18317f56ae02bae1e7e280, it actually only handled
uploading objects to a shared repository. To avoid verification when
downloading objects from a shared repository, was a lot harder.
On the plus side, if the process of downloading a file from a remote
is able to verify its content on the side, the remote can indicate this
now, and avoid the extra post-download verification.
As of yet, I don't have any remotes (except Git) using this ability.
Some more work would be needed to support it in special remotes.
It would make sense for tahoe to implicitly verify things downloaded from it;
as long as you trust your tahoe server (which typically runs locally),
there's cryptographic integrity. OTOH, despite bup being based on shas,
a bup repo under an attacker's control could have the git ref used for an
object changed, and so a bup repo shouldn't implicitly verify. Indeed,
tahoe seems unique in being trustworthy enough to implicitly verify.
2015-10-02 17:56:42 +00:00
ifM ( liftIO ( catchBoolIO ( linker src dest ) ) )
2021-02-10 17:05:35 +00:00
( ifM check
( return ( True , Verified )
2022-01-07 17:17:43 +00:00
, do
verificationOfContentFailed ( toRawFilePath dest )
return ( False , UnVerified )
2021-02-10 17:05:35 +00:00
)
2021-02-10 20:05:24 +00:00
, copier src dest k p check verifyconfig
other 80% of avoding verification when hard linking to objects in shared repo
In c6632ee5c8e66c26ef18317f56ae02bae1e7e280, it actually only handled
uploading objects to a shared repository. To avoid verification when
downloading objects from a shared repository, was a lot harder.
On the plus side, if the process of downloading a file from a remote
is able to verify its content on the side, the remote can indicate this
now, and avoid the extra post-download verification.
As of yet, I don't have any remotes (except Git) using this ability.
Some more work would be needed to support it in special remotes.
It would make sense for tahoe to implicitly verify things downloaded from it;
as long as you trust your tahoe server (which typically runs locally),
there's cryptographic integrity. OTOH, despite bup being based on shas,
a bup repo under an attacker's control could have the git ref used for an
object changed, and so a bup repo shouldn't implicitly verify. Indeed,
tahoe seems unique in being trustworthy enough to implicitly verify.
2015-10-02 17:56:42 +00:00
)
2019-08-28 15:53:10 +00:00
else return copier
2021-08-16 19:56:24 +00:00
where
copier src dest k p check verifyconfig = do
iv <- startVerifyKeyContentIncrementally verifyconfig k
2022-05-09 19:38:21 +00:00
liftIO ( fileCopier copycowtried src dest p iv ) >>= \ case
2021-08-16 19:56:24 +00:00
Copied -> ifM check
2021-08-18 17:35:53 +00:00
( finishVerifyKeyContentIncrementally iv
2022-01-07 17:17:43 +00:00
, do
verificationOfContentFailed ( toRawFilePath dest )
return ( False , UnVerified )
2021-08-16 19:56:24 +00:00
)
CopiedCoW -> unVerified check
2018-01-10 18:21:18 +00:00
2018-03-08 18:02:18 +00:00
{- Normally the UUID of a local repository is checked at startup,
- but annex - checkuuid config can prevent that . To avoid getting
- confused , a deferred check is done just before the repository
- is used .
- This returns False when the repository UUID is not as expected . - }
2018-01-10 18:21:18 +00:00
type DeferredUUIDCheck = Annex Bool
2024-06-12 14:10:11 +00:00
data State = State Ssh . P2PShellConnectionPool DeferredUUIDCheck CopyCoWTried ( Annex ( Git . Repo , GitConfig ) ) LocalRemoteAnnex
2018-06-04 20:48:26 +00:00
getRepoFromState :: State -> Annex Git . Repo
2020-04-17 21:09:29 +00:00
getRepoFromState ( State _ _ _ a _ ) = fst <$> a
removal of the rest of remoteGitConfig
In keyUrls, the GitConfig is used only by annexLocations
to support configured Differences. Since such configurations affect all
clones of a repository, the local repo's GitConfig must have the same
information as the remote's GitConfig would have. So, used getGitConfig
to get the local GitConfig, which is cached and so available cheaply.
That actually fixed a bug noone had ever noticed: keyUrls is
used for remotes accessed over http. The full git config of such a
remote is normally not available, so the remoteGitConfig that keyUrls
used would not have the necessary information in it.
In copyFromRemoteCheap', it uses gitAnnexLocation,
which does need the GitConfig of the remote repo itself in order to
check if it's crippled, supports symlinks, etc. So, made the
State include that GitConfig, cached. The use of gitAnnexLocation is
within a (not $ Git.repoIsUrl repo) guard, so it's local, and so
its git config will always be read and available.
(Note that gitAnnexLocation in turn calls annexLocations, so the
Differences config it uses in this case comes from the remote repo's
GitConfig and not from the local repo's GitConfig. As explained above
this is ok since they must have the same value.)
Not very happy with this mess of different GitConfigs not type-safe and
some read only sometimes etc. Very hairy. Think I got it this change
right. Test suite passes..
This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
2019-08-13 17:10:33 +00:00
# ifndef mingw32_HOST_OS
removal of the rest of remoteGitConfig
In keyUrls, the GitConfig is used only by annexLocations
to support configured Differences. Since such configurations affect all
clones of a repository, the local repo's GitConfig must have the same
information as the remote's GitConfig would have. So, used getGitConfig
to get the local GitConfig, which is cached and so available cheaply.
That actually fixed a bug noone had ever noticed: keyUrls is
used for remotes accessed over http. The full git config of such a
remote is normally not available, so the remoteGitConfig that keyUrls
used would not have the necessary information in it.
In copyFromRemoteCheap', it uses gitAnnexLocation,
which does need the GitConfig of the remote repo itself in order to
check if it's crippled, supports symlinks, etc. So, made the
State include that GitConfig, cached. The use of gitAnnexLocation is
within a (not $ Git.repoIsUrl repo) guard, so it's local, and so
its git config will always be read and available.
(Note that gitAnnexLocation in turn calls annexLocations, so the
Differences config it uses in this case comes from the remote repo's
GitConfig and not from the local repo's GitConfig. As explained above
this is ok since they must have the same value.)
Not very happy with this mess of different GitConfigs not type-safe and
some read only sometimes etc. Very hairy. Think I got it this change
right. Test suite passes..
This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
{- The config of the remote git repository, cached for speed. -}
getGitConfigFromState :: State -> Annex GitConfig
2020-04-17 21:09:29 +00:00
getGitConfigFromState ( State _ _ _ a _ ) = snd <$> a
2019-08-13 17:10:33 +00:00
# endif
2018-06-04 20:48:26 +00:00
mkState :: Git . Repo -> UUID -> RemoteGitConfig -> Annex State
mkState r u gc = do
2024-06-12 14:10:11 +00:00
pool <- Ssh . mkP2PShellConnectionPool
2019-07-17 18:19:00 +00:00
copycowtried <- liftIO newCopyCoWTried
2020-04-17 21:09:29 +00:00
lra <- mkLocalRemoteAnnex r
2018-06-04 20:48:26 +00:00
( duc , getrepo ) <- go
2020-04-17 21:09:29 +00:00
return $ State pool duc copycowtried getrepo lra
2018-06-04 20:48:26 +00:00
where
go
removal of the rest of remoteGitConfig
In keyUrls, the GitConfig is used only by annexLocations
to support configured Differences. Since such configurations affect all
clones of a repository, the local repo's GitConfig must have the same
information as the remote's GitConfig would have. So, used getGitConfig
to get the local GitConfig, which is cached and so available cheaply.
That actually fixed a bug noone had ever noticed: keyUrls is
used for remotes accessed over http. The full git config of such a
remote is normally not available, so the remoteGitConfig that keyUrls
used would not have the necessary information in it.
In copyFromRemoteCheap', it uses gitAnnexLocation,
which does need the GitConfig of the remote repo itself in order to
check if it's crippled, supports symlinks, etc. So, made the
State include that GitConfig, cached. The use of gitAnnexLocation is
within a (not $ Git.repoIsUrl repo) guard, so it's local, and so
its git config will always be read and available.
(Note that gitAnnexLocation in turn calls annexLocations, so the
Differences config it uses in this case comes from the remote repo's
GitConfig and not from the local repo's GitConfig. As explained above
this is ok since they must have the same value.)
Not very happy with this mess of different GitConfigs not type-safe and
some read only sometimes etc. Very hairy. Think I got it this change
right. Test suite passes..
This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
| remoteAnnexCheckUUID gc = return
git-annex config annex.largefiles
annex.largefiles can be configured by git-annex config, to more easily set
a default that will also be used by clones, without needing to shoehorn the
expression into the gitattributes file. The git config and gitattributes
override that.
Whenever something is added to git-annex config, we have to consider what
happens if a user puts a purposfully bad value in there. Or, if a new
git-annex adds some new value that an old git-annex can't parse.
In this case, a global annex.largefiles that can't be parsed currently
makes an error be thrown. That might not be ideal, but the gitattribute
behaves the same, and is almost equally repo-global.
Performance notes:
git-annex add and addurl construct a matcher once
and uses it for every file, so the added time penalty for reading the global
config log is minor. If the gitattributes annex.largefiles were deprecated,
git-annex add would get around 2% faster (excluding hashing), because
looking that up for each file is not fast. So this new way of setting
it is progress toward speeding up add.
git-annex smudge does need to load the log every time. As well as checking
the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids
both overheads.
2019-12-20 16:12:31 +00:00
( return True , return ( r , extractGitConfig FromGitConfig r ) )
2018-06-04 20:48:26 +00:00
| otherwise = do
rv <- liftIO newEmptyMVar
let getrepo = ifM ( liftIO $ isEmptyMVar rv )
( do
avoid beware of the leopard situation
* Display a warning message when a remote uses a protocol, such as
git://, that git-annex does not support. Silently skipping such a
remote was confusing behavior.
It sets annex-ignore, so the warning is only displayed once.
* Also display a warning message when a remote, without a known uuid,
is located in a directory that does not currently exist, to avoid
silently skipping such a remote.
This is a bit more debatable, since git-annex get will say,
try making repository available. And since it does not set annex-ignore,
the warning will be displayed repeatedly. It's also an extreme edge case,
I don't think I've ever seen it happen in real life.
2020-05-04 17:01:11 +00:00
r' <- tryGitConfigRead False r True
git-annex config annex.largefiles
annex.largefiles can be configured by git-annex config, to more easily set
a default that will also be used by clones, without needing to shoehorn the
expression into the gitattributes file. The git config and gitattributes
override that.
Whenever something is added to git-annex config, we have to consider what
happens if a user puts a purposfully bad value in there. Or, if a new
git-annex adds some new value that an old git-annex can't parse.
In this case, a global annex.largefiles that can't be parsed currently
makes an error be thrown. That might not be ideal, but the gitattribute
behaves the same, and is almost equally repo-global.
Performance notes:
git-annex add and addurl construct a matcher once
and uses it for every file, so the added time penalty for reading the global
config log is minor. If the gitattributes annex.largefiles were deprecated,
git-annex add would get around 2% faster (excluding hashing), because
looking that up for each file is not fast. So this new way of setting
it is progress toward speeding up add.
git-annex smudge does need to load the log every time. As well as checking
the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids
both overheads.
2019-12-20 16:12:31 +00:00
let t = ( r' , extractGitConfig FromGitConfig r' )
removal of the rest of remoteGitConfig
In keyUrls, the GitConfig is used only by annexLocations
to support configured Differences. Since such configurations affect all
clones of a repository, the local repo's GitConfig must have the same
information as the remote's GitConfig would have. So, used getGitConfig
to get the local GitConfig, which is cached and so available cheaply.
That actually fixed a bug noone had ever noticed: keyUrls is
used for remotes accessed over http. The full git config of such a
remote is normally not available, so the remoteGitConfig that keyUrls
used would not have the necessary information in it.
In copyFromRemoteCheap', it uses gitAnnexLocation,
which does need the GitConfig of the remote repo itself in order to
check if it's crippled, supports symlinks, etc. So, made the
State include that GitConfig, cached. The use of gitAnnexLocation is
within a (not $ Git.repoIsUrl repo) guard, so it's local, and so
its git config will always be read and available.
(Note that gitAnnexLocation in turn calls annexLocations, so the
Differences config it uses in this case comes from the remote repo's
GitConfig and not from the local repo's GitConfig. As explained above
this is ok since they must have the same value.)
Not very happy with this mess of different GitConfigs not type-safe and
some read only sometimes etc. Very hairy. Think I got it this change
right. Test suite passes..
This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
void $ liftIO $ tryPutMVar rv t
return t
2018-06-04 20:48:26 +00:00
, liftIO $ readMVar rv
)
cv <- liftIO newEmptyMVar
let duc = ifM ( liftIO $ isEmptyMVar cv )
( do
removal of the rest of remoteGitConfig
In keyUrls, the GitConfig is used only by annexLocations
to support configured Differences. Since such configurations affect all
clones of a repository, the local repo's GitConfig must have the same
information as the remote's GitConfig would have. So, used getGitConfig
to get the local GitConfig, which is cached and so available cheaply.
That actually fixed a bug noone had ever noticed: keyUrls is
used for remotes accessed over http. The full git config of such a
remote is normally not available, so the remoteGitConfig that keyUrls
used would not have the necessary information in it.
In copyFromRemoteCheap', it uses gitAnnexLocation,
which does need the GitConfig of the remote repo itself in order to
check if it's crippled, supports symlinks, etc. So, made the
State include that GitConfig, cached. The use of gitAnnexLocation is
within a (not $ Git.repoIsUrl repo) guard, so it's local, and so
its git config will always be read and available.
(Note that gitAnnexLocation in turn calls annexLocations, so the
Differences config it uses in this case comes from the remote repo's
GitConfig and not from the local repo's GitConfig. As explained above
this is ok since they must have the same value.)
Not very happy with this mess of different GitConfigs not type-safe and
some read only sometimes etc. Very hairy. Think I got it this change
right. Test suite passes..
This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
r' <- fst <$> getrepo
2018-06-04 20:48:26 +00:00
u' <- getRepoUUID r'
let ok = u' == u
void $ liftIO $ tryPutMVar cv ok
unless ok $
filter out control characters in warning messages
Converted warning and similar to use StringContainingQuotedPath. Most
warnings are static strings, some do refer to filepaths that need to be
quoted, and others don't need quoting.
Note that, since quote filters out control characters of even
UnquotedString, this makes all warnings safe, even when an attacker
sneaks in a control character in some other way.
When json is being output, no quoting is done, since json gets its own
quoting.
This does, as a side effect, make warning messages in json output not
be indented. The indentation is only needed to offset warning messages
underneath the display of the file they apply to, so that's ok.
Sponsored-by: Brett Eisenberg on Patreon
2023-04-10 18:47:32 +00:00
warning $ UnquotedString $ Git . repoDescribe r ++ " is not the expected repository. The remote's annex-checkuuid configuration prevented noticing the change until now. "
2018-06-04 20:48:26 +00:00
return ok
, liftIO $ readMVar cv
)
return ( duc , getrepo )
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
listProxied :: M . Map UUID ( S . Set Proxy ) -> [ Git . Repo ] -> Annex [ Git . Repo ]
listProxied proxies rs = concat <$> mapM go rs
where
go r = do
g <- Annex . gitRepo
u <- getRepoUUID r
gc <- Annex . getRemoteGitConfig r
let cu = fromMaybe u $ remoteAnnexConfigUUID gc
2024-06-12 17:24:25 +00:00
if not ( canproxy gc r ) || cu == NoUUID
2024-06-07 16:03:48 +00:00
then pure []
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
else case M . lookup cu proxies of
2024-06-07 16:03:48 +00:00
Nothing -> pure []
2024-06-26 16:56:16 +00:00
Just proxied -> catMaybes
<$> mapM ( mkproxied g r gc proxied )
( S . toList proxied )
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
2024-06-07 16:03:48 +00:00
proxiedremotename r p = do
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
n <- Git . remoteName r
pure $ n ++ " - " ++ proxyRemoteName p
2024-06-26 16:56:16 +00:00
mkproxied g r gc proxied p = case proxiedremotename r p of
2024-06-07 16:03:48 +00:00
Nothing -> pure Nothing
2024-06-26 16:56:16 +00:00
Just proxyname -> mkproxied' g r gc proxied p proxyname
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
2024-06-07 16:03:48 +00:00
-- The proxied remote is constructed by renaming the proxy remote,
2024-06-11 17:40:50 +00:00
-- changing its uuid, and setting the proxied remote's inherited
-- configs and uuid in Annex state.
2024-06-26 16:56:16 +00:00
mkproxied' g r gc proxied p proxyname
2024-06-07 16:03:48 +00:00
| any isconfig ( M . keys ( Git . config g ) ) = pure Nothing
| otherwise = do
2024-06-25 14:06:28 +00:00
clusters <- getClustersWith id
2024-06-07 16:03:48 +00:00
-- Not using addGitConfigOverride for inherited
2024-06-24 13:40:57 +00:00
-- configs, because child git processes do not
-- need them to be provided with -c.
2024-06-25 14:06:28 +00:00
Annex . adjustGitRepo ( pure . annexconfigadjuster clusters )
2024-06-07 16:03:48 +00:00
return $ Just $ renamedr
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
where
2024-06-11 17:40:50 +00:00
renamedr =
2024-06-24 13:40:57 +00:00
let c = adduuid configkeyUUID $
Git . fullconfig r
2024-06-11 17:40:50 +00:00
in r
{ Git . remoteName = Just proxyname
, Git . config = M . map Prelude . head c
, Git . fullconfig = c
}
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
2024-06-25 14:06:28 +00:00
annexconfigadjuster clusters r' =
2024-06-11 17:40:50 +00:00
let c = adduuid ( configRepoUUID renamedr ) $
2024-06-24 14:13:13 +00:00
addurl $
2024-06-26 16:56:16 +00:00
addproxiedby $
2024-06-25 14:06:28 +00:00
adjustclusternode clusters $
2024-06-11 17:40:50 +00:00
inheritconfigs $ Git . fullconfig r'
2024-06-07 16:03:48 +00:00
in r'
{ Git . config = M . map Prelude . head c
, Git . fullconfig = c
}
2024-06-11 17:40:50 +00:00
adduuid ck = M . insert ck
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
[ Git . ConfigValue $ fromUUID $ proxyRemoteUUID p ]
2024-06-25 14:06:28 +00:00
addurl = M . insert ( remoteConfig renamedr ( remoteGitConfigKey UrlField ) )
2024-06-24 13:40:57 +00:00
[ Git . ConfigValue $ encodeBS $ Git . repoLocation r ]
2024-06-24 14:13:13 +00:00
2024-06-26 16:56:16 +00:00
addproxiedby = case remoteAnnexUUID gc of
Just u -> addremoteannexfield ProxiedByField
[ Git . ConfigValue $ fromUUID u ]
Nothing -> id
2024-06-25 14:06:28 +00:00
-- A node of a cluster that is being proxied along with
-- that cluster does not need to be synced with
-- by default, because syncing with the cluster will
-- effectively sync with all of its nodes.
2024-06-27 19:21:03 +00:00
--
-- Also, give it a slightly higher cost than the
-- cluster by default, to encourage using the cluster.
2024-06-25 14:06:28 +00:00
adjustclusternode clusters =
case M . lookup ( ClusterNodeUUID ( proxyRemoteUUID p ) ) ( clusterNodeUUIDs clusters ) of
Just cs
| any ( \ c -> S . member ( fromClusterUUID c ) proxieduuids ) ( S . toList cs ) ->
2024-06-26 16:56:16 +00:00
addremoteannexfield SyncField
[ Git . ConfigValue $ Git . Config . boolConfig' False ]
2024-06-27 19:21:03 +00:00
. addremoteannexfield CostField
[ Git . ConfigValue $ encodeBS $ show $ defaultRepoCost r + 0.1 ]
2024-06-25 14:06:28 +00:00
_ -> id
proxieduuids = S . map proxyRemoteUUID proxied
2024-06-26 16:56:16 +00:00
addremoteannexfield f = M . insert
2024-06-25 14:06:28 +00:00
( remoteAnnexConfig renamedr ( remoteGitConfigKey f ) )
2024-06-24 13:40:57 +00:00
2024-06-07 16:03:48 +00:00
inheritconfigs c = foldl' inheritconfig c proxyInheritedFields
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
2024-06-07 16:03:48 +00:00
inheritconfig c k = case ( M . lookup dest c , M . lookup src c ) of
( Nothing , Just v ) -> M . insert dest v c
_ -> c
where
src = remoteAnnexConfig r k
dest = remoteAnnexConfig renamedr k
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
-- When the git config has anything set for a remote,
-- avoid making a proxied remote with the same name.
-- It is possible to set git configs of proxies, but it
-- needs both the url and uuid config to be manually set.
isconfig ( Git . ConfigKey configkey ) =
2024-06-07 16:03:48 +00:00
proxyconfigprefix ` B . isPrefixOf ` configkey
where
Git . ConfigKey proxyconfigprefix = remoteConfig proxyname mempty
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
-- Git remotes that are gcrypt or git-lfs special remotes cannot
2024-06-12 14:16:04 +00:00
-- proxy. Local git remotes cannot proxy either because
-- git-annex-shell is not used to access a local git url.
-- Proxing is also yet supported for remotes using P2P
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
-- addresses.
canproxy gc r
| remoteAnnexGitLFS gc = False
| Git . GCrypt . isEncrypted r = False
2024-06-12 14:16:04 +00:00
| Git . repoIsLocal r || Git . repoIsLocalUnknown r = False
instantiate remotes that are behind a proxy remote
Untested, but this should be close to working. The proxied remotes have
the same url but a different uuid. When talking to current
git-annex-shell, it will fail due to a uuid mismatch. Once it supports
proxies, it will know that the presented uuid is for a remote that it
proxies for.
The check for any git config settings for a remote with the same name as
the proxied remote is there for several reasons. One is security:
Writing a name to the proxy log should not cause changes to
how an existing, configured git remote operates in a different clone of
the repo.
It's possible that the user has been using a proxied remote, and decides
to set a git config for it. We can't tell the difference between that
scenario and an evil remote trying to eg, intercept a file upload
by replacing their remote with a proxied remote.
Also, if the user sets some git config, does it override the config
inherited from the proxy remote? Seems a difficult question. Luckily,
the above means we don't need to think through it.
This does mean though, that in order for a user to change the config of
a proxy remote, they have to manually set its annex-uuid and url, as
well as the config they want to change. They may also have to set any of
the inherited configs that they were relying on.
2024-06-06 21:15:32 +00:00
| otherwise = isNothing ( repoP2PAddress r )