fix git-annex sync --content with proxied remotes

Loading the remote list a second time was removing all proxied remotes.
That happened because setting up the proxied remote added some config
fields to the in-memory git config, and on the second load, it saw those
configs and decided not to overwrite them with the proxy.

Now on the second load, that still happens. But now, the proxied
git configs are used to generate a remote same as if those configs were
all set. The reason that didn't happen before was twofold,
the gitremotes cache was not dropped, and the remote's url field was not
set correctly.

The problem with the remote's url field is that while it was marked as
proxy inherited, all other proxy inherited fields are annex- configs.
And the code to inherit didn't work for the url field.

Now it all works, but git-annex sync is left running git push/pull on
the proxied remote, which doesn't work. That still needs to be fixed.
This commit is contained in:
Joey Hess 2024-06-24 09:40:57 -04:00
parent 6f94062c53
commit 0c111fc96a
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
4 changed files with 14 additions and 11 deletions

View file

@ -448,8 +448,8 @@ claimingUrl' remotefilter url = do
where where
checkclaim = maybe (pure False) (`id` url) . claimUrl checkclaim = maybe (pure False) (`id` url) . claimUrl
{- Is this a remote of a type we can sync with, or a special remote {- Is this a remote of a type that git pull and push work with?
- with an annex:: url configured? -} - That includes special remotes with an annex:: url configured. -}
gitSyncableRemote :: Remote -> Bool gitSyncableRemote :: Remote -> Bool
gitSyncableRemote r gitSyncableRemote r
| gitSyncableRemoteType (remotetype r) | gitSyncableRemoteType (remotetype r)

View file

@ -811,14 +811,14 @@ listProxied proxies rs = concat <$> mapM go rs
| any isconfig (M.keys (Git.config g)) = pure Nothing | any isconfig (M.keys (Git.config g)) = pure Nothing
| otherwise = do | otherwise = do
-- Not using addGitConfigOverride for inherited -- Not using addGitConfigOverride for inherited
-- configs other than the uuid, because child -- configs, because child git processes do not
-- git processes do not need them to be provided -- need them to be provided with -c.
-- with -c.
Annex.adjustGitRepo (pure . annexconfigadjuster) Annex.adjustGitRepo (pure . annexconfigadjuster)
return $ Just $ renamedr return $ Just $ renamedr
where where
renamedr = renamedr =
let c = adduuid configkeyUUID $ Git.fullconfig r let c = adduuid configkeyUUID $
Git.fullconfig r
in r in r
{ Git.remoteName = Just proxyname { Git.remoteName = Just proxyname
, Git.config = M.map Prelude.head c , Git.config = M.map Prelude.head c
@ -827,6 +827,7 @@ listProxied proxies rs = concat <$> mapM go rs
annexconfigadjuster r' = annexconfigadjuster r' =
let c = adduuid (configRepoUUID renamedr) $ let c = adduuid (configRepoUUID renamedr) $
addurl (remoteConfig renamedr "url") $
inheritconfigs $ Git.fullconfig r' inheritconfigs $ Git.fullconfig r'
in r' in r'
{ Git.config = M.map Prelude.head c { Git.config = M.map Prelude.head c
@ -836,6 +837,9 @@ listProxied proxies rs = concat <$> mapM go rs
adduuid ck = M.insert ck adduuid ck = M.insert ck
[Git.ConfigValue $ fromUUID $ proxyRemoteUUID p] [Git.ConfigValue $ fromUUID $ proxyRemoteUUID p]
addurl ck = M.insert ck
[Git.ConfigValue $ encodeBS $ Git.repoLocation r]
inheritconfigs c = foldl' inheritconfig c proxyInheritedFields inheritconfigs c = foldl' inheritconfig c proxyInheritedFields
inheritconfig c k = case (M.lookup dest c, M.lookup src c) of inheritconfig c k = case (M.lookup dest c, M.lookup src c) of

View file

@ -620,7 +620,7 @@ remoteGitConfigField = \case
-- Allow proxy chains. -- Allow proxy chains.
ProxyField -> inherited "proxy" ProxyField -> inherited "proxy"
ClusterNodeField -> uninherited "cluster-node" ClusterNodeField -> uninherited "cluster-node"
UrlField -> inherited "url" UrlField -> uninherited "url"
ShellField -> inherited "shell" ShellField -> inherited "shell"
SshOptionsField -> inherited "ssh-options" SshOptionsField -> inherited "ssh-options"
RsyncOptionsField -> inherited "rsync-options" RsyncOptionsField -> inherited "rsync-options"

View file

@ -26,15 +26,14 @@ In development on the `proxy` branch.
For June's work on [[design/passthrough_proxy]], remaining todos: For June's work on [[design/passthrough_proxy]], remaining todos:
* `git-annex sync --content` does not send content to clusters, or to * `git-annex sync` etc should not treat clusters as git syncable remotes.
proxied remotes. Seems that the second call to Remote.list somehow
is failing to listProxies.
* `git-annex sync` etc, when operating on clusters, should first * `git-annex sync` etc, when operating on clusters, should first
operate on the cluster as a whole, to take advantages of fanout on upload operate on the cluster as a whole, to take advantages of fanout on upload
and mass drop. Only operate on individual cluster nodes afterwards, and mass drop. Only operate on individual cluster nodes afterwards,
to handle cases such as a cluster containing a key, but some node to handle cases such as a cluster containing a key, but some node
wanting and lacking the key. wanting and lacking the key. Perhaps just setting cost for nodes slightly
higher than the cluster cost will be enough?
* On upload to cluster, send to nodes where it's preferred content, and not * On upload to cluster, send to nodes where it's preferred content, and not
to other nodes. to other nodes.