mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

thoughts on cycles

Browse source 
Rejected the idea of automatically instantiating remotes for proxies-of-proxies.
That needs cycle protection, while the current behavior, which happened
for free, is that running git-annex updateproxy on the proxy can be used
to configure it, but only for topologies that actually exist.
This commit is contained in:
Joey Hess 2024-06-25 15:27:03 -04:00
parent cec2848e8a
commit b9889917a3
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 12 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

35
doc/design/passthrough_proxy.mdwn
View file

@ -462,36 +462,19 @@ proxy, etc.
Since the proxied repo uuid is communicated to git-annex-shell via Since the proxied repo uuid is communicated to git-annex-shell via
--uuid, a repo that advertises proxying for itself will be connected to --uuid, a repo that advertises proxying for itself will be connected to
with its own uuid. No proxying is done in this case. Same happens with a with its own uuid. No proxying is done in this case.
larger cycle.
Instantiating remotes needs to identity cycles and break them. Otherwise
it would construct an infinite number of proxied remotes with names
like "foo-foo-foo-foo-..." or "foo-bar-foo-bar-..."
Once `git-annex copy --to proxy` is implemented, and the proxy decides
where to send content that is being sent directly to it, cycles will
become an issue with that as well.
What if repo A is a proxy and has repo B as a remote. Meanwhile, repo B is What if repo A is a proxy and has repo B as a remote. Meanwhile, repo B is
a proxy and has repo A as a remote? a proxy and has repo A as a remote? git-annex-shell on repo A will get
A's uuid, and so will operate on it directly without proxying. So larger
cycles are also not a problem on the proxy side.
An upload to repo A will start by checking if repo B wants the content and if so, On the client side, instantiating remotes needs to identity cycles and
start an upload to repo B. Then the same happens on repo B, leading it to break them. Otherwise it would construct an infinite number of proxied
start an upload to repo A. remotes with names like "foo-foo-foo-foo-..." or "foo-bar-foo-bar-..."
At this point, it might be possible for git-annex to detect the cycle, Clusters could also have cycles, if a cluster's UUID were configured as
if the proxy uses a transfer lock file. If repo B or repo A had some other a node of itself, or of another cluster that was a node of it.
remote that is not part of a cycle, they could deposit the upload there and
the upload still succeed. Otherwise the upload would fail, which is
probably the best that can be done with such a broken configuration.
So, it seems like proxies would need to take transfer locks for uploads,
even though the content is being proxied to elsewhere.
Dropping could have similar cycles with content presence locking, which
needs to be thought through as well. A cycle of the actual dropContent
operation might also be possible.
## exporttree=yes ## exporttree=yes

11
doc/todo/git-annex_proxies.mdwn
View file

@ -33,14 +33,9 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
* Basic proxying to special remote support (non-streaming). * Basic proxying to special remote support (non-streaming).
* Support proxies-of-proxies better, eg foo-bar-baz. * Make sure that cluster-in-cluster cycles are prevented.
Currently, it does work, but have to run `git-annex updateproxy` (Actually supporting cluster-in-cluster is optional, and it might
on foo in order for it to notice the bar-baz proxied remote exists, be added later.)
and record it as foo-bar-baz. Make it skip recording proxies of
proxies like that, and instead automatically generate those from the log.
(With cycle prevention there of course.)
* Cycle prevention including cluster-in-cluster cycles. See design.
* Optimise proxy speed. See design for ideas. * Optimise proxy speed. See design for ideas.