mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

merge from proxy branch

Browse source
This commit is contained in:
Joey Hess 2024-06-20 11:20:16 -04:00
parent d89ac8c6ee
commit 53598e5154
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 208 additions and 77 deletions
Download patch file Download diff file Expand all files Collapse all files

57
doc/todo/git-annex_proxies.mdwn
View file

@ -31,8 +31,7 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
of the remotes that the proxy is a proxy for,
from the perspective of the proxy. (done)
* Add `git-annex updateproxy` command and remote.name.annex-proxy
configuration. (done)
* Add `git-annex updateproxy` command (done)
* Remote instantiation for proxies. (done)
@ -41,12 +40,56 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
* Proxy should update location tracking information for proxied remotes,
so it is available to other users who sync with it. (done)
* Consider getting instantiated remotes into git remote list.
See design.
* Implement `git-annex updatecluster` command (done)
* Implement single upload with fanout to proxied remotes.
* Implement cluster UUID insertation on location log load, and removal
on location log store. (done)
* Implement clusters.
* Omit cluster UUIDs when constructing drop proofs, since lockcontent will
always fail on a cluster. (done)
* Don't count cluster UUID as a copy. (done)
* Tab complete proxied remotes and clusters in eg --from option. (done)
* Getting a key from a cluster should proxy from one of the nodes that has
it. (done)
* Getting a key from a cluster currently always selects the lowest cost
remote, and always the same remote if cost is the same. Should
round-robin amoung remotes, and prefer to avoid using remotes that
other git-annex processes are currently using.
* Implement upload with fanout and reporting back additional UUIDs over P2P
protocol. (done, but need to check for fencepost errors on resume of
incomplete upload with remotes at different points)
* On upload to cluster, send to nodes where it's preferred content, and not
to other nodes.
* Implement cluster drops, trying to remove from all nodes, and returning
which UUIDs it was dropped from.
Problem: May lock content on cluster
nodes to satisfy numcopies (rather than locking elsewhere) and so not be
able to drop from nodes. Avoid using cluster nodes when constructing drop
proof for cluster.
Problem: When nodes are special remotes, may
treat nodes as copies while dropping from cluster, and so violate
numcopies. (But not mincopies.)
Problem: `move --from cluster` in "does this make it worse"
check may fail to realize that dropping from multiple nodes does in fact
make it worse.
* On upload to a cluster, as well as fanout to nodes, if the key is
preferred content of the proxy repository, store it there.
(But not when preferred content is not configured.)
And on download from a cluster, if the proxy repository has the content,
get it from there to avoid the overhead of proxying to a node.
* Basic proxying to special remote support (non-streaming).
* Support proxies-of-proxies better, eg foo-bar-baz.
Currently, it does work, but have to run `git-annex updateproxy`
@ -55,7 +98,7 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
proxies like that, and instead automatically generate those from the log.
(With cycle prevention there of course.)
* Cycle prevention. See design.
* Cycle prevention including cluster-in-cluster cycles. See design.
* Optimise proxy speed. See design for ideas.