mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

dropping from clusters

Browse source 
Dropping from a cluster drops from every node of the cluster.
Including nodes that the cluster does not think have the content.
This is different from GET and CHECKPRESENT, which do trust the
cluster's location log. The difference is that removing from a cluster
should make 100% the content is gone from every node. So doing extra
work is ok. Compare with CHECKPRESENT where checking every node could
make it very expensive, and the worst that can happen in a false
negative is extra work being done.

Extended the P2P protocol with FAILURE-PLUS to handle the case where a
drop from one node succeeds, but a drop from another node fails. In that
case the entire cluster drop has failed.

Note that SUCCESS-PLUS is returned when dropping from a proxied remote
that is not a cluster, when the protocol version supports it. This is
because P2P.Proxy does not know when it's proxying for a single node
cluster vs for a remote that is not a cluster.
This commit is contained in:
Joey Hess 2024-06-23 09:28:18 -04:00
parent a6a04b7e5e
commit 5b332a87be
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
14 changed files with 144 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

14
doc/todo/git-annex_proxies.mdwn
View file

@ -55,6 +55,9 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
* Getting a key from a cluster should proxy from one of the nodes that has
it. (done)
* Implement cluster drops, trying to remove from all nodes, and returning
which UUIDs it was dropped from. (done)
* Getting a key from a cluster currently always selects the lowest cost
remote, and always the same remote if cost is the same. Should
round-robin amoung remotes, and prefer to avoid using remotes that
@ -67,13 +70,16 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
* On upload to cluster, send to nodes where it's preferred content, and not
to other nodes.
* Implement cluster drops, trying to remove from all nodes, and returning
which UUIDs it was dropped from.
Problem: `move --from cluster` in "does this make it worse"
* Problem: `move --from cluster` in "does this make it worse"
check may fail to realize that dropping from multiple nodes does in fact
make it worse.
* Bug: When a cluster has one node, copying a file to it does not update
location log to say the content is present on it. It's returning SUCCESS
rather than SUCCESS-PLUS.
* Support annex.jobs for clusters.
* On upload to a cluster, as well as fanout to nodes, if the key is
preferred content of the proxy repository, store it there.
(But not when preferred content is not configured.)