mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

more thoughts on clusters

Browse source
This commit is contained in:
Joey Hess 2024-06-13 06:41:42 -04:00
parent 555d7e52d3
commit 3cc48279ad
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
1 changed files with 58 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

94
doc/design/passthrough_proxy.mdwn
View file

@ -204,51 +204,73 @@ and let it pick which nodes to send to. And similarly,
`git-annex drop --from cluster' should drop the content from every node in `git-annex drop --from cluster' should drop the content from every node in
the cluster. the cluster.
Let's suppose there is a config to say that a repository is a proxy for For this we need a UUID for the cluster. But it is not like a usual UUID.
a cluster. The cluster gets its own UUID. This is not the same as the UUID It does not need to actually be recorded in the location tracking logs, and
of the proxy repository. it is not counted as a copy for numcopies purposes. The only point of this
UUID is to make commands like `git-annex drop --from cluster` and
`git-annex get --from cluster` talk to the cluster's frontend proxy, which
has as its UUID the cluster's UUID.
When a cluster is a remote, its annex-uuid is the cluster UUID. The cluster UUID is recorded in the git-annex branch, along with a list of
No proxied remotes are instantiated for a cluster. the UUIDs of nodes of the cluster (which can change at any time).
Copying to a cluster would cause the transfer to be proxied to one or more When reading a location log, if any UUID where content is present is part
nodes. The location log would be updated to say the key is present in the of the cluster, the cluster's UUID is added to the list of UUIDs.
cluster UUID. The cluster proxy would also record the UUIDs of the nodes
where the content was stored, since it does need to remember that.
But it would not need to expose that;the nodes might have annex-private set.
Getting from a cluster would pick a node that has the content and When writing a location log, the cluster's UUID is filtered out of the list
proxy a transfer from that node. of UUIDs.
Dropping from a cluster would drop from every node that has the The cluster's frontend proxy fans out uploads to nodes according to
content. Once the content is entirely gone from the cluster, it would preferred content. And `storeKey` is extended to be able to return a list
record it not present in the cluster's UUID. (If some drops failed, the of additional UUIDs where the content was stored. So an upload to the
overall drop would fail.) cluster will end up writing to the location log the actual nodes that it
was fanned out to.
Checkpresent to a cluster would proxy a checkpresent to nodes until it Note that to support clusters that are nodes of clusters, when a cluster's
found one does have the content. frontend proxy fans out an upload to a node, and `storeKey` returns
additional UUIDs, it should pass those UUIDs along. Of course, no cluster
can be a node of itself, and cycles have to be broken (as described in a
section below).
Lockcontent to a cluster would lock the content on one (or more?) nodes. When a file is requested from the cluster's frontend proxy, it can send its
own local copy if it has one, but otherwise it will proxy to one of its
nodes. (How to pick which node to use? Load balancing?) This behavior will
need to be added to git-annex-shell, and to Remote.Git for local paths to a
cluster.
Problem: The location log for a key that is stored in one node of a cluster The cluster's frontend proxy also fans out drops to all nodes, attempting
will show 2 copies: The UUID of the node and the UUID of the cluster. This to drop content from the whole cluster, and only indicating success if it
would cause wrong behavior when numcopies is checked. And if a cluster node can. Also needs changes to git-annex-sjell and Remote.Git.
has the cluster as a remote, and another node as a remote, this might
extend to lockcontent of both succeeding and satisfying numcopies of 2,
allowing the node to drop content, and resulting in violating numcopies.
That could be solved by publishing a list of the UUIDs of nodes of a It does not fan out lockcontent, instead the client will lock content
cluster. When loading a location log, we are either inside the cluster or on specific nodes. In fact, the cluster UUID should probably be omitted
outside the cluster. If outside the cluster, filter out the UUIDs of its when constructing a drop proof, since trying to lockcontent on it will
nodes. If inside the cluster, filter out the cluster's UUID. usually fail.
Doing that would mean that a key that is stored in several nodes Some commands like `git-annex whereis` will list content as being stored in
of a cluster will appear to have only 1 copy from outside the cluster. the cluster, as well as on whicheven of its nodes, and whereis currently
Now suppose that a node of the cluster has a remote, and numcopies = 2. says "n copies", but since the cluster doesn't count as a copy, that
The node would be able to drop a key from the remote when it and another display should probably be counted using the numcopies logic that excludes
node contain the key. But then from outside the cluster, it would appear as cluster UUIDs.
if numcopies was violated, with only the 1 copy in the cluster.
(See also [[todo/repositories_that_count_as_more_than_one_copy]]) No other protocol extensions or special cases should be needed. Except for
the strange case of content stored in the cluster's frontend proxy.
Running `git-annex fsck --fast` on the cluster's frontend proxy will look
weird: For each file, it will read the location log, and if the file is
present on any node it will add the frontend proxy's UUID. So fsck will
expect the content to be present. But it probably won't be. So it will fix
the location log... which will make no changes since the proxy's UUID will
be filtered out on write. So probably fsck will need a special case to
avoid this behavior. (Also for `git-annex fsck --from cluster --fast`)
And if a key does get stored on the cluster's frontend proxy, it will not
be possible to tell from looking at the location log that the content is
really present there. So that won't be counted as a copy. In some cases,
a cluster's frontend proxy may want to keep files, perhaps some files are
worth caching there for speed. But if a file is stored only on the
cluster's frontend proxy and not in any of its nodes, clients will not
consider the cluster to contain the file at all.
## speed ## speed