designing clusters

This commit is contained in:
Joey Hess 2024-06-12 14:45:39 -04:00
parent e70e3473b3
commit a986a20034
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 75 additions and 56 deletions

View file

@ -44,6 +44,15 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
* Consider getting instantiated remotes into git remote list.
See design.
* Make commands like `git-annex sync` not git push/pull to proxied remotes.
That doesn't work because they have no url. Or, if proxied remotes are in
git remote list, it is unncessary work because it's the same url as the
proxy.
* Implement single upload with fanout to proxied remotes.
* Implement clusters.
* Support proxies-of-proxies better, eg foo-bar-baz.
Currently, it does work, but have to run `git-annex updateproxy`
on foo in order for it to notice the bar-baz proxied remote exists,
@ -53,21 +62,6 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
* Cycle prevention. See design.
* Make `git-annex copy --from $proxy` pick a node that contains each
file, and use the instantiated remote for getting the file. Same for
similar commands.
* Make `git-annex drop --from $proxy` drop, when possible, from every
remote accessible by the proxy. Communicate partial drops somehow.
* Let `storeKey` return a list of UUIDs where content was stored,
and make proxies accept uploads directed at them, rather than a specific
instantiated remote, and fan out the upload to whatever nodes behind
the proxy want it. This will need P2P protocol extensions.
* Make commands like `git-annex push` not iterate over instantiated
remotes, and instead just send content to the proxy for fanout.
* Optimise proxy speed. See design for ideas.
* Use `sendfile()` to avoid data copying overhead when
@ -75,7 +69,7 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
* Encryption and chunking. See design for issues.
* indirect uploads (to be considered). See design.
* Indirect uploads (to be considered). See design.
* Support using a proxy when its url is a P2P address.
(Eg tor-annex remotes.)