d2576e5f1a
For NotifyChanges and also for the fallthrough case where git-annex-shell passes a command off to git-shell, proxying is currently ignored. So every remote that is accessed via a proxy will be treated as the same git repository. Every other command listed in cmdsMap will need to check if Annex.proxyremote is set, and if so handle the proxying appropriately. Probably only P2PStdio will need to support proxying. For now, everything else refuses to work when proxying. The part of that I don't like is that there's the possibility a command later gets added to the list that doesn't check proxying. When proxying is not enabled, it's important that git-annex-shell not leak information that it would not have exposed before. Such as the names or uuids of remotes. I decided that, in the case where a repository used to have proxying enabled, but no longer supports any proxies, it's ok to give the user a clear error message indicating that proxying is not configured, rather than a confusing uuid mismatch message. Similarly, if a repository has proxying enabled, but not for the requested repository, give a clear error message. A tricky thing here is how to handle the case where there is more than one remote, with proxying enabled, with the specified uuid. One way to handle that would be to plumb the proxyRemoteName all the way through from the remote git-annex to git-annex-shell, eg as a field, and use only a remote with the same name. That would be very intrusive though. Instead, I decided to let the proxy pick which remote it uses to access a given Remote. And so it picks the least expensive one. The client after all doesn't necessarily know any details about the proxy's configuration. This does mean though, that if the least expensive remote is not accessible, but another remote would have worked, an access via the proxy will fail.
67 lines
2.3 KiB
Markdown
67 lines
2.3 KiB
Markdown
This is a summary todo covering several subprojects, which would extend
|
|
git-annex to be able to use proxies which sit in front of a cluster of
|
|
repositories.
|
|
|
|
1. [[design/passthrough_proxy]]
|
|
2. [[design/p2p_protocol_over_http]]
|
|
3. [[design/balanced_preferred_content]]
|
|
4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
|
|
5. [[todo/proving_preferred_content_behavior]]
|
|
|
|
Joey has received funding to work on this.
|
|
Planned schedule of work:
|
|
|
|
* June: git-annex proxy
|
|
* July, part 1: git-annex proxy support for exporttree
|
|
* July, part 2: p2p protocol over http
|
|
* August: balanced preferred content
|
|
* September: streaming through proxy to special remotes (especially S3)
|
|
* October: proving behavior of balanced preferred content with proxies
|
|
|
|
[[!tag projects/openneuro]]
|
|
|
|
# work notes
|
|
|
|
In development on the `proxy` branch.
|
|
|
|
For June's work on [[design/passthrough_proxy]], implementation plan:
|
|
|
|
1. UUID discovery via git-annex branch. Add a log file listing UUIDs
|
|
accessible via proxy UUIDs. It also will contain the names
|
|
of the remotes that the proxy is a proxy for,
|
|
from the perspective of the proxy. (done)
|
|
|
|
1. Add `git-annex updateproxy` command and remote.name.annex-proxy
|
|
configuration. (done)
|
|
|
|
2. Remote instantiation for proxies. (done)
|
|
|
|
3. Implement proxying for Command.P2PStdIO.cmd.
|
|
|
|
4. Either implement proxying for local path remotes, or prevent
|
|
listProxied from operating on them.
|
|
|
|
4. Let `storeKey` return a list of UUIDs where content was stored,
|
|
and make proxies accept uploads directed at them, rather than a specific
|
|
instantiated remote, and fan out the upload to whatever nodes behind
|
|
the proxy want it. This will need P2P protocol extensions.
|
|
|
|
5. Make `git-annex copy --from $proxy` pick a node that contains each
|
|
file, and use the instantiated remote for getting the file. Same for
|
|
similar commands.
|
|
|
|
6. Make `git-annex drop --from $proxy` drop, when possible, from every
|
|
remote accessible by the proxy. Communicate partial drops somehow.
|
|
|
|
7. Make commands like `git-annex push` not iterate over instantiate
|
|
remotes, and instead just send content to the proxy for fanout.
|
|
|
|
8. Optimise proxy speed. See design for idea.
|
|
|
|
9. Encryption and chunking. See design for issues.
|
|
|
|
10. Cycle prevention. See design.
|
|
|
|
11. indirect uploads (to be considered). See design.
|
|
|
|
12. Support using a proxy when its url is a P2P address.
|