77 lines
2.7 KiB
Markdown
77 lines
2.7 KiB
Markdown
This is a summary todo covering several subprojects, which would extend
|
|
git-annex to be able to use proxies which sit in front of a cluster of
|
|
repositories.
|
|
|
|
1. [[design/passthrough_proxy]]
|
|
2. [[design/p2p_protocol_over_http]]
|
|
3. [[design/balanced_preferred_content]]
|
|
4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
|
|
5. [[todo/proving_preferred_content_behavior]]
|
|
|
|
Joey has received funding to work on this.
|
|
Planned schedule of work:
|
|
|
|
* June: git-annex proxy
|
|
* July, part 1: git-annex proxy support for exporttree
|
|
* July, part 2: p2p protocol over http
|
|
* August: balanced preferred content
|
|
* September: streaming through proxy to special remotes (especially S3)
|
|
* October: proving behavior of balanced preferred content with proxies
|
|
|
|
[[!tag projects/openneuro]]
|
|
|
|
# work notes
|
|
|
|
In development on the `proxy` branch.
|
|
|
|
For June's work on [[design/passthrough_proxy]], implementation plan:
|
|
|
|
1. UUID discovery via git-annex branch. Add a log file listing UUIDs
|
|
accessible via proxy UUIDs. It also will contain the names
|
|
of the remotes that the proxy is a proxy for,
|
|
from the perspective of the proxy. (done)
|
|
|
|
1. Add `git-annex updateproxy` command and remote.name.annex-proxy
|
|
configuration. (done)
|
|
|
|
2. Remote instantiation for proxies. (done)
|
|
|
|
2. Bug: In a repo cloned with ssh from a proxy repo,
|
|
running `git-annex init` sets annex-uuid for the instantiated remotes.
|
|
This prevents them being used, because instanatiation is not done
|
|
when there's any config set for a remote.
|
|
|
|
3. Implement proxying in git-annex-shell.
|
|
(Partly done, still need it for GET, PUT, CONNECT, and NOTIFYCHANGES
|
|
messages.)
|
|
|
|
4. Either implement proxying for local path remotes, or prevent
|
|
listProxied from operating on them.
|
|
|
|
4. Either implement proxying for tor-annex remotes, or prevent
|
|
listProxied from operating on them.
|
|
|
|
4. Let `storeKey` return a list of UUIDs where content was stored,
|
|
and make proxies accept uploads directed at them, rather than a specific
|
|
instantiated remote, and fan out the upload to whatever nodes behind
|
|
the proxy want it. This will need P2P protocol extensions.
|
|
|
|
5. Make `git-annex copy --from $proxy` pick a node that contains each
|
|
file, and use the instantiated remote for getting the file. Same for
|
|
similar commands.
|
|
|
|
6. Make `git-annex drop --from $proxy` drop, when possible, from every
|
|
remote accessible by the proxy. Communicate partial drops somehow.
|
|
|
|
7. Make commands like `git-annex push` not iterate over instantiate
|
|
remotes, and instead just send content to the proxy for fanout.
|
|
|
|
8. Optimise proxy speed. See design for idea.
|
|
|
|
9. Encryption and chunking. See design for issues.
|
|
|
|
10. Cycle prevention. See design.
|
|
|
|
11. indirect uploads (to be considered). See design.
|
|
|
|
12. Support using a proxy when its url is a P2P address.
|