dfdda95053
This does mean a redundant write to the git-annex branch. But, it means that two clients can be using the same proxy, and after one sends a file to a proxied remote, the other only has to pull from the proxy to learn about that. It does not need to pull from every remote behind the proxy (which it couldn't do anyway as git repo access is not currently proxied). Anyway, the overhead of this in git-annex branch writes is no worse than eg, sending a file to a repository where git-annex assistant is running, which then sends the file on to a remote, and updates the git-annex branch then. Indeed, when the assistant also drops the local copy, that results in more writes to the git-annex branch.
71 lines
2.5 KiB
Markdown
71 lines
2.5 KiB
Markdown
This is a summary todo covering several subprojects, which would extend
|
|
git-annex to be able to use proxies which sit in front of a cluster of
|
|
repositories.
|
|
|
|
1. [[design/passthrough_proxy]]
|
|
2. [[design/p2p_protocol_over_http]]
|
|
3. [[design/balanced_preferred_content]]
|
|
4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
|
|
5. [[todo/proving_preferred_content_behavior]]
|
|
|
|
Joey has received funding to work on this.
|
|
Planned schedule of work:
|
|
|
|
* June: git-annex proxy
|
|
* July, part 1: git-annex proxy support for exporttree
|
|
* July, part 2: p2p protocol over http
|
|
* August: balanced preferred content
|
|
* September: streaming through proxy to special remotes (especially S3)
|
|
* October: proving behavior of balanced preferred content with proxies
|
|
|
|
[[!tag projects/openneuro]]
|
|
|
|
# work notes
|
|
|
|
In development on the `proxy` branch.
|
|
|
|
For June's work on [[design/passthrough_proxy]], implementation plan:
|
|
|
|
1. UUID discovery via git-annex branch. Add a log file listing UUIDs
|
|
accessible via proxy UUIDs. It also will contain the names
|
|
of the remotes that the proxy is a proxy for,
|
|
from the perspective of the proxy. (done)
|
|
|
|
1. Add `git-annex updateproxy` command and remote.name.annex-proxy
|
|
configuration. (done)
|
|
|
|
2. Remote instantiation for proxies. (done)
|
|
|
|
3. Implement git-annex-shell proxying to git remotes. (done)
|
|
|
|
3. Proxy should update location tracking information for proxied remotes,
|
|
so it is available to other users who sync with it. (done)
|
|
|
|
4. Let `storeKey` return a list of UUIDs where content was stored,
|
|
and make proxies accept uploads directed at them, rather than a specific
|
|
instantiated remote, and fan out the upload to whatever nodes behind
|
|
the proxy want it. This will need P2P protocol extensions.
|
|
|
|
5. Make `git-annex copy --from $proxy` pick a node that contains each
|
|
file, and use the instantiated remote for getting the file. Same for
|
|
similar commands.
|
|
|
|
6. Make `git-annex drop --from $proxy` drop, when possible, from every
|
|
remote accessible by the proxy. Communicate partial drops somehow.
|
|
|
|
7. Make commands like `git-annex push` not iterate over instantiate
|
|
remotes, and instead just send content to the proxy for fanout.
|
|
|
|
8. Optimise proxy speed. See design for idea.
|
|
|
|
8. Use `sendfile()` to avoid data copying overhead when
|
|
`receiveBytes` is being fed right into `sendBytes`.
|
|
|
|
9. Encryption and chunking. See design for issues.
|
|
|
|
10. Cycle prevention. See design.
|
|
|
|
11. indirect uploads (to be considered). See design.
|
|
|
|
12. Support using a proxy when its url is a P2P address.
|
|
(Eg tor-annex remotes.)
|