git-annex/doc/todo/git-annex_proxies.mdwn

This is a summary todo covering several subprojects, which would extend
git-annex to be able to use proxies which sit in front of a cluster of
repositories.

1. [[design/passthrough_proxy]]
2. [[design/p2p_protocol_over_http]]
3. [[design/balanced_preferred_content]]
4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
5. [[todo/proving_preferred_content_behavior]]

Joey has received funding to work on this.
Planned schedule of work:

* June: git-annex proxy
* July, part 1: git-annex proxy support for exporttree
* July, part 2: p2p protocol over http
* August: balanced preferred content
* September: streaming through proxy to special remotes (especially S3)
* October: proving behavior of balanced preferred content with proxies

[[!tag projects/openneuro]]

# work notes

In development on the `proxy` branch.

For June's work on [[design/passthrough_proxy]], implementation plan:

1. UUID discovery via git-annex branch. Add a log file listing UUIDs
   accessible via proxy UUIDs. It also will contain the names
   of the remotes that the proxy is a proxy for, 
   from the perspective of the proxy. (done)

1. Add `git-annex updateproxy` command and remote.name.annex-proxy
   configuration. (done)

2. Remote instantiation for proxies. (done)

3. Implement git-annex-shell proxying for CONNECT and NOTIFYCHANGES.
   (For completeness, they will only be used when using tor-annex
   to access a proxy.)

3. Proxy should update location tracking information for proxied remotes,
   so it is available to other users who sync with it.

4. Either implement proxying for tor-annex remotes, or prevent
   listProxied from operating on them.

4. Let `storeKey` return a list of UUIDs where content was stored,
   and make proxies accept uploads directed at them, rather than a specific
   instantiated remote, and fan out the upload to whatever nodes behind
   the proxy want it. This will need P2P protocol extensions.

5. Make `git-annex copy --from $proxy` pick a node that contains each
   file, and use the instantiated remote for getting the file. Same for
   similar commands.

6. Make `git-annex drop --from $proxy` drop, when possible, from every
   remote accessible by the proxy. Communicate partial drops somehow.

7. Make commands like `git-annex push` not iterate over instantiate
   remotes, and instead just send content to the proxy for fanout.

8. Optimise proxy speed. See design for idea.

8. Use `sendfile()` to avoid data copying overhead when
   `receiveBytes` is being fed right into `sendBytes`.

9. Encryption and chunking. See design for issues.

10. Cycle prevention. See design.

11. indirect uploads (to be considered). See design.

12. Support using a proxy when its url is a P2P address.