git-annex/doc/todo/git-annex_proxies.mdwn

This is a summary todo covering several subprojects, which would extend
git-annex to be able to use proxies which sit in front of a cluster of
repositories.

1. [[design/passthrough_proxy]]
2. [[design/p2p_protocol_over_http]]
3. [[design/balanced_preferred_content]]
4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
5. [[todo/proving_preferred_content_behavior]]

Joey has received funding to work on this.
Planned schedule of work:

* June: git-annex proxy
* July, part 1: git-annex proxy support for exporttree
* July, part 2: p2p protocol over http
* August: balanced preferred content
* September: streaming through proxy to special remotes (especially S3)
* October: proving behavior of balanced preferred content with proxies

[[!tag projects/openneuro]]

# work notes

In development on the `proxy` branch.

For June's work on [[design/passthrough_proxy]], remaining todos:

* Since proxying to special remotes is not supported yet, and won't be for
  the first release, make it fail in a reasonable way.

* Getting a key from a cluster currently always selects the lowest cost
  remote, and always the same remote if cost is the same. Should
  round-robin amoung remotes, and prefer to avoid using remotes that
  other git-annex processes are currently using.

* Optimise proxy speed. See design for ideas.

* Use `sendfile()` to avoid data copying overhead when
  `receiveBytes` is being fed right into `sendBytes`.
  Library to use:
  <https://hackage.haskell.org/package/hsyscall-0.4/docs/System-Syscall.html>

* Encryption and chunking. See design for issues.

* Indirect uploads (to be considered). See design.

* Support using a proxy when its url is a P2P address.
  (Eg tor-annex remotes.)

* `viconfig` support for setting preferred content, group, 
  and description of clusters


# completed items for June's work on [[design/passthrough_proxy]]:

* UUID discovery via git-annex branch. Add a log file listing UUIDs
  accessible via proxy UUIDs. It also will contain the names
  of the remotes that the proxy is a proxy for, 
  from the perspective of the proxy. (done)

* Add `git-annex updateproxy` command (done)

* Remote instantiation for proxies. (done)

* Implement git-annex-shell proxying to git remotes. (done)

* Proxy should update location tracking information for proxied remotes,
  so it is available to other users who sync with it. (done)

* Implement `git-annex updatecluster` command (done)

* Implement cluster UUID insertation on location log load, and removal
  on location log store. (done)

* Omit cluster UUIDs when constructing drop proofs, since lockcontent will
  always fail on a cluster. (done)

* Don't count cluster UUID as a copy in numcopies checking etc. (done)

* Tab complete proxied remotes and clusters in eg --from option. (done)

* Getting a key from a cluster should proxy from one of the nodes that has
  it. (done)

* Implement upload with fanout to multiple cluster nodes and reporting back
  additional UUIDs over P2P protocol. (done)

* Implement cluster drops, trying to remove from all nodes, and returning
  which UUIDs it was dropped from. (done)

* `git-annex testremote` works against proxied remote and cluster. (done)

* Avoid `git-annex sync --content` etc from operating on cluster nodes by
  default since syncing with a cluster implicitly syncs with its nodes. (done)

* On upload to cluster, send to nodes where its preferred content, and not
  to other nodes. (done)

* Support annex.jobs for clusters. (done)

* Add `git-annex extendcluster` command and extend `git-annex updatecluster` 
  to support clusters with multiple gateways. (done)

* Support proxying for a remote that is proxied by another gateway of
  a cluster. (done)

* Support distributed clusters: Make a proxy for a cluster repeat
  protocol messages on to any remotes that have the same UUID as
  the cluster. Needs extension to P2P protocol to avoid cycles.
  (done)