a2f4a8eddf
Memory use is small and constant; receiveBytes returns a lazy bytestring and it does stream. Comparing speed of a get of a 500 mb file over proxy from origin-origin, vs from the same remote over a direct ssh: joey@darkstar:~/tmp/bench/client>/usr/bin/time git-annex get bigfile --from origin-origin get bigfile (from origin-origin...) ok (recording state in git...) 1.89user 0.67system 0:10.79elapsed 23%CPU (0avgtext+0avgdata 68716maxresident)k 0inputs+984320outputs (0major+10779minor)pagefaults 0swaps joey@darkstar:~/tmp/bench/client>/usr/bin/time git-annex get bigfile --from direct-ssh get bigfile (from direct-ssh...) ok 1.79user 0.63system 0:10.49elapsed 23%CPU (0avgtext+0avgdata 65776maxresident)k 0inputs+1024312outputs (0major+9773minor)pagefaults 0swaps So the proxy doesn't add much overhead even when run on the same machine as the client and remote. Still, piping receiveBytes into sendBytes like this does suggest that the proxy could be made to use less CPU resouces by using `sendfile()`.
73 lines
2.6 KiB
Markdown
73 lines
2.6 KiB
Markdown
This is a summary todo covering several subprojects, which would extend
|
|
git-annex to be able to use proxies which sit in front of a cluster of
|
|
repositories.
|
|
|
|
1. [[design/passthrough_proxy]]
|
|
2. [[design/p2p_protocol_over_http]]
|
|
3. [[design/balanced_preferred_content]]
|
|
4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
|
|
5. [[todo/proving_preferred_content_behavior]]
|
|
|
|
Joey has received funding to work on this.
|
|
Planned schedule of work:
|
|
|
|
* June: git-annex proxy
|
|
* July, part 1: git-annex proxy support for exporttree
|
|
* July, part 2: p2p protocol over http
|
|
* August: balanced preferred content
|
|
* September: streaming through proxy to special remotes (especially S3)
|
|
* October: proving behavior of balanced preferred content with proxies
|
|
|
|
[[!tag projects/openneuro]]
|
|
|
|
# work notes
|
|
|
|
In development on the `proxy` branch.
|
|
|
|
For June's work on [[design/passthrough_proxy]], implementation plan:
|
|
|
|
1. UUID discovery via git-annex branch. Add a log file listing UUIDs
|
|
accessible via proxy UUIDs. It also will contain the names
|
|
of the remotes that the proxy is a proxy for,
|
|
from the perspective of the proxy. (done)
|
|
|
|
1. Add `git-annex updateproxy` command and remote.name.annex-proxy
|
|
configuration. (done)
|
|
|
|
2. Remote instantiation for proxies. (done)
|
|
|
|
3. Implement proxying in git-annex-shell.
|
|
(Partly done, still need it for PUT, CONNECT, and NOTIFYCHANGES
|
|
messages.)
|
|
|
|
4. Either implement proxying for local path remotes, or prevent
|
|
listProxied from operating on them. Currently it seems to work,
|
|
but doesn't work right.
|
|
|
|
4. Either implement proxying for tor-annex remotes, or prevent
|
|
listProxied from operating on them.
|
|
|
|
4. Let `storeKey` return a list of UUIDs where content was stored,
|
|
and make proxies accept uploads directed at them, rather than a specific
|
|
instantiated remote, and fan out the upload to whatever nodes behind
|
|
the proxy want it. This will need P2P protocol extensions.
|
|
|
|
5. Make `git-annex copy --from $proxy` pick a node that contains each
|
|
file, and use the instantiated remote for getting the file. Same for
|
|
similar commands.
|
|
|
|
6. Make `git-annex drop --from $proxy` drop, when possible, from every
|
|
remote accessible by the proxy. Communicate partial drops somehow.
|
|
|
|
7. Make commands like `git-annex push` not iterate over instantiate
|
|
remotes, and instead just send content to the proxy for fanout.
|
|
|
|
8. Optimise proxy speed. See design for idea.
|
|
|
|
9. Encryption and chunking. See design for issues.
|
|
|
|
10. Cycle prevention. See design.
|
|
|
|
11. indirect uploads (to be considered). See design.
|
|
|
|
12. Support using a proxy when its url is a P2P address.
|