f98605bce7
Prevent listProxied from listing anything when the proxy remote's url is a local directory. Proxying does not work in that situation, because the proxied remotes have the same url, and so git-annex-shell is not run when accessing them, instead the proxy remote is accessed directly. I don't think there is any good way to support this. Even if the instantiated git repos for the proxied remotes somehow used an url that caused it to use git-annex-shell to access them, planned features like `git-annex copy --to proxy` accepting a key and sending it on to nodes behind the proxy would not work, since git-annex-shell is not used to access the proxy. So it would need to use something to access the proxy that causes git-annex-shell to be run and speaks P2P protocol over it. And we have that. It's a ssh connection to localhost. Of course, it would be possible to take ssh out of that mix, and swap in something that does not have encryption overhead and authentication complications, but otherwise behaves the same as ssh. And if the user wants to do that, GIT_SSH does exist.
75 lines
2.7 KiB
Markdown
75 lines
2.7 KiB
Markdown
This is a summary todo covering several subprojects, which would extend
|
|
git-annex to be able to use proxies which sit in front of a cluster of
|
|
repositories.
|
|
|
|
1. [[design/passthrough_proxy]]
|
|
2. [[design/p2p_protocol_over_http]]
|
|
3. [[design/balanced_preferred_content]]
|
|
4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
|
|
5. [[todo/proving_preferred_content_behavior]]
|
|
|
|
Joey has received funding to work on this.
|
|
Planned schedule of work:
|
|
|
|
* June: git-annex proxy
|
|
* July, part 1: git-annex proxy support for exporttree
|
|
* July, part 2: p2p protocol over http
|
|
* August: balanced preferred content
|
|
* September: streaming through proxy to special remotes (especially S3)
|
|
* October: proving behavior of balanced preferred content with proxies
|
|
|
|
[[!tag projects/openneuro]]
|
|
|
|
# work notes
|
|
|
|
In development on the `proxy` branch.
|
|
|
|
For June's work on [[design/passthrough_proxy]], implementation plan:
|
|
|
|
1. UUID discovery via git-annex branch. Add a log file listing UUIDs
|
|
accessible via proxy UUIDs. It also will contain the names
|
|
of the remotes that the proxy is a proxy for,
|
|
from the perspective of the proxy. (done)
|
|
|
|
1. Add `git-annex updateproxy` command and remote.name.annex-proxy
|
|
configuration. (done)
|
|
|
|
2. Remote instantiation for proxies. (done)
|
|
|
|
3. Implement git-annex-shell proxying for CONNECT and NOTIFYCHANGES.
|
|
(For completeness, they will only be used when using tor-annex
|
|
to access a proxy.)
|
|
|
|
3. Proxy should update location tracking information for proxied remotes,
|
|
so it is available to other users who sync with it.
|
|
|
|
4. Either implement proxying for tor-annex remotes, or prevent
|
|
listProxied from operating on them.
|
|
|
|
4. Let `storeKey` return a list of UUIDs where content was stored,
|
|
and make proxies accept uploads directed at them, rather than a specific
|
|
instantiated remote, and fan out the upload to whatever nodes behind
|
|
the proxy want it. This will need P2P protocol extensions.
|
|
|
|
5. Make `git-annex copy --from $proxy` pick a node that contains each
|
|
file, and use the instantiated remote for getting the file. Same for
|
|
similar commands.
|
|
|
|
6. Make `git-annex drop --from $proxy` drop, when possible, from every
|
|
remote accessible by the proxy. Communicate partial drops somehow.
|
|
|
|
7. Make commands like `git-annex push` not iterate over instantiate
|
|
remotes, and instead just send content to the proxy for fanout.
|
|
|
|
8. Optimise proxy speed. See design for idea.
|
|
|
|
8. Use `sendfile()` to avoid data copying overhead when
|
|
`receiveBytes` is being fed right into `sendBytes`.
|
|
|
|
9. Encryption and chunking. See design for issues.
|
|
|
|
10. Cycle prevention. See design.
|
|
|
|
11. indirect uploads (to be considered). See design.
|
|
|
|
12. Support using a proxy when its url is a P2P address.
|