0c111fc96a
Loading the remote list a second time was removing all proxied remotes. That happened because setting up the proxied remote added some config fields to the in-memory git config, and on the second load, it saw those configs and decided not to overwrite them with the proxy. Now on the second load, that still happens. But now, the proxied git configs are used to generate a remote same as if those configs were all set. The reason that didn't happen before was twofold, the gitremotes cache was not dropped, and the remote's url field was not set correctly. The problem with the remote's url field is that while it was marked as proxy inherited, all other proxy inherited fields are annex- configs. And the code to inherit didn't work for the url field. Now it all works, but git-annex sync is left running git push/pull on the proxied remote, which doesn't work. That still needs to be fixed.
118 lines
4.3 KiB
Markdown
118 lines
4.3 KiB
Markdown
This is a summary todo covering several subprojects, which would extend
|
|
git-annex to be able to use proxies which sit in front of a cluster of
|
|
repositories.
|
|
|
|
1. [[design/passthrough_proxy]]
|
|
2. [[design/p2p_protocol_over_http]]
|
|
3. [[design/balanced_preferred_content]]
|
|
4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
|
|
5. [[todo/proving_preferred_content_behavior]]
|
|
|
|
Joey has received funding to work on this.
|
|
Planned schedule of work:
|
|
|
|
* June: git-annex proxy
|
|
* July, part 1: git-annex proxy support for exporttree
|
|
* July, part 2: p2p protocol over http
|
|
* August: balanced preferred content
|
|
* September: streaming through proxy to special remotes (especially S3)
|
|
* October: proving behavior of balanced preferred content with proxies
|
|
|
|
[[!tag projects/openneuro]]
|
|
|
|
# work notes
|
|
|
|
In development on the `proxy` branch.
|
|
|
|
For June's work on [[design/passthrough_proxy]], remaining todos:
|
|
|
|
* `git-annex sync` etc should not treat clusters as git syncable remotes.
|
|
|
|
* `git-annex sync` etc, when operating on clusters, should first
|
|
operate on the cluster as a whole, to take advantages of fanout on upload
|
|
and mass drop. Only operate on individual cluster nodes afterwards,
|
|
to handle cases such as a cluster containing a key, but some node
|
|
wanting and lacking the key. Perhaps just setting cost for nodes slightly
|
|
higher than the cluster cost will be enough?
|
|
|
|
* On upload to cluster, send to nodes where it's preferred content, and not
|
|
to other nodes.
|
|
|
|
* Getting a key from a cluster currently always selects the lowest cost
|
|
remote, and always the same remote if cost is the same. Should
|
|
round-robin amoung remotes, and prefer to avoid using remotes that
|
|
other git-annex processes are currently using.
|
|
|
|
* Support annex.jobs for clusters.
|
|
|
|
* On upload to a cluster, as well as fanout to nodes, if the key is
|
|
preferred content of the proxy repository, store it there.
|
|
(But not when preferred content is not configured.)
|
|
And on download from a cluster, if the proxy repository has the content,
|
|
get it from there to avoid the overhead of proxying to a node.
|
|
|
|
* Basic proxying to special remote support (non-streaming).
|
|
|
|
* Support proxies-of-proxies better, eg foo-bar-baz.
|
|
Currently, it does work, but have to run `git-annex updateproxy`
|
|
on foo in order for it to notice the bar-baz proxied remote exists,
|
|
and record it as foo-bar-baz. Make it skip recording proxies of
|
|
proxies like that, and instead automatically generate those from the log.
|
|
(With cycle prevention there of course.)
|
|
|
|
* Cycle prevention including cluster-in-cluster cycles. See design.
|
|
|
|
* Optimise proxy speed. See design for ideas.
|
|
|
|
* Use `sendfile()` to avoid data copying overhead when
|
|
`receiveBytes` is being fed right into `sendBytes`.
|
|
|
|
* Encryption and chunking. See design for issues.
|
|
|
|
* Indirect uploads (to be considered). See design.
|
|
|
|
* Support using a proxy when its url is a P2P address.
|
|
(Eg tor-annex remotes.)
|
|
|
|
* `viconfig` support for setting preferred content, group,
|
|
and description of clusters
|
|
|
|
|
|
# completed items for June's work on [[design/passthrough_proxy]]:
|
|
|
|
* UUID discovery via git-annex branch. Add a log file listing UUIDs
|
|
accessible via proxy UUIDs. It also will contain the names
|
|
of the remotes that the proxy is a proxy for,
|
|
from the perspective of the proxy. (done)
|
|
|
|
* Add `git-annex updateproxy` command (done)
|
|
|
|
* Remote instantiation for proxies. (done)
|
|
|
|
* Implement git-annex-shell proxying to git remotes. (done)
|
|
|
|
* Proxy should update location tracking information for proxied remotes,
|
|
so it is available to other users who sync with it. (done)
|
|
|
|
* Implement `git-annex updatecluster` command (done)
|
|
|
|
* Implement cluster UUID insertation on location log load, and removal
|
|
on location log store. (done)
|
|
|
|
* Omit cluster UUIDs when constructing drop proofs, since lockcontent will
|
|
always fail on a cluster. (done)
|
|
|
|
* Don't count cluster UUID as a copy in numcopies checking etc. (done)
|
|
|
|
* Tab complete proxied remotes and clusters in eg --from option. (done)
|
|
|
|
* Getting a key from a cluster should proxy from one of the nodes that has
|
|
it. (done)
|
|
|
|
* Implement upload with fanout to multiple cluster nodes and reporting back
|
|
additional UUIDs over P2P protocol. (done)
|
|
|
|
* Implement cluster drops, trying to remove from all nodes, and returning
|
|
which UUIDs it was dropped from. (done)
|
|
|
|
* `git-annex testremote` works against proxied remote and cluster. (done)
|