copied over some changes from proxy branch

doc/design/passthrough_proxy.mdwn

@@ -15,7 +15,7 @@ existing remotes to keep up with changes are made on the server side.
 A proxy would avoid this complexity. It also allows limiting network
 ingress to a single point.
 
-Ideally a proxy would look like any other git-annex remote. All the files
+A proxy can be the frontend to a cluster. All the files
 stored anywhere in the cluster would be available to retrieve from the
 proxy. When a file is sent to the proxy, it would store it somewhere in the
 cluster.
@@ -108,55 +108,169 @@ The only real difference seems to be that the UUID of a remote is cached,
 so A could only do this the first time we accessed it, and not later.
 With UUID discovery, A can do that at any time.
 
-## user interface
+## proxied remote names
 
 What to name the instantiated remotes? Probably the best that could
 be done is to use the proxy's own remote names as suffixes on the client.
 Eg, the proxy's "node1" remote is "proxy-node1".
 
-But the user probably doesn't want to pick which node to send content to.
+But, the user might have their own "proxy-node1" remote configured that
-They don't necessarily know anything about the nodes. Ideally the user
+points to something else. To avoid a proxy changing the configuration of
-would `git-annex copy --to proxy` or `git-annex push` and let it pick
+the user's remote to point to its remote, git-annex must avoid
-which instantiated remote(s) to send to.
+instantiating a proxied remote when there's already a configuration for a
+remote with that same name.
 
-To make `git-annex copy --to proxy` work, `storeKey` could be changed to
+That does mean that, if a user wants to set a git config for a proxy
-allow returning a UUID (or UUIDs) where the content was actually stored.
+remote, they will need to manually set its annex-uuid and its url.
-That would also allow a single upload to the proxy to fan out and be stored
+Which is awkward. Many git configs of the proxy remote can be inherited by
-in multiple nodes. The proxy would use preferred content to pick which of
+the instantiated remotes, so users won't often need to do that.
-its nodes to store on.
 
-Instantiated remotes would still be needed for `git-annex get` and similar
+A user can also set up a remote with another name that they
-to work.
+prefer, that points at a remote behind a proxy. They just need to set
+its annex-uuid and its url. Perhaps there should be a git-annex command
+that eases setting up a remote like that?
 
-To make `git-annex copy --from proxy` work, the proxy would need to pick
+## proxied remotes in git remote list
-a node and stream content from it. That's doable, but how to handle a case
-where a node gets corrupted? The best it could do is mark that node as no
-longer containing the content (as if a fsck failed) and try another one
-next time. This complication might not be necessary. Consider that
-while `git-annex copy --to foo` followed later by `git-annex copy --from foo`
-will usually work, it doesn't work when eg first copying to a transfer
-remote, which then sends the content elsewhere and drops its copy.
 
-What about dropping? `git-annex drop --from proxy` could be made to work,
+Should instantiated remotes have enough configured in git so that
-by having `removeKey` return a list of UUIDs that the content was dropped
+`git remote list` will list them? This would make things like tab
-from. What should that do if it's able to drop from some nodes but not
+completion of proxied remotes work, and would generally let the user
-others? Perhaps it would need to be able to return a list of UUIDs that
+discover that there *are* proxied remotes.
-content was dropped from but still indicate it overall failed to drop.
+
-(Note that it's entirely possible that dropping from one node of the proxy
+This could be done by a config like remote.name.annex-proxied = true.
-involves lockContent on another node of the proxy in order to satisfy
+That makes other configs of the remote not prevent it being used as an
-numcopies.)
+instantiated remote. So remote.name.annex-uuid can be changed when
+the uuid behind a proxy changes. And it allows updating remote.name.url
+to keep it the same as the proxy remote's url. (Or possibly to set it to
+something else?)
+
+Configuring the instantiated remotes like that would let anyone who can
+write to the git-annex branch flood other people's repos with configs
+for any number of git remotes. Which might be obnoxious.
+
+## single upload with fanout
+
+If we want to send a file to multiple repositories that are behind the same
+proxy, it would be wasteful to upload it through the proxy repeatedly.
+
+Perhaps a good user interface to this is `git-annex copy --to proxy`.
+The proxy could fan out the upload and store it in one or more nodes behind
+it. Using preferred content to select which nodes to use.
+This would need `storeKey` to be changed to allow returning a UUID (or UUIDs)
+where the content was actually stored.
+
+Alternatively, `git-annex copy --to proxy-foo` could notice that proxy-bar
+also wants the content, and fan out a copy to there. Then it could 
+record in its git-annex branch that the content is present in proxy-bar.
+If the user later does `git-annex copy --to proxy-bar`, it would avoid
+another upload (and the user would learn at that point that it was in
+proxy-bar). This avoids needing to change the `storeKey` interface.
+
+Should a proxy always fanout? if `git-annex copy --to proxy` is what does
+fanout, and `git-annex copy --to proxy-foo` doesn't, then the user has
+content. But if the latter does fanout, that might be annoying to users who
+want to use proxies, but want full control over what lands where, and don't
+want to use preferred content to do it. So probably fanout should be
+configurable. But it can't be configured client side, because the fanout
+happens on the proxy. Seems like remote.name.annex-fanout could be set to
+false to prevent fanout to a specific remote. (This is analagous to a
+remote having `git-annex assistant` running on it, it might fan out uploads
+to it to other repos, and only the owner of that repo can control it.)
 
 A command like `git-annex push` would see all the instantiated remotes and
-would pick one to send content to. Seems like the proxy might choose to
+would pick ones to send content to. If the proxy does fanout, this would
-`storeKey` the content on other node(s) than the requested one. Which would
+lead to `git-annex push` doing extra work iterating over instantiated
-be fine. But, `git-annex push` would still do considerable extra work in
+remotes that have already received content via fanout. Could this extra
-iterating over all the instantiated remotes. So it might be better to make
+work be avoided?
-such commands not operate on instantiated remotes for sending content but
-only on the proxy. 
 
-Commands like `git-annex push` and `git-annex pull`
+## clusters
-should also skip the instantiated remotes when pushing or pulling the git
+
-repo, because that would be extra work that accomplishes nothing.
+One way to use a proxy is just as a convenient way to access a group of
+remotes that are behind it. Some remotes may only be reachable by the
+proxy, but you still know what the individual remotes are. Eg, one might be
+a S3 bucket that can only be written via the proxy, but is globally
+readable without going through the proxy. Another might be a drive that is
+sometimes located behind the proxy, but other times connected directly.
+Using a proxy this way just involves using the instantiated proxied remotes.
+
+Or a proxy can be the frontend for a cluster. In this situation, the user
+doesn't know anything much about the nodes in the cluster, perhaps not even
+that they exist, or perhaps what keys are stored on which nodes.
+
+In the cluster case, the user would like to not need to pick a specific
+node to send content to. While they could use preferred content to pick a
+node, or nodes, they would prefer to be able to say `git-annex copy --to cluster` 
+and let it pick which nodes to send to. And similarly,
+`git-annex drop --from cluster' should drop the content from every node in
+the cluster.
+
+For this we need a UUID for the cluster. But it is not like a usual UUID.
+It does not need to actually be recorded in the location tracking logs, and
+it is not counted as a copy for numcopies purposes. The only point of this
+UUID is to make commands like `git-annex drop --from cluster` and
+`git-annex get --from cluster` talk to the cluster's frontend proxy, which
+has as its UUID the cluster's UUID.
+
+The cluster UUID is recorded in the git-annex branch, along with a list of
+the UUIDs of nodes of the cluster (which can change at any time).
+
+When reading a location log, if any UUID where content is present is part
+of the cluster, the cluster's UUID is added to the list of UUIDs.
+
+When writing a location log, the cluster's UUID is filtered out of the list
+of UUIDs.
+
+The cluster's frontend proxy fans out uploads to nodes according to
+preferred content. And `storeKey` is extended to be able to return a list
+of additional UUIDs where the content was stored. So an upload to the
+cluster will end up writing to the location log the actual nodes that it
+was fanned out to. 
+
+Note that to support clusters that are nodes of clusters, when a cluster's
+frontend proxy fans out an upload to a node, and `storeKey` returns
+additional UUIDs, it should pass those UUIDs along. Of course, no cluster
+can be a node of itself, and cycles have to be broken (as described in a
+section below).
+
+When a file is requested from the cluster's frontend proxy, it can send its
+own local copy if it has one, but otherwise it will proxy to one of its
+nodes. (How to pick which node to use? Load balancing?) This behavior will
+need to be added to git-annex-shell, and to Remote.Git for local paths to a
+cluster.
+
+The cluster's frontend proxy also fans out drops to all nodes, attempting
+to drop content from the whole cluster, and only indicating success if it
+can. Also needs changes to git-annex-sjell and Remote.Git.
+
+It does not fan out lockcontent, instead the client will lock content
+on specific nodes. In fact, the cluster UUID should probably be omitted
+when constructing a drop proof, since trying to lockcontent on it will
+usually fail.
+
+Some commands like `git-annex whereis` will list content as being stored in
+the cluster, as well as on whicheven of its nodes, and whereis currently
+says "n copies", but since the cluster doesn't count as a copy, that
+display should probably be counted using the numcopies logic that excludes
+cluster UUIDs.
+
+No other protocol extensions or special cases should be needed. Except for
+the strange case of content stored in the cluster's frontend proxy.
+
+Running `git-annex fsck --fast` on the cluster's frontend proxy will look
+weird: For each file, it will read the location log, and if the file is
+present on any node it will add the frontend proxy's UUID. So fsck will
+expect the content to be present. But it probably won't be. So it will fix
+the location log... which will make no changes since the proxy's UUID will
+be filtered out on write. So probably fsck will need a special case to
+avoid this behavior. (Also for `git-annex fsck --from cluster --fast`)
+
+And if a key does get stored on the cluster's frontend proxy, it will not
+be possible to tell from looking at the location log that the content is
+really present there. So that won't be counted as a copy. In some cases,
+a cluster's frontend proxy may want to keep files, perhaps some files are
+worth caching there for speed. But if a file is stored only on the
+cluster's frontend proxy and not in any of its nodes, clients will not
+consider the cluster to contain the file at all.
 
 ## speed
 
@@ -246,6 +360,23 @@ in front of the proxy.
 
 ## cycles
 
+A repo can advertise that it proxies for a repo which has the same uuid as
+itself. Or there can be a larger cycle involving a proxy that proxies to a
+proxy, etc.
+
+Since the proxied repo uuid is communicated to git-annex-shell via 
+--uuid, a repo that advertises proxying for itself will be connected to
+with its own uuid. No proxying is done in this case. Same happens with a
+larger cycle.
+
+Instantiating remotes needs to identity cycles and break them. Otherwise
+it would construct an infinite number of proxied remotes with names
+like "foo-foo-foo-foo-..." or "foo-bar-foo-bar-..."
+
+Once `git-annex copy --to proxy` is implemented, and the proxy decides
+where to send content that is being sent directly to it, cycles will
+become an issue with that as well.
+
 What if repo A is a proxy and has repo B as a remote. Meanwhile, repo B is
 a proxy and has repo A as a remote?
 
@@ -259,7 +390,7 @@ remote that is not part of a cycle, they could deposit the upload there and
 the upload still succeed. Otherwise the upload would fail, which is
 probably the best that can be done with such a broken configuration.
 
-So, it seems like proxies will need to take transfer locks for uploads,
+So, it seems like proxies would need to take transfer locks for uploads,
 even though the content is being proxied to elsewhere.
 
 Dropping could have similar cycles with content presence locking, which

doc/todo/git-annex_proxies.mdwn

@@ -26,52 +26,45 @@ In development on the `proxy` branch.
 
 For June's work on [[design/passthrough_proxy]], implementation plan:
 
-1. UUID discovery via git-annex branch. Add a log file listing UUIDs
+* UUID discovery via git-annex branch. Add a log file listing UUIDs
   accessible via proxy UUIDs. It also will contain the names
   of the remotes that the proxy is a proxy for, 
   from the perspective of the proxy. (done)
 
-1. Add `git-annex updateproxy` command and remote.name.annex-proxy
+* Add `git-annex updateproxy` command and remote.name.annex-proxy
   configuration. (done)
 
-2. Remote instantiation for proxies. (done)
+* Remote instantiation for proxies. (done)
 
-2. Bug: In a repo cloned with ssh from a proxy repo,
+* Implement git-annex-shell proxying to git remotes. (done)
-   running `git-annex init` sets annex-uuid for the instantiated remotes.
-   This prevents them being used, because instanatiation is not done
-   when there's any config set for a remote.
 
-3. Implement proxying in git-annex-shell.
+* Proxy should update location tracking information for proxied remotes,
-   (Partly done, still need it for GET, PUT, CONNECT, and NOTIFYCHANGES
+  so it is available to other users who sync with it. (done)
-   messages.)
 
-4. Either implement proxying for local path remotes, or prevent
+* Consider getting instantiated remotes into git remote list.
-   listProxied from operating on them.
+  See design.
 
-4. Either implement proxying for tor-annex remotes, or prevent
+* Implement single upload with fanout to proxied remotes.
-   listProxied from operating on them.
 
-4. Let `storeKey` return a list of UUIDs where content was stored,
+* Implement clusters.
-   and make proxies accept uploads directed at them, rather than a specific
-   instantiated remote, and fan out the upload to whatever nodes behind
-   the proxy want it. This will need P2P protocol extensions.
 
-5. Make `git-annex copy --from $proxy` pick a node that contains each
+* Support proxies-of-proxies better, eg foo-bar-baz.
-   file, and use the instantiated remote for getting the file. Same for
+  Currently, it does work, but have to run `git-annex updateproxy`
-   similar commands.
+  on foo in order for it to notice the bar-baz proxied remote exists,
+  and record it as foo-bar-baz. Make it skip recording proxies of
+  proxies like that, and instead automatically generate those from the log.
+  (With cycle prevention there of course.)
 
-6. Make `git-annex drop --from $proxy` drop, when possible, from every
+* Cycle prevention. See design.
-   remote accessible by the proxy. Communicate partial drops somehow.
 
-7. Make commands like `git-annex push` not iterate over instantiate
+* Optimise proxy speed. See design for ideas.
-   remotes, and instead just send content to the proxy for fanout.
 
-8. Optimise proxy speed. See design for idea.
+* Use `sendfile()` to avoid data copying overhead when
+  `receiveBytes` is being fed right into `sendBytes`.
 
-9. Encryption and chunking. See design for issues.
+* Encryption and chunking. See design for issues.
 
-10. Cycle prevention. See design.
+* Indirect uploads (to be considered). See design.
 
-11. indirect uploads (to be considered). See design.
+* Support using a proxy when its url is a P2P address.
-
+  (Eg tor-annex remotes.)
-12. Support using a proxy when its url is a P2P address.