mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions
git-annex/doc/todo/git-annex_proxies.mdwn

265 lines
10 KiB
Text
Raw Normal View History

add todo linking to two design docs and some related todos Tagging with projects/openneuro as Christopher Markiewicz has oked them funding at least the initial design work on this.
2024-05-01 15:04:20 +00:00
This is a summary todo covering several subprojects, which would extend
git-annex to be able to use proxies which sit in front of a cluster of
repositories.
number list
2024-05-01 16:19:12 +00:00
1. [[design/passthrough_proxy]]
started on a design for P2P protocol over HTTP Added to git-annex_proxies todo because this is something OpenNeuro would need in order to use the git-annex proxy. Sponsored-by: Dartmouth College's OpenNeuro project
2024-05-01 19:26:51 +00:00
2. [[design/p2p_protocol_over_http]]
3. [[design/balanced_preferred_content]]
4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
5. [[todo/proving_preferred_content_behavior]]
add todo linking to two design docs and some related todos Tagging with projects/openneuro as Christopher Markiewicz has oked them funding at least the initial design work on this.
2024-05-01 15:04:20 +00:00
update
2024-07-01 15:38:29 +00:00
## table of contents
[[!toc ]]
## planned schedule
recieved funding to work on this, which comes with a schedule
2024-06-04 10:53:59 +00:00
Joey has received funding to work on this.
Planned schedule of work:
update
2024-06-27 19:28:10 +00:00
* June: git-annex proxies and clusters
fix negative DATA when 1 node of a cluster has a partial transfer
2024-07-30 15:42:17 +00:00
* July: p2p protocol over http
* August, part 1: git-annex proxy support for exporttree
update
2024-08-30 15:14:45 +00:00
* August, part 2: balanced preferred content
update
2024-10-28 17:46:57 +00:00
* September: proving behavior of balanced preferred content with proxies
* October: streaming through proxy to special remotes (especially S3)
recieved funding to work on this, which comes with a schedule
2024-06-04 10:53:59 +00:00
add todo linking to two design docs and some related todos Tagging with projects/openneuro as Christopher Markiewicz has oked them funding at least the initial design work on this.
2024-05-01 15:04:20 +00:00
[[!tag projects/openneuro]]
implementation plan
2024-06-04 11:51:33 +00:00
update
2024-10-28 17:46:57 +00:00
## remaining things to do in October
implementation plan
2024-06-04 11:51:33 +00:00
planning
2024-10-22 15:09:47 +00:00
* Streaming uploads to special remotes via the proxy. Possibly; if a
workable design can be developed. It seems difficult without changing the
external special remote protocol, unless a fifo is used. Make ORDERED
response in p2p protocol allow using a fifo?
update
2024-10-15 14:31:42 +00:00
planning
2024-10-22 15:09:47 +00:00
* Indirect uploads when proxying for special remote is an alternative that
would work for OpenNeuro's use case.
* If not implementing upload streaming to proxied special remotes,
this needs to be addressed:
When an upload to a cluster is distributed to multiple special remotes,
a temporary file is written for each one, which may even happen in
parallel. This is a lot of extra work and may use excess disk space.
It should be possible to only write a single temp file.
(With streaming this wouldn't be an issue.)
tried a blind alley on streaming special remote download via proxy This didn't work. In case I want to revisit, here's what I tried. diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs index 48222872c1..e4e526d3dd 100644 --- a/Annex/Proxy.hs +++ b/Annex/Proxy.hs @@ -26,16 +26,21 @@ import Logs.UUID import Logs.Location import Utility.Tmp.Dir import Utility.Metered +import Utility.ThreadScheduler +import Utility.OpenFd import Git.Types import qualified Database.Export as Export import Control.Concurrent.STM import Control.Concurrent.Async +import Control.Concurrent.MVar import qualified Data.ByteString as B +import qualified Data.ByteString as BS import qualified Data.ByteString.Lazy as L import qualified System.FilePath.ByteString as P import qualified Data.Map as M import qualified Data.Set as S +import System.IO.Unsafe proxyRemoteSide :: ProtocolVersion -> Bypass -> Remote -> Annex RemoteSide proxyRemoteSide clientmaxversion bypass r @@ -240,21 +245,99 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv oclosedv mexportdb = go writeVerifyChunk iv h b storetofile iv h (n - fromIntegral (B.length b)) bs - proxyget offset af k = withproxytmpfile k $ \tmpfile -> do + proxyget offset af k = withproxytmpfile k $ \tmpfile -> + let retrieve = tryNonAsync $ Remote.retrieveKeyFile r k af + (fromRawFilePath tmpfile) nullMeterUpdate vc + in case fromKey keySize k of + Just size | size > 0 -> do + cancelv <- liftIO newEmptyMVar + donev <- liftIO newEmptyMVar + streamer <- liftIO $ async $ + streamdata offset tmpfile size cancelv donev + retrieve >>= \case + Right _ -> liftIO $ do + putMVar donev () + wait streamer + Left err -> liftIO $ do + putMVar cancelv () + wait streamer + propagateerror err + _ -> retrieve >>= \case + Right _ -> liftIO $ senddata offset tmpfile + Left err -> liftIO $ propagateerror err + where -- Don't verify the content from the remote, -- because the client will do its own verification. - let vc = Remote.NoVerify - tryNonAsync (Remote.retrieveKeyFile r k af (fromRawFilePath tmpfile) nullMeterUpdate vc) >>= \case - Right _ -> liftIO $ senddata offset tmpfile - Left err -> liftIO $ propagateerror err + vc = Remote.NoVerify + streamdata (Offset offset) f size cancelv donev = do + sendlen offset size + waitforfile + x <- tryNonAsync $ do + fd <- openFdWithMode f ReadOnly Nothing defaultFileFlags + h <- fdToHandle fd + hSeek h AbsoluteSeek offset + senddata' h (getcontents size) + case x of + Left err -> do + throwM err + Right res -> return res + where + -- The file doesn't exist at the start. + -- Wait for some data to be written to it as well, + -- in case an empty file is first created and then + -- overwritten. When there is an offset, wait for + -- the file to get that large. Note that this is not used + -- when the size is 0. + waitforfile = tryNonAsync (fromIntegral <$> getFileSize f) >>= \case + Right sz | sz > 0 && sz >= offset -> return () + _ -> ifM (isEmptyMVar cancelv) + ( do + threadDelaySeconds (Seconds 1) + waitforfile + , do + return () + ) + + getcontents n h = unsafeInterleaveIO $ do + isdone <- isEmptyMVar donev <||> isEmptyMVar cancelv + c <- BS.hGet h defaultChunkSize + let n' = n - fromIntegral (BS.length c) + let c' = L.fromChunks [BS.take (fromIntegral n) c] + if BS.null c + then if isdone + then return mempty + else do + -- Wait for more data to be + -- written to the file. + threadDelaySeconds (Seconds 1) + getcontents n h + else if n' > 0 + then do + -- unsafeInterleaveIO causes + -- this to be deferred until + -- data is read from the lazy + -- ByteString. + cs <- getcontents n' h + return $ L.append c' cs + else return c' + senddata (Offset offset) f = do size <- fromIntegral <$> getFileSize f - let n = max 0 (size - offset) - sendmessage $ DATA (Len n) + sendlen offset size withBinaryFile (fromRawFilePath f) ReadMode $ \h -> do hSeek h AbsoluteSeek offset - sendbs =<< L.hGetContents h + senddata' h L.hGetContents + + senddata' h getcontents = do + sendbs =<< getcontents h -- Important to keep the handle open until -- the client responds. The bytestring -- could still be lazily streaming out to @@ -272,6 +355,11 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv oclosedv mexportdb = go Just FAILURE -> return () Just _ -> giveup "protocol error" Nothing -> return () + + sendlen offset size = do + let n = max 0 (size - offset) + sendmessage $ DATA (Len n) + {- Check if this repository can proxy for a specified remote uuid, - and if so enable proxying for it. -}
2024-10-07 17:54:04 +00:00
update
2024-10-28 17:46:57 +00:00
* Possibly some of the deferred items listed in following sections:
update
2024-09-25 18:26:32 +00:00
update
2024-08-30 15:14:45 +00:00
## items deferred until later for balanced preferred content and maxsize tracking
locking in checkLiveUpdate This makes sure that two threads don't check balanced preferred content at the same time, so each thread always sees a consistent picture of what is happening. This does add a fairly expensive file level lock to every check of preferred content, in commands that use prepareLiveUpdate. It would be good to only do that when live updates are actually needed, eg when the preferred content expression uses balanced preferred content.
2024-08-27 17:07:06 +00:00
punt on LiveUpdate plumbing through assistant for now
2024-08-24 13:37:24 +00:00
* The assistant is using NoLiveUpdate, but it should be posssible to plumb
a LiveUpdate through it from preferred content checking to location log
updating.
plumb in LiveUpdate (WIP) Each command that first checks preferred content (and/or required content) and then does something that can change the sizes of repositories needs to call prepareLiveUpdate, and plumb it through the preferred content check and the location log update. So far, only Command.Drop is done. Many other commands that don't need to do this have been updated to keep working. There may be some calls to NoLiveUpdate in places where that should be done. All will need to be double checked. Not currently in a compilable state.
2024-08-23 20:35:12 +00:00
consistently don't show sizes of empty repositories This used to be the case, and when matching options are used, that code path still omits them, so also omit them in the getRepoSize code path.
2024-08-17 18:58:36 +00:00
* `git-annex info` in the limitedcalc path in cachedAllRepoData
double-counts redundant information from the journal due to using
combine 2 queries
2024-08-28 15:00:59 +00:00
overLocationLogs. In the other path it does not (any more; it used to),
and this should be fixed for consistency and correctness.
consistently don't show sizes of empty repositories This used to be the case, and when matching options are used, that code path still omits them, so also omit them in the getRepoSize code path.
2024-08-17 18:58:36 +00:00
started work on getLiveRepoSizes Doesn't quite compile
2024-08-26 18:50:09 +00:00
* getLiveRepoSizes has a filterM getRecentChange over the live updates.
This could be optimised to a single sql join. There are usually not many
live updates, but sometimes there will be a great many recent changes,
update
2024-08-30 15:14:45 +00:00
so it might be worth doing this optimisation. Persistent is not capable
of this, would need dependency added on esquelito.
started work on getLiveRepoSizes Doesn't quite compile
2024-08-26 18:50:09 +00:00
update
2024-07-28 18:22:44 +00:00
## items deferred until later for p2p protocol over http
documentation for p2phttp
2024-07-28 21:19:27 +00:00
* `git-annex p2phttp` should support serving several repositories at the same
time (not as proxied remotes), so that eg, every git-annex repository
on a server can be served on the same port.
idea
2024-07-29 13:11:27 +00:00
* Support proxying to git remotes that use annex+http urls. This needs a
translation from P2P protocol to servant-client to P2P protocol.
* Should be possible to use a git-remote-annex annex::$uuid url as
remote.foo.url with remote.foo.annexUrl using annex+http, and so
not need a separate web server to serve the git repository. Doesn't work
currently because git-remote-annex urls only support special remotes.
It would need a new form of git-remote-annex url, eg:
annex::$uuid?annex+http://example.com/git-annex/
update
2024-07-28 18:22:44 +00:00
* `git-annex p2phttp` could support systemd socket activation. This would
allow making a systemd unit that listens on port 80.
update
2024-07-02 20:16:37 +00:00
## items deferred until later for [[design/passthrough_proxy]]
dropping from clusters Dropping from a cluster drops from every node of the cluster. Including nodes that the cluster does not think have the content. This is different from GET and CHECKPRESENT, which do trust the cluster's location log. The difference is that removing from a cluster should make 100% the content is gone from every node. So doing extra work is ok. Compare with CHECKPRESENT where checking every node could make it very expensive, and the worst that can happen in a false negative is extra work being done. Extended the P2P protocol with FAILURE-PLUS to handle the case where a drop from one node succeeds, but a drop from another node fails. In that case the entire cluster drop has failed. Note that SUCCESS-PLUS is returned when dropping from a proxied remote that is not a cluster, when the protocol version supports it. This is because P2P.Proxy does not know when it's proxying for a single node cluster vs for a remote that is not a cluster.
2024-06-23 13:28:18 +00:00
update
2024-07-01 15:29:04 +00:00
* Check annex.diskreserve when proxying for special remotes
to avoid the proxy's disk filling up with the temporary object file
cached there.
GET from proxied special remote Working, but lots of room for improvement... Without streaming, so there is a delay before download begins as the file is retreived from the special remote. And when resuming it retrieves the whole file from the special remote *again*. Also, if the special remote throws an exception, currently it shows as "protocol error".
2024-06-28 19:32:00 +00:00
* Resuming an interrupted download from proxied special remote makes the proxy
re-download the whole content. It could instead keep some of the
object files around when the client does not send SUCCESS. This would
planning
2024-10-22 15:09:47 +00:00
use more disk, but could minimize to eg, the last 2 or so.
PUT to proxied special remote working Still needs some work. The reason that the waitv is necessary is because without it, runNet loops back around and reads the next protocol message. But it's not finished reading the whole bytestring yet, and so it reads some part of it.
2024-06-28 21:07:01 +00:00
The design doc has some more thoughts about this.
P2P protocol version 2, adding SUCCESS-PLUS and ALREADY-HAVE-PLUS Client side support for SUCCESS-PLUS and ALREADY-HAVE-PLUS is complete, when a PUT stores to additional repositories than the expected on, the location log is updated with the additional UUIDs that contain the content. Started implementing PUT fanout to multiple remotes for clusters. It is untested, and I fear fencepost errors in the relative offset calculations. And it is missing proxying for the protocol after DATA.
2024-06-18 16:07:01 +00:00
GET and CHECKPRESENT amoung lowest cost cluster nodes Before it was using a node that might have had a higher cost. Also threw in a random selection from amoung the low cost nodes. Of course this is a poor excuse for load balancing, but it's better than nothing. Most of the time...
2024-06-27 18:36:55 +00:00
* Getting a key from a cluster currently picks from amoung
the lowest cost remotes at random. This could be smarter,
eg prefer to avoid using remotes that are doing other transfers at the
same time.
give proxied cluster nodes a higher cost than the cluster gateway This makes eg git-annex get default to using the cluster rather than an arbitrary node, which is better UI. The actual cost of accessing a proxied node vs using the cluster is basically the same. But using the cluster allows smarter load-balancing to be done on the cluster.
2024-06-27 19:21:03 +00:00
* The cost of a proxied node that is accessed via an intermediate gateway
is currently the same as a node accessed via the cluster gateway.
To fix this, there needs to be some way to tell how many hops through
gateways it takes to get to a node. Currently the only way is to
guess based on number of dashes in the node name, which is not satisfying.
Even counting hops is not very satisfying, one cluster gateway could
be much more expensive to traverse than another one.
If seriously tackling this, it might be worth making enough information
available to use spanning tree protocol for routing inside clusters.
improve docs
2024-06-25 21:50:22 +00:00
fix http proxying for a local git remote with a relative path git-annex-shell expects an absolute path
2024-07-28 17:31:30 +00:00
* Speed: A proxy to a local git repository spawns git-annex-shell
to communicate with it. It would be more efficient to operate
directly on the Remote. Especially when transferring content to/from it.
But: When a cluster has several nodes that are local git repositories,
and is sending data to all of them, this would need an alternate
interface than `storeKey`, which supports streaming, of chunks
of a ByteString.
updates
2024-06-12 15:55:18 +00:00
* Use `sendfile()` to avoid data copying overhead when
`receiveBytes` is being fed right into `sendBytes`.
update
2024-06-25 21:26:26 +00:00
Library to use:
<https://hackage.haskell.org/package/hsyscall-0.4/docs/System-Syscall.html>
implementation plan
2024-06-04 11:51:33 +00:00
updates
2024-06-12 15:55:18 +00:00
* Support using a proxy when its url is a P2P address.
(Eg tor-annex remotes.)
update
2024-06-23 16:31:00 +00:00
update
2024-10-28 17:46:57 +00:00
## completed items for October's work on streaming through proxy to special remotes
* Stream downloads through proxy for all special remotes that indicate
they download in order.
* Added ORDERED message to external special remote protocol.
## completed items for September's work on proving behavior of preferred content
* Static analysis to detect "not present", "not balanced", and similar
unstable preferred content expressions and avoid problems with them.
* Implemented `git-annex sim` command.
* Simulated a variety of repository networks, and random preferred content
expressions, checking that a stable state is always reached.
* Fix bug that prevented anything being stored in an empty
repository whose preferred content expression uses sizebalanced.
(Identified via `git-annex sim`)
## completed items for August's work on balanced preferred content
* Balanced preferred content basic implementation, including --rebalance
option.
* Implemented [[track_free_space_in_repos_via_git-annex_branch]]
* Implemented tracking of live changes to repository sizes.
* `git-annex maxsize`
* annex.fullybalancedthreshhold
## completed items for August's work on git-annex proxy support for exporttre
* Special remotes configured with exporttree=yes annexobjects=yes
can store objects in .git/annex/objects, as well as an exported tree.
* Support proxying to special remotes configured with
exporttree=yes annexobjects=yes.
* post-retrieve: When proxying is enabled for an exporttree=yes
special remote and the configured remote.name.annex-tracking-branch
is received, the tree is exported to the special remote.
* When getting from a P2P HTTP remote, prompt for credentials when
required, instead of failing.
* Prevent `updateproxy` and `updatecluster` from adding
an exporttree=yes special remote that does not have
annexobjects=yes, to avoid foot shooting.
* Implement `git-annex export treeish --to=foo --from=bar`, which
gets from bar as needed to send to foo. Make post-retrieve use
`--to=r --from=r` to handle the multiple files case.
## completed items for July's work on p2p protocol over http
* HTTP P2P protocol design [[design/p2p_protocol_over_http]].
* addressed [[doc/todo/P2P_locking_connection_drop_safety]]
* implemented server and client for HTTP P2P protocol
* added git-annex p2phttp command to serve HTTP P2P protocol
* Make git-annex p2phttp support https.
* Allow using annex+http urls in remote.name.annexUrl
* Make http server support proxying.
* Make http server support serving a cluster.
update
2024-07-01 15:38:29 +00:00
## completed items for June's work on [[design/passthrough_proxy]]:
update
2024-06-23 20:38:01 +00:00
* UUID discovery via git-annex branch. Add a log file listing UUIDs
accessible via proxy UUIDs. It also will contain the names
of the remotes that the proxy is a proxy for,
from the perspective of the proxy. (done)
* Add `git-annex updateproxy` command (done)
* Remote instantiation for proxies. (done)
* Implement git-annex-shell proxying to git remotes. (done)
* Proxy should update location tracking information for proxied remotes,
so it is available to other users who sync with it. (done)
update
2024-06-27 19:28:10 +00:00
* Implement `git-annex initcluster` and `git-annex updatecluster` commands (done)
update
2024-06-23 20:38:01 +00:00
* Implement cluster UUID insertation on location log load, and removal
on location log store. (done)
* Omit cluster UUIDs when constructing drop proofs, since lockcontent will
always fail on a cluster. (done)
* Don't count cluster UUID as a copy in numcopies checking etc. (done)
* Tab complete proxied remotes and clusters in eg --from option. (done)
* Getting a key from a cluster should proxy from one of the nodes that has
it. (done)
* Implement upload with fanout to multiple cluster nodes and reporting back
additional UUIDs over P2P protocol. (done)
* Implement cluster drops, trying to remove from all nodes, and returning
which UUIDs it was dropped from. (done)
* `git-annex testremote` works against proxied remote and cluster. (done)
don't sync with cluster nodes by default Avoid `git-annex sync --content` etc from operating on cluster nodes by default since syncing with a cluster implicitly syncs with its nodes. This avoids a lot of unncessary work when a cluster has a lot of nodes just in checking if each node's preferred content is satisfied. And it avoids content being sent to nodes individually, so instead syncing with clusters always fanout uploads to nodes. The downside is that there are situations where a cluster's preferred content settings can be met, but those of its nodes are not. Or where a node does not contain a key, but the cluster does, and there are not enough copies of the key yet, so it would be desirable the send it there. I think that's an acceptable tradeoff. These kind of situations are ones where the cluster itself should probably be responsible for copying content to the node. Which it can do much less expensively than a client can. Part of the balanced preferred content design that I will be working on in a couple of months involves rebalancing clusters, so I expect to revisit this. The use of annex-sync config does allow running git-annex sync with a specific node, or nodes, and it will sync with it. And it's also possible to set annex-sync git configs to make it sync with a node by default. (Although that will require setting up an explicit git remote for the node rather than relying on the proxied remote.) Logs.Cluster.Basic is needed because Remote.Git cannot import Logs.Cluster due to a cycle. And the Annex.Startup load of clusters happens too late for Remote.Git to use that. This does mean one redundant load of the cluster log, though only when there is a proxy.
2024-06-25 14:06:28 +00:00
* Avoid `git-annex sync --content` etc from operating on cluster nodes by
default since syncing with a cluster implicitly syncs with its nodes. (done)
honor preferred content settings of cluster nodes Except when no nodes want a file, it has to be stored somewhere, so store it on all. Which is not really desirable, but neither is having to pick one. ProtoAssociatedFile deserialization is rather broken, and this could possibly affect preferred content expressions that match on filenames. The inability to roundtrip whitespace like tabs and newlines through is not a problem because preferred content expressions can't be written that match on whitespace such as a tab. For example: joey@darkstar:~/tmp/bench/z>git-annex wanted origin-node2 'exclude=*CTRL-VTab*' wanted origin-node2 git-annex: Parse error: Parse failure: near "*" But, the filtering of control characters could perhaps be a problem. I think that filtering is now obsolete, git-annex has comprehensive filtering of control characters when displaying filenames, that happens at a higher level. However, I don't want to risk a security hole so am leaving in that filtering in ProtoAssociatedFile deserialization for now.
2024-06-25 15:35:41 +00:00
* On upload to cluster, send to nodes where its preferred content, and not
to other nodes. (done)
support annex.jobs for clusters
2024-06-25 18:52:47 +00:00
* Support annex.jobs for clusters. (done)
git-annex-shell: proxy nodes located beyond remote cluster gateways Walking a tightrope between security and convenience here, because git-annex-shell needs to only proxy for things when there has been an explicit, local action to configure them. In this case, the user has to have run `git-annex extendcluster`, which now sets annex-cluster-gateway on the remote. Note that any repositories that the gateway is recorded to proxy for will be proxied onward. This is not limited to cluster nodes, because checking the node log would not add any security; someone could add any uuid to it. The gateway of course then does its own checking to determine if it will allow proxying for the remote.
2024-06-26 16:56:16 +00:00
* Add `git-annex extendcluster` command and extend `git-annex updatecluster`
to support clusters with multiple gateways. (done)
* Support proxying for a remote that is proxied by another gateway of
a cluster. (done)
distributed cluster cycle prevention Added BYPASS to P2P protocol, and use it to avoid cycling between cluster gateways. Distributed clusters are working well now!
2024-06-27 16:20:22 +00:00
* Support distributed clusters: Make a proxy for a cluster repeat
protocol messages on to any remotes that have the same UUID as
the cluster. Needs extension to P2P protocol to avoid cycles.
(done)
give proxied cluster nodes a higher cost than the cluster gateway This makes eg git-annex get default to using the cluster rather than an arbitrary node, which is better UI. The actual cost of accessing a proxied node vs using the cluster is basically the same. But using the cluster allows smarter load-balancing to be done on the cluster.
2024-06-27 19:21:03 +00:00
* Proxied cluster nodes should have slightly higher cost than the cluster
gateway. (done)
GET from proxied special remote Working, but lots of room for improvement... Without streaming, so there is a delay before download begins as the file is retreived from the special remote. And when resuming it retrieves the whole file from the special remote *again*. Also, if the special remote throws an exception, currently it shows as "protocol error".
2024-06-28 19:32:00 +00:00
* Basic support for proxying special remotes. (But not exporttree=yes ones
yet.) (done)
update
2024-07-01 15:29:04 +00:00
* Tab complete remotes in all relevant commands (done)
* Display cluster and proxy information in git-annex info (done)
Reference in a new issue Copy permalink