mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions
git-annex/doc/todo/git-annex_proxies.mdwn

267 lines
10 KiB
Text
Raw Normal View History

add todo linking to two design docs and some related todos Tagging with projects/openneuro as Christopher Markiewicz has oked them funding at least the initial design work on this.
2024-05-01 15:04:20 +00:00
This is a summary todo covering several subprojects, which would extend
git-annex to be able to use proxies which sit in front of a cluster of
repositories.
number list
2024-05-01 16:19:12 +00:00
1. [[design/passthrough_proxy]]
started on a design for P2P protocol over HTTP Added to git-annex_proxies todo because this is something OpenNeuro would need in order to use the git-annex proxy. Sponsored-by: Dartmouth College's OpenNeuro project
2024-05-01 19:26:51 +00:00
2. [[design/p2p_protocol_over_http]]
3. [[design/balanced_preferred_content]]
4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
5. [[todo/proving_preferred_content_behavior]]
add todo linking to two design docs and some related todos Tagging with projects/openneuro as Christopher Markiewicz has oked them funding at least the initial design work on this.
2024-05-01 15:04:20 +00:00
update
2024-07-01 15:38:29 +00:00
## table of contents
[[!toc ]]
## planned schedule
recieved funding to work on this, which comes with a schedule
2024-06-04 10:53:59 +00:00
Joey has received funding to work on this.
Planned schedule of work:
update
2024-06-27 19:28:10 +00:00
* June: git-annex proxies and clusters
fix negative DATA when 1 node of a cluster has a partial transfer
2024-07-30 15:42:17 +00:00
* July: p2p protocol over http
* August, part 1: git-annex proxy support for exporttree
update
2024-08-30 15:14:45 +00:00
* August, part 2: balanced preferred content
* October: proving behavior of balanced preferred content with proxies
update
2024-09-03 15:52:54 +00:00
* September: streaming through proxy to special remotes (especially S3)
recieved funding to work on this, which comes with a schedule
2024-06-04 10:53:59 +00:00
add todo linking to two design docs and some related todos Tagging with projects/openneuro as Christopher Markiewicz has oked them funding at least the initial design work on this.
2024-05-01 15:04:20 +00:00
[[!tag projects/openneuro]]
implementation plan
2024-06-04 11:51:33 +00:00
update
2024-07-01 15:38:29 +00:00
## work notes
implementation plan
2024-06-04 11:51:33 +00:00
streaming uploads
2024-10-15 20:02:19 +00:00
* Currently working on streaming special remotes via proxy
working in streamproxy branch
2024-10-15 16:26:53 +00:00
in the `streamproxy` branch.
update
2024-10-15 14:31:42 +00:00
streaming uploads
2024-10-15 20:02:19 +00:00
* Downloads from special remotes can stream (though using a temp file on
the proxy). Next: Streaming uploads via the proxy.
tried a blind alley on streaming special remote download via proxy This didn't work. In case I want to revisit, here's what I tried. diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs index 48222872c1..e4e526d3dd 100644 --- a/Annex/Proxy.hs +++ b/Annex/Proxy.hs @@ -26,16 +26,21 @@ import Logs.UUID import Logs.Location import Utility.Tmp.Dir import Utility.Metered +import Utility.ThreadScheduler +import Utility.OpenFd import Git.Types import qualified Database.Export as Export import Control.Concurrent.STM import Control.Concurrent.Async +import Control.Concurrent.MVar import qualified Data.ByteString as B +import qualified Data.ByteString as BS import qualified Data.ByteString.Lazy as L import qualified System.FilePath.ByteString as P import qualified Data.Map as M import qualified Data.Set as S +import System.IO.Unsafe proxyRemoteSide :: ProtocolVersion -> Bypass -> Remote -> Annex RemoteSide proxyRemoteSide clientmaxversion bypass r @@ -240,21 +245,99 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv oclosedv mexportdb = go writeVerifyChunk iv h b storetofile iv h (n - fromIntegral (B.length b)) bs - proxyget offset af k = withproxytmpfile k $ \tmpfile -> do + proxyget offset af k = withproxytmpfile k $ \tmpfile -> + let retrieve = tryNonAsync $ Remote.retrieveKeyFile r k af + (fromRawFilePath tmpfile) nullMeterUpdate vc + in case fromKey keySize k of + Just size | size > 0 -> do + cancelv <- liftIO newEmptyMVar + donev <- liftIO newEmptyMVar + streamer <- liftIO $ async $ + streamdata offset tmpfile size cancelv donev + retrieve >>= \case + Right _ -> liftIO $ do + putMVar donev () + wait streamer + Left err -> liftIO $ do + putMVar cancelv () + wait streamer + propagateerror err + _ -> retrieve >>= \case + Right _ -> liftIO $ senddata offset tmpfile + Left err -> liftIO $ propagateerror err + where -- Don't verify the content from the remote, -- because the client will do its own verification. - let vc = Remote.NoVerify - tryNonAsync (Remote.retrieveKeyFile r k af (fromRawFilePath tmpfile) nullMeterUpdate vc) >>= \case - Right _ -> liftIO $ senddata offset tmpfile - Left err -> liftIO $ propagateerror err + vc = Remote.NoVerify + streamdata (Offset offset) f size cancelv donev = do + sendlen offset size + waitforfile + x <- tryNonAsync $ do + fd <- openFdWithMode f ReadOnly Nothing defaultFileFlags + h <- fdToHandle fd + hSeek h AbsoluteSeek offset + senddata' h (getcontents size) + case x of + Left err -> do + throwM err + Right res -> return res + where + -- The file doesn't exist at the start. + -- Wait for some data to be written to it as well, + -- in case an empty file is first created and then + -- overwritten. When there is an offset, wait for + -- the file to get that large. Note that this is not used + -- when the size is 0. + waitforfile = tryNonAsync (fromIntegral <$> getFileSize f) >>= \case + Right sz | sz > 0 && sz >= offset -> return () + _ -> ifM (isEmptyMVar cancelv) + ( do + threadDelaySeconds (Seconds 1) + waitforfile + , do + return () + ) + + getcontents n h = unsafeInterleaveIO $ do + isdone <- isEmptyMVar donev <||> isEmptyMVar cancelv + c <- BS.hGet h defaultChunkSize + let n' = n - fromIntegral (BS.length c) + let c' = L.fromChunks [BS.take (fromIntegral n) c] + if BS.null c + then if isdone + then return mempty + else do + -- Wait for more data to be + -- written to the file. + threadDelaySeconds (Seconds 1) + getcontents n h + else if n' > 0 + then do + -- unsafeInterleaveIO causes + -- this to be deferred until + -- data is read from the lazy + -- ByteString. + cs <- getcontents n' h + return $ L.append c' cs + else return c' + senddata (Offset offset) f = do size <- fromIntegral <$> getFileSize f - let n = max 0 (size - offset) - sendmessage $ DATA (Len n) + sendlen offset size withBinaryFile (fromRawFilePath f) ReadMode $ \h -> do hSeek h AbsoluteSeek offset - sendbs =<< L.hGetContents h + senddata' h L.hGetContents + + senddata' h getcontents = do + sendbs =<< getcontents h -- Important to keep the handle open until -- the client responds. The bytestring -- could still be lazily streaming out to @@ -272,6 +355,11 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv oclosedv mexportdb = go Just FAILURE -> return () Just _ -> giveup "protocol error" Nothing -> return () + + sendlen offset size = do + let n = max 0 (size - offset) + sendmessage $ DATA (Len n) + {- Check if this repository can proxy for a specified remote uuid, - and if so enable proxying for it. -}
2024-10-07 17:54:04 +00:00
added retrieveKeyFileInOrder and ORDERED to external special remote protocol I anticipate lots of external special remote programs will neglect implementing this. Still, it's the right thing to do to assume that some of them may write files out of order. Probably most external special remotes will not be used with a proxy. When someone is using one with a proxy, they can always get it fixed to send ORDERED.
2024-10-15 19:35:09 +00:00
## completed items for October's work on streaming through proxy to special remotes
* Stream downloads through proxy for all special remotes that indicate
they download in order.
* Added ORDERED message to external special remote protocol.
update
2024-09-25 18:26:32 +00:00
## completed items for September's work on proving behavior of preferred content
make --rebalance of balanced use fullysizebalanced when useful When the specified number of copies is > 1, and some repositories are too full, it can be better to move content from them to other less full repositories, in order to make space for new content. annex.fullybalancedthreshhold is documented, but not implemented yet This is not tested very well yet, and is known to sometimes take several runs to stabalize.
2024-08-21 21:56:06 +00:00
update
2024-09-25 18:26:32 +00:00
* Static analysis to detect "not present", "not balanced", and similar
unstable preferred content expressions and avoid problems with them.
* Implemented `git-annex sim` command.
* Simulated a variety of repository networks, and random preferred content
expressions, checking that a stable state is always reached.
* Fix bug that prevented anything being stored in an empty
repository whose preferred content expression uses sizebalanced.
(Identified via `git-annex sim`)
update
2024-08-30 15:14:45 +00:00
## items deferred until later for balanced preferred content and maxsize tracking
locking in checkLiveUpdate This makes sure that two threads don't check balanced preferred content at the same time, so each thread always sees a consistent picture of what is happening. This does add a fairly expensive file level lock to every check of preferred content, in commands that use prepareLiveUpdate. It would be good to only do that when live updates are actually needed, eg when the preferred content expression uses balanced preferred content.
2024-08-27 17:07:06 +00:00
punt on LiveUpdate plumbing through assistant for now
2024-08-24 13:37:24 +00:00
* The assistant is using NoLiveUpdate, but it should be posssible to plumb
a LiveUpdate through it from preferred content checking to location log
updating.
plumb in LiveUpdate (WIP) Each command that first checks preferred content (and/or required content) and then does something that can change the sizes of repositories needs to call prepareLiveUpdate, and plumb it through the preferred content check and the location log update. So far, only Command.Drop is done. Many other commands that don't need to do this have been updated to keep working. There may be some calls to NoLiveUpdate in places where that should be done. All will need to be double checked. Not currently in a compilable state.
2024-08-23 20:35:12 +00:00
consistently don't show sizes of empty repositories This used to be the case, and when matching options are used, that code path still omits them, so also omit them in the getRepoSize code path.
2024-08-17 18:58:36 +00:00
* `git-annex info` in the limitedcalc path in cachedAllRepoData
double-counts redundant information from the journal due to using
combine 2 queries
2024-08-28 15:00:59 +00:00
overLocationLogs. In the other path it does not (any more; it used to),
and this should be fixed for consistency and correctness.
consistently don't show sizes of empty repositories This used to be the case, and when matching options are used, that code path still omits them, so also omit them in the getRepoSize code path.
2024-08-17 18:58:36 +00:00
started work on getLiveRepoSizes Doesn't quite compile
2024-08-26 18:50:09 +00:00
* getLiveRepoSizes has a filterM getRecentChange over the live updates.
This could be optimised to a single sql join. There are usually not many
live updates, but sometimes there will be a great many recent changes,
update
2024-08-30 15:14:45 +00:00
so it might be worth doing this optimisation. Persistent is not capable
of this, would need dependency added on esquelito.
started work on getLiveRepoSizes Doesn't quite compile
2024-08-26 18:50:09 +00:00
balanced preferred content and --rebalance This all works fine. But it doesn't check repository sizes yet, and without repository size checking, once a repository gets full, there will be no other repository that will want its files. Use of sha2 seems unncessary, probably alder2 or md5 or crc would have been enough. Possibly just summing up the bytes of the key mod the number of repositories would have sufficed. But sha2 is there, and probably hardware accellerated. I doubt very much there is any security benefit to using it though. If someone wants to construct a key that will be balanced onto a given repository, sha2 is certianly not going to stop them.
2024-08-09 18:16:09 +00:00
## completed items for August's work on balanced preferred content
* Balanced preferred content basic implementation, including --rebalance
option.
update RepoSize database from git-annex branch incrementally The use of catObjectStream is optimally fast. Although it might be possible to combine this with git-annex branch merge to avoid some redundant work. Benchmarking, a git-annex branch that had 100000 files changed took less than 1.88 seconds to run through this.
2024-08-17 17:30:24 +00:00
* Implemented [[track_free_space_in_repos_via_git-annex_branch]]
closing in on finishing live reposizes Fixed successfullyFinishedLiveSizeChange to not update the rolling total when a redundant change is in RecentChanges. Made setRepoSizes clear RecentChanges that are no longer needed. It might be possible to clear those earlier, this is only a convenient point to do it. The reason it's safe to clear RecentChanges here is that, in order for a live update to call successfullyFinishedLiveSizeChange, a change must be made to a location log. If a RecentChange gets cleared, and just after that a new live update is started, making the same change, the location log has already been changed (since the RecentChange exists), and so when the live update succeeds, it won't call successfullyFinishedLiveSizeChange. The reason it doesn't clear RecentChanges when there is a reduntant live update is because I didn't want to think through whether or not all races are avoided in that case. The rolling total in SizeChanges is never cleared. Instead, calcJournalledRepoSizes gets the initial value of it, and then getLiveRepoSizes subtracts that initial value from the current value. Since the rolling total can only be updated by updateRepoSize, which is called with the journal locked, locking the journal in calcJournalledRepoSizes ensures that the database does not change while reading the journal.
2024-08-27 15:04:27 +00:00
* Implemented tracking of live changes to repository sizes.
update
2024-08-22 11:53:56 +00:00
* `git-annex maxsize`
* annex.fullybalancedthreshhold
proxying to exporttree=yes annexobjects=yes basically working It works when using git-annex sync/push/assist, or when manually sending all content to the proxied remote before pushing to the proxy remote. But when the push comes before the content is sent, sending content does not update the exported tree.
2024-08-06 18:18:30 +00:00
update
2024-08-08 19:34:36 +00:00
## completed items for August's work on git-annex proxy support for exporttre
update
2024-08-07 15:49:53 +00:00
* Special remotes configured with exporttree=yes annexobjects=yes
can store objects in .git/annex/objects, as well as an exported tree.
* Support proxying to special remotes configured with
exporttree=yes annexobjects=yes.
* post-retrieve: When proxying is enabled for an exporttree=yes
special remote and the configured remote.name.annex-tracking-branch
is received, the tree is exported to the special remote.
* When getting from a P2P HTTP remote, prompt for credentials when
required, instead of failing.
updateproxy, updatecluster check annexobjects=yes updateproxy, updatecluster: Prevent using an exporttree=yes special remote that does not have annexobjects=yes, since it will not work.
2024-08-07 16:27:24 +00:00
* Prevent `updateproxy` and `updatecluster` from adding
an exporttree=yes special remote that does not have
annexobjects=yes, to avoid foot shooting.
update
2024-08-08 18:05:05 +00:00
* Implement `git-annex export treeish --to=foo --from=bar`, which
gets from bar as needed to send to foo. Make post-retrieve use
`--to=r --from=r` to handle the multiple files case.
post-receive: use the exporttree=yes remote as a source This handles cases where a single key is used by multiple files in the exported tree. When using `git-annex push`, the key's content gets stored in the annexobjects location, and then when the branch is pushed, it gets renamed from the annexobjects location to the first exported file. For subsequent exported files, a copy of the content needs to be made. This causes it to download the key from the remote in order to upload another copy to it. This is not needed when using `git push` followed by `git-annex copy --to` the proxied remote, because the received key is stored at all export locations then. Also, fixed handling of the synced branch push, it was exporting master when synced/master was pushed. Note that currently, the first push to the remote does not see that it is able to get a key from it in order to upload it back. It displays "(not available)". The second push is able to. Since git-annex push pushes first the synced branch and then the branch, this does end up with a full export being made, but it is not quite right.
2024-08-08 16:28:12 +00:00
update
2024-07-28 18:22:44 +00:00
## items deferred until later for p2p protocol over http
documentation for p2phttp
2024-07-28 21:19:27 +00:00
* `git-annex p2phttp` should support serving several repositories at the same
time (not as proxied remotes), so that eg, every git-annex repository
on a server can be served on the same port.
idea
2024-07-29 13:11:27 +00:00
* Support proxying to git remotes that use annex+http urls. This needs a
translation from P2P protocol to servant-client to P2P protocol.
* Should be possible to use a git-remote-annex annex::$uuid url as
remote.foo.url with remote.foo.annexUrl using annex+http, and so
not need a separate web server to serve the git repository. Doesn't work
currently because git-remote-annex urls only support special remotes.
It would need a new form of git-remote-annex url, eg:
annex::$uuid?annex+http://example.com/git-annex/
update
2024-07-28 18:22:44 +00:00
* `git-annex p2phttp` could support systemd socket activation. This would
allow making a systemd unit that listens on port 80.
update
2024-07-04 19:18:06 +00:00
## completed items for July's work on p2p protocol over http
update
2024-07-24 16:19:53 +00:00
* HTTP P2P protocol design [[design/p2p_protocol_over_http]].
update
2024-07-04 19:18:06 +00:00
P2P protocol is finalized
2024-07-22 23:50:08 +00:00
* addressed [[doc/todo/P2P_locking_connection_drop_safety]]
finalizing HTTP P2p protocol some more Added v2-v0 endpoints. These are tedious, but will be needed in order to use the HTTP protocol to proxy to repositories with older git-annex, where git-annex-shell will be speaking an older version of the protocol. Changed GET to use 422 when the content is not present. 404 is needed to detect when a protocol version is not supported.
2024-07-05 19:34:58 +00:00
simplify
2024-07-11 15:50:44 +00:00
* implemented server and client for HTTP P2P protocol
implemented servePutOffset and clientPutOffset But, it's buggy: the server hangs without processing the VALIDITY, and I can't seem to work out why. As far as I can see, storefile is getting as far as running the validitycheck, which is supposed to read that, but never does. This is especially strange because what seems like the same protocol doesn't hang when servePut runs it. This made me think that it needed to use inAnnexWorker to be more like servePut, but that didn't help. Another small problem with this is that it does create an empty .git/annex/tmp/ file for the key. Since this will usually be used in combination with servePut, that doesn't seem worth worrying about much.
2024-07-22 19:02:08 +00:00
* added git-annex p2phttp command to serve HTTP P2P protocol
p2phttp support https
2024-07-23 19:37:36 +00:00
* Make git-annex p2phttp support https.
update
2024-07-24 16:19:53 +00:00
* Allow using annex+http urls in remote.name.annexUrl
http p2p proxy is now largely working
2024-07-26 14:24:23 +00:00
* Make http server support proxying.
add --clusterjobs option and default to 1 The default of 1 is not ideal at all, but it avoids an accidental M*N causing so much concurrency it becomes unusable.
2024-07-28 14:36:22 +00:00
* Make http server support serving a cluster.
update
2024-07-02 20:16:37 +00:00
## items deferred until later for [[design/passthrough_proxy]]
dropping from clusters Dropping from a cluster drops from every node of the cluster. Including nodes that the cluster does not think have the content. This is different from GET and CHECKPRESENT, which do trust the cluster's location log. The difference is that removing from a cluster should make 100% the content is gone from every node. So doing extra work is ok. Compare with CHECKPRESENT where checking every node could make it very expensive, and the worst that can happen in a false negative is extra work being done. Extended the P2P protocol with FAILURE-PLUS to handle the case where a drop from one node succeeds, but a drop from another node fails. In that case the entire cluster drop has failed. Note that SUCCESS-PLUS is returned when dropping from a proxied remote that is not a cluster, when the protocol version supports it. This is because P2P.Proxy does not know when it's proxying for a single node cluster vs for a remote that is not a cluster.
2024-06-23 13:28:18 +00:00
update
2024-07-01 15:29:04 +00:00
* Check annex.diskreserve when proxying for special remotes
to avoid the proxy's disk filling up with the temporary object file
cached there.
GET from proxied special remote Working, but lots of room for improvement... Without streaming, so there is a delay before download begins as the file is retreived from the special remote. And when resuming it retrieves the whole file from the special remote *again*. Also, if the special remote throws an exception, currently it shows as "protocol error".
2024-06-28 19:32:00 +00:00
* Resuming an interrupted download from proxied special remote makes the proxy
re-download the whole content. It could instead keep some of the
object files around when the client does not send SUCCESS. This would
use more disk, but without streaming, proxying a special remote already
needs some disk. And it could minimize to eg, the last 2 or so.
PUT to proxied special remote working Still needs some work. The reason that the waitv is necessary is because without it, runNet loops back around and reads the next protocol message. But it's not finished reading the whole bytestring yet, and so it reads some part of it.
2024-06-28 21:07:01 +00:00
The design doc has some more thoughts about this.
P2P protocol version 2, adding SUCCESS-PLUS and ALREADY-HAVE-PLUS Client side support for SUCCESS-PLUS and ALREADY-HAVE-PLUS is complete, when a PUT stores to additional repositories than the expected on, the location log is updated with the additional UUIDs that contain the content. Started implementing PUT fanout to multiple remotes for clusters. It is untested, and I fear fencepost errors in the relative offset calculations. And it is missing proxying for the protocol after DATA.
2024-06-18 16:07:01 +00:00
GET from proxied special remote Working, but lots of room for improvement... Without streaming, so there is a delay before download begins as the file is retreived from the special remote. And when resuming it retrieves the whole file from the special remote *again*. Also, if the special remote throws an exception, currently it shows as "protocol error".
2024-06-28 19:32:00 +00:00
* Streaming download from proxied special remotes. See design.
update
2024-07-01 15:29:04 +00:00
(Planned for September)
updates
2024-06-27 16:57:08 +00:00
update
2024-07-01 15:33:07 +00:00
* When an upload to a cluster is distributed to multiple special remotes,
a temporary file is written for each one, which may even happen in
parallel. This is a lot of extra work and may use excess disk space.
It should be possible to only write a single temp file.
(With streaming this won't be an issue.)
update
2024-06-27 17:40:09 +00:00
* Indirect uploads when proxying for special remote
(to be considered). See design.
GET and CHECKPRESENT amoung lowest cost cluster nodes Before it was using a node that might have had a higher cost. Also threw in a random selection from amoung the low cost nodes. Of course this is a poor excuse for load balancing, but it's better than nothing. Most of the time...
2024-06-27 18:36:55 +00:00
* Getting a key from a cluster currently picks from amoung
the lowest cost remotes at random. This could be smarter,
eg prefer to avoid using remotes that are doing other transfers at the
same time.
give proxied cluster nodes a higher cost than the cluster gateway This makes eg git-annex get default to using the cluster rather than an arbitrary node, which is better UI. The actual cost of accessing a proxied node vs using the cluster is basically the same. But using the cluster allows smarter load-balancing to be done on the cluster.
2024-06-27 19:21:03 +00:00
* The cost of a proxied node that is accessed via an intermediate gateway
is currently the same as a node accessed via the cluster gateway.
To fix this, there needs to be some way to tell how many hops through
gateways it takes to get to a node. Currently the only way is to
guess based on number of dashes in the node name, which is not satisfying.
Even counting hops is not very satisfying, one cluster gateway could
be much more expensive to traverse than another one.
If seriously tackling this, it might be worth making enough information
available to use spanning tree protocol for routing inside clusters.
improve docs
2024-06-25 21:50:22 +00:00
updates
2024-06-12 15:55:18 +00:00
* Optimise proxy speed. See design for ideas.
implementation plan
2024-06-04 11:51:33 +00:00
fix http proxying for a local git remote with a relative path git-annex-shell expects an absolute path
2024-07-28 17:31:30 +00:00
* Speed: A proxy to a local git repository spawns git-annex-shell
to communicate with it. It would be more efficient to operate
directly on the Remote. Especially when transferring content to/from it.
But: When a cluster has several nodes that are local git repositories,
and is sending data to all of them, this would need an alternate
interface than `storeKey`, which supports streaming, of chunks
of a ByteString.
updates
2024-06-12 15:55:18 +00:00
* Use `sendfile()` to avoid data copying overhead when
`receiveBytes` is being fed right into `sendBytes`.
update
2024-06-25 21:26:26 +00:00
Library to use:
<https://hackage.haskell.org/package/hsyscall-0.4/docs/System-Syscall.html>
implementation plan
2024-06-04 11:51:33 +00:00
updates
2024-06-12 15:55:18 +00:00
* Support using a proxy when its url is a P2P address.
(Eg tor-annex remotes.)
update
2024-06-23 16:31:00 +00:00
update
2024-07-01 15:38:29 +00:00
## completed items for June's work on [[design/passthrough_proxy]]:
update
2024-06-23 20:38:01 +00:00
* UUID discovery via git-annex branch. Add a log file listing UUIDs
accessible via proxy UUIDs. It also will contain the names
of the remotes that the proxy is a proxy for,
from the perspective of the proxy. (done)
* Add `git-annex updateproxy` command (done)
* Remote instantiation for proxies. (done)
* Implement git-annex-shell proxying to git remotes. (done)
* Proxy should update location tracking information for proxied remotes,
so it is available to other users who sync with it. (done)
update
2024-06-27 19:28:10 +00:00
* Implement `git-annex initcluster` and `git-annex updatecluster` commands (done)
update
2024-06-23 20:38:01 +00:00
* Implement cluster UUID insertation on location log load, and removal
on location log store. (done)
* Omit cluster UUIDs when constructing drop proofs, since lockcontent will
always fail on a cluster. (done)
* Don't count cluster UUID as a copy in numcopies checking etc. (done)
* Tab complete proxied remotes and clusters in eg --from option. (done)
* Getting a key from a cluster should proxy from one of the nodes that has
it. (done)
* Implement upload with fanout to multiple cluster nodes and reporting back
additional UUIDs over P2P protocol. (done)
* Implement cluster drops, trying to remove from all nodes, and returning
which UUIDs it was dropped from. (done)
* `git-annex testremote` works against proxied remote and cluster. (done)
don't sync with cluster nodes by default Avoid `git-annex sync --content` etc from operating on cluster nodes by default since syncing with a cluster implicitly syncs with its nodes. This avoids a lot of unncessary work when a cluster has a lot of nodes just in checking if each node's preferred content is satisfied. And it avoids content being sent to nodes individually, so instead syncing with clusters always fanout uploads to nodes. The downside is that there are situations where a cluster's preferred content settings can be met, but those of its nodes are not. Or where a node does not contain a key, but the cluster does, and there are not enough copies of the key yet, so it would be desirable the send it there. I think that's an acceptable tradeoff. These kind of situations are ones where the cluster itself should probably be responsible for copying content to the node. Which it can do much less expensively than a client can. Part of the balanced preferred content design that I will be working on in a couple of months involves rebalancing clusters, so I expect to revisit this. The use of annex-sync config does allow running git-annex sync with a specific node, or nodes, and it will sync with it. And it's also possible to set annex-sync git configs to make it sync with a node by default. (Although that will require setting up an explicit git remote for the node rather than relying on the proxied remote.) Logs.Cluster.Basic is needed because Remote.Git cannot import Logs.Cluster due to a cycle. And the Annex.Startup load of clusters happens too late for Remote.Git to use that. This does mean one redundant load of the cluster log, though only when there is a proxy.
2024-06-25 14:06:28 +00:00
* Avoid `git-annex sync --content` etc from operating on cluster nodes by
default since syncing with a cluster implicitly syncs with its nodes. (done)
honor preferred content settings of cluster nodes Except when no nodes want a file, it has to be stored somewhere, so store it on all. Which is not really desirable, but neither is having to pick one. ProtoAssociatedFile deserialization is rather broken, and this could possibly affect preferred content expressions that match on filenames. The inability to roundtrip whitespace like tabs and newlines through is not a problem because preferred content expressions can't be written that match on whitespace such as a tab. For example: joey@darkstar:~/tmp/bench/z>git-annex wanted origin-node2 'exclude=*CTRL-VTab*' wanted origin-node2 git-annex: Parse error: Parse failure: near "*" But, the filtering of control characters could perhaps be a problem. I think that filtering is now obsolete, git-annex has comprehensive filtering of control characters when displaying filenames, that happens at a higher level. However, I don't want to risk a security hole so am leaving in that filtering in ProtoAssociatedFile deserialization for now.
2024-06-25 15:35:41 +00:00
* On upload to cluster, send to nodes where its preferred content, and not
to other nodes. (done)
support annex.jobs for clusters
2024-06-25 18:52:47 +00:00
* Support annex.jobs for clusters. (done)
git-annex-shell: proxy nodes located beyond remote cluster gateways Walking a tightrope between security and convenience here, because git-annex-shell needs to only proxy for things when there has been an explicit, local action to configure them. In this case, the user has to have run `git-annex extendcluster`, which now sets annex-cluster-gateway on the remote. Note that any repositories that the gateway is recorded to proxy for will be proxied onward. This is not limited to cluster nodes, because checking the node log would not add any security; someone could add any uuid to it. The gateway of course then does its own checking to determine if it will allow proxying for the remote.
2024-06-26 16:56:16 +00:00
* Add `git-annex extendcluster` command and extend `git-annex updatecluster`
to support clusters with multiple gateways. (done)
* Support proxying for a remote that is proxied by another gateway of
a cluster. (done)
distributed cluster cycle prevention Added BYPASS to P2P protocol, and use it to avoid cycling between cluster gateways. Distributed clusters are working well now!
2024-06-27 16:20:22 +00:00
* Support distributed clusters: Make a proxy for a cluster repeat
protocol messages on to any remotes that have the same UUID as
the cluster. Needs extension to P2P protocol to avoid cycles.
(done)
give proxied cluster nodes a higher cost than the cluster gateway This makes eg git-annex get default to using the cluster rather than an arbitrary node, which is better UI. The actual cost of accessing a proxied node vs using the cluster is basically the same. But using the cluster allows smarter load-balancing to be done on the cluster.
2024-06-27 19:21:03 +00:00
* Proxied cluster nodes should have slightly higher cost than the cluster
gateway. (done)
GET from proxied special remote Working, but lots of room for improvement... Without streaming, so there is a delay before download begins as the file is retreived from the special remote. And when resuming it retrieves the whole file from the special remote *again*. Also, if the special remote throws an exception, currently it shows as "protocol error".
2024-06-28 19:32:00 +00:00
* Basic support for proxying special remotes. (But not exporttree=yes ones
yet.) (done)
update
2024-07-01 15:29:04 +00:00
* Tab complete remotes in all relevant commands (done)
* Display cluster and proxy information in git-annex info (done)
Reference in a new issue Copy permalink