mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/RemoteDaemon/Types.hs

113 lines
3.4 KiB
Haskell
Raw Normal View History

basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
{- git-remote-daemon data types.
-
update my email address and homepage url
2015-01-21 16:50:09 +00:00
- Copyright 2014 Joey Hess <id@joeyh.name>
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
-
update licenses from GPL to AGPL This does not change the overall license of the git-annex program, which was already AGPL due to a number of sources files being AGPL already. Legally speaking, I'm adding a new license under which these files are now available; I already released their current contents under the GPL license. Now they're dual licensed GPL and AGPL. However, I intend for all my future changes to these files to only be released under the AGPL license, and I won't be tracking the dual licensing status, so I'm simply changing the license statement to say it's AGPL. (In some cases, others wrote parts of the code of a file and released it under the GPL; but in all cases I have contributed a significant portion of the code in each file and it's that code that is getting the AGPL license; the GPL license of other contributors allows combining with AGPL code.)
2019-03-13 19:48:14 +00:00
- Licensed under the GNU AGPL version 3 or higher.
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
-}
{-# OPTIONS_GHC -fno-warn-orphans #-}
module RemoteDaemon.Types where
make git-remote-daemon ssh transport robust * Remote system might be available, and connection get lost. Should reconnect, but needs to avoid bad behavior (ie, constant reconnect attempts.) Use exponential backoff. * Detect if old system had a too old git-annex-shell, and show the user a nice message in the webapp. Required parsing error messages, so perhaps this code shoudl be removed once enough time has passed.. * Switch the protocol to using remote URI's, rather than remote names. Names change. Also avoids issues with serialization of names containing whitespace. This is nearly ready for merge into master now. I'd still like to make the ssh transport smarter about reusing ssh connection caching during git pull. This commit was sponsored by Jim Paris.
2014-04-09 18:10:29 +00:00
import Common
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
import qualified Annex
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
import qualified Git.Types as Git
import qualified Utility.SimpleProtocol as Proto
propigate ssh-options everywhere ssh caching is used * sync: Use the ssh-options git config when doing git pull and push. * remotedaemon: Use the ssh-options git config. Note that the rename env var means that if a new git-annex calls an old one for git-annex ssh, or a new calls an old, nothing much will go wrong; just ssh caching won't happen.
2015-02-12 20:12:32 +00:00
import Types.GitConfig
refactor ref change watching Added to change notification to P2P protocol. Switched to a TBChan so that a single long-running thread can be started, and serve perhaps intermittent requests for change notifications, without buffering all changes in memory. The P2P runner currently starts up a new thread each times it waits for a change, but that should allow later reusing a thread. Although each connection from a peer will still need a new watcher thread to run. The dependency on stm-chans is more or less free; some stuff in yesod uses it, so it was already indirectly pulled in when building with the webapp. This commit was sponsored by Francois Marier on Patreon.
2016-12-09 18:52:38 +00:00
import Annex.ChangedRefs (ChangedRefs)
Windows: Support urls like "file:///c:/path" That is a legal url, but parseUrl parses it to "/c:/path" which is not a valid path on Windows. So as a workaround, use parseURIPortable everywhere, which removes the leading slash when run on windows. Note that if an url is parsed like this and then serialized back to a string, it will be different from the input. Which could potentially be a problem, but is probably not in practice. An alternative way to do it would be to have an uriPathPortable that fixes up the path after parsing. But it would be harder to make sure that is used everywhere, since uriPath is also used when constructing an URI. It's also worth noting that System.FilePath.normalize "/c:/path" yields "c:/path". The reason I didn't use it is that it also may change "/" to "\" in the path and I wanted to keep the url changes minimal. Also noticed that convertToWindowsNativeNamespace handles "/c:/path" the same as "c:/path". Sponsored-By: the NIH-funded NICEMAN (ReproNim TR&D3) project
2023-03-27 17:38:02 +00:00
import Utility.Url
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
make git-remote-daemon ssh transport robust * Remote system might be available, and connection get lost. Should reconnect, but needs to avoid bad behavior (ie, constant reconnect attempts.) Use exponential backoff. * Detect if old system had a too old git-annex-shell, and show the user a nice message in the webapp. Required parsing error messages, so perhaps this code shoudl be removed once enough time has passed.. * Switch the protocol to using remote URI's, rather than remote names. Names change. Also avoids issues with serialization of names containing whitespace. This is nearly ready for merge into master now. I'd still like to make the ssh transport smarter about reusing ssh connection caching during git pull. This commit was sponsored by Jim Paris.
2014-04-09 18:10:29 +00:00
import Network.URI
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
import Control.Concurrent
remotedaemon: Fix problem that could prevent ssh connections being made after two LOSTNET messages were received in a row Perhaps due to two different network interfaces being brought down. Since there is no reliable way to drain a Chan, I switched to STM TChan.
2015-01-15 19:37:48 +00:00
import Control.Concurrent.STM
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
make git-remote-daemon ssh transport robust * Remote system might be available, and connection get lost. Should reconnect, but needs to avoid bad behavior (ie, constant reconnect attempts.) Use exponential backoff. * Detect if old system had a too old git-annex-shell, and show the user a nice message in the webapp. Required parsing error messages, so perhaps this code shoudl be removed once enough time has passed.. * Switch the protocol to using remote URI's, rather than remote names. Names change. Also avoids issues with serialization of names containing whitespace. This is nearly ready for merge into master now. I'd still like to make the ssh transport smarter about reusing ssh connection caching during git pull. This commit was sponsored by Jim Paris.
2014-04-09 18:10:29 +00:00
-- The URI of a remote is used to uniquely identify it (names change..)
newtype RemoteURI = RemoteURI URI
indent with tabs not spaces Found these with: git grep "^ " $(find -type f -name \*.hs) |grep -v ': where' Unfortunately there is some inline hamlet that cannot use tabs for indentation. Also, Assistant/WebApp/Bootstrap3.hs is a copy of a module and so I'm leaving it as-is.
2014-10-09 19:09:26 +00:00
deriving (Show)
make git-remote-daemon ssh transport robust * Remote system might be available, and connection get lost. Should reconnect, but needs to avoid bad behavior (ie, constant reconnect attempts.) Use exponential backoff. * Detect if old system had a too old git-annex-shell, and show the user a nice message in the webapp. Required parsing error messages, so perhaps this code shoudl be removed once enough time has passed.. * Switch the protocol to using remote URI's, rather than remote names. Names change. Also avoids issues with serialization of names containing whitespace. This is nearly ready for merge into master now. I'd still like to make the ssh transport smarter about reusing ssh connection caching during git pull. This commit was sponsored by Jim Paris.
2014-04-09 18:10:29 +00:00
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
-- A Transport for a particular git remote consumes some messages
-- from a Chan, and emits others to another Chan.
remotedaemon: Fix problem that could prevent ssh connections being made after two LOSTNET messages were received in a row Perhaps due to two different network interfaces being brought down. Since there is no reliable way to drain a Chan, I switched to STM TChan.
2015-01-15 19:37:48 +00:00
type Transport = RemoteRepo -> RemoteURI -> TransportHandle -> TChan Consumed -> TChan Emitted -> IO ()
cleaned up AnnexState handling in transports
2014-04-08 17:41:36 +00:00
remotedaemon: notice on RELOAD when tor hidden service has been enabled and start serving it. This makes the webapp wormhole tor pairing work 100%! This commit was sponsored by Andrea Rota.
2016-12-28 16:21:52 +00:00
-- A server for a Transport consumes some messages from a Chan in
-- order to learn about network changes, reloads, etc.
type Server = TChan Consumed -> TransportHandle -> IO ()
propigate ssh-options everywhere ssh caching is used * sync: Use the ssh-options git config when doing git pull and push. * remotedaemon: Use the ssh-options git config. Note that the rename env var means that if a new git-annex calls an old one for git-annex ssh, or a new calls an old, nothing much will go wrong; just ssh caching won't happen.
2015-02-12 20:12:32 +00:00
data RemoteRepo = RemoteRepo Git.Repo RemoteGitConfig
make LocalRepo a newtype
2016-06-02 20:34:52 +00:00
newtype LocalRepo = LocalRepo Git.Repo
cleaned up AnnexState handling in transports
2014-04-08 17:41:36 +00:00
start splitting out readonly values from AnnexState Values in AnnexRead can be read more efficiently, without MVar overhead. Only a few things have been moved into there, and the performance increase so far is not likely to be noticable. This is groundwork for putting more stuff in there, particularly a value that indicates if debugging is enabled. The obvious next step is to change option parsing to not run in the Annex monad to set values in AnnexState, and instead return a pure value that gets stored in AnnexRead.
2021-04-02 19:26:21 +00:00
-- All Transports share a single AnnexState MVar and an AnnexRead.
fix RELOAD
2014-04-20 19:45:14 +00:00
--
-- Different TransportHandles may have different versions of the LocalRepo.
-- (For example, the ssh transport modifies it to enable ssh connection
-- caching.)
start splitting out readonly values from AnnexState Values in AnnexRead can be read more efficiently, without MVar overhead. Only a few things have been moved into there, and the performance increase so far is not likely to be noticable. This is groundwork for putting more stuff in there, particularly a value that indicates if debugging is enabled. The obvious next step is to change option parsing to not run in the Annex monad to set values in AnnexState, and instead return a pure value that gets stored in AnnexRead.
2021-04-02 19:26:21 +00:00
data TransportHandle = TransportHandle LocalRepo (MVar Annex.AnnexState) Annex.AnnexRead
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
-- Messages that the daemon emits.
data Emitted
make git-remote-daemon ssh transport robust * Remote system might be available, and connection get lost. Should reconnect, but needs to avoid bad behavior (ie, constant reconnect attempts.) Use exponential backoff. * Detect if old system had a too old git-annex-shell, and show the user a nice message in the webapp. Required parsing error messages, so perhaps this code shoudl be removed once enough time has passed.. * Switch the protocol to using remote URI's, rather than remote names. Names change. Also avoids issues with serialization of names containing whitespace. This is nearly ready for merge into master now. I'd still like to make the ssh transport smarter about reusing ssh connection caching during git pull. This commit was sponsored by Jim Paris.
2014-04-09 18:10:29 +00:00
= CONNECTED RemoteURI
| DISCONNECTED RemoteURI
| SYNCING RemoteURI
| DONESYNCING RemoteURI Bool
| WARNING RemoteURI String
send remote-daemon a RELOAD after making a ssh remote This doesn't work yet, because RELOAD is buggy and does not notice the new remote.
2014-04-20 19:30:39 +00:00
deriving (Show)
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
Apply codespell -w throughout
2023-03-14 02:39:16 +00:00
-- Messages that the daemon consumes.
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
data Consumed
= PAUSE
remotedaemon: When network connection is lost, close all cached ssh connections. This commit was sponsored by Cedric Staub.
2014-04-12 20:32:59 +00:00
| LOSTNET
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
| RESUME
refactor ref change watching Added to change notification to P2P protocol. Switched to a TBChan so that a single long-running thread can be started, and serve perhaps intermittent requests for change notifications, without buffering all changes in memory. The P2P runner currently starts up a new thread each times it waits for a change, but that should allow later reusing a thread. Although each connection from a peer will still need a new watcher thread to run. The dependency on stm-chans is more or less free; some stuff in yesod uses it, so it was already indirectly pulled in when building with the webapp. This commit was sponsored by Francois Marier on Patreon.
2016-12-09 18:52:38 +00:00
| CHANGED ChangedRefs
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
| RELOAD
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
| STOP
send remote-daemon a RELOAD after making a ssh remote This doesn't work yet, because RELOAD is buggy and does not notice the new remote.
2014-04-20 19:30:39 +00:00
deriving (Show)
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
instance Proto.Sendable Emitted where
git-annex-shell: Added notifychanges command. This will be used by the remote-daemon to quickly tell when changes have been pushed from some other repository into a ssh remote. Adjusted the remote-daemon protocol to communicate changed shas, rather than git branch refs. This way, it can easily check if a sha is new. This commit was sponsored by Carlos Trijueque Albarran.
2014-04-05 20:04:37 +00:00
formatMessage (CONNECTED remote) =
["CONNECTED", Proto.serialize remote]
formatMessage (DISCONNECTED remote) =
["DISCONNECTED", Proto.serialize remote]
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
formatMessage (SYNCING remote) =
["SYNCING", Proto.serialize remote]
make git-remote-daemon ssh transport robust * Remote system might be available, and connection get lost. Should reconnect, but needs to avoid bad behavior (ie, constant reconnect attempts.) Use exponential backoff. * Detect if old system had a too old git-annex-shell, and show the user a nice message in the webapp. Required parsing error messages, so perhaps this code shoudl be removed once enough time has passed.. * Switch the protocol to using remote URI's, rather than remote names. Names change. Also avoids issues with serialization of names containing whitespace. This is nearly ready for merge into master now. I'd still like to make the ssh transport smarter about reusing ssh connection caching during git pull. This commit was sponsored by Jim Paris.
2014-04-09 18:10:29 +00:00
formatMessage (DONESYNCING remote status) =
["DONESYNCING", Proto.serialize remote, Proto.serialize status]
formatMessage (WARNING remote message) =
["WARNING", Proto.serialize remote, Proto.serialize message]
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
instance Proto.Sendable Consumed where
formatMessage PAUSE = ["PAUSE"]
remotedaemon: When network connection is lost, close all cached ssh connections. This commit was sponsored by Cedric Staub.
2014-04-12 20:32:59 +00:00
formatMessage LOSTNET = ["LOSTNET"]
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
formatMessage RESUME = ["RESUME"]
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
formatMessage (CHANGED refs) =["CHANGED", Proto.serialize refs]
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
formatMessage RELOAD = ["RELOAD"]
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
formatMessage STOP = ["STOP"]
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
instance Proto.Receivable Emitted where
git-annex-shell: Added notifychanges command. This will be used by the remote-daemon to quickly tell when changes have been pushed from some other repository into a ssh remote. Adjusted the remote-daemon protocol to communicate changed shas, rather than git branch refs. This way, it can easily check if a sha is new. This commit was sponsored by Carlos Trijueque Albarran.
2014-04-05 20:04:37 +00:00
parseCommand "CONNECTED" = Proto.parse1 CONNECTED
parseCommand "DISCONNECTED" = Proto.parse1 DISCONNECTED
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
parseCommand "SYNCING" = Proto.parse1 SYNCING
parseCommand "DONESYNCING" = Proto.parse2 DONESYNCING
make git-remote-daemon ssh transport robust * Remote system might be available, and connection get lost. Should reconnect, but needs to avoid bad behavior (ie, constant reconnect attempts.) Use exponential backoff. * Detect if old system had a too old git-annex-shell, and show the user a nice message in the webapp. Required parsing error messages, so perhaps this code shoudl be removed once enough time has passed.. * Switch the protocol to using remote URI's, rather than remote names. Names change. Also avoids issues with serialization of names containing whitespace. This is nearly ready for merge into master now. I'd still like to make the ssh transport smarter about reusing ssh connection caching during git pull. This commit was sponsored by Jim Paris.
2014-04-09 18:10:29 +00:00
parseCommand "WARNING" = Proto.parse2 WARNING
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
parseCommand _ = Proto.parseFail
instance Proto.Receivable Consumed where
parseCommand "PAUSE" = Proto.parse0 PAUSE
remotedaemon: When network connection is lost, close all cached ssh connections. This commit was sponsored by Cedric Staub.
2014-04-12 20:32:59 +00:00
parseCommand "LOSTNET" = Proto.parse0 LOSTNET
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
parseCommand "RESUME" = Proto.parse0 RESUME
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
parseCommand "CHANGED" = Proto.parse1 CHANGED
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
parseCommand "RELOAD" = Proto.parse0 RELOAD
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
parseCommand "STOP" = Proto.parse0 STOP
basic support for git-remote-daemon protocol
2014-04-05 18:30:42 +00:00
parseCommand _ = Proto.parseFail
make git-remote-daemon ssh transport robust * Remote system might be available, and connection get lost. Should reconnect, but needs to avoid bad behavior (ie, constant reconnect attempts.) Use exponential backoff. * Detect if old system had a too old git-annex-shell, and show the user a nice message in the webapp. Required parsing error messages, so perhaps this code shoudl be removed once enough time has passed.. * Switch the protocol to using remote URI's, rather than remote names. Names change. Also avoids issues with serialization of names containing whitespace. This is nearly ready for merge into master now. I'd still like to make the ssh transport smarter about reusing ssh connection caching during git pull. This commit was sponsored by Jim Paris.
2014-04-09 18:10:29 +00:00
instance Proto.Serializable RemoteURI where
serialize (RemoteURI u) = show u
Windows: Support urls like "file:///c:/path" That is a legal url, but parseUrl parses it to "/c:/path" which is not a valid path on Windows. So as a workaround, use parseURIPortable everywhere, which removes the leading slash when run on windows. Note that if an url is parsed like this and then serialized back to a string, it will be different from the input. Which could potentially be a problem, but is probably not in practice. An alternative way to do it would be to have an uriPathPortable that fixes up the path after parsing. But it would be harder to make sure that is used everywhere, since uriPath is also used when constructing an URI. It's also worth noting that System.FilePath.normalize "/c:/path" yields "c:/path". The reason I didn't use it is that it also may change "/" to "\" in the path and I wanted to keep the url changes minimal. Also noticed that convertToWindowsNativeNamespace handles "/c:/path" the same as "c:/path". Sponsored-By: the NIH-funded NICEMAN (ReproNim TR&D3) project
2023-03-27 17:38:02 +00:00
deserialize = RemoteURI <$$> parseURIPortable
make git-remote-daemon ssh transport robust * Remote system might be available, and connection get lost. Should reconnect, but needs to avoid bad behavior (ie, constant reconnect attempts.) Use exponential backoff. * Detect if old system had a too old git-annex-shell, and show the user a nice message in the webapp. Required parsing error messages, so perhaps this code shoudl be removed once enough time has passed.. * Switch the protocol to using remote URI's, rather than remote names. Names change. Also avoids issues with serialization of names containing whitespace. This is nearly ready for merge into master now. I'd still like to make the ssh transport smarter about reusing ssh connection caching during git pull. This commit was sponsored by Jim Paris.
2014-04-09 18:10:29 +00:00
added git-annex remotedaemon So far, handling connecting to git-annex-shell notifychanges, and pulling immediately when a change is pushed to a remote. A little bit buggy (crashes after the first pull), but it already works! This commit was sponsored by Mark Sheppard.
2014-04-06 23:06:03 +00:00
instance Proto.Serializable Bool where
serialize False = "0"
serialize True = "1"
deserialize "0" = Just False
deserialize "1" = Just True
deserialize _ = Nothing
Reference in a new issue Copy permalink