2010-10-27 20:53:54 +00:00
|
|
|
|
{- git-annex monad
|
|
|
|
|
-
|
Switch to MonadCatchIO-transformers for better handling of state while catching exceptions.
As seen in this bug report, the lifted exception handling using the StateT
monad throws away state changes when an action throws an exception.
http://git-annex.branchable.com/bugs/git_annex_fork_bombs_on_gpg_file/
.. Which can result in cached values being redundantly calculated, or other
possibly worse bugs when the annex state gets out of sync with reality.
This switches from a StateT AnnexState to a ReaderT (MVar AnnexState).
All changes to the state go via the MVar. So when an Annex action is
running inside an exception handler, and it makes some changes, they
immediately go into affect in the MVar. If it then throws an exception
(or even crashes its thread!), the state changes are still in effect.
The MonadCatchIO-transformers change is actually only incidental.
I could have kept on using lifted-base for the exception handling.
However, I'd have needed to write a new instance of MonadBaseControl
for the new monad.. and I didn't write the old instance.. I begged Bas
and he kindly sent it to me. Happily, MonadCatchIO-transformers is
able to derive a MonadCatchIO instance for my monad.
This is a deep level change. It passes the test suite! What could it break?
Well.. The most likely breakage would be to code that runs an Annex action
in an exception handler, and *wants* state changes to be thrown away.
Perhaps the state changes leaves the state inconsistent, or wrong. Since
there are relatively few places in git-annex that catch exceptions in the
Annex monad, and the AnnexState is generally just used to cache calculated
data, this is unlikely to be a problem.
Oh yeah, this change also makes Assistant.Types.ThreadedMonad a bit
redundant. It's now entirely possible to run concurrent Annex actions in
different threads, all sharing access to the same state! The ThreadedMonad
just adds some extra work on top of that, with its own MVar, and avoids
such actions possibly stepping on one-another's toes. I have not gotten
rid of it, but might try that later. Being able to run concurrent Annex
actions would simplify parts of the Assistant code.
2013-05-19 18:16:36 +00:00
|
|
|
|
- Copyright 2010-2013 Joey Hess <joey@kitenet.net>
|
2010-10-27 20:53:54 +00:00
|
|
|
|
-
|
|
|
|
|
- Licensed under the GNU GPL version 3 or higher.
|
|
|
|
|
-}
|
2010-10-10 19:04:07 +00:00
|
|
|
|
|
2013-11-22 16:21:37 +00:00
|
|
|
|
{-# LANGUAGE GeneralizedNewtypeDeriving, PackageImports #-}
|
2011-08-19 18:28:07 +00:00
|
|
|
|
|
2010-10-11 21:52:46 +00:00
|
|
|
|
module Annex (
|
2011-01-26 01:49:04 +00:00
|
|
|
|
Annex,
|
|
|
|
|
AnnexState(..),
|
2012-10-04 19:48:59 +00:00
|
|
|
|
PreferredContentMap,
|
2010-10-14 07:18:11 +00:00
|
|
|
|
new,
|
|
|
|
|
run,
|
2010-11-01 03:24:16 +00:00
|
|
|
|
eval,
|
2011-01-26 04:17:38 +00:00
|
|
|
|
getState,
|
|
|
|
|
changeState,
|
more command-specific options
Made --from and --to command-specific options.
Added generic storage for values of command-specific options,
which allows removing some of the special case fields in AnnexState.
(Also added generic storage for command-specific flags, although there are
not yet any.)
Note that this storage uses a Map, so repeatedly looking up the same value
is slightly more expensive than looking up an AnnexState field. But, the
value can be looked up once in the seek stage, transformed as necessary,
and passed in a closure to the start stage, and this avoids that overhead.
Still, I'm hesitant to use this for things like force or fast flags.
It's probably best to reserve it for flags that are only used by a few
commands, or options like --from and --to that it's important only be
allowed to be used with commands that implement them, to avoid user
confusion.
2012-01-06 07:06:25 +00:00
|
|
|
|
setFlag,
|
|
|
|
|
setField,
|
2012-04-30 17:59:05 +00:00
|
|
|
|
setOutput,
|
more command-specific options
Made --from and --to command-specific options.
Added generic storage for values of command-specific options,
which allows removing some of the special case fields in AnnexState.
(Also added generic storage for command-specific flags, although there are
not yet any.)
Note that this storage uses a Map, so repeatedly looking up the same value
is slightly more expensive than looking up an AnnexState field. But, the
value can be looked up once in the seek stage, transformed as necessary,
and passed in a closure to the start stage, and this avoids that overhead.
Still, I'm hesitant to use this for things like force or fast flags.
It's probably best to reserve it for flags that are only used by a few
commands, or options like --from and --to that it's important only be
allowed to be used with commands that implement them, to avoid user
confusion.
2012-01-06 07:06:25 +00:00
|
|
|
|
getFlag,
|
|
|
|
|
getField,
|
2012-02-25 22:02:49 +00:00
|
|
|
|
addCleanup,
|
2011-11-08 19:34:10 +00:00
|
|
|
|
gitRepo,
|
|
|
|
|
inRepo,
|
|
|
|
|
fromRepo,
|
2013-04-04 19:46:33 +00:00
|
|
|
|
calcRepo,
|
2013-01-01 17:52:47 +00:00
|
|
|
|
getGitConfig,
|
|
|
|
|
changeGitConfig,
|
2012-12-30 03:10:18 +00:00
|
|
|
|
changeGitRepo,
|
2013-03-12 20:41:54 +00:00
|
|
|
|
withCurrentState,
|
2010-10-11 21:52:46 +00:00
|
|
|
|
) where
|
2010-10-10 19:04:07 +00:00
|
|
|
|
|
Switch to MonadCatchIO-transformers for better handling of state while catching exceptions.
As seen in this bug report, the lifted exception handling using the StateT
monad throws away state changes when an action throws an exception.
http://git-annex.branchable.com/bugs/git_annex_fork_bombs_on_gpg_file/
.. Which can result in cached values being redundantly calculated, or other
possibly worse bugs when the annex state gets out of sync with reality.
This switches from a StateT AnnexState to a ReaderT (MVar AnnexState).
All changes to the state go via the MVar. So when an Annex action is
running inside an exception handler, and it makes some changes, they
immediately go into affect in the MVar. If it then throws an exception
(or even crashes its thread!), the state changes are still in effect.
The MonadCatchIO-transformers change is actually only incidental.
I could have kept on using lifted-base for the exception handling.
However, I'd have needed to write a new instance of MonadBaseControl
for the new monad.. and I didn't write the old instance.. I begged Bas
and he kindly sent it to me. Happily, MonadCatchIO-transformers is
able to derive a MonadCatchIO instance for my monad.
This is a deep level change. It passes the test suite! What could it break?
Well.. The most likely breakage would be to code that runs an Annex action
in an exception handler, and *wants* state changes to be thrown away.
Perhaps the state changes leaves the state inconsistent, or wrong. Since
there are relatively few places in git-annex that catch exceptions in the
Annex monad, and the AnnexState is generally just used to cache calculated
data, this is unlikely to be a problem.
Oh yeah, this change also makes Assistant.Types.ThreadedMonad a bit
redundant. It's now entirely possible to run concurrent Annex actions in
different threads, all sharing access to the same state! The ThreadedMonad
just adds some extra work on top of that, with its own MVar, and avoids
such actions possibly stepping on one-another's toes. I have not gotten
rid of it, but might try that later. Being able to run concurrent Annex
actions would simplify parts of the Assistant code.
2013-05-19 18:16:36 +00:00
|
|
|
|
import "mtl" Control.Monad.Reader
|
|
|
|
|
import "MonadCatchIO-transformers" Control.Monad.CatchIO
|
|
|
|
|
import Control.Concurrent
|
2010-10-16 20:20:49 +00:00
|
|
|
|
|
2011-10-04 02:24:57 +00:00
|
|
|
|
import Common
|
2011-06-30 17:16:57 +00:00
|
|
|
|
import qualified Git
|
2011-12-13 19:05:07 +00:00
|
|
|
|
import qualified Git.Config
|
2013-11-26 22:11:37 +00:00
|
|
|
|
import Annex.Direct.Fixup
|
2011-09-28 19:15:42 +00:00
|
|
|
|
import Git.CatFile
|
2012-02-14 03:42:44 +00:00
|
|
|
|
import Git.CheckAttr
|
gitignore support for the assistant and watcher
Requires git 1.8.4 or newer. When it's installed, a background
git check-ignore process is run, and used to efficiently check ignores
whenever a new file is added.
Thanks to Adam Spiers, for getting the necessary support into git for this.
A complication is what to do about files that are gitignored but have
been checked into git anyway. git commands assume the ignore has been
overridden in this case, and not need any more overriding to commit a
changed version.
However, for the assistant to do the same, it would have to run git ls-files
to check if the ignored file is in git. This is somewhat expensive. Or it
could use the running git-cat-file process to query the file that way,
but that requires transferring the whole file content over a pipe, so it
can be quite expensive too, for files that are not git-annex
symlinks.
Now imagine if the user knows that a file or directory tree will be getting
frequent changes, and doesn't want the assistant to sync it, so gitignores
it. The assistant could overload the system with repeated ls-files checks!
So, I've decided that the assistant will not automatically commit changes
to files that are gitignored. This is a tradeoff. Hopefully it won't be a
problem to adjust .gitignore settings to not ignore files you want the
assistant to autocommit, or to manually git annex add files that are listed
in .gitignore.
(This could be revisited if git-annex gets access to an interface to check
the content of the index w/o forking a git command. This could be libgit2,
or perhaps a separate git cat-file --batch-check process, so it wouldn't
need to ship over the whole file content.)
This commit was sponsored by Francois Marier. Thanks!
2013-08-02 23:31:55 +00:00
|
|
|
|
import Git.CheckIgnore
|
2012-04-21 23:42:49 +00:00
|
|
|
|
import Git.SharedRepository
|
2011-12-20 18:37:53 +00:00
|
|
|
|
import qualified Git.Queue
|
2014-01-22 20:35:32 +00:00
|
|
|
|
import Types.Key
|
2011-06-02 01:56:04 +00:00
|
|
|
|
import Types.Backend
|
2013-01-01 17:52:47 +00:00
|
|
|
|
import Types.GitConfig
|
2011-09-19 00:11:39 +00:00
|
|
|
|
import qualified Types.Remote
|
2011-06-02 01:56:04 +00:00
|
|
|
|
import Types.Crypto
|
2011-06-22 19:58:30 +00:00
|
|
|
|
import Types.BranchState
|
2011-06-24 01:25:39 +00:00
|
|
|
|
import Types.TrustLevel
|
2012-10-01 19:12:04 +00:00
|
|
|
|
import Types.Group
|
2012-04-27 17:23:52 +00:00
|
|
|
|
import Types.Messages
|
2012-10-04 19:48:59 +00:00
|
|
|
|
import Types.UUID
|
2013-05-25 03:07:26 +00:00
|
|
|
|
import Types.FileMatcher
|
2014-01-21 20:08:19 +00:00
|
|
|
|
import Types.NumCopies
|
2014-01-28 18:17:14 +00:00
|
|
|
|
import Types.LockPool
|
2011-09-18 21:47:49 +00:00
|
|
|
|
import qualified Utility.Matcher
|
2011-12-08 20:01:46 +00:00
|
|
|
|
import qualified Data.Map as M
|
2012-10-05 20:52:44 +00:00
|
|
|
|
import qualified Data.Set as S
|
2011-01-26 01:49:04 +00:00
|
|
|
|
|
Switch to MonadCatchIO-transformers for better handling of state while catching exceptions.
As seen in this bug report, the lifted exception handling using the StateT
monad throws away state changes when an action throws an exception.
http://git-annex.branchable.com/bugs/git_annex_fork_bombs_on_gpg_file/
.. Which can result in cached values being redundantly calculated, or other
possibly worse bugs when the annex state gets out of sync with reality.
This switches from a StateT AnnexState to a ReaderT (MVar AnnexState).
All changes to the state go via the MVar. So when an Annex action is
running inside an exception handler, and it makes some changes, they
immediately go into affect in the MVar. If it then throws an exception
(or even crashes its thread!), the state changes are still in effect.
The MonadCatchIO-transformers change is actually only incidental.
I could have kept on using lifted-base for the exception handling.
However, I'd have needed to write a new instance of MonadBaseControl
for the new monad.. and I didn't write the old instance.. I begged Bas
and he kindly sent it to me. Happily, MonadCatchIO-transformers is
able to derive a MonadCatchIO instance for my monad.
This is a deep level change. It passes the test suite! What could it break?
Well.. The most likely breakage would be to code that runs an Annex action
in an exception handler, and *wants* state changes to be thrown away.
Perhaps the state changes leaves the state inconsistent, or wrong. Since
there are relatively few places in git-annex that catch exceptions in the
Annex monad, and the AnnexState is generally just used to cache calculated
data, this is unlikely to be a problem.
Oh yeah, this change also makes Assistant.Types.ThreadedMonad a bit
redundant. It's now entirely possible to run concurrent Annex actions in
different threads, all sharing access to the same state! The ThreadedMonad
just adds some extra work on top of that, with its own MVar, and avoids
such actions possibly stepping on one-another's toes. I have not gotten
rid of it, but might try that later. Being able to run concurrent Annex
actions would simplify parts of the Assistant code.
2013-05-19 18:16:36 +00:00
|
|
|
|
{- git-annex's monad is a ReaderT around an AnnexState stored in a MVar.
|
|
|
|
|
- This allows modifying the state in an exception-safe fashion.
|
|
|
|
|
- The MVar is not exposed outside this module.
|
|
|
|
|
-}
|
|
|
|
|
newtype Annex a = Annex { runAnnex :: ReaderT (MVar AnnexState) IO a }
|
2011-08-19 18:28:07 +00:00
|
|
|
|
deriving (
|
|
|
|
|
Monad,
|
|
|
|
|
MonadIO,
|
Switch to MonadCatchIO-transformers for better handling of state while catching exceptions.
As seen in this bug report, the lifted exception handling using the StateT
monad throws away state changes when an action throws an exception.
http://git-annex.branchable.com/bugs/git_annex_fork_bombs_on_gpg_file/
.. Which can result in cached values being redundantly calculated, or other
possibly worse bugs when the annex state gets out of sync with reality.
This switches from a StateT AnnexState to a ReaderT (MVar AnnexState).
All changes to the state go via the MVar. So when an Annex action is
running inside an exception handler, and it makes some changes, they
immediately go into affect in the MVar. If it then throws an exception
(or even crashes its thread!), the state changes are still in effect.
The MonadCatchIO-transformers change is actually only incidental.
I could have kept on using lifted-base for the exception handling.
However, I'd have needed to write a new instance of MonadBaseControl
for the new monad.. and I didn't write the old instance.. I begged Bas
and he kindly sent it to me. Happily, MonadCatchIO-transformers is
able to derive a MonadCatchIO instance for my monad.
This is a deep level change. It passes the test suite! What could it break?
Well.. The most likely breakage would be to code that runs an Annex action
in an exception handler, and *wants* state changes to be thrown away.
Perhaps the state changes leaves the state inconsistent, or wrong. Since
there are relatively few places in git-annex that catch exceptions in the
Annex monad, and the AnnexState is generally just used to cache calculated
data, this is unlikely to be a problem.
Oh yeah, this change also makes Assistant.Types.ThreadedMonad a bit
redundant. It's now entirely possible to run concurrent Annex actions in
different threads, all sharing access to the same state! The ThreadedMonad
just adds some extra work on top of that, with its own MVar, and avoids
such actions possibly stepping on one-another's toes. I have not gotten
rid of it, but might try that later. Being able to run concurrent Annex
actions would simplify parts of the Assistant code.
2013-05-19 18:16:36 +00:00
|
|
|
|
MonadReader (MVar AnnexState),
|
|
|
|
|
MonadCatchIO,
|
2011-08-25 04:28:55 +00:00
|
|
|
|
Functor,
|
|
|
|
|
Applicative
|
2011-08-19 18:28:07 +00:00
|
|
|
|
)
|
2011-01-26 01:49:04 +00:00
|
|
|
|
|
2011-12-31 08:19:10 +00:00
|
|
|
|
type Matcher a = Either [Utility.Matcher.Token a] (Utility.Matcher.Matcher a)
|
2014-01-18 18:51:55 +00:00
|
|
|
|
type PreferredContentMap = M.Map UUID (Utility.Matcher.Matcher (S.Set UUID -> MatchInfo -> Annex Bool))
|
2012-10-04 19:48:59 +00:00
|
|
|
|
|
2011-01-26 01:49:04 +00:00
|
|
|
|
-- internal state storage
|
2011-01-26 04:17:38 +00:00
|
|
|
|
data AnnexState = AnnexState
|
|
|
|
|
{ repo :: Git.Repo
|
2013-01-01 17:52:47 +00:00
|
|
|
|
, gitconfig :: GitConfig
|
2011-12-31 08:11:39 +00:00
|
|
|
|
, backends :: [BackendA Annex]
|
|
|
|
|
, remotes :: [Types.Remote.RemoteA Annex]
|
2012-04-27 17:23:52 +00:00
|
|
|
|
, output :: MessageState
|
2011-01-26 04:17:38 +00:00
|
|
|
|
, force :: Bool
|
2011-03-22 21:41:06 +00:00
|
|
|
|
, fast :: Bool
|
2011-09-15 17:30:04 +00:00
|
|
|
|
, auto :: Bool
|
2013-05-25 04:37:41 +00:00
|
|
|
|
, daemon :: Bool
|
2011-06-22 19:58:30 +00:00
|
|
|
|
, branchstate :: BranchState
|
2012-02-15 15:13:13 +00:00
|
|
|
|
, repoqueue :: Maybe Git.Queue.Queue
|
2013-05-15 22:46:38 +00:00
|
|
|
|
, catfilehandles :: M.Map FilePath CatFileHandle
|
2012-02-14 03:42:44 +00:00
|
|
|
|
, checkattrhandle :: Maybe CheckAttrHandle
|
gitignore support for the assistant and watcher
Requires git 1.8.4 or newer. When it's installed, a background
git check-ignore process is run, and used to efficiently check ignores
whenever a new file is added.
Thanks to Adam Spiers, for getting the necessary support into git for this.
A complication is what to do about files that are gitignored but have
been checked into git anyway. git commands assume the ignore has been
overridden in this case, and not need any more overriding to commit a
changed version.
However, for the assistant to do the same, it would have to run git ls-files
to check if the ignored file is in git. This is somewhat expensive. Or it
could use the running git-cat-file process to query the file that way,
but that requires transferring the whole file content over a pipe, so it
can be quite expensive too, for files that are not git-annex
symlinks.
Now imagine if the user knows that a file or directory tree will be getting
frequent changes, and doesn't want the assistant to sync it, so gitignores
it. The assistant could overload the system with repeated ls-files checks!
So, I've decided that the assistant will not automatically commit changes
to files that are gitignored. This is a tradeoff. Hopefully it won't be a
problem to adjust .gitignore settings to not ignore files you want the
assistant to autocommit, or to manually git annex add files that are listed
in .gitignore.
(This could be revisited if git-annex gets access to an interface to check
the content of the index w/o forking a git command. This could be libgit2,
or perhaps a separate git cat-file --batch-check process, so it wouldn't
need to ship over the whole file content.)
This commit was sponsored by Francois Marier. Thanks!
2013-08-02 23:31:55 +00:00
|
|
|
|
, checkignorehandle :: Maybe (Maybe CheckIgnoreHandle)
|
2011-05-18 23:34:46 +00:00
|
|
|
|
, forcebackend :: Maybe String
|
2014-01-21 20:08:19 +00:00
|
|
|
|
, globalnumcopies :: Maybe NumCopies
|
2014-01-21 21:08:49 +00:00
|
|
|
|
, forcenumcopies :: Maybe NumCopies
|
2014-01-18 18:51:55 +00:00
|
|
|
|
, limit :: Matcher (MatchInfo -> Annex Bool)
|
add ConfigMonitor thread
Monitors git-annex branch for changes, which are noticed by the Merger
thread whenever the branch ref is changed (either due to an incoming push,
or a local change), and refreshes cached config values for modified config
files.
Rate limited to run no more often than once per minute. This is important
because frequent git-annex branch changes happen when files are being
added, or transferred, etc.
A primary use case is that, when preferred content changes are made,
and get pushed to remotes, the remotes start honoring those settings.
Other use cases include propigating repository description and trust
changes to remotes, and learning when a remote has added a new special
remote, so the webapp can present the GUI to enable that special remote
locally.
Also added a uuid.log cache. All other config files already had caches.
2012-10-20 20:37:06 +00:00
|
|
|
|
, uuidmap :: Maybe UUIDMap
|
2012-10-04 19:48:59 +00:00
|
|
|
|
, preferredcontentmap :: Maybe PreferredContentMap
|
2012-04-21 23:42:49 +00:00
|
|
|
|
, shared :: Maybe SharedRepository
|
2012-01-10 03:31:44 +00:00
|
|
|
|
, forcetrust :: TrustMap
|
2011-06-24 01:25:39 +00:00
|
|
|
|
, trustmap :: Maybe TrustMap
|
2012-10-01 19:12:04 +00:00
|
|
|
|
, groupmap :: Maybe GroupMap
|
2012-04-29 18:02:18 +00:00
|
|
|
|
, ciphers :: M.Map StorableCipher Cipher
|
2014-01-28 18:17:14 +00:00
|
|
|
|
, lockpool :: LockPool
|
more command-specific options
Made --from and --to command-specific options.
Added generic storage for values of command-specific options,
which allows removing some of the special case fields in AnnexState.
(Also added generic storage for command-specific flags, although there are
not yet any.)
Note that this storage uses a Map, so repeatedly looking up the same value
is slightly more expensive than looking up an AnnexState field. But, the
value can be looked up once in the seek stage, transformed as necessary,
and passed in a closure to the start stage, and this avoids that overhead.
Still, I'm hesitant to use this for things like force or fast flags.
It's probably best to reserve it for flags that are only used by a few
commands, or options like --from and --to that it's important only be
allowed to be used with commands that implement them, to avoid user
confusion.
2012-01-06 07:06:25 +00:00
|
|
|
|
, flags :: M.Map String Bool
|
|
|
|
|
, fields :: M.Map String String
|
2012-02-25 22:02:49 +00:00
|
|
|
|
, cleanup :: M.Map String (Annex ())
|
2013-02-19 20:26:07 +00:00
|
|
|
|
, inodeschanged :: Maybe Bool
|
2013-09-28 18:35:21 +00:00
|
|
|
|
, useragent :: Maybe String
|
fix inversion of control in CommandSeek (no behavior changes)
I've been disliking how the command seek actions were written for some
time, with their inversion of control and ugly workarounds.
The last straw to fix it was sync --content, which didn't fit the
Annex [CommandStart] interface well at all. I have not yet made it take
advantage of the changed interface though.
The crucial change, and probably why I didn't do it this way from the
beginning, is to make each CommandStart action be run with exceptions
caught, and if it fails, increment a failure counter in annex state.
So I finally remove the very first code I wrote for git-annex, which
was before I had exception handling in the Annex monad, and so ran outside
that monad, passing state explicitly as it ran each CommandStart action.
This was a real slog from 1 to 5 am.
Test suite passes.
Memory usage is lower than before, sometimes by a couple of megabytes, and
remains constant, even when running in a large repo, and even when
repeatedly failing and incrementing the error counter. So no accidental
laziness space leaks.
Wall clock speed is identical, even in large repos.
This commit was sponsored by an anonymous bitcoiner.
2014-01-20 08:11:42 +00:00
|
|
|
|
, errcounter :: Integer
|
2014-01-22 20:35:32 +00:00
|
|
|
|
, unusedkeys :: Maybe (S.Set Key)
|
2011-04-16 20:41:46 +00:00
|
|
|
|
}
|
2011-01-26 04:17:38 +00:00
|
|
|
|
|
2013-11-26 22:11:37 +00:00
|
|
|
|
newState :: GitConfig -> Git.Repo -> AnnexState
|
|
|
|
|
newState c r = AnnexState
|
|
|
|
|
{ repo = r
|
2013-11-05 18:24:28 +00:00
|
|
|
|
, gitconfig = c
|
2011-01-26 04:17:38 +00:00
|
|
|
|
, backends = []
|
2011-03-27 20:17:56 +00:00
|
|
|
|
, remotes = []
|
2012-04-27 17:23:52 +00:00
|
|
|
|
, output = defaultMessageState
|
2011-01-26 04:17:38 +00:00
|
|
|
|
, force = False
|
2011-03-22 21:41:06 +00:00
|
|
|
|
, fast = False
|
2011-09-15 17:30:04 +00:00
|
|
|
|
, auto = False
|
2013-05-25 04:37:41 +00:00
|
|
|
|
, daemon = False
|
2011-06-22 19:58:30 +00:00
|
|
|
|
, branchstate = startBranchState
|
2012-02-15 15:13:13 +00:00
|
|
|
|
, repoqueue = Nothing
|
2013-05-15 22:46:38 +00:00
|
|
|
|
, catfilehandles = M.empty
|
2012-02-14 03:42:44 +00:00
|
|
|
|
, checkattrhandle = Nothing
|
gitignore support for the assistant and watcher
Requires git 1.8.4 or newer. When it's installed, a background
git check-ignore process is run, and used to efficiently check ignores
whenever a new file is added.
Thanks to Adam Spiers, for getting the necessary support into git for this.
A complication is what to do about files that are gitignored but have
been checked into git anyway. git commands assume the ignore has been
overridden in this case, and not need any more overriding to commit a
changed version.
However, for the assistant to do the same, it would have to run git ls-files
to check if the ignored file is in git. This is somewhat expensive. Or it
could use the running git-cat-file process to query the file that way,
but that requires transferring the whole file content over a pipe, so it
can be quite expensive too, for files that are not git-annex
symlinks.
Now imagine if the user knows that a file or directory tree will be getting
frequent changes, and doesn't want the assistant to sync it, so gitignores
it. The assistant could overload the system with repeated ls-files checks!
So, I've decided that the assistant will not automatically commit changes
to files that are gitignored. This is a tradeoff. Hopefully it won't be a
problem to adjust .gitignore settings to not ignore files you want the
assistant to autocommit, or to manually git annex add files that are listed
in .gitignore.
(This could be revisited if git-annex gets access to an interface to check
the content of the index w/o forking a git command. This could be libgit2,
or perhaps a separate git cat-file --batch-check process, so it wouldn't
need to ship over the whole file content.)
This commit was sponsored by Francois Marier. Thanks!
2013-08-02 23:31:55 +00:00
|
|
|
|
, checkignorehandle = Nothing
|
2011-05-18 23:34:46 +00:00
|
|
|
|
, forcebackend = Nothing
|
2014-01-20 20:47:56 +00:00
|
|
|
|
, globalnumcopies = Nothing
|
2014-01-21 21:08:49 +00:00
|
|
|
|
, forcenumcopies = Nothing
|
2011-09-18 21:47:49 +00:00
|
|
|
|
, limit = Left []
|
add ConfigMonitor thread
Monitors git-annex branch for changes, which are noticed by the Merger
thread whenever the branch ref is changed (either due to an incoming push,
or a local change), and refreshes cached config values for modified config
files.
Rate limited to run no more often than once per minute. This is important
because frequent git-annex branch changes happen when files are being
added, or transferred, etc.
A primary use case is that, when preferred content changes are made,
and get pushed to remotes, the remotes start honoring those settings.
Other use cases include propigating repository description and trust
changes to remotes, and learning when a remote has added a new special
remote, so the webapp can present the GUI to enable that special remote
locally.
Also added a uuid.log cache. All other config files already had caches.
2012-10-20 20:37:06 +00:00
|
|
|
|
, uuidmap = Nothing
|
2012-10-04 19:48:59 +00:00
|
|
|
|
, preferredcontentmap = Nothing
|
2012-04-21 23:42:49 +00:00
|
|
|
|
, shared = Nothing
|
2012-01-10 03:31:44 +00:00
|
|
|
|
, forcetrust = M.empty
|
2011-06-24 01:25:39 +00:00
|
|
|
|
, trustmap = Nothing
|
2012-10-01 19:12:04 +00:00
|
|
|
|
, groupmap = Nothing
|
2011-12-08 20:01:46 +00:00
|
|
|
|
, ciphers = M.empty
|
2012-01-20 19:34:52 +00:00
|
|
|
|
, lockpool = M.empty
|
more command-specific options
Made --from and --to command-specific options.
Added generic storage for values of command-specific options,
which allows removing some of the special case fields in AnnexState.
(Also added generic storage for command-specific flags, although there are
not yet any.)
Note that this storage uses a Map, so repeatedly looking up the same value
is slightly more expensive than looking up an AnnexState field. But, the
value can be looked up once in the seek stage, transformed as necessary,
and passed in a closure to the start stage, and this avoids that overhead.
Still, I'm hesitant to use this for things like force or fast flags.
It's probably best to reserve it for flags that are only used by a few
commands, or options like --from and --to that it's important only be
allowed to be used with commands that implement them, to avoid user
confusion.
2012-01-06 07:06:25 +00:00
|
|
|
|
, flags = M.empty
|
|
|
|
|
, fields = M.empty
|
2012-02-25 22:02:49 +00:00
|
|
|
|
, cleanup = M.empty
|
2013-02-19 20:26:07 +00:00
|
|
|
|
, inodeschanged = Nothing
|
2013-09-28 18:35:21 +00:00
|
|
|
|
, useragent = Nothing
|
fix inversion of control in CommandSeek (no behavior changes)
I've been disliking how the command seek actions were written for some
time, with their inversion of control and ugly workarounds.
The last straw to fix it was sync --content, which didn't fit the
Annex [CommandStart] interface well at all. I have not yet made it take
advantage of the changed interface though.
The crucial change, and probably why I didn't do it this way from the
beginning, is to make each CommandStart action be run with exceptions
caught, and if it fails, increment a failure counter in annex state.
So I finally remove the very first code I wrote for git-annex, which
was before I had exception handling in the Annex monad, and so ran outside
that monad, passing state explicitly as it ran each CommandStart action.
This was a real slog from 1 to 5 am.
Test suite passes.
Memory usage is lower than before, sometimes by a couple of megabytes, and
remains constant, even when running in a large repo, and even when
repeatedly failing and incrementing the error counter. So no accidental
laziness space leaks.
Wall clock speed is identical, even in large repos.
This commit was sponsored by an anonymous bitcoiner.
2014-01-20 08:11:42 +00:00
|
|
|
|
, errcounter = 0
|
2014-01-22 20:35:32 +00:00
|
|
|
|
, unusedkeys = Nothing
|
2011-01-26 04:17:38 +00:00
|
|
|
|
}
|
2010-10-14 07:18:11 +00:00
|
|
|
|
|
2012-05-18 22:20:53 +00:00
|
|
|
|
{- Makes an Annex state object for the specified git repo.
|
|
|
|
|
- Ensures the config is read, if it was not already. -}
|
2011-07-05 22:31:46 +00:00
|
|
|
|
new :: Git.Repo -> IO AnnexState
|
2013-11-26 22:11:37 +00:00
|
|
|
|
new r = do
|
|
|
|
|
r' <- Git.Config.read r
|
|
|
|
|
let c = extractGitConfig r'
|
|
|
|
|
newState c <$> if annexDirect c then fixupDirect r' else return r'
|
2010-10-14 07:18:11 +00:00
|
|
|
|
|
Switch to MonadCatchIO-transformers for better handling of state while catching exceptions.
As seen in this bug report, the lifted exception handling using the StateT
monad throws away state changes when an action throws an exception.
http://git-annex.branchable.com/bugs/git_annex_fork_bombs_on_gpg_file/
.. Which can result in cached values being redundantly calculated, or other
possibly worse bugs when the annex state gets out of sync with reality.
This switches from a StateT AnnexState to a ReaderT (MVar AnnexState).
All changes to the state go via the MVar. So when an Annex action is
running inside an exception handler, and it makes some changes, they
immediately go into affect in the MVar. If it then throws an exception
(or even crashes its thread!), the state changes are still in effect.
The MonadCatchIO-transformers change is actually only incidental.
I could have kept on using lifted-base for the exception handling.
However, I'd have needed to write a new instance of MonadBaseControl
for the new monad.. and I didn't write the old instance.. I begged Bas
and he kindly sent it to me. Happily, MonadCatchIO-transformers is
able to derive a MonadCatchIO instance for my monad.
This is a deep level change. It passes the test suite! What could it break?
Well.. The most likely breakage would be to code that runs an Annex action
in an exception handler, and *wants* state changes to be thrown away.
Perhaps the state changes leaves the state inconsistent, or wrong. Since
there are relatively few places in git-annex that catch exceptions in the
Annex monad, and the AnnexState is generally just used to cache calculated
data, this is unlikely to be a problem.
Oh yeah, this change also makes Assistant.Types.ThreadedMonad a bit
redundant. It's now entirely possible to run concurrent Annex actions in
different threads, all sharing access to the same state! The ThreadedMonad
just adds some extra work on top of that, with its own MVar, and avoids
such actions possibly stepping on one-another's toes. I have not gotten
rid of it, but might try that later. Being able to run concurrent Annex
actions would simplify parts of the Assistant code.
2013-05-19 18:16:36 +00:00
|
|
|
|
{- Performs an action in the Annex monad from a starting state,
|
|
|
|
|
- returning a new state. -}
|
2011-01-11 22:13:26 +00:00
|
|
|
|
run :: AnnexState -> Annex a -> IO (a, AnnexState)
|
Switch to MonadCatchIO-transformers for better handling of state while catching exceptions.
As seen in this bug report, the lifted exception handling using the StateT
monad throws away state changes when an action throws an exception.
http://git-annex.branchable.com/bugs/git_annex_fork_bombs_on_gpg_file/
.. Which can result in cached values being redundantly calculated, or other
possibly worse bugs when the annex state gets out of sync with reality.
This switches from a StateT AnnexState to a ReaderT (MVar AnnexState).
All changes to the state go via the MVar. So when an Annex action is
running inside an exception handler, and it makes some changes, they
immediately go into affect in the MVar. If it then throws an exception
(or even crashes its thread!), the state changes are still in effect.
The MonadCatchIO-transformers change is actually only incidental.
I could have kept on using lifted-base for the exception handling.
However, I'd have needed to write a new instance of MonadBaseControl
for the new monad.. and I didn't write the old instance.. I begged Bas
and he kindly sent it to me. Happily, MonadCatchIO-transformers is
able to derive a MonadCatchIO instance for my monad.
This is a deep level change. It passes the test suite! What could it break?
Well.. The most likely breakage would be to code that runs an Annex action
in an exception handler, and *wants* state changes to be thrown away.
Perhaps the state changes leaves the state inconsistent, or wrong. Since
there are relatively few places in git-annex that catch exceptions in the
Annex monad, and the AnnexState is generally just used to cache calculated
data, this is unlikely to be a problem.
Oh yeah, this change also makes Assistant.Types.ThreadedMonad a bit
redundant. It's now entirely possible to run concurrent Annex actions in
different threads, all sharing access to the same state! The ThreadedMonad
just adds some extra work on top of that, with its own MVar, and avoids
such actions possibly stepping on one-another's toes. I have not gotten
rid of it, but might try that later. Being able to run concurrent Annex
actions would simplify parts of the Assistant code.
2013-05-19 18:16:36 +00:00
|
|
|
|
run s a = do
|
|
|
|
|
mvar <- newMVar s
|
|
|
|
|
r <- runReaderT (runAnnex a) mvar
|
|
|
|
|
s' <- takeMVar mvar
|
|
|
|
|
return (r, s')
|
|
|
|
|
|
|
|
|
|
{- Performs an action in the Annex monad from a starting state,
|
|
|
|
|
- and throws away the new state. -}
|
2011-01-11 22:13:26 +00:00
|
|
|
|
eval :: AnnexState -> Annex a -> IO a
|
Switch to MonadCatchIO-transformers for better handling of state while catching exceptions.
As seen in this bug report, the lifted exception handling using the StateT
monad throws away state changes when an action throws an exception.
http://git-annex.branchable.com/bugs/git_annex_fork_bombs_on_gpg_file/
.. Which can result in cached values being redundantly calculated, or other
possibly worse bugs when the annex state gets out of sync with reality.
This switches from a StateT AnnexState to a ReaderT (MVar AnnexState).
All changes to the state go via the MVar. So when an Annex action is
running inside an exception handler, and it makes some changes, they
immediately go into affect in the MVar. If it then throws an exception
(or even crashes its thread!), the state changes are still in effect.
The MonadCatchIO-transformers change is actually only incidental.
I could have kept on using lifted-base for the exception handling.
However, I'd have needed to write a new instance of MonadBaseControl
for the new monad.. and I didn't write the old instance.. I begged Bas
and he kindly sent it to me. Happily, MonadCatchIO-transformers is
able to derive a MonadCatchIO instance for my monad.
This is a deep level change. It passes the test suite! What could it break?
Well.. The most likely breakage would be to code that runs an Annex action
in an exception handler, and *wants* state changes to be thrown away.
Perhaps the state changes leaves the state inconsistent, or wrong. Since
there are relatively few places in git-annex that catch exceptions in the
Annex monad, and the AnnexState is generally just used to cache calculated
data, this is unlikely to be a problem.
Oh yeah, this change also makes Assistant.Types.ThreadedMonad a bit
redundant. It's now entirely possible to run concurrent Annex actions in
different threads, all sharing access to the same state! The ThreadedMonad
just adds some extra work on top of that, with its own MVar, and avoids
such actions possibly stepping on one-another's toes. I have not gotten
rid of it, but might try that later. Being able to run concurrent Annex
actions would simplify parts of the Assistant code.
2013-05-19 18:16:36 +00:00
|
|
|
|
eval s a = do
|
|
|
|
|
mvar <- newMVar s
|
|
|
|
|
runReaderT (runAnnex a) mvar
|
|
|
|
|
|
|
|
|
|
getState :: (AnnexState -> v) -> Annex v
|
|
|
|
|
getState selector = do
|
|
|
|
|
mvar <- ask
|
|
|
|
|
s <- liftIO $ readMVar mvar
|
|
|
|
|
return $ selector s
|
|
|
|
|
|
|
|
|
|
changeState :: (AnnexState -> AnnexState) -> Annex ()
|
|
|
|
|
changeState modifier = do
|
|
|
|
|
mvar <- ask
|
|
|
|
|
liftIO $ modifyMVar_ mvar $ return . modifier
|
2010-10-14 07:18:11 +00:00
|
|
|
|
|
more command-specific options
Made --from and --to command-specific options.
Added generic storage for values of command-specific options,
which allows removing some of the special case fields in AnnexState.
(Also added generic storage for command-specific flags, although there are
not yet any.)
Note that this storage uses a Map, so repeatedly looking up the same value
is slightly more expensive than looking up an AnnexState field. But, the
value can be looked up once in the seek stage, transformed as necessary,
and passed in a closure to the start stage, and this avoids that overhead.
Still, I'm hesitant to use this for things like force or fast flags.
It's probably best to reserve it for flags that are only used by a few
commands, or options like --from and --to that it's important only be
allowed to be used with commands that implement them, to avoid user
confusion.
2012-01-06 07:06:25 +00:00
|
|
|
|
{- Sets a flag to True -}
|
|
|
|
|
setFlag :: String -> Annex ()
|
|
|
|
|
setFlag flag = changeState $ \s ->
|
2012-02-25 22:02:49 +00:00
|
|
|
|
s { flags = M.insertWith' const flag True $ flags s }
|
more command-specific options
Made --from and --to command-specific options.
Added generic storage for values of command-specific options,
which allows removing some of the special case fields in AnnexState.
(Also added generic storage for command-specific flags, although there are
not yet any.)
Note that this storage uses a Map, so repeatedly looking up the same value
is slightly more expensive than looking up an AnnexState field. But, the
value can be looked up once in the seek stage, transformed as necessary,
and passed in a closure to the start stage, and this avoids that overhead.
Still, I'm hesitant to use this for things like force or fast flags.
It's probably best to reserve it for flags that are only used by a few
commands, or options like --from and --to that it's important only be
allowed to be used with commands that implement them, to avoid user
confusion.
2012-01-06 07:06:25 +00:00
|
|
|
|
|
|
|
|
|
{- Sets a field to a value -}
|
|
|
|
|
setField :: String -> String -> Annex ()
|
|
|
|
|
setField field value = changeState $ \s ->
|
2012-02-25 22:02:49 +00:00
|
|
|
|
s { fields = M.insertWith' const field value $ fields s }
|
|
|
|
|
|
|
|
|
|
{- Adds a cleanup action to perform. -}
|
|
|
|
|
addCleanup :: String -> Annex () -> Annex ()
|
|
|
|
|
addCleanup uid a = changeState $ \s ->
|
|
|
|
|
s { cleanup = M.insertWith' const uid a $ cleanup s }
|
more command-specific options
Made --from and --to command-specific options.
Added generic storage for values of command-specific options,
which allows removing some of the special case fields in AnnexState.
(Also added generic storage for command-specific flags, although there are
not yet any.)
Note that this storage uses a Map, so repeatedly looking up the same value
is slightly more expensive than looking up an AnnexState field. But, the
value can be looked up once in the seek stage, transformed as necessary,
and passed in a closure to the start stage, and this avoids that overhead.
Still, I'm hesitant to use this for things like force or fast flags.
It's probably best to reserve it for flags that are only used by a few
commands, or options like --from and --to that it's important only be
allowed to be used with commands that implement them, to avoid user
confusion.
2012-01-06 07:06:25 +00:00
|
|
|
|
|
2012-04-30 17:59:05 +00:00
|
|
|
|
{- Sets the type of output to emit. -}
|
|
|
|
|
setOutput :: OutputType -> Annex ()
|
|
|
|
|
setOutput o = changeState $ \s ->
|
|
|
|
|
s { output = (output s) { outputType = o } }
|
|
|
|
|
|
more command-specific options
Made --from and --to command-specific options.
Added generic storage for values of command-specific options,
which allows removing some of the special case fields in AnnexState.
(Also added generic storage for command-specific flags, although there are
not yet any.)
Note that this storage uses a Map, so repeatedly looking up the same value
is slightly more expensive than looking up an AnnexState field. But, the
value can be looked up once in the seek stage, transformed as necessary,
and passed in a closure to the start stage, and this avoids that overhead.
Still, I'm hesitant to use this for things like force or fast flags.
It's probably best to reserve it for flags that are only used by a few
commands, or options like --from and --to that it's important only be
allowed to be used with commands that implement them, to avoid user
confusion.
2012-01-06 07:06:25 +00:00
|
|
|
|
{- Checks if a flag was set. -}
|
|
|
|
|
getFlag :: String -> Annex Bool
|
|
|
|
|
getFlag flag = fromMaybe False . M.lookup flag <$> getState flags
|
|
|
|
|
|
|
|
|
|
{- Gets the value of a field. -}
|
|
|
|
|
getField :: String -> Annex (Maybe String)
|
|
|
|
|
getField field = M.lookup field <$> getState fields
|
|
|
|
|
|
2011-11-08 19:34:10 +00:00
|
|
|
|
{- Returns the annex's git repository. -}
|
2010-10-14 07:18:11 +00:00
|
|
|
|
gitRepo :: Annex Git.Repo
|
2011-01-26 01:49:04 +00:00
|
|
|
|
gitRepo = getState repo
|
2011-11-08 19:34:10 +00:00
|
|
|
|
|
|
|
|
|
{- Runs an IO action in the annex's git repository. -}
|
|
|
|
|
inRepo :: (Git.Repo -> IO a) -> Annex a
|
2011-11-12 18:24:07 +00:00
|
|
|
|
inRepo a = liftIO . a =<< gitRepo
|
2011-11-08 19:34:10 +00:00
|
|
|
|
|
|
|
|
|
{- Extracts a value from the annex's git repisitory. -}
|
|
|
|
|
fromRepo :: (Git.Repo -> a) -> Annex a
|
|
|
|
|
fromRepo a = a <$> gitRepo
|
2012-12-30 03:10:18 +00:00
|
|
|
|
|
Switch to MonadCatchIO-transformers for better handling of state while catching exceptions.
As seen in this bug report, the lifted exception handling using the StateT
monad throws away state changes when an action throws an exception.
http://git-annex.branchable.com/bugs/git_annex_fork_bombs_on_gpg_file/
.. Which can result in cached values being redundantly calculated, or other
possibly worse bugs when the annex state gets out of sync with reality.
This switches from a StateT AnnexState to a ReaderT (MVar AnnexState).
All changes to the state go via the MVar. So when an Annex action is
running inside an exception handler, and it makes some changes, they
immediately go into affect in the MVar. If it then throws an exception
(or even crashes its thread!), the state changes are still in effect.
The MonadCatchIO-transformers change is actually only incidental.
I could have kept on using lifted-base for the exception handling.
However, I'd have needed to write a new instance of MonadBaseControl
for the new monad.. and I didn't write the old instance.. I begged Bas
and he kindly sent it to me. Happily, MonadCatchIO-transformers is
able to derive a MonadCatchIO instance for my monad.
This is a deep level change. It passes the test suite! What could it break?
Well.. The most likely breakage would be to code that runs an Annex action
in an exception handler, and *wants* state changes to be thrown away.
Perhaps the state changes leaves the state inconsistent, or wrong. Since
there are relatively few places in git-annex that catch exceptions in the
Annex monad, and the AnnexState is generally just used to cache calculated
data, this is unlikely to be a problem.
Oh yeah, this change also makes Assistant.Types.ThreadedMonad a bit
redundant. It's now entirely possible to run concurrent Annex actions in
different threads, all sharing access to the same state! The ThreadedMonad
just adds some extra work on top of that, with its own MVar, and avoids
such actions possibly stepping on one-another's toes. I have not gotten
rid of it, but might try that later. Being able to run concurrent Annex
actions would simplify parts of the Assistant code.
2013-05-19 18:16:36 +00:00
|
|
|
|
{- Calculates a value from an annex's git repository and its GitConfig. -}
|
2013-04-04 19:46:33 +00:00
|
|
|
|
calcRepo :: (Git.Repo -> GitConfig -> IO a) -> Annex a
|
|
|
|
|
calcRepo a = do
|
|
|
|
|
s <- getState id
|
|
|
|
|
liftIO $ a (repo s) (gitconfig s)
|
|
|
|
|
|
2013-01-01 17:52:47 +00:00
|
|
|
|
{- Gets the GitConfig settings. -}
|
|
|
|
|
getGitConfig :: Annex GitConfig
|
|
|
|
|
getGitConfig = getState gitconfig
|
2012-12-30 03:10:18 +00:00
|
|
|
|
|
2013-01-01 17:52:47 +00:00
|
|
|
|
{- Modifies a GitConfig setting. -}
|
|
|
|
|
changeGitConfig :: (GitConfig -> GitConfig) -> Annex ()
|
|
|
|
|
changeGitConfig a = changeState $ \s -> s { gitconfig = a (gitconfig s) }
|
2012-12-30 03:10:18 +00:00
|
|
|
|
|
2013-01-01 17:52:47 +00:00
|
|
|
|
{- Changing the git Repo data also involves re-extracting its GitConfig. -}
|
2012-12-30 03:10:18 +00:00
|
|
|
|
changeGitRepo :: Git.Repo -> Annex ()
|
|
|
|
|
changeGitRepo r = changeState $ \s -> s
|
|
|
|
|
{ repo = r
|
2013-01-01 17:52:47 +00:00
|
|
|
|
, gitconfig = extractGitConfig r
|
2012-12-30 03:10:18 +00:00
|
|
|
|
}
|
2013-03-12 20:41:54 +00:00
|
|
|
|
|
|
|
|
|
{- Converts an Annex action into an IO action, that runs with a copy
|
|
|
|
|
- of the current Annex state.
|
|
|
|
|
-
|
|
|
|
|
- Use with caution; the action should not rely on changing the
|
|
|
|
|
- state, as it will be thrown away. -}
|
|
|
|
|
withCurrentState :: Annex a -> Annex (IO a)
|
|
|
|
|
withCurrentState a = do
|
|
|
|
|
s <- getState id
|
|
|
|
|
return $ eval s a
|