mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Annex/Queue.hs

95 lines
2.4 KiB
Haskell
Raw Normal View History

Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
{- git-annex command queue
-
Improved locking when multiple git-annex processes are writing to the .git/index file
2019-05-06 19:15:12 +00:00
- Copyright 2011, 2012, 2019 Joey Hess <id@joeyh.name>
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
-
update licenses from GPL to AGPL This does not change the overall license of the git-annex program, which was already AGPL due to a number of sources files being AGPL already. Legally speaking, I'm adding a new license under which these files are now available; I already released their current contents under the GPL license. Now they're dual licensed GPL and AGPL. However, I intend for all my future changes to these files to only be released under the AGPL license, and I won't be tracking the dual licensing status, so I'm simply changing the license statement to say it's AGPL. (In some cases, others wrote parts of the code of a file and released it under the GPL; but in all cases I have contributed a significant portion of the code in each file and it's that code that is getting the AGPL license; the GPL license of other contributors allows combining with AGPL code.)
2019-03-13 19:48:14 +00:00
- Licensed under the GNU AGPL version 3 or higher.
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
-}
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
{-# LANGUAGE BangPatterns #-}
rename
2011-10-04 04:40:47 +00:00
module Annex.Queue (
extend Git.Queue to be able to queue more than simple git commands While I was in there, I noticed and fixed a bug in the queue size calculations. It was never encountered only because Queue.add was only ever run with 1 file in the list.
2012-06-07 19:19:44 +00:00
addCommand,
queueing of internal IO actions on files This would be better if getInternalFiles were more polymorphic, but I can't see a good way to accomplish that without messing with Data.Typeable, which seemed like overkill. Reverted CommandAction back to the simpler version. This commit was sponsored by Eric Drechsel on Patreon.
2018-08-17 17:24:52 +00:00
addInternalAction,
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
addUpdateIndex,
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
flush,
queue size fix Increase queue size for update-index actions, because otherwise they'll never be flushed.
2012-06-10 17:56:04 +00:00
flushWhenFull,
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
size,
withAltRepo needs a separate queue of changes The queue could potentially contain changes from before withAltRepo, and get flushed inside the call, which would apply the changes to the modified repo. Or, changes could be queued in withAltRepo that were intended to affect the modified repo, but don't get flushed until later. I don't know of any cases where either happens, but better safe than sorry. Note that this affect withIndexFile, which is used in git-annex branch updates. So, it potentially makes things slower. Should not be by much; the overhead consists only of querying the current queue a couple of times, and potentially flushing changes queued within withAltRepo earlier, that could have maybe been bundled with other later changes. Notice in particular that the existing queue is not flushed when calling withAltRepo. So eg when git annex add needs to stage files in the index, it will still bundle them together efficiently.
2016-06-03 17:48:14 +00:00
get,
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
mergeFrom,
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
) where
remove 163 lines of code without changing anything except imports
2016-01-20 20:36:33 +00:00
import Annex.Common
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
import Annex hiding (new)
Improved locking when multiple git-annex processes are writing to the .git/index file
2019-05-06 19:15:12 +00:00
import Annex.LockFile
rename GitQueue to Git.Queue
2011-06-30 17:25:37 +00:00
import qualified Git.Queue
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
import qualified Git.UpdateIndex
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
outdated comment
2011-08-21 16:59:49 +00:00
{- Adds a git command to the queue. -}
extend Git.Queue to be able to queue more than simple git commands While I was in there, I noticed and fixed a bug in the queue size calculations. It was never encountered only because Queue.add was only ever run with 1 file in the list.
2012-06-07 19:19:44 +00:00
addCommand :: String -> [CommandParam] -> [FilePath] -> Annex ()
addCommand command params files = do
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
q <- get
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
store =<< flushWhenFull =<<
(Git.Queue.addCommand command params files q =<< gitRepo)
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
addInternalAction :: Git.Queue.InternalActionRunner Annex -> [(FilePath, IO Bool)] -> Annex ()
queueing of internal IO actions on files This would be better if getInternalFiles were more polymorphic, but I can't see a good way to accomplish that without messing with Data.Typeable, which seemed like overkill. Reverted CommandAction back to the simpler version. This commit was sponsored by Eric Drechsel on Patreon.
2018-08-17 17:24:52 +00:00
addInternalAction runner files = do
narrow the race where a file gets modified before update-index Check just before running update-index if the worktree file's content is still the same, don't update it when it's been modified. This narrows the race window a lot, from possibly minutes or hours, to seconds or less. (Use replaceFile so that the worktree update happens atomically, allowing the InodeCache of the new worktree file to itself be gathered w/o any other race.) This doesn't eliminate the race; it can still occur in the window before update-index runs. When annex.queue is large, a lot of files will be statted by the checks, and so the window may still be large enough to be a problem. When only a few files are being processed, the window is as small as it is in the race where a modification gets overwritten by git-annex when it updates the worktree. Or maybe as small as whatever race git checkout/pull/merge may have when the worktree gets modified during it. Still, I've kept a todo about this race. This commit was supported by the NSF-funded DataLad project.
2018-08-16 19:15:20 +00:00
q <- get
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
store =<< flushWhenFull =<<
(Git.Queue.addInternalAction runner files q =<< gitRepo)
narrow the race where a file gets modified before update-index Check just before running update-index if the worktree file's content is still the same, don't update it when it's been modified. This narrows the race window a lot, from possibly minutes or hours, to seconds or less. (Use replaceFile so that the worktree update happens atomically, allowing the InodeCache of the new worktree file to itself be gathered w/o any other race.) This doesn't eliminate the race; it can still occur in the window before update-index runs. When annex.queue is large, a lot of files will be statted by the checks, and so the window may still be large enough to be a problem. When only a few files are being processed, the window is as small as it is in the race where a modification gets overwritten by git-annex when it updates the worktree. Or maybe as small as whatever race git checkout/pull/merge may have when the worktree gets modified during it. Still, I've kept a todo about this race. This commit was supported by the NSF-funded DataLad project.
2018-08-16 19:15:20 +00:00
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
{- Adds an update-index stream to the queue. -}
addUpdateIndex :: Git.UpdateIndex.Streamer -> Annex ()
addUpdateIndex streamer = do
q <- get
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
store =<< flushWhenFull =<<
(Git.Queue.addUpdateIndex streamer q =<< gitRepo)
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
{- Runs the queue if it is full. -}
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
flushWhenFull :: Git.Queue.Queue Annex -> Annex (Git.Queue.Queue Annex)
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
flushWhenFull q
| Git.Queue.full q = flush' q
| otherwise = return q
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
{- Runs (and empties) the queue. -}
display "Recording state in git..." when staging the journal A bit tricky to avoid printing it twice in a row when there are queued git commands to run and journal to stage. Added a generic way to run an action that may output multiple side messages, with only the first displayed.
2012-04-27 17:23:52 +00:00
flush :: Annex ()
flush = do
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
q <- get
rename GitQueue to Git.Queue
2011-06-30 17:25:37 +00:00
unless (0 == Git.Queue.size q) $ do
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
store =<< flush' q
fix git command queue to be concurrency safe Probably not noticed until now because the queue is large enough that two threads each filling theirs at the same time and flushing is unlikely to happen. Also made explicit that each worker thread gets its own queue. I think that was the case before, but if something was put in the queue before worker threads were forked off, they could have each inherited the same queue. Could have gone with a single shared queue, but per-worker queues is more efficient, because a worker can add lots of stuff to its own queue without any locking. This commit was sponsored by Ole-Morten Duesund on Patreon.
2018-08-28 17:14:44 +00:00
{- When there are multiple worker threads, each has its own queue.
Improved locking when multiple git-annex processes are writing to the .git/index file
2019-05-06 19:15:12 +00:00
- And of course multiple git-annex processes may be running each with its
- own queue.
fix git command queue to be concurrency safe Probably not noticed until now because the queue is large enough that two threads each filling theirs at the same time and flushing is unlikely to happen. Also made explicit that each worker thread gets its own queue. I think that was the case before, but if something was put in the queue before worker threads were forked off, they could have each inherited the same queue. Could have gone with a single shared queue, but per-worker queues is more efficient, because a worker can add lots of stuff to its own queue without any locking. This commit was sponsored by Ole-Morten Duesund on Patreon.
2018-08-28 17:14:44 +00:00
-
- But, flushing two queues at the same time could lead to failures due to
- git locking files. So, only one queue is allowed to flush at a time.
-}
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
flush' :: Git.Queue.Queue Annex -> Annex (Git.Queue.Queue Annex)
Improved locking when multiple git-annex processes are writing to the .git/index file
2019-05-06 19:15:12 +00:00
flush' q = withExclusiveLock gitAnnexGitQueueLock $ do
showStoringStateAction
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
Git.Queue.flush q =<< gitRepo
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
queue size fix Increase queue size for update-index actions, because otherwise they'll never be flushed.
2012-06-10 17:56:04 +00:00
{- Gets the size of the queue. -}
size :: Annex Int
size = Git.Queue.size <$> get
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
get :: Annex (Git.Queue.Queue Annex)
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
get = maybe new return =<< getState repoqueue
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
new :: Annex (Git.Queue.Queue Annex)
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
new = do
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
q <- Git.Queue.new . annexQueueSize <$> getGitConfig
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
store q
return q
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
store :: Git.Queue.Queue Annex -> Annex ()
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
store q = changeState $ \s -> s { repoqueue = Just q }
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
mergeFrom :: AnnexState -> Annex ()
mergeFrom st = case repoqueue st of
Nothing -> noop
Just newq -> do
q <- get
let !q' = Git.Queue.merge q newq
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
store =<< flushWhenFull q'
Reference in a new issue Copy permalink