mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Annex/Queue.hs

80 lines
1.8 KiB
Haskell
Raw Normal View History

Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
{- git-annex command queue
-
update my email address and homepage url
2015-01-21 16:50:09 +00:00
- Copyright 2011, 2012 Joey Hess <id@joeyh.name>
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
-
- Licensed under the GNU GPL version 3 or higher.
-}
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
{-# LANGUAGE BangPatterns #-}
rename
2011-10-04 04:40:47 +00:00
module Annex.Queue (
extend Git.Queue to be able to queue more than simple git commands While I was in there, I noticed and fixed a bug in the queue size calculations. It was never encountered only because Queue.add was only ever run with 1 file in the list.
2012-06-07 19:19:44 +00:00
addCommand,
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
addUpdateIndex,
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
flush,
queue size fix Increase queue size for update-index actions, because otherwise they'll never be flushed.
2012-06-10 17:56:04 +00:00
flushWhenFull,
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
size,
withAltRepo needs a separate queue of changes The queue could potentially contain changes from before withAltRepo, and get flushed inside the call, which would apply the changes to the modified repo. Or, changes could be queued in withAltRepo that were intended to affect the modified repo, but don't get flushed until later. I don't know of any cases where either happens, but better safe than sorry. Note that this affect withIndexFile, which is used in git-annex branch updates. So, it potentially makes things slower. Should not be by much; the overhead consists only of querying the current queue a couple of times, and potentially flushing changes queued within withAltRepo earlier, that could have maybe been bundled with other later changes. Notice in particular that the existing queue is not flushed when calling withAltRepo. So eg when git annex add needs to stage files in the index, it will still bundle them together efficiently.
2016-06-03 17:48:14 +00:00
get,
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
mergeFrom,
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
) where
remove 163 lines of code without changing anything except imports
2016-01-20 20:36:33 +00:00
import Annex.Common
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
import Annex hiding (new)
rename GitQueue to Git.Queue
2011-06-30 17:25:37 +00:00
import qualified Git.Queue
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
import qualified Git.UpdateIndex
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
outdated comment
2011-08-21 16:59:49 +00:00
{- Adds a git command to the queue. -}
extend Git.Queue to be able to queue more than simple git commands While I was in there, I noticed and fixed a bug in the queue size calculations. It was never encountered only because Queue.add was only ever run with 1 file in the list.
2012-06-07 19:19:44 +00:00
addCommand :: String -> [CommandParam] -> [FilePath] -> Annex ()
addCommand command params files = do
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
q <- get
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
store <=< flushWhenFull <=< inRepo $
Git.Queue.addCommand command params files q
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
{- Adds an update-index stream to the queue. -}
addUpdateIndex :: Git.UpdateIndex.Streamer -> Annex ()
addUpdateIndex streamer = do
q <- get
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
store <=< flushWhenFull <=< inRepo $
Git.Queue.addUpdateIndex streamer q
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
{- Runs the queue if it is full. -}
flushWhenFull :: Git.Queue.Queue -> Annex Git.Queue.Queue
flushWhenFull q
| Git.Queue.full q = flush' q
| otherwise = return q
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
{- Runs (and empties) the queue. -}
display "Recording state in git..." when staging the journal A bit tricky to avoid printing it twice in a row when there are queued git commands to run and journal to stage. Added a generic way to run an action that may output multiple side messages, with only the first displayed.
2012-04-27 17:23:52 +00:00
flush :: Annex ()
flush = do
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
q <- get
rename GitQueue to Git.Queue
2011-06-30 17:25:37 +00:00
unless (0 == Git.Queue.size q) $ do
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
store =<< flush' q
flush' :: Git.Queue.Queue -> Annex Git.Queue.Queue
flush' q = do
showStoringStateAction
inRepo $ Git.Queue.flush q
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
queue size fix Increase queue size for update-index actions, because otherwise they'll never be flushed.
2012-06-10 17:56:04 +00:00
{- Gets the size of the queue. -}
size :: Annex Int
size = Git.Queue.size <$> get
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
get :: Annex Git.Queue.Queue
get = maybe new return =<< getState repoqueue
new :: Annex Git.Queue.Queue
new = do
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
q <- Git.Queue.new . annexQueueSize <$> getGitConfig
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
store q
return q
rename GitQueue to Git.Queue
2011-06-30 17:25:37 +00:00
store :: Git.Queue.Queue -> Annex ()
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
store q = changeState $ \s -> s { repoqueue = Just q }
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
mergeFrom :: AnnexState -> Annex ()
mergeFrom st = case repoqueue st of
Nothing -> noop
Just newq -> do
q <- get
let !q' = Git.Queue.merge q newq
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
store =<< flushWhenFull q'
Reference in a new issue Copy permalink