mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Annex/Queue.hs

96 lines
2.5 KiB
Haskell
Raw Normal View History

Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
{- git-annex command queue
-
Git.Queue: allow providing git common options like -c
2021-01-04 16:51:55 +00:00
- Copyright 2011-2021 Joey Hess <id@joeyh.name>
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
-
update licenses from GPL to AGPL This does not change the overall license of the git-annex program, which was already AGPL due to a number of sources files being AGPL already. Legally speaking, I'm adding a new license under which these files are now available; I already released their current contents under the GPL license. Now they're dual licensed GPL and AGPL. However, I intend for all my future changes to these files to only be released under the AGPL license, and I won't be tracking the dual licensing status, so I'm simply changing the license statement to say it's AGPL. (In some cases, others wrote parts of the code of a file and released it under the GPL; but in all cases I have contributed a significant portion of the code in each file and it's that code that is getting the AGPL license; the GPL license of other contributors allows combining with AGPL code.)
2019-03-13 19:48:14 +00:00
- Licensed under the GNU AGPL version 3 or higher.
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
-}
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
{-# LANGUAGE BangPatterns #-}
rename
2011-10-04 04:40:47 +00:00
module Annex.Queue (
extend Git.Queue to be able to queue more than simple git commands While I was in there, I noticed and fixed a bug in the queue size calculations. It was never encountered only because Queue.add was only ever run with 1 file in the list.
2012-06-07 19:19:44 +00:00
addCommand,
queueing of internal IO actions on files This would be better if getInternalFiles were more polymorphic, but I can't see a good way to accomplish that without messing with Data.Typeable, which seemed like overkill. Reverted CommandAction back to the simpler version. This commit was sponsored by Eric Drechsel on Patreon.
2018-08-17 17:24:52 +00:00
addInternalAction,
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
addUpdateIndex,
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
flush,
queue size fix Increase queue size for update-index actions, because otherwise they'll never be flushed.
2012-06-10 17:56:04 +00:00
flushWhenFull,
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
size,
withAltRepo needs a separate queue of changes The queue could potentially contain changes from before withAltRepo, and get flushed inside the call, which would apply the changes to the modified repo. Or, changes could be queued in withAltRepo that were intended to affect the modified repo, but don't get flushed until later. I don't know of any cases where either happens, but better safe than sorry. Note that this affect withIndexFile, which is used in git-annex branch updates. So, it potentially makes things slower. Should not be by much; the overhead consists only of querying the current queue a couple of times, and potentially flushing changes queued within withAltRepo earlier, that could have maybe been bundled with other later changes. Notice in particular that the existing queue is not flushed when calling withAltRepo. So eg when git annex add needs to stage files in the index, it will still bundle them together efficiently.
2016-06-03 17:48:14 +00:00
get,
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
mergeFrom,
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
) where
remove 163 lines of code without changing anything except imports
2016-01-20 20:36:33 +00:00
import Annex.Common
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
import Annex hiding (new)
Improved locking when multiple git-annex processes are writing to the .git/index file
2019-05-06 19:15:12 +00:00
import Annex.LockFile
rename GitQueue to Git.Queue
2011-06-30 17:25:37 +00:00
import qualified Git.Queue
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
import qualified Git.UpdateIndex
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
outdated comment
2011-08-21 16:59:49 +00:00
{- Adds a git command to the queue. -}
Git.Queue: allow providing git common options like -c
2021-01-04 16:51:55 +00:00
addCommand :: [CommandParam] -> String -> [CommandParam] -> [FilePath] -> Annex ()
addCommand commonparams command params files = do
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
q <- get
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
store =<< flushWhenFull =<<
Git.Queue: allow providing git common options like -c
2021-01-04 16:51:55 +00:00
(Git.Queue.addCommand commonparams command params files q =<< gitRepo)
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
dynamically disable filter-process in restagePointerFile when it would be slower Based on my earlier benchmark, I have a rough cost model for how expensive it is for git-annex smudge to be run on a file, vs how expensive it is for a gigabyte of a file's content to be read and piped through to filter-process. So, using that cost model, it can decide if using filter-process will be more or less expensive than running the smudge filter on the files to be restaged. It turned out to be *really* annoying to temporarily disable filter-process. I did find a way, but urk, this is horrible. Notice that, if it's interrupted with it disabled, it will remain disabled until the next time restagePointerFile runs. Which could be some time later. If the user runs `git add` or `git checkout` on a lot of small files before that, they will see slower than expected performance. (This commit also deletes where I wrote down the benchmark results earlier.) Sponsored-by: Noam Kremen on Patreon
2021-11-08 19:55:27 +00:00
addInternalAction :: Git.Queue.InternalActionRunner Annex -> [(RawFilePath, IO Bool, FileSize)] -> Annex ()
queueing of internal IO actions on files This would be better if getInternalFiles were more polymorphic, but I can't see a good way to accomplish that without messing with Data.Typeable, which seemed like overkill. Reverted CommandAction back to the simpler version. This commit was sponsored by Eric Drechsel on Patreon.
2018-08-17 17:24:52 +00:00
addInternalAction runner files = do
narrow the race where a file gets modified before update-index Check just before running update-index if the worktree file's content is still the same, don't update it when it's been modified. This narrows the race window a lot, from possibly minutes or hours, to seconds or less. (Use replaceFile so that the worktree update happens atomically, allowing the InodeCache of the new worktree file to itself be gathered w/o any other race.) This doesn't eliminate the race; it can still occur in the window before update-index runs. When annex.queue is large, a lot of files will be statted by the checks, and so the window may still be large enough to be a problem. When only a few files are being processed, the window is as small as it is in the race where a modification gets overwritten by git-annex when it updates the worktree. Or maybe as small as whatever race git checkout/pull/merge may have when the worktree gets modified during it. Still, I've kept a todo about this race. This commit was supported by the NSF-funded DataLad project.
2018-08-16 19:15:20 +00:00
q <- get
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
store =<< flushWhenFull =<<
(Git.Queue.addInternalAction runner files q =<< gitRepo)
narrow the race where a file gets modified before update-index Check just before running update-index if the worktree file's content is still the same, don't update it when it's been modified. This narrows the race window a lot, from possibly minutes or hours, to seconds or less. (Use replaceFile so that the worktree update happens atomically, allowing the InodeCache of the new worktree file to itself be gathered w/o any other race.) This doesn't eliminate the race; it can still occur in the window before update-index runs. When annex.queue is large, a lot of files will be statted by the checks, and so the window may still be large enough to be a problem. When only a few files are being processed, the window is as small as it is in the race where a modification gets overwritten by git-annex when it updates the worktree. Or maybe as small as whatever race git checkout/pull/merge may have when the worktree gets modified during it. Still, I've kept a todo about this race. This commit was supported by the NSF-funded DataLad project.
2018-08-16 19:15:20 +00:00
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
{- Adds an update-index stream to the queue. -}
addUpdateIndex :: Git.UpdateIndex.Streamer -> Annex ()
addUpdateIndex streamer = do
q <- get
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
store =<< flushWhenFull =<<
(Git.Queue.addUpdateIndex streamer q =<< gitRepo)
make watch use the queue May not work. Certianly needs to flush the queue from time to time when only symlink changes are being made.
2012-06-07 19:40:44 +00:00
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
{- Runs the queue if it is full. -}
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
flushWhenFull :: Git.Queue.Queue Annex -> Annex (Git.Queue.Queue Annex)
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
flushWhenFull q
| Git.Queue.full q = flush' q
| otherwise = return q
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
{- Runs (and empties) the queue. -}
display "Recording state in git..." when staging the journal A bit tricky to avoid printing it twice in a row when there are queued git commands to run and journal to stage. Added a generic way to run an action that may output multiple side messages, with only the first displayed.
2012-04-27 17:23:52 +00:00
flush :: Annex ()
flush = do
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
q <- get
rename GitQueue to Git.Queue
2011-06-30 17:25:37 +00:00
unless (0 == Git.Queue.size q) $ do
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
store =<< flush' q
fix git command queue to be concurrency safe Probably not noticed until now because the queue is large enough that two threads each filling theirs at the same time and flushing is unlikely to happen. Also made explicit that each worker thread gets its own queue. I think that was the case before, but if something was put in the queue before worker threads were forked off, they could have each inherited the same queue. Could have gone with a single shared queue, but per-worker queues is more efficient, because a worker can add lots of stuff to its own queue without any locking. This commit was sponsored by Ole-Morten Duesund on Patreon.
2018-08-28 17:14:44 +00:00
{- When there are multiple worker threads, each has its own queue.
Improved locking when multiple git-annex processes are writing to the .git/index file
2019-05-06 19:15:12 +00:00
- And of course multiple git-annex processes may be running each with its
- own queue.
fix git command queue to be concurrency safe Probably not noticed until now because the queue is large enough that two threads each filling theirs at the same time and flushing is unlikely to happen. Also made explicit that each worker thread gets its own queue. I think that was the case before, but if something was put in the queue before worker threads were forked off, they could have each inherited the same queue. Could have gone with a single shared queue, but per-worker queues is more efficient, because a worker can add lots of stuff to its own queue without any locking. This commit was sponsored by Ole-Morten Duesund on Patreon.
2018-08-28 17:14:44 +00:00
-
- But, flushing two queues at the same time could lead to failures due to
- git locking files. So, only one queue is allowed to flush at a time.
-}
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
flush' :: Git.Queue.Queue Annex -> Annex (Git.Queue.Queue Annex)
Improved locking when multiple git-annex processes are writing to the .git/index file
2019-05-06 19:15:12 +00:00
flush' q = withExclusiveLock gitAnnexGitQueueLock $ do
showStoringStateAction
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
Git.Queue.flush q =<< gitRepo
Periodically flush git command queue, to avoid boating memory usage too much. Since the queue is flushed in between subcommand actions being run, there should be no issues with actions that expect to queue up some stuff and have it run after they do other stuff. So I didn't have to audit for such assumptions.
2011-04-07 17:59:31 +00:00
queue size fix Increase queue size for update-index actions, because otherwise they'll never be flushed.
2012-06-10 17:56:04 +00:00
{- Gets the size of the queue. -}
size :: Annex Int
size = Git.Queue.size <$> get
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
get :: Annex (Git.Queue.Queue Annex)
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
get = maybe new return =<< getState repoqueue
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
new :: Annex (Git.Queue.Queue Annex)
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
new = do
improve git command queue flushing with time limit So that eg, addurl of several large files that take time to download will update the index for each file, rather than deferring the index updates to the end. In cases like an add of many smallish files, where a new file is being added every few seconds. In that case, the queue will still build up a lot of changes which are flushed at once, for best performance. Since the default queue size is 10240, often it only gets flushed once at the end, same as before. (Notice that updateQueue updated _lastchanged when adding a new item to the queue without flushing it; that is necessary to avoid it flushing the queue every 5 minutes in this case.) But, when it takes more than a 5 minutes to add a file, the overhead of updating the index immediately is probably small, so do it after each file. This avoids git-annex potentially taking a very very long time indeed to stage newly added files, which can be annoying to the user who would like to get on with doing something with the files it's already added, eg using git mv to rename them to a better name. This is only likely to cause a problem if it takes say, 30 seconds to update the index; doing an extra 30 seconds of work after every 5 minute file add would be less optimal. Normally, updating the index takes significantly less time than that. On a SSD with 100k files it takes less than 1 second, and the index write time is bound by disk read and write so is not too much worse on a hard drive. So I hope this will not impact users, although if it does turn out to, the time limit could be made configurable. A perhaps better way to do it would be to have a background worker thread that wakes up every 60 seconds or so and flushes the queue. That is made somewhat difficult because the queue can contain Annex actions and so this would add a new source of concurrency issues. So I'm trying to avoid that approach if possible. Sponsored-by: Erik Bjäreholt on Patreon
2021-12-14 15:48:07 +00:00
sz <- annexQueueSize <$> getGitConfig
q <- liftIO $ Git.Queue.new sz Nothing
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
store q
return q
remove one more warningIO Had to generalize Git.Queue so it can run an Annex action, yipes. Only remaining warningIO are in the legacy chunk code.
2019-11-12 14:44:51 +00:00
store :: Git.Queue.Queue Annex -> Annex ()
Added a annex.queuesize setting useful when adding hundreds of thousands of files on a system with plenty of memory. git add gets quite slow in such a large repository, so if the system has more than the ~32 mb of memory the queue can use by default, it's a useful optimisation to increase the queue size, in order to decrease the number of times git add is run.
2012-02-15 15:13:13 +00:00
store q = changeState $ \s -> s { repoqueue = Just q }
merge git command queue when joining with concurrent thread
2015-11-05 22:21:48 +00:00
mergeFrom :: AnnexState -> Annex ()
mergeFrom st = case repoqueue st of
Nothing -> noop
Just newq -> do
q <- get
let !q' = Git.Queue.merge q newq
immediate queue flushing when annex.queuesize=1 Previously, it only flushed when the queue got larger than 1. Also, make the queue auto-flush when items are added, rather than needing to be flushed as a separate step. This simplifies the code and make it more efficient too, as it avoids needing to read the queue out of the state to check if it should be flushed.
2016-01-13 18:55:01 +00:00
store =<< flushWhenFull q'
Reference in a new issue Copy permalink