separate queue for cleanup actions
When running multiple concurrent actions, the cleanup phase is run in a separate queue than the main action queue. This can make some commands faster, because less time is spent on bookkeeping in between each file transfer. But as far as I can see, nothing will be sped up much by this yet, because all the existing cleanup actions are very light-weight. This is just groundwork for deferring checksum verification to cleanup time. This change does mean that if the user expects -J2 will mean that they see no more than 2 jobs running at a time, they may be surprised to see 4 in some cases (if the cleanup actions are slow enough to notice). It might also make sense to enable background cleanup without the -J, for at least one cleanup action. Indeed, that's the behavior that -J1 has now. At some point in the future, it make make sense to make the behavior with no -J the same as -J1. The only reason it's not currently is that git-annex can build w/o concurrent-output, and also any bugs in concurrent-output (such as perhaps misbehaving on non-VT100 compatible terminals) are avoided by default by only using it when -J is used.
This commit is contained in:
parent
c04b2af3e1
commit
659640e224
6 changed files with 128 additions and 46 deletions
|
@ -17,8 +17,7 @@ are still some things that could be improved, tracked here:
|
|||
can still end up stuck doing checksum verification at the same time,
|
||||
so the pipe to the remote is not saturated.
|
||||
|
||||
Running cleanup actions in a separate queue from the main job queue
|
||||
wouldn't be sufficient for this, because verification is done as part
|
||||
of the same action that transfers content. That needs to somehow be
|
||||
refactored to a cleanup action that ingests the file, and then
|
||||
the cleanup action can be run in a separate queue.
|
||||
Now that cleanup actions don't occupy space in the main worker queue,
|
||||
all that needs to be done is make checksum verification be done as the
|
||||
cleanup action. Currently, it's bundled into the same action that
|
||||
transfers content.
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue