mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

separate queue for cleanup actions

Browse source 
When running multiple concurrent actions, the cleanup phase is run in a
separate queue than the main action queue. This can make some commands
faster, because less time is spent on bookkeeping in between each file
transfer.

But as far as I can see, nothing will be sped up much by this yet, because
all the existing cleanup actions are very light-weight. This is just groundwork
for deferring checksum verification to cleanup time.

This change does mean that if the user expects -J2 will mean that they see no
more than 2 jobs running at a time, they may be surprised to see 4 in some
cases (if the cleanup actions are slow enough to notice).

It might also make sense to enable background cleanup without the -J,
for at least one cleanup action. Indeed, that's the behavior that -J1
has now. At some point in the future, it make make sense to make the
behavior with no -J the same as -J1. The only reason it's not currently
is that git-annex can build w/o concurrent-output, and also any bugs
in concurrent-output (such as perhaps misbehaving on non-VT100 compatible
terminals) are avoided by default by only using it when -J is used.
This commit is contained in:
Joey Hess 2019-06-05 17:54:35 -04:00
parent c04b2af3e1
commit 659640e224
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
6 changed files with 128 additions and 46 deletions
Download patch file Download diff file Expand all files Collapse all files

9
doc/todo/parallel_possibilities.mdwn
View file

@ -17,8 +17,7 @@ are still some things that could be improved, tracked here:
can still end up stuck doing checksum verification at the same time,
so the pipe to the remote is not saturated.
Running cleanup actions in a separate queue from the main job queue
wouldn't be sufficient for this, because verification is done as part
of the same action that transfers content. That needs to somehow be
refactored to a cleanup action that ingests the file, and then
the cleanup action can be run in a separate queue.
Now that cleanup actions don't occupy space in the main worker queue,
all that needs to be done is make checksum verification be done as the
cleanup action. Currently, it's bundled into the same action that
transfers content.