8e5ea28c26
The hoped for optimisation of CommandStart with -J did not materialize. In fact, not runnign CommandStart in parallel is slower than -J3. So, CommandStart are still run in parallel. (The actual bad performance I've been seeing with -J in my big repo has to do with building the remoteList.) But, this is still progress toward making -J faster, because it gets rid of the onlyActionOn roadblock in the way of making CommandCleanup jobs run separate from CommandPerform jobs. Added OnlyActionOn constructor for ActionItem which fixes the onlyActionOn breakage in the last commit. Made CustomOutput include an ActionItem, so even things using it can specify OnlyActionOn. In Command.Move and Command.Sync, there were CommandStarts that used includeCommandAction, so output messages, which is no longer allowed. Fixed by using startingCustomOutput, but that's still not quite right, since it prevents message display for the includeCommandAction run inside it too.
27 lines
1.3 KiB
Markdown
27 lines
1.3 KiB
Markdown
git-annex has good support for running commands in parallel, but there
|
|
are still some things that could be improved, tracked here:
|
|
|
|
* Maybe support -Jn in more commands. Just needs changing a few lines of code
|
|
and testing each.
|
|
|
|
* Maybe extend --jobs/annex.jobs for more control. `--jobs=cpus` is already
|
|
supported; it might be good to have `--jobs=cpus-1` to leave a spare
|
|
cpu to avoid contention, or `--jobs=remotes*2` to run 2 jobs per remote.
|
|
|
|
* Parallelism is often used when the user wants to full saturate the pipe
|
|
to a remote, since having some extra transfers running avoid being
|
|
delayed while git-annex runs cleanup actions, checksum verification,
|
|
and other non-transfer stuff.
|
|
|
|
But, the user will sometimes be disappointed, because every job
|
|
can still end up stuck doing checksum verification at the same time,
|
|
so the pipe to the remote is not saturated.
|
|
|
|
Now that cleanup actions don't occupy space in the main worker queue,
|
|
all that needs to be done is make checksum verification be done as the
|
|
cleanup action. Currently, it's bundled into the same action that
|
|
transfers content.
|
|
|
|
* Using -J can sometimes lead to a slowdown while a rsync special remote
|
|
runs Remote.Rsync.rsyncTransport, which sets up a ssh connection to the
|
|
remote. This is done even when the remote is not used.
|