mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

finish CommandStart transition

Browse source 
The hoped for optimisation of CommandStart with -J did not materialize.
In fact, not runnign CommandStart in parallel is slower than -J3.
So, CommandStart are still run in parallel.

(The actual bad performance I've been seeing with -J in my big repo
has to do with building the remoteList.)

But, this is still progress toward making -J faster, because it gets rid
of the onlyActionOn roadblock in the way of making CommandCleanup jobs
run separate from CommandPerform jobs.

Added OnlyActionOn constructor for ActionItem which fixes the
onlyActionOn breakage in the last commit.

Made CustomOutput include an ActionItem, so even things using it can
specify OnlyActionOn.

In Command.Move and Command.Sync, there were CommandStarts that used
includeCommandAction, so output messages, which is no longer allowed.
Fixed by using startingCustomOutput, but that's still not quite right,
since it prevents message display for the includeCommandAction run
inside it too.
This commit is contained in:
Joey Hess 2019-06-12 09:23:26 -04:00
parent 436f107715
commit 8e5ea28c26
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
26 changed files with 142 additions and 97 deletions
Download patch file Download diff file Expand all files Collapse all files

18
doc/todo/parallel_possibilities.mdwn
View file

@ -22,18 +22,6 @@ are still some things that could be improved, tracked here:
cleanup action. Currently, it's bundled into the same action that
transfers content.
* onlyActionOn collapses the cleanup action into the start action,
and so prevents use of the separate cleanup queue.
* Don't parallelize start stage actions. They are supposed to run fast,
and often a huge number of them don't print out anything. The overhead of
bookkeeping for parallizing those swamps the benefit of parallelizing by
what seems to be a large degree. Compare `git annex get` in a directory
where the first several thousand files are already present with and
without -J.
Only once the start stage has decided
something needs to be done should a job be started up.
This probably needs display of any output to be moved out of the start
stage, because no console region will be allocated for it.
* Using -J can sometimes lead to a slowdown while a rsync special remote
runs Remote.Rsync.rsyncTransport, which sets up a ssh connection to the
remote. This is done even when the remote is not used.