use fine-grained WorkerStages when transferring and verifying

This means that Command.Move and Command.Get don't need to
manually set the stage, and is a lot cleaner conceptually.

Also, this makes Command.Sync.syncFile use the worker pool better.
In the scenario where it first downloads content and then uploads it to
some other remotes, it will start in TransferStage, then enter VerifyStage
and then go back to TransferStage for each transfer to the remotes.
Before, it entered CleanupStage after the download, and stayed in it for
the upload, so too many transfer jobs could run at the same time.

Note that, in Remote.Git, it uses runTransfer and also verifyKeyContent
inside onLocal. That has a Annex state for the remote, with no worker pool.
So the resulting calls to enteringStage won't block in there.

While Remote.Git.copyToRemote does do checksum verification, I
realized that should not use a verification slot in the WorkerPool
to do it. Because, it's reading back from eg, a removable disk to checksum.
That will contend with other writes to that disk. It's best to treat
that checksum verification as just part of the transer. So, removed the todo
item about that, as there's nothing needing to be done.
This commit is contained in:
Joey Hess 2019-06-19 13:09:26 -04:00
parent 53882ab4a7
commit 9d36c826c0
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
9 changed files with 13 additions and 34 deletions

View file

@ -105,24 +105,6 @@ stop = return Nothing
stopUnless :: Annex Bool -> Annex (Maybe a) -> Annex (Maybe a)
stopUnless c a = ifM c ( a , stop )
{- This can be used in the perform stage to run the action that is the bulk
- of the work to do in that stage. If the action succeeds, then any actions
- run after it will be scheduled as if they were run in the cleanup stage
- instead of the perform stage.
-
- This is not needed for a perform stage that uses `next` to run the
- cleanup stage action. But sometimes a larger action is being built up
- and it's not practical to separate out the cleanup stage part from the
- rest of the action.
-}
performJob :: Observable a => Annex a -> Annex a
performJob a = do
r <- a
if observeBool r
then changeStageTo CleanupStage
else noop
return r
{- When acting on a failed transfer, stops unless it was in the specified
- direction. -}
checkFailedTransferDirection :: ActionItem -> Direction -> Annex (Maybe a) -> Annex (Maybe a)