better handling of finishing up incomplete incremental verify

Now it's run in VerifyStage.

I thought about keeping the file handle open, and resuming reading where
tailVerify left off. But that risks leaking open file handles, until the
GC closes them, if the deferred verification does not get resumed. Since
that could perhaps happen if there's an exception somewhere, I decided
that was too unsafe.

Instead, re-open the file, seek, and resume.

Sponsored-by: Dartmouth College's DANDI project

Types/Backend.hs

@@ -52,6 +52,10 @@ data IncrementalVerifier = IncrementalVerifier
 	-- if the hash verified.
 	, failIncremental :: IO ()
 	-- ^ Call if the incremental verification needs to fail.
+	, positionIncremental :: IO (Maybe Integer)
+	-- ^ Returns the number of bytes that have been fed to this
+	-- incremental verifier so far. (Nothing if failIncremental was
+	-- called.)
 	, descVerify :: String
 	-- ^ A description of what is done to verify the content.
 	}

Types/Remote.hs

@@ -2,7 +2,7 @@
  -
  - Most things should not need this, using Types instead
  -
- - Copyright 2011-2020 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2021 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -41,6 +41,7 @@ import Types.NumCopies
 import Types.Export
 import Types.Import
 import Types.RemoteConfig
+import Types.Backend (IncrementalVerifier)
 import Config.Cost
 import Utility.Metered
 import Git.Types (RemoteName)
@@ -203,7 +204,9 @@ data Verification
 	| MustVerify
 	-- ^ Content likely to have been altered during transfer,
 	-- verify even if verification is normally disabled
-	deriving (Show)
+	| IncompleteVerify IncrementalVerifier
+	-- ^ Content was partially verified during transfer, but
+	-- the verification is not complete.
 
 unVerified :: Monad m => m a -> m (a, Verification)
 unVerified a = do

Types/StoreRetrieve.hs

@@ -35,8 +35,8 @@ type Storer = Key -> ContentSource -> MeterUpdate -> Annex ()
 --
 -- When it retrieves FileContent, it is responsible for updating the
 -- MeterUpdate. And when the IncrementalVerifier is passed to it,
--- and it retrieves FileContent, it should feed the content to the
--- verifier before running the callback. 
+-- and it retrieves FileContent, it can feed some or all of the file's
+-- content to the verifier before running the callback.
 -- This should not be done when it retrieves ByteContent.
 type Retriever = forall a.
 	Key -> MeterUpdate -> Maybe IncrementalVerifier