incremental hashing for fileRetriever

It uses tailVerify to hash the file while it's being written.

This is able to sometimes avoid a separate checksum step. Although
if the file gets written quickly enough, tailVerify may not see it
get created before the write finishes, and the checksum still happens.

Testing with the directory special remote, incremental checksumming did
not happen. But then I disabled the copy CoW probing, and it did work.
What's going on with that is the CoW probe creates an empty file on
failure, then deletes it, and then the file is created again. tailVerify
will open the first, empty file, and so fails to read the content that
gets written to the file that replaces it.

The directory special remote really ought to be able to avoid needing to
use tailVerify, and while other special remotes could do things that
cause similar problems, they probably don't. And if they do, it just
means the checksum doesn't get done incrementally.

Sponsored-by: Dartmouth College's DANDI project

Types/Backend.hs

@@ -52,4 +52,6 @@ data IncrementalVerifier = IncrementalVerifier
 	-- if the hash verified.
 	, failIncremental :: IO ()
 	-- ^ Call if the incremental verification needs to fail.
+	, descVerify :: String
+	-- ^ A description of what is done to verify the content.
 	}

Types/StoreRetrieve.hs

@@ -1,6 +1,6 @@
 {- Types for Storer and Retriever actions for remotes.
  -
- - Copyright 2014 Joey Hess <id@joeyh.name>
+ - Copyright 2014-2021 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -11,6 +11,7 @@ module Types.StoreRetrieve where
 
 import Annex.Common
 import Utility.Metered
+import Types.Backend (IncrementalVerifier)
 
 import qualified Data.ByteString.Lazy as L
 
@@ -29,8 +30,17 @@ type Storer = Key -> ContentSource -> MeterUpdate -> Annex ()
 
 -- Action that retrieves a Key's content from a remote, passing it to a
 -- callback, which will fully consume the content before returning.
+--
 -- Throws exception if key is not present, or remote is not accessible.
-type Retriever = forall a. Key -> MeterUpdate -> (ContentSource -> Annex a) -> Annex a
+--
+-- When it retrieves FileContent, it is responsible for updating the
+-- MeterUpdate. And when the IncrementalVerifier is passed to it,
+-- and it retrieves FileContent, it should feed the content to the
+-- verifier before running the callback. 
+-- This should not be done when it retrieves ByteContent.
+type Retriever = forall a.
+	Key -> MeterUpdate -> Maybe IncrementalVerifier
+		-> (ContentSource -> Annex a) -> Annex a
 
 -- Action that removes a Key's content from a remote.
 -- Succeeds if key is already not present.