incremental hashing for fileRetriever
It uses tailVerify to hash the file while it's being written. This is able to sometimes avoid a separate checksum step. Although if the file gets written quickly enough, tailVerify may not see it get created before the write finishes, and the checksum still happens. Testing with the directory special remote, incremental checksumming did not happen. But then I disabled the copy CoW probing, and it did work. What's going on with that is the CoW probe creates an empty file on failure, then deletes it, and then the file is created again. tailVerify will open the first, empty file, and so fails to read the content that gets written to the file that replaces it. The directory special remote really ought to be able to avoid needing to use tailVerify, and while other special remotes could do things that cause similar problems, they probably don't. And if they do, it just means the checksum doesn't get done incrementally. Sponsored-by: Dartmouth College's DANDI project
This commit is contained in:
parent
ff2dc5eb18
commit
dadbb510f6
8 changed files with 80 additions and 49 deletions
|
@ -124,7 +124,7 @@ checkKeyChecksum hash key file = catchIOErrorType HardwareFault hwfault $ do
|
|||
exists <- liftIO $ R.doesPathExist file
|
||||
case (exists, fast) of
|
||||
(True, False) -> do
|
||||
showAction "checksum"
|
||||
showAction descChecksum
|
||||
sameCheckSum key
|
||||
<$> hashFile hash file nullMeterUpdate
|
||||
_ -> return True
|
||||
|
@ -293,8 +293,12 @@ mkIncrementalVerifier ctx key = do
|
|||
return $ sameCheckSum key (show digest)
|
||||
Nothing -> return False
|
||||
, failIncremental = writeIORef v Nothing
|
||||
, descVerify = descChecksum
|
||||
}
|
||||
|
||||
descChecksum :: String
|
||||
descChecksum = "checksum"
|
||||
|
||||
{- A varient of the SHA256E backend, for testing that needs special keys
|
||||
- that cannot collide with legitimate keys in the repository.
|
||||
-
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue