incremental verify for byteRetriever special remotes
Several special remotes verify content while it is being retrieved, avoiding a separate checksum pass. They are: S3, bup, ddar, and gcrypt (with a local repository). Not done when using chunking, yet. Complicated by Retriever needing to change to be polymorphic. Which in turn meant RankNTypes is needed, and also needed some code changes. The change in Remote.External does not change behavior at all but avoids the type checking failing because of a "rigid, skolem type" which "would escape its scope". So I refactored slightly to make the type checker's job easier there. Unfortunately, directory uses fileRetriever (except when chunked), so it is not amoung the improved ones. Fixing that would need a way for FileRetriever to return a Verification. But, since the file retrieved may be encrypted or chunked, it would be extra work to always incrementally checksum the file while retrieving it. Hm. Some other special remotes use fileRetriever, and so don't get incremental verification, but could be converted to byteRetriever later. One is GitLFS, which uses downloadConduit, which writes to the file, so could verify as it goes. Other special remotes like web could too, but don't use Remote.Helper.Special and so will need to be addressed separately. Sponsored-by: Dartmouth College's DANDI project
This commit is contained in:
parent
9518aca2f5
commit
c20358b671
16 changed files with 111 additions and 50 deletions
|
@ -5,6 +5,7 @@
|
|||
- Licensed under the GNU AGPL version 3 or higher.
|
||||
-}
|
||||
|
||||
{-# LANGUAGE RankNTypes #-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module Remote.Helper.Special (
|
||||
|
@ -111,7 +112,7 @@ fileRetriever a k m callback = do
|
|||
-- A Retriever that generates a lazy ByteString containing the Key's
|
||||
-- content, and passes it to a callback action which will fully consume it
|
||||
-- before returning.
|
||||
byteRetriever :: (Key -> (L.ByteString -> Annex ()) -> Annex ()) -> Retriever
|
||||
byteRetriever :: (Key -> (L.ByteString -> Annex a) -> Annex a) -> Key -> MeterUpdate -> (ContentSource -> Annex a) -> Annex a
|
||||
byteRetriever a k _m callback = a k (callback . ByteContent)
|
||||
|
||||
{- The base Remote that is provided to specialRemote needs to have
|
||||
|
@ -216,11 +217,12 @@ specialRemote' cfg c storer retriever remover checkpresent baser = encr
|
|||
enck = maybe id snd enc
|
||||
|
||||
-- call retriever to get chunks; decrypt them; stream to dest file
|
||||
retrieveKeyFileGen k dest p enc = do
|
||||
retrieveKeyFileGen k dest p enc =
|
||||
displayprogress p k Nothing $ \p' ->
|
||||
retrieveChunks retriever (uuid baser) chunkconfig
|
||||
enck k dest p' enc encr
|
||||
return UnVerified
|
||||
retrieveChunks retriever
|
||||
(uuid baser)
|
||||
(RemoteVerify baser)
|
||||
chunkconfig enck k dest p' enc encr
|
||||
where
|
||||
enck = maybe id snd enc
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue