incremental verify for byteRetriever special remotes
Several special remotes verify content while it is being retrieved, avoiding a separate checksum pass. They are: S3, bup, ddar, and gcrypt (with a local repository). Not done when using chunking, yet. Complicated by Retriever needing to change to be polymorphic. Which in turn meant RankNTypes is needed, and also needed some code changes. The change in Remote.External does not change behavior at all but avoids the type checking failing because of a "rigid, skolem type" which "would escape its scope". So I refactored slightly to make the type checker's job easier there. Unfortunately, directory uses fileRetriever (except when chunked), so it is not amoung the improved ones. Fixing that would need a way for FileRetriever to return a Verification. But, since the file retrieved may be encrypted or chunked, it would be extra work to always incrementally checksum the file while retrieving it. Hm. Some other special remotes use fileRetriever, and so don't get incremental verification, but could be converted to byteRetriever later. One is GitLFS, which uses downloadConduit, which writes to the file, so could verify as it goes. Other special remotes like web could too, but don't use Remote.Helper.Special and so will need to be addressed separately. Sponsored-by: Dartmouth College's DANDI project
This commit is contained in:
parent
9518aca2f5
commit
c20358b671
16 changed files with 111 additions and 50 deletions
|
@ -7,6 +7,7 @@
|
|||
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
{-# LANGUAGE BangPatterns #-}
|
||||
{-# LANGUAGE RankNTypes #-}
|
||||
|
||||
module Remote.External (remote) where
|
||||
|
||||
|
@ -66,17 +67,19 @@ gen r u rc gc rs
|
|||
| externaltype == "readonly" = do
|
||||
c <- parsedRemoteConfig remote rc
|
||||
cst <- remoteCost gc expensiveRemoteCost
|
||||
mk c cst GloballyAvailable
|
||||
readonlyStorer
|
||||
retrieveUrl
|
||||
readonlyRemoveKey
|
||||
checkKeyUrl
|
||||
let rmt = mk c cst GloballyAvailable
|
||||
Nothing
|
||||
(externalInfo externaltype)
|
||||
Nothing
|
||||
Nothing
|
||||
exportUnsupported
|
||||
exportUnsupported
|
||||
return $ Just $ specialRemote c
|
||||
readonlyStorer
|
||||
retrieveUrl
|
||||
readonlyRemoveKey
|
||||
checkKeyUrl
|
||||
rmt
|
||||
| otherwise = do
|
||||
c <- parsedRemoteConfig remote rc
|
||||
external <- newExternal externaltype (Just u) c (Just gc)
|
||||
|
@ -103,20 +106,22 @@ gen r u rc gc rs
|
|||
let cheapexportsupported = if exportsupported
|
||||
then exportIsSupported
|
||||
else exportUnsupported
|
||||
mk c cst avail
|
||||
(storeKeyM external)
|
||||
(retrieveKeyFileM external)
|
||||
(removeKeyM external)
|
||||
(checkPresentM external)
|
||||
let rmt = mk c cst avail
|
||||
(Just (whereisKeyM external))
|
||||
(getInfoM external)
|
||||
(Just (claimUrlM external))
|
||||
(Just (checkUrlM external))
|
||||
exportactions
|
||||
cheapexportsupported
|
||||
return $ Just $ specialRemote c
|
||||
(storeKeyM external)
|
||||
(retrieveKeyFileM external)
|
||||
(removeKeyM external)
|
||||
(checkPresentM external)
|
||||
rmt
|
||||
where
|
||||
mk c cst avail tostore toretrieve toremove tocheckkey towhereis togetinfo toclaimurl tocheckurl exportactions cheapexportsupported = do
|
||||
let rmt = Remote
|
||||
mk c cst avail towhereis togetinfo toclaimurl tocheckurl exportactions cheapexportsupported =
|
||||
Remote
|
||||
{ uuid = u
|
||||
, cost = cst
|
||||
, name = Git.repoDescribe r
|
||||
|
@ -154,12 +159,6 @@ gen r u rc gc rs
|
|||
, checkUrl = tocheckurl
|
||||
, remoteStateHandle = rs
|
||||
}
|
||||
return $ Just $ specialRemote c
|
||||
tostore
|
||||
toretrieve
|
||||
toremove
|
||||
tocheckkey
|
||||
rmt
|
||||
externaltype = fromMaybe (giveup "missing externaltype") (remoteAnnexExternalType gc)
|
||||
|
||||
externalSetup :: SetupStage -> Maybe UUID -> Maybe CredPair -> RemoteConfig -> RemoteGitConfig -> Annex (RemoteConfig, UUID)
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue