don't untrust appendonly exports

Make exporttree=yes remotes that are appendonly not be untrusted, and not force
verification of content, since the usual concerns about losing data when an
export is updated by someone else don't apply.

Note that all the remote operations on keys are left as usual for
appendonly export remotes, except for storing content.

This commit was supported by the NSF-funded DataLad project.

Logs/Trust.hs

@@ -66,8 +66,12 @@ trustMapLoad :: Annex TrustMap
 trustMapLoad = do
 	overrides <- Annex.getState Annex.forcetrust
 	l <- remoteList
-	-- Exports are never trusted, since they are not key/value stores.
+	-- Exports are not trusted, since they are not key/value stores.
-	exports <- filterM Types.Remote.isExportSupported l
+	-- This does not apply to appendonly exports, which are key/value
+	-- stores.
+	let untrustworthy r = pure (not (Types.Remote.appendonly r)) 
+		<&&> Types.Remote.isExportSupported r 
+	exports <- filterM untrustworthy l
 	let exportoverrides = M.fromList $
 		map (\r -> (Types.Remote.uuid r, UnTrusted)) exports
 	logged <- trustMapRaw

Remote/Helper/Export.hs

@@ -113,24 +113,20 @@ adjustExportable r = case M.lookup "exporttree" (config r) of
 			{ storeKey = \_ _ _ -> do
 				warning "remote is configured with exporttree=yes; use `git-annex export` to store content on it"
 				return False
-			-- Keys can be retrieved, but since an export
+			-- Keys can be retrieved using retrieveExport, 
-			-- is not a true key/value store, the content of
+			-- but since that retrieves from a path in the
-			-- the key has to be able to be strongly verified.
+			-- remote that another writer could have replaced
-			, retrieveKeyFile = \k _af dest p -> unVerified $
+			-- with content not of the requested key,
-				if maybe False (isJust . verifyKeyContent) (maybeLookupBackendVariety (keyVariety k))
+			-- the content has to be strongly verified.
-					then do
+			--
-						locs <- getexportlocs k
+			-- But, appendonly remotes have a key/value store,
-						case locs of
+			-- so don't need to use retrieveExport.
-							[] -> do
+			, retrieveKeyFile = if appendonly r
-								warning "unknown export location"
+				then retrieveKeyFile r
-								return False
+				else retrieveKeyFileFromExport getexportlocs
-							(l:_) -> do
+			, retrieveKeyFileCheap = if appendonly r
-								ea <- exportActions r
+				then retrieveKeyFileCheap r
-								retrieveExport ea k l dest p
+				else \_ _ _ -> return False
-					else do
-						warning $ "exported content cannot be verified due to using the " ++ formatKeyVariety (keyVariety k) ++ " backend"
-						return False
-			, retrieveKeyFileCheap = \_ _ _ -> return False
 			-- Removing a key from an export would need to
 			-- change the tree in the export log to not include
 			-- the file. Otherwise, conflicts when removing
@@ -143,16 +139,40 @@ adjustExportable r = case M.lookup "exporttree" (config r) of
 			-- Can't lock content on exports, since they're
 			-- not key/value stores, and someone else could
 			-- change what's exported to a file at any time.
-			, lockContent = Nothing
+			--
-			-- Check if any of the files a key was exported
+			-- (except for appendonly remotes)
-			-- to are present. This doesn't guarantee the
+			, lockContent = if appendonly r
-			-- export contains the right content.
+				then lockContent r
-			, checkPresent = \k -> do
+				else Nothing
-				ea <- exportActions r
+			-- Check if any of the files a key was exported to
-				anyM (checkPresentExport ea k)
+			-- are present. This doesn't guarantee the export
-					=<< getexportlocs k
+			-- contains the right content, which is why export
+			-- remotes are untrusted.
+			--
+			-- (but appendonly remotes work the same as any
+			-- non-export remote)
+			, checkPresent = if appendonly r
+				then checkPresent r
+				else \k -> do
+					ea <- exportActions r
+					anyM (checkPresentExport ea k)
+						=<< getexportlocs k
 			, mkUnavailable = return Nothing
 			, getInfo = do
 				is <- getInfo r
 				return (is++[("export", "yes")])
 			}
+	retrieveKeyFileFromExport getexportlocs k _af dest p = unVerified $
+		if maybe False (isJust . verifyKeyContent) (maybeLookupBackendVariety (keyVariety k))
+			then do
+				locs <- getexportlocs k
+				case locs of
+					[] -> do
+						warning "unknown export location"
+						return False
+					(l:_) -> do
+						ea <- exportActions r
+						retrieveExport ea k l dest p
+			else do
+				warning $ "exported content cannot be verified due to using the " ++ formatKeyVariety (keyVariety k) ++ " backend"
+				return False

doc/todo/versioning_in_export_remotes.mdwn

@@ -33,6 +33,17 @@ won't be updated, as discussed above.
 Add an "appendOnly" field to Remote, indicating it retains all content stored
 in it. done
 
+Make `git annex export` check appendOnly when removing a file from an
+export, and not update the location log, since the remote still contains
+the content. done
+
+Make git-annex sync and the assistant skip trying to drop from appendOnly
+remotes since it's just going to fail. done
+
+Make exporttree=yes remotes that are appendOnly not be untrusted, and not force
+verification of content, since the usual concerns about losing data when an
+export is updated by someone else don't apply. done
+
 Let S3 remotes be configured with versioned=yes or something like that
 (what does S3 call the feature?) which enables appendOnly.
 
@@ -41,17 +52,6 @@ configured, and use them for when retrieving keys and for checkpresent.
 
 Make S3 refuse to removeKey when configured appendOnly, failing with an error.
 
-Make `git annex export` check appendOnly when removing a file from an
-export, and not update the location log, since the remote still contains
-the content. done
-
-Make git-annex sync and the assistant skip trying to drop from appendOnly
-remotes since it's just going to fail.
-
-Make exporttree=yes remotes that are appendOnly be trusted, and not force
-verification of content, since the usual concerns about losing data when an
-export is updated by someone else don't apply.
-
 When a file was deleted from an exported tree, and then put back
 in a later exported tree, it might get re-uploaded even though the content
 is still retained in the versioned remote. S3 might have a way to avoid