all keys are still present on versioned remote after import of a tree

When importing from versioned remotes, fix tracking of the content of
deleted files.

Only S3 supports versioning so far, so only it was affected.

But, the draft import/export interface for external remotes also seemed to
need a change, so that versionedExport could be set.

Annex/Import.hs

@@ -184,10 +184,13 @@ recordImportTree remote importtreeconfig importable = do
 				unlessM (stillpresent db oldkey) $
 					logChange oldkey (Remote.uuid remote) InfoMissing
 			_ -> noop
-		db <- Export.openDb (Remote.uuid remote)
+		-- When the remote is versioned, it still contains keys
-		forM_ (exportedTreeishes oldexport) $ \oldtree ->
+		-- that are not present in the new tree.
-			Export.runExportDiffUpdater updater db oldtree finaltree
+		unless (Remote.versionedExport (Remote.exportActions remote)) $ do
-		Export.closeDb db
+			db <- Export.openDb (Remote.uuid remote)
+			forM_ (exportedTreeishes oldexport) $ \oldtree ->
+				Export.runExportDiffUpdater updater db oldtree finaltree
+			Export.closeDb db
 
 buildImportCommit' :: Remote -> ImportCommitConfig -> Maybe Sha -> History Sha -> Annex (Maybe Sha)
 buildImportCommit' remote importcommitconfig mtrackingcommit imported@(History ti _) =

CHANGELOG

@@ -6,8 +6,10 @@ git-annex (10.20221004) UNRELEASED; urgency=medium
     do not operate on a repository that has an empty name.
   * move: Fix openFile crash with -J
     (Fixes a reversion in 8.20201103)
-  * S3: Speed up importing from a large bucket when fileprefix= is set
+  * S3: Speed up importing from a large bucket when fileprefix= is set,
     by only asking for files under the prefix.
+  * When importing from versioned remotes, fix tracking of the content
+    of deleted files.
 
  -- Joey Hess <id@joeyh.name>  Mon, 03 Oct 2022 13:36:42 -0400
 

Command/Export.hs

@@ -378,7 +378,7 @@ cleanupUnexport r db eks loc = do
 			removeExportedLocation db ek loc
 		flushDbQueue db
 
-	-- An versionedExport remote supports removeExportLocation to remove
+	-- A versionedExport remote supports removeExportLocation to remove
 	-- the file from the exported tree, but still retains the content
 	-- and allows retrieving it.
 	unless (versionedExport (exportActions r)) $ do

Types/Remote.hs

@@ -270,9 +270,10 @@ data ExportActions a = ExportActions
 	-- Can throw exception if unable to access remote, or if remote
 	-- refuses to remove the content.
 	, removeExport :: Key -> ExportLocation -> a ()
-	-- Set when the content of a Key stored in the remote to an
+	-- Set when the remote is versioned, so once a Key is stored
-	-- ExportLocation and then removed with removeExport remains
+	-- to an ExportLocation, a subsequent deletion of that
-	-- accessible to retrieveKeyFile and checkPresent.
+	-- ExportLocation leaves the key still accessible to retrieveKeyFile
+	-- and checkPresent.
 	, versionedExport :: Bool
 	-- Removes an exported directory. Typically the directory will be
 	-- empty, but it could possibly contain files or other directories,

doc/bugs/annex_import_does_not_account_for_versioning_on_S3.mdwn

@@ -13,3 +13,4 @@ Observed with several versions from 8-10 on linux. As laid out above, I strongly
 
 Lots. I love git-annex.
 
+> [[fixed|done]] --[[Joey]]

doc/bugs/annex_import_does_not_account_for_versioning_on_S3/comment_1_a04fbb936785456ed99512ea4c29fd53._comment

@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-10-11T16:28:39Z"
+ content="""
+This looks like a simple fix. After importing from a versioned remote,
+it can just skip updating the location logs to remove the keys that are not
+present in the current tree. The same as is already done when exporting
+to a versioned remote. I've made that change.
+"""]]

doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn

@@ -150,6 +150,13 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
     Indicates that `IMPORTKEY` can be used.
   * `IMPORTKEYSUPPORTED-FAILURE`  
     Indicates that `IMPORTKEY` cannot be used.
+ * `VERSIONED`
+  Used to check if the special remote is versioned.
+  Note that this request may be made before or after `PREPARE`.
+  * `ISVERSIONED`  
+    Indicates that the remote is versioned.
+  * `NOTVERSIONED`  
+    Indicates that the remote is not versioned.
 * `LISTIMPORTABLECONTENTS`  
   Used to get a list of all the files that are stored in the special 
   remote. A block of responses
@@ -170,6 +177,8 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
     be nested multiple levels deep.  
     This should only be used when the remote supports using
     "TRANSFER RECEIVE Key" to retrieve historical versions of files.
+    And, it should only be used when the remote replies `ISVERSIONED`
+    to the `VERSIONED` message.
   * `END`  
     Indicates the end of a block of responses.
 * `LOCATION Name`