webdav: deal with buggy webdav servers in renameExport

box.com already had a special case, since its renaming was known buggy.
In its case, renaming to the temp file succeeds, but then renaming the temp
file to final destination fails.

Then this 4shared server has buggy handling of renames across directories.
While already worked around with for the temp files when storing exports
now being in the same directory as the final filename, that also affected
renameExport when the file moves between directories.

I'm not entirely clear what happens on the 4shared server when it fails
this way. It kind of looks like it may rename the file to destination and
then still fail.

To handle both, when rename fails, delete both the source and the
destination, and fall back to uploading the content again. In the box.com
case, the temp file is the source, and deleting it makes sure the temp file
gets cleaned up. In the 4shared case, the file may have been renamed to the
destination and so cleaning that up avoids any interference with the
re-upload to the destination.

CHANGELOG

@@ -14,6 +14,7 @@ git-annex (8.20210311) UNRELEASED; urgency=medium
     special remote, fix a bug that caused the url to to not be removed.
   * unregisterurl: Fix a bug that caused an url to not be unregistered
     when it is claimed by a special remote other than the web.
+  * Work around some buggy webdav server behavior involving renaming files.
 
  -- Joey Hess <id@joeyh.name>  Fri, 12 Mar 2021 12:06:37 -0400
 

Remote/WebDAV.hs

@@ -247,15 +247,24 @@ removeExportDirectoryDav hdl dir = withDavHandle hdl $ \h -> runExport h $ \_dav
 
 renameExportDav :: DavHandleVar -> Key -> ExportLocation -> ExportLocation -> Annex (Maybe ())
 renameExportDav hdl _k src dest = case (exportLocation src, exportLocation dest) of
-	(Right srcl, Right destl) -> withDavHandle hdl $ \h -> 
-		-- box.com's DAV endpoint has buggy handling of renames,
-		-- so avoid renaming when using it.
-		if boxComUrl `isPrefixOf` baseURL h 
-			then return Nothing
-			else runExport h $ \dav -> do
-				maybe noop (void . mkColRecursive) (locationParent destl)
-				moveDAV (baseURL dav) srcl destl
-				return (Just ())
+	(Right srcl, Right destl) -> withDavHandle hdl $ \h -> do
+		-- Several webdav servers have buggy handing of renames,
+		-- and fail to rename in some circumstances.
+		-- Since after a failure it's not clear where the file ended
+		-- up, recover by deleting both the source and destination.
+		-- The file will later be re-uploaded to the destination,
+		-- so this deletion is ok.
+		let go = runExport h $ \dav -> do
+			maybe noop (void . mkColRecursive) (locationParent destl)
+			moveDAV (baseURL dav) srcl destl
+			return (Just ())
+		let recover = do
+			void $ runExport h $ \_dav -> safely $
+				inLocation srcl delContentM
+			void $ runExport h $ \_dav -> safely $
+				inLocation destl delContentM
+			return Nothing
+		catchNonAsync go (const recover)
 	(Left err, _) -> giveup err
 	(_, Left err) -> giveup err
 

Types/Remote.hs

@@ -280,7 +280,10 @@ data ExportActions a = ExportActions
 	, checkPresentExport :: Key -> ExportLocation -> a Bool
 	-- Renames an already exported file.
 	--
-	-- If the remote does not support renames, it can return Nothing.
+	-- If the remote does not support the requested rename,
+	-- it can return Nothing. It's ok if the remove deletes
+	-- the file in such a situation too; it will be re-exported to
+	-- recover.
 	--
 	-- Throws an exception if the remote cannot be accessed, or
 	-- the file doesn't exist or cannot be renamed.
@@ -393,3 +396,4 @@ data ImportActions a = ImportActions
 		-> [ContentIdentifier]
 		-> a Bool
 	}
+

doc/bugs/WEBDAV_export_has_wrong_subdirectory_content.mdwn

@@ -55,3 +55,4 @@ Thx!
 
 [[!tag projects/datalad]]
 
+> [[fixed|done]] --[[Joey]]

doc/bugs/WEBDAV_export_has_wrong_subdirectory_content/comment_11_03ea7014854be4fa9f40a8815616b81d._comment

@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 11"""
+ date="2021-03-22T16:27:10Z"
+ content="""
+Ok, failed kind of like I expected, although I have to say the filenames
+don't entirely make sense to me.
+("git-annex-webdav-tmp" is not used in filenames when renaming an exported
+file, only when storing it in the first place)
+
+So what seems to make sense for git-annex to do when renaming an exported
+file is: Try to rename, and if it fails, delete the source and the
+destination -- since it has no idea what may have been left in either place
+-- and then fall back to uploading the file again instead. I have
+implemented that.
+"""]]