S3: Support signature=v4

To use S3 Signature Version 4. Some S3 services seem to require v4, while
others may only support v2, which remains the default.

I'm also not sure if v4 works correctly in all cases, there is this
upstream bug report: https://github.com/aristidb/aws/issues/262
I've only tested it against the default S3 endpoint.

CHANGELOG

@@ -10,6 +10,9 @@ git-annex (8.20200502) UNRELEASED; urgency=medium
   * sync: Avoid an ugly error message when nothing has been committed to
     master yet and there is a synced master branch to merge from.
   * upgrade: When upgrade fails due to an exception, display it.
+  * S3: Support signature=v4, to use S3 Signature Version 4.
+    Some S3 services seem to require v4, while others may only
+    support v2, which remains the default.
 
  -- Joey Hess <id@joeyh.name>  Mon, 04 May 2020 12:46:11 -0400
 

Remote/S3.hs

@@ -99,6 +99,8 @@ remote = specialRemoteType $ RemoteType
 				(FieldDesc "port to connect to")
 			, optionalStringParser requeststyleField
 				(FieldDesc "for path-style requests, set to \"path\"")
+			, signatureVersionParser signatureField
+				(FieldDesc "S3 signature version")
 			, optionalStringParser mungekeysField HiddenField
 			, optionalStringParser AWS.s3credsField HiddenField
 			]
@@ -148,6 +150,22 @@ protocolField = Accepted "protocol"
 requeststyleField :: RemoteConfigField
 requeststyleField = Accepted "requeststyle"
 
+signatureField :: RemoteConfigField
+signatureField = Accepted "signature"
+
+newtype SignatureVersion = SignatureVersion Int
+
+signatureVersionParser :: RemoteConfigField -> FieldDesc -> RemoteConfigFieldParser
+signatureVersionParser f fd =
+	genParser go f defver fd
+		(Just (ValueDesc "v2 or v4"))
+  where
+	go "v2" = Just (SignatureVersion 2)
+	go "v4" = Just (SignatureVersion 4)
+	go _ = Nothing
+
+	defver = SignatureVersion 2
+
 portField :: RemoteConfigField
 portField = Accepted "port"
 
@@ -877,7 +895,10 @@ s3Configuration c = cfg
 		Nothing
 			| port == 443 -> AWS.HTTPS
 			| otherwise -> AWS.HTTP
-	cfg = S3.s3 proto endpoint False
+	cfg = case getRemoteConfigValue signatureField c of
+		Just (SignatureVersion 4) -> 
+			S3.s3v4 proto endpoint False S3.SignWithEffort
+		_ -> S3.s3 proto endpoint False
 
 data S3Info = S3Info
 	{ bucket :: S3.Bucket

doc/bugs/S3_special_remote_support_for_DigitalOcean_Spaces/comment_2_3bbdf23c8a4a480f4f6b8e8a2f8ddecd._comment

@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2020-05-07T16:46:14Z"
+ content="""
+I have added to git-annex a way to use v4 authentication signatures.
+
+You will need a daily build, or the next release of git-annex.
+
+Give it a try by adding signature=v4 to your initremote
+or enableremote, and please let me know if it works or how it fails with
+that.
+"""]]

doc/forum/backblaze_s3/comment_1_854390b9a781da82ecb85ad85eecad04._comment

@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-05-07T16:23:26Z"
+ content="""
+It may be that they are only supporting a newer version of the S3
+protocol that does authorization differently. The S3 library
+git-annex uses defaults to V2 not V4.
+
+I thought that library didn't properly support V4, but it seems it does,
+although I don't know if it works in all cases. So, I've added an
+initremote option for S3, signature=v4 .. give it a try.
+"""]]

doc/special_remotes/S3.mdwn

@@ -77,6 +77,10 @@ the S3 remote.
   If you get an error about a host name not existing, it's a good
   indication that you need to use this.
 
+* `signature` - This controls the S3 signature version to use.
+  "v2" is currently the default, "v4" is needed to use some S3 services.
+  If you get some kind of authentication error, try "v4".
+
 * `bucket` - S3 requires that buckets have a globally unique name, 
   so by default, a bucket name is chosen based on the remote name
   and UUID. This can be specified to pick a bucket name.

doc/special_remotes/S3/comment_34_cf57e8dbd9fdc7c487565b61808b6bb2._comment

@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""V4"""
+ date="2020-05-07T16:49:05Z"
+ content="""
+@bec.watson, I've now added a way to use V4 authentication, the authentication=v4
+option. 
+
+Please file a bug report if it still doesn't work.
+"""]]