S3: Added protocol= initremote setting, to allow https to be used on a non-standard port

protocol=https implies port=443 and
port=443 implies protocol=https
-- this was necessary because the existing configs set port=443, but
with a protocol setting, users will naturally want to use it, and then
there's no need for them to supply the default https port. So we keep
back-compat, add a nicer way to enable https, and also add support for
non-standard https ports.

CHANGELOG

@@ -37,6 +37,8 @@ git-annex (7.20190220) UNRELEASED; urgency=medium
   * import: Let --force overwrite symlinks, not only regular files.
   * Android: Fix typo of name of armv7l in installation script.
     Thanks, 4omecha.
+  * S3: Added protocol= initremote setting, to allow https to be used
+    on a non-standard port.
 
  -- Joey Hess <id@joeyh.name>  Wed, 20 Feb 2019 14:20:59 -0400
 

Remote/S3.hs

@@ -146,7 +146,6 @@ s3Setup' ss u mcreds c gc
 		[ ("datacenter", T.unpack $ AWS.defaultRegion AWS.S3)
 		, ("storageclass", "STANDARD")
 		, ("host", AWS.s3DefaultHost)
-		, ("port", "80")
 		, ("bucket", defbucket)
 		]
 		
@@ -571,9 +570,6 @@ s3Configuration c = cfg
 		Nothing -> S3.s3RequestStyle cfg
 	}
   where
-	proto
-		| port == 443 = AWS.HTTPS
-		| otherwise = AWS.HTTP
 	h = fromJust $ M.lookup "host" c
 	datacenter = fromJust $ M.lookup "datacenter" c
 	-- When the default S3 host is configured, connect directly to
@@ -582,10 +578,25 @@ s3Configuration c = cfg
 	endpoint
 		| h == AWS.s3DefaultHost = AWS.s3HostName $ T.pack datacenter
 		| otherwise = T.encodeUtf8 $ T.pack h
-	port = let s = fromJust $ M.lookup "port" c in
+	port = case M.lookup "port" c of
+		Just s -> 
 			case reads s of
 				[(p, _)] -> p
 				_ -> giveup $ "bad S3 port value: " ++ s
+		Nothing -> case cfgproto of
+			Just AWS.HTTPS -> 443
+			Just AWS.HTTP -> 80
+			Nothing -> 80
+	cfgproto = case M.lookup "protocol" c of
+		Just "https" -> Just AWS.HTTPS
+		Just "http" -> Just AWS.HTTP
+		Just _ -> giveup $ "bad S3 protocol value"
+		Nothing -> Nothing
+	proto = case cfgproto of
+		Just v -> v
+		Nothing
+			| port == 443 -> AWS.HTTPS
+			| otherwise -> AWS.HTTP
 	cfg = S3.s3 proto endpoint False
 
 data S3Info = S3Info
@@ -735,6 +746,7 @@ s3Info c info = catMaybes
 	[ Just ("bucket", fromMaybe "unknown" (getBucketName c))
 	, Just ("endpoint", w82s (BS.unpack (S3.s3Endpoint s3c)))
 	, Just ("port", show (S3.s3Port s3c))
+	, Just ("protocol", map toLower (show (S3.s3Protocol s3c)))
 	, Just ("storage class", showstorageclass (getStorageClass c))
 	, if configIA c
 		then Just ("internet archive item", iaItemUrl $ fromMaybe "unknown" $ getBucketName c)

doc/bugs/The_S3_special_remote_limits_HTTPS_support_to_port_443.mdwn

@@ -9,3 +9,5 @@ Version 7.20181121 on MacOS and 7.20190130-g024120065 on FreeBSD
 
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 Yes. I only encountered the problem because git-annex works well enough for me that I want to put a lot more data into it.
+
+> [[fixed|done]] --[[Joey]]

doc/bugs/The_S3_special_remote_limits_HTTPS_support_to_port_443/comment_1_f6de7a30714d6281a81ae218bb92dfd6._comment

@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-03-22T15:48:24Z"
+ content="""
+I've added a protocol= setting, so you can use port=N protocol=https
+"""]]

doc/special_remotes/S3.mdwn

@@ -58,9 +58,15 @@ the S3 remote.
   affect new objects sent to the remote, but not objects already
   stored there.
 
-* `host` and `port` - Specify in order to use a different, S3 compatable
+* `host` - Specify in order to use a different, S3 compatable
   service.
 
+* `protocol` - Either "http" (the default) or "https". Setting
+  protocol=https implies port=443.
+
+* `port` - Specify the port to connect to. Only needed when using a service
+  on an unusual port. Setting port=443 implies protocol=https.
+
 * `requeststyle` - Set to "path" to use path style requests, instead of the
   default DNS style requests. This is needed with some S3 services.