Use http-conduit for url downloads by default, annex.web-options enables curl

* For url downloads, git-annex now defaults to using a http library,
  rather than wget or curl. But, if annex.web-options is set, it will
  use curl. To use the .netrc file, run:
    git config annex.web-options --netrc
* git-annex no longer uses wget (and wget is no longer shipped with
  git-annex builds).

Note that curl is always run in silent mode, since the new API for
download has a MeterUpdate and doesn't make way for curl progress
output. It might be worth writing a parser for curl's progress output
to update the meter when using it, but I didn't bother with this edge
case for now.

This commit was supported by the NSF-funded DataLad project.

Remote/Web.hs

@@ -17,6 +17,7 @@ import Annex.Content
 import Config.Cost
 import Logs.Web
 import Annex.UUID
+import Messages.Progress
 import Utility.Metered
 import qualified Annex.Url as Url
 import Annex.YoutubeDl
@@ -74,13 +75,14 @@ downloadKey key _af dest p = unVerified $ get =<< getWebUrls key
 	get [] = do
 		warning "no known url"
 		return False
-	get urls = do
-		showOutput -- make way for download progress bar
-		untilTrue urls $ \u -> do
-			let (u', downloader) = getDownloader u
-			case downloader of
-				YoutubeDownloader -> youtubeDlTo key u' dest
-				_ -> downloadUrl key p [u'] dest
+	get urls = untilTrue urls $ \u -> do
+		let (u', downloader) = getDownloader u
+		case downloader of
+			YoutubeDownloader -> do
+				showOutput
+				youtubeDlTo key u' dest
+			_ -> metered (Just p) key (pure Nothing) $ \_ p' -> 
+				downloadUrl key p' [u'] dest
 
 downloadKeyCheap :: Key -> AssociatedFile -> FilePath -> Annex Bool
 downloadKeyCheap _ _ _ = return False