don't force use of conduit in withUrlOptionsPromptingCreds

Use curl for downloads from git remotes when annex.url-options and other
git configs are set.

If the url needs a password, curl will fail, and git credential will not be
used to prompt for it. But the user can set --netrc in url-options and
put the password in the netrc file.

This also means that url-options settings like -4 will take effect.
That was the case before commit 1883f7ef8f
forced conduit to be used.

Annex/Url.hs

@@ -147,6 +147,10 @@ withUrlOptions a = a =<< getUrlOptions
 
 -- When downloading an url, if authentication is needed, uses
 -- git-credential to prompt for username and password.
+--
+-- Note that, when the downloader is curl, it will not use git-credential.
+-- If the user wants to, they can configure curl to use a netrc file that
+-- handles authentication.
 withUrlOptionsPromptingCreds :: (U.UrlOptions -> Annex a) -> Annex a
 withUrlOptionsPromptingCreds a = do
 	g <- Annex.gitRepo
@@ -156,12 +160,6 @@ withUrlOptionsPromptingCreds a = do
 	a $ uo
 		{ U.getBasicAuth = \u -> prompter $
 			getBasicAuthFromCredential g cc u
-		-- Can't download with curl and handle basic auth,
-		-- so make sure it uses conduit.
-		, U.urlDownloader = case U.urlDownloader uo of
-			U.DownloadWithCurl _ -> U.DownloadWithConduit $
-				U.DownloadWithCurlRestricted mempty
-			v -> v
 		}
 
 checkBoth :: U.URLString -> Maybe Integer -> U.UrlOptions -> Annex Bool

CHANGELOG

@@ -8,6 +8,8 @@ git-annex (10.20220823) UNRELEASED; urgency=medium
   * When accessing a git remote over http needs a git credential
     prompt for a password, cache it for the lifetime of the git-annex
     process, rather than repeatedly prompting.
+  * Use curl for downloads from git remotes when annex.url-options
+    and other git configs are set.
 
  -- Joey Hess <id@joeyh.name>  Mon, 29 Aug 2022 15:03:04 -0400
 

doc/bugs/http_remotes_ignore_annex.web-options_--netrc/comment_2_599a66a1833000dc0e54a33511891b36._comment

@@ -0,0 +1,33 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2022-09-09T19:19:44Z"
+ content="""
+Confirmed this behavior.
+
+It is due to withUrlOptionsPromptingCreds, which forces use of conduit
+rather than curl. The idea there was to use git credentials when basic
+auth is needed. Since those can be provided to conduit but not to curl
+(securely).
+
+But I do think that, if the user has forced use of curl, it ought to use curl.
+Even if the user only set options to `-4`, and so curl is not going to use
+the netrc and will fail the download. I have changed it to do so.
+
+----
+
+This bug report also suggests making git-annex read the netrc file itself.
+Note that git does *not* read the netrc file itself. What it does do is use
+libcurl. git-annex has good reasons to not use libcurl though.
+
+I am not thrilled by the prospect of implementing a parser for netrc
+in git-annex. The file is not even documented on my debian system;
+curl's man page links to a `netrc(5)` but that does not exist.
+
+Aside from git-credential-netrc, there is not a single mention of 
+the netrc file in git's documentation. This is arguably surprising behavior
+on the part of git. 
+
+I feel that git's support for netrc is vestigal and mostly supersceded by
+git credentials.
+"""]]

doc/git-annex.mdwn

@@ -1698,12 +1698,15 @@ Remotes are configured using these settings in `.git/config`.
   (rather than the default built-in url downloader).
 
   For example, to force IPv4 only, set it to "-4".
-  Or to make curl use your ~/.netrc file, set it to "--netrc".
 
   Setting this option makes git-annex use curl, but only
   when annex.security.allowed-ip-addresses is configured in a
   specific way. See its documentation.
 
+  Setting this option prevents git-annex from using git-credential
+  for prompting for http passwords. Instead, you can include "--netrc"
+  to make curl use your ~/.netrc file and record the passwords there.
+
 * `annex.youtube-dl-options`
 
   Options to pass to youtube-dl when using it to find the url to download