limit url downloads to whitelisted schemes

Security fix! Allowing any schemes, particularly file: and
possibly others like scp: allowed file exfiltration by anyone who had
write access to the git repository, since they could add an annexed file
using such an url, or using an url that redirected to such an url,
and wait for the victim to get it into their repository and send them a copy.

* Added annex.security.allowed-url-schemes setting, which defaults
  to only allowing http and https URLs. Note especially that file:/
  is no longer enabled by default.

* Removed annex.web-download-command, since its interface does not allow
  supporting annex.security.allowed-url-schemes across redirects.
  If you used this setting, you may want to instead use annex.web-options
  to pass options to curl.

With annex.web-download-command removed, nearly all url accesses in
git-annex are made via Utility.Url via http-client or curl. http-client
only supports http and https, so no problem there.
(Disabling one and not the other is not implemented.)

Used curl --proto to limit the allowed url schemes.

Note that this will cause git annex fsck --from web to mark files using
a disallowed url scheme as not being present in the web. That seems
acceptable; fsck --from web also does that when a web server is not available.

youtube-dl already disabled file: itself (probably for similar
reasons). The scheme check was also added to youtube-dl urls for
completeness, although that check won't catch any redirects it might
follow. But youtube-dl goes off and does its own thing with other
protocols anyway, so that's fine.

Special remotes that support other domain-specific url schemes are not
affected by this change. In the bittorrent remote, aria2c can still
download magnet: links. The download of the .torrent file is
otherwise now limited by annex.security.allowed-url-schemes.

This does not address any external special remotes that might download
an url themselves. Current thinking is all external special remotes will
need to be audited for this problem, although many of them will use
http libraries that only support http and not curl's menagarie.

The related problem of accessing private localhost and LAN urls is not
addressed by this commit.

This commit was sponsored by Brett Eisenberg on Patreon.

doc/git-annex.mdwn

@@ -1387,12 +1387,20 @@ Here are all the supported configuration settings.
   If set, the command is run and each line of its output is used as a HTTP
   header. This overrides annex.http-headers.
 
-* `annex.web-download-command`
+* `annex.security.allowed-url-schemes`
 
-  Use to specify a command to run to download a file from the web.
+  List of URL schemes that git-annex is allowed to download content from.
+  The default is "http https".
 
-  In the command line, %url is replaced with the url to download,
-  and %file is replaced with the file that it should be saved to.
+  Think very carefully before changing this; there are security
+  implications. For example, if it's changed to allow "file" URLs,
+  then anyone who can get a commit into your git-annex repository
+  could add a pointer to a private file located outside that repository, 
+  risking it being copied into the repository and transferred on to other
+  remotes, exposing its content.
+
+  Some special remotes support their own domain-specific URL
+  schemes; those are not affected by this configuration setting.
 
 * `annex.secure-erase-command`
 

doc/special_remotes/external/git-annex-remote-torrent

@@ -123,7 +123,7 @@ while read line; do
 			url="$2"
 			# List contents of torrent.
 			tmp=$(mktemp)
-			if ! runcmd curl -o "$tmp" "$url"; then
+			if ! runcmd curl --proto -all,http,https -o "$tmp" "$url"; then
 				echo CHECKURL-FAILURE
 			else
 				oldIFS="$IFS"
@@ -166,7 +166,7 @@ while read line; do
 						echo TRANSFER-FAILURE RETRIEVE "$key" "no known torrent urls for this key"
 					else
 						tmp=$(mktemp)
-						if ! runcmd curl -o "$tmp" "$url"; then
+						if ! runcmd curl --proto -all,http,https -o "$tmp" "$url"; then
 							echo TRANSFER-FAILURE RETRIEVE "$key" "failed downloading torrent file from $url"
 						else
 							filenum="$(echo "$url" | sed 's/(.*#\(\d*\)/\1/')"

doc/special_remotes/web.mdwn

@@ -6,6 +6,6 @@ See [[tips/using_the_web_as_a_special_remote]] for usage examples.
 Currently git-annex only supports downloading content from the web; 
 it cannot upload to it or remove content.
 
-This special remote uses arbitrary urls on the web as the source for content.
+This special remote uses urls on the web as the source for content.
 git-annex can also download content from a normal git remote, accessible by
 http.

doc/special_remotes/web/comment_2_333141cc9ec6c26ffd19aa95303a91e3._comment

@@ -5,4 +5,7 @@
  date="2013-08-17T08:59:11Z"
  content="""
 When it says \"arbitrary urls\", it means it. The only requirement is that the url be well formed and that wget or whatever command you have it configured to use via annex.web-download-command knows how to download it.
+
+Update 2018: That used to be the case, but it's now limited by default to
+http and https urls.
 """]]

doc/tips/Using_Git-annex_as_a_web_browsing_assistant.mdwn

@@ -11,12 +11,6 @@ The first step is to install the Firefox plugin
 [FlashGot](http://flashgot.net/). We will use it to provide the Firefox
 shortcuts to add things to our annex.
 
-We also need a normal download manager, if we want to get status updates as
-the download is done. We'll need to configure git-annex to use it by
-setting `annex.web-download-command` as Joey describes in his comment on
-[[todo/wishlist: allow configuration of downloader for addurl]]. See the
-manpage [[git-annex]] for more information on setting configuration.
-
 Once we have installed all that, we need a script that has an interface
 which FlashGot can treat as a downloader, but which calls git-annex to do
 the actual downloading. Such a script is available from

doc/tips/Using_Git-annex_as_a_web_browsing_assistant/comment_1_74167f9fff400f148916003468c77de4._comment

@@ -1,8 +0,0 @@
-[[!comment format=mdwn
- username="http://joeyh.name/"
- nickname="joey"
- subject="comment 1"
- date="2013-04-11T20:16:02Z"
- content="""
-As of my last commit, you don't really need a separate download manager. The webapp will now display urls that `git annex addurl` is downloading in among the other transfers.
-"""]]

doc/todo/easy_way_to_reproduce_normal_download_command.mdwn

@@ -22,3 +22,10 @@ what about the other settings, is it okay to hardcode those?
 
 maybe this would be easier if there would be an options override just
 like rsync, but separate ones for curl and wget... --[[anarcat]]
+
+> git-annex now only uses curl, and defaults to a built-in http downloader.
+> The annex.web-download-command is no longer supported. annex.web-options
+> can be used to pass options to curl.
+> 
+> So, I don't think this todo is relevant anymore, closing [[done]].
+> --[[Joey]]