CI to automate build of cargo lockfiles on different Alpine releases for git-annex aports https://gitlab.alpinelinux.org/alpine/aports/-/tree/master/community/git-annex
Find a file
Joey Hess 28720c795f
limit url downloads to whitelisted schemes
Security fix! Allowing any schemes, particularly file: and
possibly others like scp: allowed file exfiltration by anyone who had
write access to the git repository, since they could add an annexed file
using such an url, or using an url that redirected to such an url,
and wait for the victim to get it into their repository and send them a copy.

* Added annex.security.allowed-url-schemes setting, which defaults
  to only allowing http and https URLs. Note especially that file:/
  is no longer enabled by default.

* Removed annex.web-download-command, since its interface does not allow
  supporting annex.security.allowed-url-schemes across redirects.
  If you used this setting, you may want to instead use annex.web-options
  to pass options to curl.

With annex.web-download-command removed, nearly all url accesses in
git-annex are made via Utility.Url via http-client or curl. http-client
only supports http and https, so no problem there.
(Disabling one and not the other is not implemented.)

Used curl --proto to limit the allowed url schemes.

Note that this will cause git annex fsck --from web to mark files using
a disallowed url scheme as not being present in the web. That seems
acceptable; fsck --from web also does that when a web server is not available.

youtube-dl already disabled file: itself (probably for similar
reasons). The scheme check was also added to youtube-dl urls for
completeness, although that check won't catch any redirects it might
follow. But youtube-dl goes off and does its own thing with other
protocols anyway, so that's fine.

Special remotes that support other domain-specific url schemes are not
affected by this change. In the bittorrent remote, aria2c can still
download magnet: links. The download of the .torrent file is
otherwise now limited by annex.security.allowed-url-schemes.

This does not address any external special remotes that might download
an url themselves. Current thinking is all external special remotes will
need to be audited for this problem, although many of them will use
http libraries that only support http and not curl's menagarie.

The related problem of accessing private localhost and LAN urls is not
addressed by this commit.

This commit was sponsored by Brett Eisenberg on Patreon.
2018-06-16 11:57:50 -04:00
Annex limit url downloads to whitelisted schemes 2018-06-16 11:57:50 -04:00
Assistant change Remote.repo to Remote.getRepo 2018-06-04 15:30:26 -04:00
Backend fix migration bug and make fsck warn 2018-05-23 14:07:51 -04:00
Build fix build 2018-04-27 12:59:09 -04:00
CmdLine GIT_ANNEX_SHELL_APPENDONLY 2018-05-25 13:17:56 -04:00
Command limit url downloads to whitelisted schemes 2018-06-16 11:57:50 -04:00
Config use DynamicConfig to handle cost-command 2017-08-17 14:04:29 -04:00
Database fix build with old version of persistent 2017-09-25 09:57:41 -04:00
debian BF: deb standalone patch - force use of gzip compression 2018-05-23 11:48:29 -04:00
doc limit url downloads to whitelisted schemes 2018-06-16 11:57:50 -04:00
Git Fix build with ghc 8.4+, which broke due to the Semigroup Monoid change 2018-05-30 12:28:43 -04:00
Limit AssociatedFile newtype 2017-03-10 13:35:31 -04:00
Logs avoid git annex info remote buffering list of keys 2018-04-26 16:13:05 -04:00
Messages Fix mangling of --json output of utf-8 characters when not running in a utf-8 locale 2018-04-16 16:21:21 -04:00
P2P improve indent 2018-06-14 11:40:23 -04:00
Remote remove unused import 2018-06-14 12:33:00 -04:00
RemoteDaemon move protocol version stuff to the Net free monad 2018-03-12 15:20:51 -04:00
standalone use proot to support Android 8 2018-05-08 13:55:10 -04:00
static Revert "remove newlines from static js and css" 2014-06-13 02:20:39 -04:00
templates version: Added "dependency versions" line. 2017-04-07 18:16:11 -04:00
Test Split Test.hs and avoid optimising it much, to need less memory to compile. 2018-02-18 11:48:48 -04:00
Types limit url downloads to whitelisted schemes 2018-06-16 11:57:50 -04:00
Upgrade squelch a couple of warnings about moveAnnex return code 2017-02-28 12:49:17 -04:00
Utility limit url downloads to whitelisted schemes 2018-06-16 11:57:50 -04:00
.ghci workaround for ghci fragility in reusing objects compiled with ghc 2012-10-20 16:20:42 -04:00
.gitattributes update changelog location 2016-08-22 23:54:11 -04:00
.gitignore ignore generated file Build/SysConfig.hs 2018-02-28 11:58:53 -04:00
.mailmap alias for Yaroslav @ yahoo to mailmap 2015-09-09 12:38:59 -04:00
Annex.hs avoid insertWith' depreaction warning 2018-04-22 13:28:31 -04:00
Assistant.hs fold Build/SysConfig.hs into BuildInfo via include 2017-12-14 12:46:57 -04:00
Backend.hs more lambda-case conversion 2017-12-05 15:00:50 -04:00
bash-completion.bash update from optparse-applicative output 2017-06-09 11:30:24 -04:00
build.bat let build.bat also be run from cygwin terminal 2015-04-21 14:12:14 -04:00
BuildFlags.hs split BuildInfo and BuildFlags 2018-01-02 13:47:51 -04:00
BuildInfo.hs split BuildInfo and BuildFlags 2018-01-02 13:47:51 -04:00
CHANGELOG limit url downloads to whitelisted schemes 2018-06-16 11:57:50 -04:00
CmdLine.hs Make --json and --quiet suppress automatic init messages 2016-09-05 15:34:38 -04:00
Command.hs add --json-error-messages (not yet implemented) 2018-02-19 14:32:15 -04:00
Common.hs finally really add back custom-setup stanza 2017-12-31 16:36:39 -04:00
Config.hs configuration and docs for tracking exports 2017-09-19 13:05:43 -04:00
COPYRIGHT add GETINFO to external protocol (for ronnypfa) 2018-06-08 11:56:24 -04:00
Creds.hs finish git-annex enable-tor 2016-11-29 17:30:27 -04:00
Crypto.hs add KeyVariety type 2017-02-24 15:16:56 -04:00
ghci avoid tab warnings from ghc 8 2016-11-17 13:39:30 -04:00
git-annex.cabal releasing package git-annex version 6.20180529 2018-05-29 13:06:56 -04:00
git-annex.hs windows build fix 2018-01-09 11:51:17 -04:00
git-union-merge.hs Always use filesystem encoding for all file and handle reads and writes. 2016-12-24 14:46:31 -04:00
Git.hs Propigate GIT_DIR and GIT_WORK_TREE environment to external special remotes. 2016-05-06 12:26:44 -04:00
Jenkinsfile Revert "debugging strange old version of git-annex in windows installer bundle" 2017-10-26 11:32:04 -04:00
Key.hs Fix mangling of --json output of utf-8 characters when not running in a utf-8 locale 2018-04-16 16:21:21 -04:00
Limit.hs followup 2018-06-04 12:12:56 -04:00
Logs.hs implement export.log and resolve export conflicts 2017-08-31 15:47:23 -04:00
Makefile android: try harder to force PIE for android 5+ 2018-04-16 17:30:21 -04:00
Messages.hs Added INFO to external special remote protocol. 2018-02-06 13:03:55 -04:00
NEWS limit url downloads to whitelisted schemes 2018-06-16 11:57:50 -04:00
README use https 2014-11-06 14:20:10 -04:00
Remote.hs change Remote.repo to Remote.getRepo 2018-06-04 15:30:26 -04:00
Setup.hs Added git-remote-tor-annex, which allows git pull and push to the tor hidden service. 2016-11-21 17:27:38 -04:00
stack-windows.yaml Removed the testsuite build flag 2017-12-20 12:25:03 -04:00
stack.yaml Revert "response" 2018-05-30 11:27:54 -04:00
Test.hs limit url downloads to whitelisted schemes 2018-06-16 11:57:50 -04:00
Types.hs better dup key with -J fix 2017-10-17 18:48:53 -04:00
Upgrade.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00

git-annex allows managing files with git, without checking the file
contents into git. While that may seem paradoxical, it is useful when
dealing with files larger than git can currently easily handle, whether due
to limitations in memory, checksumming time, or disk space.

For documentation, see doc/ or <https://git-annex.branchable.com/>