Refuse to upgrade direct mode repositories when git is older than 2.22

That git fixed a memory leak that could cause an OOM during the upgrade.

Most git-annex builds have a new enough git already.
OSX git was upgraded with brew.

Linux i386ancient build's git was too old. Upgrading it to a fixed
git didn't work (due to the newer git not working with the old ssh,
https://bugs.chromium.org/p/git/issues/detail?id=7 )

Choices to deal with that were:

* Somehow make direct mode upgrade work with the old git, avoiding its
  OOM problem. One way would be to switch the repo to indirect mode
  first, and so upgrade to a repo with locked files. Not good when
  the filesystem does not support symlinks.
* backport the OOM fix from git 2.22
  (And do what about the version number so git-annex knows it's fixed?)
* backport openssh (and possibly more stuff)
* move the i386ancient build to at least Debian stretch (still backporting git)
  But this will make it no longer work with some of the ancient kernels it
  targets.

Of those, backporting the OOM fix seemed the best approach. Put "oomfix"
in the git version number to indicate it.

I have not automated building the git backport, so here's the patch I
used:

diff -ur orig/git-2.1.4/convert.c git-2.1.4/convert.c
--- orig/git-2.1.4/convert.c	2014-12-18 18:42:18.000000000 +0000
+++ git-2.1.4/convert.c	2019-08-29 20:05:04.371872338 +0100
@@ -404,7 +404,7 @@
 	if (start_async(&async))
 		return 0;	/* error was already reported */

-	if (strbuf_read(&nbuf, async.out, len) < 0) {
+	if (strbuf_read(&nbuf, async.out, 0) < 0) {
 		error("read from external filter %s failed", cmd);
 		ret = 0;
 	}
diff -ur orig/git-2.1.4/GIT-VERSION-GEN git-2.1.4/GIT-VERSION-GEN
--- orig/git-2.1.4/GIT-VERSION-GEN	2014-12-18 18:42:18.000000000 +0000
+++ git-2.1.4/GIT-VERSION-GEN	2019-08-29 20:06:39.132743228 +0100
@@ -1,7 +1,7 @@
 #!/bin/sh

 GVF=GIT-VERSION-FILE
-DEF_VER=v2.1.4
+DEF_VER=v2.1.4.oomfix

 LF='
 '
diff -ur orig/git-2.1.4/configure git-2.1.4/configure
--- orig/git-2.1.4/configure	2014-12-18 18:42:19.000000000 +0000
+++ git-2.1.4/configure	2019-08-29 20:27:45.896380015 +0100
@@ -580,8 +580,8 @@
 # Identity of this package.
 PACKAGE_NAME='git'
 PACKAGE_TARNAME='git'
-PACKAGE_VERSION='2.1.4'
-PACKAGE_STRING='git 2.1.4'
+PACKAGE_VERSION='2.1.4.oomfix'
+PACKAGE_STRING='git 2.1.4.oomfix'
 PACKAGE_BUGREPORT='git@vger.kernel.org'
 PACKAGE_URL=''

diff -ur orig/git-2.1.4/version git-2.1.4/version
--- orig/git-2.1.4/version	2014-12-18 18:42:19.000000000 +0000
+++ git-2.1.4/version	2019-08-29 20:06:17.572545210 +0100
@@ -1 +1 @@
-2.1.4
+2.1.4.oomfix

CHANGELOG

@@ -2,6 +2,8 @@ git-annex (7.20190826) UNRELEASED; urgency=medium
 
   * Automatically convert direct mode repositories to v7 with adjusted
     unlocked branches and set annex.thin.
+  * Refuse to upgrade direct mode repositories when git is older than 2.22,
+    which fixed a memory leak that could cause an OOM during the upgrade.
   * assistant: When creating a new repository, no longer use direct
     mode, instead use v7 adjusted branches with annex.thin.
   * init: When run on a crippled filesystem with --version=5,

Upgrade/V5.hs

@@ -23,11 +23,13 @@ import qualified Database.Keys
 import qualified Git
 import qualified Git.LsFiles
 import qualified Git.Branch
+import qualified Git.Version
 import Git.FilePath
 import Git.FileMode
 import Git.Config
 import Git.Ref
 import Utility.InodeCache
+import Utility.DottedVersion
 import Annex.AdjustedBranch
 
 import qualified Data.ByteString as S
@@ -38,11 +40,14 @@ upgrade automatic = flip catchNonAsync (const $ return False) $ do
 		showAction "v5 to v6"
 	ifM isDirect
 		( do
+			checkGitVersionForDirectUpgrade
 			convertDirect
 			-- Worktree files are already populated, so don't
 			-- have this try to populate them again.
 			scanUnlockedFiles False
-		, scanUnlockedFiles True
+		, do
+			checkGitVersionForIndirectUpgrade
+			scanUnlockedFiles True
 		)
 	configureSmudgeFilter
 	-- Inode sentinal file was only used in direct mode and when
@@ -53,6 +58,27 @@ upgrade automatic = flip catchNonAsync (const $ return False) $ do
 		createInodeSentinalFile True
 	return True
 
+-- git before 2.22 would OOM running git status on a large file.
+--
+-- Older versions of git that are patched (with
+-- commit 02156ab031e430bc45ce6984dfc712de9962dec8)
+-- can include "oomfix" in their version to indicate it.
+gitWillOOM :: Annex Bool
+gitWillOOM = liftIO $ do
+	v <- Git.Version.installed
+	return $ v < Git.Version.normalize "2.22" &&
+		not ("oomfix" `isInfixOf` fromDottedVersion v)
+
+-- configureSmudgeFilter has to run git status, and direct mode files
+-- are unlocked, so avoid the upgrade failing half way through.
+checkGitVersionForDirectUpgrade :: Annex ()
+checkGitVersionForDirectUpgrade = whenM gitWillOOM $
+	giveup "You must upgrade git to version 2.22 or newer in order to use this version of git-annex in this repository."
+
+checkGitVersionForIndirectUpgrade :: Annex ()
+checkGitVersionForIndirectUpgrade = whenM gitWillOOM $
+	warning "Git is older than version 2.22 and so it has a memory leak that affects using unlocked files. Recommend you upgrade git before unlocking any files in your repository."
+
 convertDirect :: Annex ()
 convertDirect = do
 	{- Direct mode makes the same tradeoff of using less disk

Utility/DottedVersion.hs

@@ -18,7 +18,10 @@ instance Ord DottedVersion where
 	compare (DottedVersion _ x) (DottedVersion _ y) = compare x y
 
 instance Show DottedVersion where
-	show (DottedVersion s _) = s
+	show = fromDottedVersion
+
+fromDottedVersion :: DottedVersion -> String
+fromDottedVersion (DottedVersion s _) = s
 
 {- To compare dotted versions like 1.7.7 and 1.8, they are normalized to
  - a somewhat arbitrary integer representation. -}

doc/bugs/git-annex_upgrade_fails.mdwn

@@ -43,3 +43,5 @@ Would be happy if anybody could share some insight on how to get that repo to v7
 
 Regards,
 Felix
+
+> [[fixed|done]] --[[Joey]]

doc/bugs/git-annex_upgrade_fails/comment_3_acdfcd2bb40820cf69748fe4df956366._comment

@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2019-08-29T17:02:02Z"
+ content="""
+It doesn't seem practical to get around the old git's OOM problem.
+I think that the best thing to do is to not allow direct mode upgrade to v7 
+until git has been upgraded.
+
+The minimum git version is 2.22. Unfortunately, Debian stable shipped 2.20.
+So direct mode users there who upgrade git-annex will need to also upgrade
+git in order for git-annex to convert their repo -- and since the next
+git-annex release drops support for direct mode, they'll have to upgrade
+git (or downgrade git-annex) in order to use git-annex in their repo at all.
+"""]]