Slightly sped up the linux standalone bundle

Reduce the number of directories listed in libdirs, which makes the linker
check a lot less dead ends looking for directories.

Eliminated some directories that didn't really contain shared libraries,
or only contained the linker.

That left only 2, one in lib and one in usr/lib, so consolidate those two.

Doing it this way, rather than just consolidating all libs that might exist
into a single directory means that, if there are optimised versions of some
libs, eg in lib/subarch/foo.so, and lib/subarch2/foo.so, they don't get
moved around in a way that would make the linker pick the wrong one.

Build/LinuxMkLibs.hs

@@ -36,18 +36,51 @@ mklibs top = do
 	fs <- dirContentsRecursive top
 	exes <- filterM checkExe fs
 	libs <- parseLdd <$> readProcess "ldd" exes
+	
 	glibclibs <- glibcLibs
 	let libs' = nub $ libs ++ glibclibs
-	libdirs <- nub . catMaybes <$> mapM (installLib installFile top) libs'
+	let (linkers, otherlibs) = partition ("ld-linux" `isInfixOf`) libs'
+	libdirs <- nub . catMaybes <$> mapM (installLib installFile top) otherlibs
+	libdirs' <- consolidateUsrLib top libdirs
+
+	gconvlibs <- gconvLibs
+	mapM_ (installLib installFile top) gconvlibs
 
 	-- Various files used by runshell to set up env vars used by the
 	-- linker shims.
-	writeFile (top </> "libdirs") (unlines libdirs)
-	writeFile (top </> "gconvdir")
-		(parentDir $ Prelude.head $ filter ("/gconv/" `isInfixOf`) glibclibs)
+	writeFile (top </> "libdirs") (unlines libdirs')
+	writeFile (top </> "gconvdir") (parentDir $ Prelude.head gconvlibs)
 	
-	let linker = Prelude.head $ filter ("ld-linux" `isInfixOf`) libs'
+	mapM_ (installLib installFile top) linkers
+	let linker = Prelude.head linkers
 	mapM_ (installLinkerShim top linker) exes
+	
+{- If there are two libdirs that are the same except one is in
+ - usr/lib and the other is in lib/, move the contents of the usr/lib one
+ - into the lib/ one. This reduces the number of directories the linker
+ - needs to look in, and so reduces the number of failed stats
+ - and improves startup time.
+ -}
+consolidateUsrLib :: FilePath -> [FilePath] -> IO [FilePath]
+consolidateUsrLib top libdirs = map reverse <$> go [] (map reverse libdirs)
+  where
+	go c [] = return c
+	go c (x:[]) = return (x:c)
+	go c (x:y:rest)
+		| x == y ++ reverse ("/usr") = do
+			let x' = reverse x
+			let y' = reverse y
+			fs <- getDirectoryContents (inTop top x')
+			forM_ fs $ \f -> do
+				print f
+				unless (dirCruft f) $
+					renameFile 
+						(inTop top (x' </> f))
+						(inTop top (y' </> f))
+			go (y:c) rest
+		| otherwise = do
+			print (x,y)
+			go (x:c) (y:rest)
 
 {- Installs a linker shim script around a binary.
  -

CHANGELOG

@@ -17,6 +17,7 @@ git-annex (8.20200720.2) UNRELEASED; urgency=medium
   * Deal with unusual IFS settings in the shell scripts for linux
     standalone and OSX app.
     Thanks, Yaroslav Halchenko
+  * Slightly sped up the linux standalone bundle.
 
  -- Joey Hess <id@joeyh.name>  Tue, 21 Jul 2020 12:58:30 -0400
 

Utility/LinuxMkLibs.hs

@@ -1,6 +1,6 @@
 {- Linux library copier and binary shimmer
  -
- - Copyright 2013 Joey Hess <id@joeyh.name>
+ - Copyright 2013-2020 Joey Hess <id@joeyh.name>
  -
  - License: BSD-2-clause
  -}
@@ -9,6 +9,7 @@ module Utility.LinuxMkLibs (
 	installLib,
 	parseLdd,
 	glibcLibs,
+	gconvLibs,
 	inTop,
 ) where
 
@@ -59,9 +60,15 @@ parseLdd = mapMaybe (getlib . dropWhile isSpace) . lines
   where
 	getlib l = headMaybe . words =<< lastMaybe (split " => " l)
 
-{- Get all glibc libs and other support files, including gconv files
+{- Get all glibc libs.
  -
  - XXX Debian specific. -}
 glibcLibs :: IO [FilePath]
 glibcLibs = lines <$> readProcess "sh"
-	["-c", "dpkg -L libc6:$(dpkg --print-architecture) libgcc1:$(dpkg --print-architecture) | egrep '\\.so|gconv'"]
+	["-c", "dpkg -L libc6:$(dpkg --print-architecture) libgcc1:$(dpkg --print-architecture) | egrep '\\.so' | grep -v /gconv/ | grep -v ld.so.conf | grep -v sotruss-lib"]
+
+{- Get gblibc's gconv libs, which are handled specially.. -}
+gconvLibs :: IO [FilePath]
+gconvLibs = lines <$> readProcess "sh"
+	["-c", "dpkg -L libc6:$(dpkg --print-architecture) | grep /gconv/"]
+

doc/todo/may_be___40__again__41___to_prelink_or_somehow_avoid_all_the_failing_opens__63__/comment_13_5d65192366f7690c22d1a209fb1e8f78._comment

@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 13"""
+ date="2020-07-31T17:41:55Z"
+ content="""
+`git-annex init` runs git something like 30 times, so it's close to the worst
+case for a single git-annex command, other than when smudge filters are run.
+
+I tried inlining runshell into both git and git-annex scripts, thinking
+that the overhead of starting the second shell script might be measurable.
+It was not; I saw `git-annex init` taking 0.10-0.14s before and after.
+
+I also tried trimming out some parts of the script that normally
+don't run, like the android support, but that didn't speed it up.
+
+With the consolidated lib dirs, I did see `git-annex init` drop to 0.07-0.10s.
+
+Ok, I found a way to consolidate the directories that will not include
+directories for optimised libs. Implemented that.
+"""]]