Submodules are now supported by git-annex!

Seems to work, but still experimental until it's been tested more.

When repositories are on filesystems not supporting symlinks, the .git dir
symlink trick cannot be used. Since we're going to be in direct mode
anyway, the .git dir symlink is not strictly needed.

However, I have not fixed the code that creates new annex symlinks to
handle this case -- the committed symlinks will be wrong.

git annex sync happens to currently fail in a submodule using direct mode,
because there's no HEAD ref. That also needs to be dealt with to get
this fully working in crippled filesystems.

Leaving http://github.com/datalad/datalad/issues/44 open until these issues
are dealt with.

Annex.hs

@@ -37,7 +37,7 @@ module Annex (
 import Common
 import qualified Git
 import qualified Git.Config
-import Annex.Direct.Fixup
+import Annex.Fixup
 import Git.CatFile
 import Git.CheckAttr
 import Git.CheckIgnore
@@ -183,12 +183,13 @@ newState c r = AnnexState
 	}
 
 {- Makes an Annex state object for the specified git repo.
- - Ensures the config is read, if it was not already. -}
+ - Ensures the config is read, if it was not already, and performs
+ - any necessary git repo fixups. -}
 new :: Git.Repo -> IO AnnexState
 new r = do
 	r' <- Git.Config.read =<< Git.relPath r
 	let c = extractGitConfig r'
-	newState c <$> if annexDirect c then fixupDirect r' else return r'
+	newState c <$> fixupRepo r' c
 
 {- Performs an action in the Annex monad from a starting state,
  - returning a new state. -}

Annex/Direct.hs

@@ -406,7 +406,25 @@ setDirect wantdirect = do
 	Annex.changeGitConfig $ \c -> c { annexDirect = wantdirect }
   where
 	val = Git.Config.boolConfig wantdirect
-	setbare = setConfig (ConfigKey Git.Config.coreBare) val
+	coreworktree = ConfigKey "core.worktree"
+	indirectworktree = ConfigKey "core.indirect-worktree"
+	setbare = do
+		-- core.worktree is not compatable with
+		-- core.bare; git does not allow both to be set, so
+		-- unset it when enabling direct mode, caching in
+		-- core.indirect-worktree
+		if wantdirect
+			then moveconfig coreworktree indirectworktree
+			else moveconfig indirectworktree coreworktree
+		setConfig (ConfigKey Git.Config.coreBare) val
+	moveconfig src dest = do
+		v <- getConfigMaybe src
+		case v of
+			Nothing -> noop
+			Just wt -> do
+				unsetConfig src
+				setConfig dest wt
+				reloadConfig
 
 {- Since direct mode sets core.bare=true, incoming pushes could change
  - the currently checked out branch. To avoid this problem, HEAD

Annex/Direct/Fixup.hs

@@ -1,31 +0,0 @@
-{- git-annex direct mode guard fixup
- -
- - Copyright 2013 Joey Hess <id@joeyh.name>
- -
- - Licensed under the GNU GPL version 3 or higher.
- -}
-
-module Annex.Direct.Fixup where
-
-import Git.Types
-import Git.Config
-import qualified Git.Construct as Construct
-import Utility.Path
-import Utility.SafeCommand
-
-{- Direct mode repos have core.bare=true, but are not really bare.
- - Fix up the Repo to be a non-bare repo, and arrange for git commands
- - run by git-annex to be passed parameters that override this setting. -}
-fixupDirect :: Repo -> IO Repo
-fixupDirect r@(Repo { location = l@(Local { gitdir = d, worktree = Nothing }) }) = do
-	let r' = r
-		{ location = l { worktree = Just (parentDir d) }
-		, gitGlobalOpts = gitGlobalOpts r ++
-			[ Param "-c"
-			, Param $ coreBare ++ "=" ++ boolConfig False
-			]
-		}
-	-- Recalc now that the worktree is correct.
-	rs' <- Construct.fromRemotes r'
-	return $ r' { remotes = rs' }
-fixupDirect r = return r

Annex/Fixup.hs

@@ -0,0 +1,86 @@
+{- git-annex repository fixups
+ -
+ - Copyright 2013, 2015 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU GPL version 3 or higher.
+ -}
+
+module Annex.Fixup where
+
+import Git.Types
+import Git.Config
+import Types.GitConfig
+import qualified Git.Construct as Construct
+import Utility.Path
+import Utility.SafeCommand
+import Utility.Directory
+import Utility.PosixFiles
+import Utility.Exception
+
+import System.IO
+import System.FilePath
+import System.Directory
+import Data.List
+import Control.Monad
+import Control.Monad.IfElse
+import qualified Data.Map as M
+
+fixupRepo :: Repo -> GitConfig -> IO Repo
+fixupRepo r c = do
+	r' <- fixupSubmodule r c
+	if annexDirect c
+		then fixupDirect r'
+		else return r'
+
+{- Direct mode repos have core.bare=true, but are not really bare.
+ - Fix up the Repo to be a non-bare repo, and arrange for git commands
+ - run by git-annex to be passed parameters that override this setting. -}
+fixupDirect :: Repo -> IO Repo
+fixupDirect r@(Repo { location = l@(Local { gitdir = d, worktree = Nothing }) }) = do
+	let r' = r
+		{ location = l { worktree = Just (parentDir d) }
+		, gitGlobalOpts = gitGlobalOpts r ++
+			[ Param "-c"
+			, Param $ coreBare ++ "=" ++ boolConfig False
+			]
+		}
+	-- Recalc now that the worktree is correct.
+	rs' <- Construct.fromRemotes r'
+	return $ r' { remotes = rs' }
+fixupDirect r = return r
+
+{- Submodules have their gitdir containing ".git/modules/", and
+ - have core.worktree set, and also have a .git file in the top
+ - of the repo. 
+ -
+ - We need to unset core.worktree, and change the .git file into a
+ - symlink to the git directory. This way, annex symlinks will be
+ - of the usual .git/annex/object form, and will consistently work
+ - whether a repo is used as a submodule or not, and wheverever the
+ - submodule is mounted.
+ -
+ - When the filesystem doesn't support symlinks, we cannot make .git
+ - into a symlink. In this case, we merely adjust the Repo so that
+ - symlinks to objects that get checked in will be in the right form.
+ -}
+fixupSubmodule :: Repo -> GitConfig -> IO Repo
+fixupSubmodule r@(Repo { location = l@(Local { worktree = Just w, gitdir = d }) }) c
+	| (".git" </> "modules") `isInfixOf` d = do
+		when (coreSymlinks c) $
+			replacedotgit
+				`catchNonAsync` \_e -> hPutStrLn stderr
+					"warning: unable to convert submodule to form that will work with git-annex"
+		return $ r
+			{ location = if coreSymlinks c
+				then l { gitdir = dotgit }
+				else l
+			, config = M.delete "core.worktree" (config r)
+			}
+  where
+	dotgit = w </> ".git"
+	replacedotgit = whenM (doesFileExist dotgit) $ do
+		nukeFile dotgit
+		createSymbolicLink (w </> d) dotgit
+		maybe (error "unset core.worktree failed") (\_ -> return ())
+			=<< Git.Config.unset "core.worktree" r
+fixupSubmodule r _ = return r

Config.hs

@@ -37,13 +37,9 @@ setConfig (ConfigKey key) value = do
 reloadConfig :: Annex ()
 reloadConfig = Annex.changeGitRepo =<< inRepo Git.Config.reRead
 
-{- Unsets a git config setting. (Leaves it in state currently.) -}
+{- Unsets a git config setting. (Leaves it in state.) -}
 unsetConfig :: ConfigKey -> Annex ()
-unsetConfig ck@(ConfigKey key) = ifM (isJust <$> getConfigMaybe ck)
+unsetConfig (ConfigKey key) = void $ inRepo $ Git.Config.unset key
-	( inRepo $ Git.Command.run
-		[Param "config", Param "--unset", Param key]
-	, noop -- avoid unsetting something not set; that would fail
-	)
 
 {- A per-remote config setting in git config. -}
 remoteConfig :: Git.Repo -> UnqualifiedConfigKey -> ConfigKey

Git/Config.hs

@@ -14,6 +14,7 @@ import Common
 import Git
 import Git.Types
 import qualified Git.Construct
+import qualified Git.Command
 import Utility.UserInfo
 
 {- Returns a single git config setting, or a default value if not set. -}
@@ -193,3 +194,17 @@ changeFile f k v = boolSystem "git"
 	, Param k
 	, Param v
 	]
+
+{- Unsets a git config setting, in both the git repo,
+ - and the cached config in the Repo.
+ -
+ - If unsetting the config fails, including in a read-only repo, or
+ - when the config is not set, returns Nothing.
+ -}
+unset :: String -> Repo -> IO (Maybe Repo)
+unset k r = ifM (Git.Command.runBool ps r)
+	( return $ Just $ r { config = M.delete k (config r) }
+	, return Nothing
+	)
+  where
+	ps = [Param "config", Param "--unset-all", Param k]

debian/changelog

@@ -21,6 +21,7 @@ git-annex (5.2015022) UNRELEASED; urgency=medium
     (This used to be done, but it forgot to do it since version 4.20130909.)
   * When re-execing git-annex, use current program location, rather than
     ~/.config/git-annex/program, when possible.
+  * Submodules are now supported by git-annex!
 
  -- Joey Hess <id@joeyh.name>  Thu, 19 Feb 2015 14:16:03 -0400
 

doc/bugs/Git_annexed_files_symlink_are_wrong_when_submodule_is_not_in_the_same_path.mdwn

@@ -63,3 +63,9 @@ I tried playing with making the repository direct and then indirect, hoping that
 ### Please provide any additional information below.
 
 [[!tag confirmed]]
+
+> [[fixed|done]] -- with a current version of git, git-annex now supports
+> [[/submodules]]. NB: Filesystem must support symlinks, or this won't
+> work.
+> 
+> -- [[Joey]]

doc/links/the_details.mdwn

@@ -3,6 +3,7 @@
 * [[encryption]]
 * [[key-value backends|backends]]
 * [[bare_repositories]]
+* [[submodules]]
 * [[internals]]
 * [[scalability]]
 * [[design]]

doc/submodules.mdwn

@@ -0,0 +1,21 @@
+[Git submodules](http://git-scm.com/book/en/v2/Git-Tools-Submodules) are
+supported by git-annex since version 5.20150303.
+
+Git normally makes a `.git` **file** in a
+submodule, that points to the real git repository under `.git/modules/`.
+This presents problems for git-annex. So, when used in a submodule,
+git-annex will automatically replace the `.git` file with a symlink
+pointing at the git repository.
+
+With that taken care of, git-annex should work ok in submodules. Although
+this is a new and somewhat experimental feature.
+
+The conversion of .git file to .git symlink mostly won't bother git.
+
+Known problems:
+
+* If you want to delete a whole submodule, `git rm submodule`
+  will refuse to delete it, complaining that the
+  submodule "uses a .git directory". Workaround: Use `rm -rf`
+  to delete the tree, and then `git commit`.
+* This won't work on filesystems not supporting symlinks.