annex.dotfiles

The git add behavior changes could be avoided if it turns out to be
really annoying, but then it would need to behave the old way when
annex.dotfiles=false and the new way when annex.dotfiles=true. I'd
rather not have the config option result in such divergent behavior as
`git annex add .` skipping a dotfile (old) vs adding to annex (new).

Note that the assistant always adds dotfiles to the annex.
This is surprising, but not new behavior. Might be worth making it also
honor annex.dotfiles, but I wonder if perhaps some user somewhere uses
it and keeps large files in a directory that happens to begin with a
dot. Since dotfiles and dotdirs are a unix culture thing, and the
assistant users may not be part of that culture, it seems best to keep
its current behavior for now.

CHANGELOG

@@ -13,6 +13,15 @@ git-annex (8.20191107) UNRELEASED; urgency=medium
     Microbenchmarks show around 10-25% speedup of sqlite database operations.
   * Improved serialization of filenames and keys to the sqlite databases,
     avoiding encoding problems.
+  * annex.dotfiles configuration setting can be used to let dotfiles
+    be added to the annex the same as other files. When it's set,
+    annex.largefiles can be used to match whatever dotfiles should be added.
+  * add: Explicitly passing dotfiles no longer adds them to the annex,
+    unless you set annex.dotfiles=true.
+  * add: When adding a whole directory, any dotfiles found in it will
+    not be skipped, but will be added to git by default. This behavior
+    can be configured with annex.dotfiles.
+  * add: Removed the --include-dotfiles option.
 
  -- Joey Hess <id@joeyh.name>  Tue, 29 Oct 2019 15:13:03 -0400
 

CmdLine/Seek.hs

@@ -59,23 +59,14 @@ withFilesInGitNonRecursive needforce a l = ifM (Annex.getState Annex.force)
 				getfiles c ps
 			_ -> giveup needforce
 
-withFilesNotInGit :: Bool -> (RawFilePath -> CommandSeek) -> [WorkTreeItem] -> CommandSeek
-withFilesNotInGit skipdotfiles a l
-	| skipdotfiles = do
-		{- dotfiles are not acted on unless explicitly listed -}
-		files <- filter (not . dotfile . fromRawFilePath) <$>
-			seekunless (null ps && not (null l)) ps
-		dotfiles <- seekunless (null dotps) dotps
-		go (files++dotfiles)
-	| otherwise = go =<< seekunless False l
+withFilesNotInGit :: (RawFilePath -> CommandSeek) -> [WorkTreeItem] -> CommandSeek
+withFilesNotInGit  a l = go =<< seek
   where
-	(dotps, ps) = partition (\(WorkTreeItem f) -> dotfile f) l
-	seekunless True _ = return []
-	seekunless _ l' = do
+	seek = do
 		force <- Annex.getState Annex.force
 		g <- gitRepo
 		liftIO $ Git.Command.leaveZombie
-			<$> LsFiles.notInRepo force (map (\(WorkTreeItem f) -> toRawFilePath f) l') g
+			<$> LsFiles.notInRepo force (map (\(WorkTreeItem f) -> toRawFilePath f) l) g
 	go fs = seekActions $ prepFiltered a $
 		return $ concat $ segmentPaths (map (\(WorkTreeItem f) -> toRawFilePath f) l) fs
 

Command/Add.hs

@@ -1,6 +1,6 @@
 {- git-annex command
  -
- - Copyright 2010-2017 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2019 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -19,6 +19,7 @@ import Annex.Link
 import Annex.Tmp
 import Messages.Progress
 import Git.FilePath
+import Config.GitConfig
 import qualified Utility.RawFilePath as R
 
 cmd :: Command
@@ -29,7 +30,6 @@ cmd = notBareRepo $
 
 data AddOptions = AddOptions
 	{ addThese :: CmdParams
-	, includeDotFiles :: Bool
 	, batchOption :: BatchMode
 	, updateOnly :: Bool
 	}
@@ -37,10 +37,6 @@ data AddOptions = AddOptions
 optParser :: CmdParamsDesc -> Parser AddOptions
 optParser desc = AddOptions
 	<$> cmdParams desc
-	<*> switch
-		( long "include-dotfiles"
-		<> help "don't skip dotfiles"
-		)
 	<*> parseBatchOption
 	<*> switch
 		( long "update"
@@ -52,13 +48,16 @@ seek :: AddOptions -> CommandSeek
 seek o = startConcurrency commandStages $ do
 	largematcher <- largeFilesMatcher
 	addunlockedmatcher <- addUnlockedMatcher
-	let gofile file = ifM (checkFileMatcher largematcher (fromRawFilePath file) <||> Annex.getState Annex.force)
-		( start file addunlockedmatcher
-		, ifM (annexAddSmallFiles <$> Annex.getGitConfig)
-			( startSmall file
-			, stop
+	annexdotfiles <- getGitConfigVal annexDotFiles 
+	let gofile file =
+		let file' = fromRawFilePath file
+		in ifM (pure (annexdotfiles || not (dotfile file')) <&&> (checkFileMatcher largematcher file' <||> Annex.getState Annex.force))
+			( start file addunlockedmatcher
+			, ifM (annexAddSmallFiles <$> Annex.getGitConfig)
+				( startSmall file
+				, stop
+				)
 			)
-		)
 	case batchOption o of
 		Batch fmt
 			| updateOnly o ->
@@ -68,7 +67,7 @@ seek o = startConcurrency commandStages $ do
 			l <- workTreeItems (addThese o)
 			let go a = a (commandAction . gofile) l
 			unless (updateOnly o) $
-				go (withFilesNotInGit (not $ includeDotFiles o))
+				go withFilesNotInGit
 			go withFilesMaybeModified
 			go withUnmodifiedUnlockedPointers
 

Command/Smudge.hs

@@ -24,6 +24,7 @@ import Backend
 import Utility.Metered
 import Annex.InodeSentinal
 import Utility.InodeCache
+import Config.GitConfig
 
 import qualified Data.ByteString as S
 import qualified Data.ByteString.Lazy as L
@@ -166,9 +167,16 @@ shouldAnnex file moldkey = ifM (annexGitAddToAnnex <$> Annex.getGitConfig)
 	, checkheuristics
 	)
   where
-	checkmatcher d = do
-		matcher <- largeFilesMatcher
-		checkFileMatcher' matcher file d
+	checkmatcher d
+		| dotfile file = ifM (getGitConfigVal annexDotFiles)
+			( go
+			, return False
+			)
+		| otherwise = go
+	  where
+		go = do
+			matcher <- largeFilesMatcher
+			checkFileMatcher' matcher file d
 	
 	checkheuristics = case moldkey of
 		Just _ -> return True

Command/Uninit.hs

@@ -42,7 +42,7 @@ check = do
 seek :: CmdParams -> CommandSeek
 seek ps = do
 	l <- workTreeItems ps
-	withFilesNotInGit False (commandAction . whenAnnexed (startCheckIncomplete . fromRawFilePath)) l
+	withFilesNotInGit (commandAction . whenAnnexed (startCheckIncomplete . fromRawFilePath)) l
 	Annex.changeState $ \s -> s { Annex.fast = True }
 	withFilesInGit (commandAction . whenAnnexed Command.Unannex.start) l
 	finish

Config/Smudge.hs

@@ -42,7 +42,6 @@ configureSmudgeFilter = unlessM (fromRepo Git.repoIsLocalBare) $ do
 stdattr :: [String]
 stdattr =
 	[ "* filter=annex"
-	, ".* !filter"
 	]
 
 -- Note that this removes the local git attributes for filtering, 

NEWS

@@ -5,9 +5,8 @@ git-annex (8.20191107) upstream; urgency=medium
   Existing repositories will be automatically upgraded by default.
   You can prevent this, by runing: git config annex.autoupgraderepository false
  
-  There is no user-visible change between v7 and v8. However, the upgrade
-  process involves regenerating some sqlite databases. There are a couple
-  consequences of the upgrade to keep in mind:
+  The upgrade process involves regenerating some sqlite databases. There are a
+  couple consequences of the upgrade to keep in mind:
 
   * Any incremental fscks that were started in v7 won't resume where
     they left off in v8, but will instead begin again from the first file.
@@ -16,6 +15,11 @@ git-annex (8.20191107) upstream; urgency=medium
   * After the upgrade, git-annex will in some situations have to do extra
     work while it finishes populating its sqlite databases.
 
+  Also, there are some behavior changes around adding dotfiles. While before
+  git-annex add skipped adding dotfiles when operating on whole directories,
+  and added dotfiles that were explicitly listed to the annex, it now adds
+  dotfiles to git by default, unless annex.dotfiles is set to true.
+
  -- Joey Hess <id@joeyh.name>  Thu, 07 Nov 2019 13:21:11 -0400
 
 git-annex (7.20191024) upstream; urgency=medium

Types/GitConfig.hs

@@ -86,6 +86,7 @@ data GitConfig = GitConfig
 	, annexAriaTorrentOptions :: [String]
 	, annexCrippledFileSystem :: Bool
 	, annexLargeFiles :: Configurable (Maybe String)
+	, annexDotFiles :: Configurable Bool
 	, annexGitAddToAnnex :: Bool
 	, annexAddSmallFiles :: Bool
 	, annexFsckNudge :: Bool
@@ -158,6 +159,7 @@ extractGitConfig configsource r = GitConfig
 	, annexCrippledFileSystem = getbool (annex "crippledfilesystem") False
 	, annexLargeFiles = configurable Nothing $
 		fmap Just $ getmaybe (annex "largefiles")
+	, annexDotFiles = configurable False $ getmaybebool (annex "dotfiles")
 	, annexGitAddToAnnex = getbool (annex "gitaddtoannex") True
 	, annexAddSmallFiles = getbool (annex "addsmallfiles") True
 	, annexFsckNudge = getbool (annex "fscknudge") True
@@ -232,6 +234,7 @@ mergeGitConfig gitconfig repoglobals = gitconfig
 	, annexSyncContent = merge annexSyncContent
 	, annexResolveMerge = merge annexResolveMerge
 	, annexLargeFiles = merge annexLargeFiles
+	, annexDotFiles = merge annexDotFiles
 	, annexAddUnlocked = merge annexAddUnlocked
 	}
   where

Upgrade/V7.hs

@@ -9,6 +9,7 @@
 
 module Upgrade.V7 where
 
+import qualified Annex
 import Annex.Common
 import Annex.CatFile
 import qualified Database.Keys
@@ -43,6 +44,8 @@ upgrade automatic = do
 	liftIO . nukeFile =<< fromRepo gitAnnexKeysDbIndexCacheOld
 	liftIO . nukeFile =<< fromRepo gitAnnexKeysDbLockOld
 	
+	updateSmudgeFilter
+
 	return True
 
 gitAnnexKeysDbOld :: Git.Repo -> FilePath
@@ -109,3 +112,21 @@ populateKeysDb = do
 	liftIO $ void cleanup
 	Database.Keys.closeDb
 
+-- The gitatrributes used to have a line that prevented filtering dotfiles,
+-- but now they are filtered and annex.dotfiles controls whether they get
+-- added to the annex.
+--
+-- Only done on local gitattributes, not any gitatrributes that might be
+-- checked into the repository.
+updateSmudgeFilter :: Annex ()
+updateSmudgeFilter = do
+	lf <- Annex.fromRepo Git.attributesLocal
+	ls <- liftIO $ lines <$> catchDefaultIO "" (readFileStrict lf)
+	let ls' = removedotfilter ls
+	when (ls /= ls') $
+		liftIO $ writeFile lf (unlines ls')
+  where
+	removedotfilter ("* filter=annex":".* !filter":rest) =
+		"* filter=annex" : removedotfilter rest
+	removedotfilter (l:ls) = l : removedotfilter ls
+	removedotfilter [] = []

doc/git-annex-add.mdwn

@@ -18,6 +18,8 @@ git has been configured to ignore will be silently skipped.
 If annex.largefiles is configured, and does not match a file, 
 `git annex add` will behave the same as `git add` and add the
 non-large file directly to the git repository, instead of to the annex.
+(By default dotfiles are assumed to not be large, and are added directly
+to git, but annex.dotfiles can be configured to annex those too.)
 
 Large files are added to the annex in locked form, which prevents further
 modification of their content unless unlocked by [[git-annex-unlock]](1).
@@ -30,11 +32,6 @@ annexed content, and other symlinks.
 
 # OPTIONS
 
-* `--include-dotfiles`
-
-  Dotfiles are skipped unless explicitly listed, or unless this option is
-  used.
-
 * `--force`
 
   Add gitignored files.

doc/git-annex-config.mdwn

@@ -37,6 +37,16 @@ These settings can be overridden on a per-repository basis using
    This sets a default, which can be overridden by annex.largefiles
    attributes in `.gitattributes` files, or by `git config`.
 
+* `annex.dotfiles`
+
+   Normally, dotfiles are assumed to be files like .gitignore,
+   whose content should always be part of the git repository, so 
+   they will not be added to the annex. Setting annex.dotfiles to true
+   makes dotfiles be added to the annex the same as any other file. 
+
+   This sets a default, which can be overridden by annex.dotfiles
+   in `git config`.
+
 * `annex.addunlocked`
 
    Commands like `git-annex add` default to adding files to the repository

doc/git-annex-matching-expression.mdwn

@@ -26,6 +26,11 @@ The following terms can be used:
 
   The glob can contain `*` and `?` to match arbitrary characters.
 
+  Note that this matches on the whole filename, relative to the top 
+  of the git directory. So, `include=foo` will include a file `foo`
+  in the top, but not `subdir/foo`. To include both, use 
+  `include=foo or include=*/foo`
+
 * `smallerthan=size` / `largerthan=size`
 
   Matches only files smaller than, or larger than the specified size.

doc/git-annex.mdwn

@@ -901,8 +901,9 @@ Like other git commands, git-annex is configured via `.git/config`.
 
   This configures the behavior of both git-annex and git when adding
   files to the repository. By default, `git-annex add` adds all files
-  to the annex, and `git add` adds files to git (unless they were added
-  to the annex previously). When annex.largefiles is configured, both
+  to the annex (except dotfiles), and `git add` adds files to git
+  (unless they were added to the annex previously).
+  When annex.largefiles is configured, both
   `git annex add` and `git add` will add matching large files to the
   annex, and the other files to git.
 
@@ -910,6 +911,23 @@ Like other git commands, git-annex is configured via `.git/config`.
   `git annex import`, `git annex addurl`, `git annex importfeed`
   and the assistant.
 
+* `annex.dotfiles`
+
+  Normally, dotfiles are assumed to be files like .gitignore,
+  whose content should always be part of the git repository, so 
+  they will not be added to the annex. Setting annex.dotfiles to true
+  makes dotfiles be added to the annex the same as any other file. 
+
+  To annex only some dotfiles, set this and configure annex.largefiles
+  to match the ones you want. For example, to match only dotfiles ending 
+  in ".big"
+
+	git config annex.largefiles "(include=.*.big or include=*/.*.big) or (exclude=.* and exclude=*/.*)"
+  	git config annex.dotfiles true
+  
+  To configure a default annex.dotfiles for all clones of the repository,
+  this can be set in [[git-annex-config]](1).
+
 * `annex.gitaddtoannex`
 
   Setting this to false will prevent `git add` from adding

doc/todo/making_it_easier_to_smudge_dotfiles.mdwn

@@ -3,3 +3,5 @@ I want to add some dotfiles in the root of my repository to git-annex as unlocke
 I was thinking that it might make sense to have a `git annex config` option to tell git-annex not to add the `.* !filter` line to `.git/info/attributes` when initialising other clones of this repo.  In the meantime, I've worked around it using a `post_checkout` hook in my `~/.mrconfig` which edits `.git/info/attributes`.
 
 --spwhitton
+
+> annex.dotfiles added, [[done]] --[[Joey]] 

doc/upgrades.mdwn

@@ -67,6 +67,11 @@ work while it finishes populating the new databases. After this one-time
 speed cost, git-annex's speed will improve when using the new and improved
 databases.
   
+Also, there are some behavior changes around adding dotfiles. While before
+git-annex add skipped adding dotfiles when operating on whole directories,
+and added dotfiles that were explicitly listed to the annex, it now adds
+dotfiles to git by default, unless annex.dotfiles is set to true.
+
 ## v6 -> v7 (git-annex version 7.x)
 
 v6 repositories are automatically upgraded to v7.