annex.cachecreds: New config to allow disabling of credentials caching for special remotes.

Note that it does not prevent storing p2p access tokens or multicast
encryption keys, since those are not cached; the previous commit
established the distinction.

How well this works depends on how often getRemoteCredPair is called and
how expensive it is. In some cases setting this will result in an annoying
number of gpg password prompts and/or slowdowns due to reading creds
from the git-annex branch and decrypting, which could be improved by calling
getRemoteCredPair less often.

This commit was sponsored by Ilya Shlyakhter on Patreon.

CHANGELOG

@@ -15,6 +15,8 @@ git-annex (7.20181122) UNRELEASED; urgency=medium
   * dropunused: When an unused object file has gotten modified, eg due to
     annex.thin being set, don't silently skip it, but display a warning
     and let --force drop it.
+  * annex.cachecreds: New config to allow disabling of credentials caching
+    for special remotes.
 
  -- Joey Hess <id@joeyh.name>  Tue, 27 Nov 2018 12:29:27 -0400
 

Creds.hs

@@ -139,8 +139,10 @@ getEnvCredPair storage = liftM2 (,)
   where
 	(uenv, penv) = credPairEnvironment storage
 
+{- Writes a cred pair to local cache, unless prevented by configuration. -}
 writeCacheCredPair :: CredPair -> CredPairStorage -> Annex ()
 writeCacheCredPair credpair storage = 
+	whenM (annexCacheCreds <$> Annex.getGitConfig) $
 		writeCreds (encodeCredPair credpair) (credPairFile storage)
 
 readCacheCredPair :: CredPairStorage -> Annex (Maybe CredPair)

Types/GitConfig.hs

@@ -102,6 +102,7 @@ data GitConfig = GitConfig
 	, annexAllowUnverifiedDownloads :: Bool
 	, annexMaxExtensionLength :: Maybe Int
 	, annexJobs :: Concurrency
+	, annexCacheCreds :: Bool
 	, coreSymlinks :: Bool
 	, coreSharedRepository :: SharedRepository
 	, receiveDenyCurrentBranch :: DenyCurrentBranch
@@ -177,6 +178,7 @@ extractGitConfig r = GitConfig
 		getmaybe (annex "security.allow-unverified-downloads")
 	, annexMaxExtensionLength = getmayberead (annex "maxextensionlength")
 	, annexJobs = maybe NonConcurrent Concurrent $ getmayberead (annex "jobs")
+	, annexCacheCreds = getbool (annex "cachecreds") True
 	, coreSymlinks = getbool "core.symlinks" True
 	, coreSharedRepository = getSharedRepository r
 	, receiveDenyCurrentBranch = getDenyCurrentBranch r

doc/encryption.mdwn

@@ -129,3 +129,16 @@ of the special remote with the option `mac=HMACSHA512`. The available
 MAC algorithms are HMACSHA1, HMACSHA224, HMACSHA256, HMACSHA384, and
 HMACSHA512. Note that it is not possible to change algorithm for a
 non-empty remote.
+
+## credentials storage
+
+Special remotes that need some form of credentials, such as a password,
+may support embedding the credentials in the git repository, using
+embedcreds=yes. See individual special remotes' documentation for details.
+When credentials are embedded in the repository, they're also encrypted using
+whatever encryption setting has been selected for the repository.
+
+Such credentials are also cached locally in a file only you can read,
+in `.git/annex/creds/`. If you prefer to not expose the credentials on disk
+in unencrypted form, you can disable this cache, by setting the
+`annex.cachecreds` config to `false`.

doc/git-annex.mdwn

@@ -1160,6 +1160,16 @@ Here are all the supported configuration settings.
   git-annex will wait up to this many seconds for the pid lock
   file to go away, and will then abort if it cannot continue. Default: 300
 
+* `annex.cachecreds`
+
+  When "true" (the default), git-annex will cache credentials used to
+  access special remotes in files in .git/annex/creds/
+  that only you can read. To disable that caching, set to "false",
+  and credentials will only be read from the environment, or if
+  they have been embedded in encrypted form in the git repository, will
+  be extracted and decrypted each time git-annex needs to access the
+  remote.
+
 * `remote.<name>.annex-cost`
 
   When determining which repository to

doc/todo/encrypt_only_the_credentials/comment_1_7c3911e8fbc981c7e1f32b1464ce4122._comment

@@ -0,0 +1,30 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2018-12-04T17:18:19Z"
+ content="""
+I agree it would make sense to have some way to embedcreds without
+encrypting content stored on the remote.
+
+I suppose one way to express it is as encryption=onlycreds embedcreds=yes
+with one or more keyids.
+
+Note that the tahoe special remote supports embedcreds,
+but disallows setting any encryption (because tahoe handles that)
+so the encryptions can only be stored in the clear. It would make sense for
+tahoe to support encryption=onlycreds while disallowing other encryption
+methods.
+
+----
+
+As for storing creds locally only in encrypted form, it would suffice to
+have an option that makes git-annex not write anything to
+.git/annex/creds/, so it would not use those files as a cache, and would
+pull the creds out of the repository and decrypt each time needed
+(or use environment varibles for creds when applicable.) In some cases
+that would cause more gpg prompts. I think that S3 and WebDAV special
+remotes only call getRemoteCredPair once per run, but external may
+call it repeatedly, and glacier calls it once per request.
+
+Implemented as annex.cachecreds.
+"""]]