mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Types/GitConfig.hs

701 lines
26 KiB
Haskell
Raw Normal View History

type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
{- git-annex configuration
-
add directional stalldetection and bwlimit configs Sponsored-by: Dartmouth College's DANDI project
2024-01-19 19:14:26 +00:00
- Copyright 2012-2024 Joey Hess <id@joeyh.name>
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
-
update licenses from GPL to AGPL This does not change the overall license of the git-annex program, which was already AGPL due to a number of sources files being AGPL already. Legally speaking, I'm adding a new license under which these files are now available; I already released their current contents under the GPL license. Now they're dual licensed GPL and AGPL. However, I intend for all my future changes to these files to only be released under the AGPL license, and I won't be tracking the dual licensing status, so I'm simply changing the license statement to say it's AGPL. (In some cases, others wrote parts of the code of a file and released it under the GPL; but in all cases I have contributed a significant portion of the code in each file and it's that code that is getting the AGPL license; the GPL license of other contributors allows combining with AGPL code.)
2019-03-13 19:48:14 +00:00
- Licensed under the GNU AGPL version 3 or higher.
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
-}
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
{-# LANGUAGE TypeSynonymInstances, FlexibleInstances #-}
use ByteString for git config The parser and looking up config keys in the map should both be faster due to using ByteString. I had hoped this would speed up startup time, but any improvement to that was too small to measure. Seems worth keeping though. Note that the parser breaks up the ByteString, but a config map ends up pointing to the config as read, which is retained in memory until every value from it is no longer used. This can change memory usage patterns marginally, but won't affect git-annex.
2019-11-27 20:54:11 +00:00
{-# LANGUAGE OverloadedStrings #-}
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
module Types.GitConfig (
rename Configurable for clarity
2020-09-16 15:16:48 +00:00
GlobalConfigurable(..),
git-annex config annex.largefiles annex.largefiles can be configured by git-annex config, to more easily set a default that will also be used by clones, without needing to shoehorn the expression into the gitattributes file. The git config and gitattributes override that. Whenever something is added to git-annex config, we have to consider what happens if a user puts a purposfully bad value in there. Or, if a new git-annex adds some new value that an old git-annex can't parse. In this case, a global annex.largefiles that can't be parsed currently makes an error be thrown. That might not be ideal, but the gitattribute behaves the same, and is almost equally repo-global. Performance notes: git-annex add and addurl construct a matcher once and uses it for every file, so the added time penalty for reading the global config log is minor. If the gitattributes annex.largefiles were deprecated, git-annex add would get around 2% faster (excluding hashing), because looking that up for each file is not fast. So this new way of setting it is progress toward speeding up add. git-annex smudge does need to load the log every time. As well as checking the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids both overheads.
2019-12-20 16:12:31 +00:00
ConfigSource(..),
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
GitConfig(..),
extractGitConfig,
annex.autocommit can be configured via git-annex config ... to control the default behavior in all clones of a repository. This includes a new Configurable data type, so the GitConfig type indicates which values can be configured this way. The implementation should be quite efficient; the config log is only read once, and only when a Configurable value has not already been set by git-config. Indeed, it would be nice in the future to extend this, so that git-config is itself only read on demand. Some commands may not need to look at the git configuration at all. This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 17:40:14 +00:00
mergeGitConfig,
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
globalConfigs,
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
RemoteGitConfig(..),
extractRemoteGitConfig,
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
dummyRemoteGitConfig,
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
annexConfig,
RemoteNameable(..),
remoteAnnexConfig,
remoteConfig,
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
proxyInheritedFields
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
) where
import Common
import qualified Git
import qualified Git.Config
The ssh-options git config is now used by gcrypt, rsync, and ddar special remotes that use ssh as a transport.
2015-02-12 19:44:10 +00:00
import qualified Git.Construct
removal of the rest of remoteGitConfig In keyUrls, the GitConfig is used only by annexLocations to support configured Differences. Since such configurations affect all clones of a repository, the local repo's GitConfig must have the same information as the remote's GitConfig would have. So, used getGitConfig to get the local GitConfig, which is cached and so available cheaply. That actually fixed a bug noone had ever noticed: keyUrls is used for remotes accessed over http. The full git config of such a remote is normally not available, so the remoteGitConfig that keyUrls used would not have the necessary information in it. In copyFromRemoteCheap', it uses gitAnnexLocation, which does need the GitConfig of the remote repo itself in order to check if it's crippled, supports symlinks, etc. So, made the State include that GitConfig, cached. The use of gitAnnexLocation is within a (not $ Git.repoIsUrl repo) guard, so it's local, and so its git config will always be read and available. (Note that gitAnnexLocation in turn calls annexLocations, so the Differences config it uses in this case comes from the remote repo's GitConfig and not from the local repo's GitConfig. As explained above this is ok since they must have the same value.) Not very happy with this mess of different GitConfigs not type-safe and some read only sometimes etc. Very hairy. Think I got it this change right. Test suite passes.. This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
import Git.Types
post-recive hook to make updateInstead work in direct mode and adjusted branches * Added post-recieve hook, which makes updateInstead work with direct mode and adjusted branches. * init: Set up the post-receive hook. This commit was sponsored by Fernando Jimenez on Patreon.
2017-02-17 18:04:43 +00:00
import Git.ConfigTypes
add git-annex updatecluster command Seems to work fine, making the right changes to the git-annex branch.
2024-06-14 18:21:50 +00:00
import Git.Remote (isRemoteKey, isLegalName, remoteKeyToRemoteName)
refactor Better to not have a single function module, and better to have a more specific type than Bool. This commit was sponsored by Jack Hill on Patreon
2019-11-11 22:20:35 +00:00
import Git.Branch (CommitMode(..))
rename Git.Filename to Git.Quote
2023-04-12 21:18:29 +00:00
import Git.Quote (QuotePath(..))
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
import Utility.DataUnits
split cost out into its own module Added a function to insert a new cost into a list, which could be used to asjust costs after a drag and drop.
2013-03-13 20:16:01 +00:00
import Config.Cost
optimise getUUID This avoids a Map lookup each time it's called, instead the GitConfig field lazily looks it up once and then caches.
2016-01-20 20:55:06 +00:00
import Types.UUID
annex.autoupgrade setting
2013-11-22 20:04:20 +00:00
import Types.Distribution
annex.jobs Added annex.jobs setting, which is like using the -J option. Of course, -J overrides annex.jobs. This commit was sponsored by Trenton Cronholm on Patreon.
2018-10-04 16:47:27 +00:00
import Types.Concurrency
numcopies cleanup, part 2 This includes several bug fixes.
2014-01-21 21:08:49 +00:00
import Types.NumCopies
Repository tuning parameters can now be passed when initializing a repository for the first time. * init: Repository tuning parameters can now be passed when initializing a repository for the first time. For details, see http://git-annex.branchable.com/tuning/ * merge: Refuse to merge changes from a git-annex branch of a repo that has been tuned in incompatable ways.
2015-01-27 21:38:06 +00:00
import Types.Difference
add annex.used-refspec
2015-05-14 19:44:08 +00:00
import Types.RefSpec
v7 Install new git hooks in this version. This does beg the question of what to do if git later gets eg a post-smudge hook, that could run git-annex smudge --update. I think the thing to do in that case would be to make git-annex smudge --update install the new hooks. That way, as the user uses git-annex, the hook would be created pretty quickly and without needing any extra syscalls except for when git-annex smudge --update is called. I considered doing something like that for installation of the post-checkout and post-merge hooks, which would have avoided the need for v7. But the only place it was cheap to do it would be in git-annex smudge which could cheaply notice that smudge.log didn't exist yet and so know the hooks needed to be installed. But since smudge used to populate pointer files, it would be quite surprising if a single git checkout/merge failed to update the work tree, and so that idea didn't work out. The other reason for v7 is psychological -- users don't need to worry about whether they might be running an old version of git-annex that doesn't support their v7 repository very well. And bug reports about "v6" have gotten a bit of a bad association in my head since they often hit one of the known limitations and didn't realize it was experimental. newtyped RepoVersion Int to avoid needing 2 comparisons in versionSupportsUnlockedPointers etc. Also it's just nicer. This commit was sponsored by John Pellman on Patreon.
2018-10-25 21:23:53 +00:00
import Types.RepoVersion
stall detection is working New config annex.stalldetection, remote.name.annex-stalldetection, which can be used to deal with remotes that stall during transfers, or are sometimes too slow to want to use. This commit was sponsored by Luke Shumaker on Patreon.
2020-12-08 19:22:18 +00:00
import Types.StallDetection
add directory to views for files that lack specified metadata * view: New field?=glob and ?tag syntax that includes a directory "_" in the view for files that do not have the specified metadata set. * Added annex.viewunsetdirectory git config to change the name of the "_" directory in a view. When in a view using the new syntax, old git-annex will fail to parse the view log. It errors with "Not in a view.", which is not ideal. But that only affects view commands. annex.viewunsetdirectory is included in the View for a couple of reasons. One is to avoid needing to warn the user that it should not be changed when in a view, since that would confuse git-annex. Another reason is that it helped with plumbing the value through to some pure functions. annex.viewunsetdirectory is actually mangled the same as any other view directory. So if it's configured to something like "N/A", there won't be multiple levels of directories, which would also confuse git-annex. Sponsored-By: Jack Hill on Patreon
2023-02-07 20:28:46 +00:00
import Types.View
add git configs for clusters
2024-06-14 15:42:32 +00:00
import Types.Cluster
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
import Config.DynamicConfig
assistant unused file handling Make sanity checker run git annex unused daily, and queue up transfers of unused files to any remotes that will have them. The transfer retrying code works for us here, so eg when a backup disk remote is plugged in, any transfers to it are done. Once the unused files reach a remote, they'll be removed locally as unwanted. If the setup does not cause unused files to go to a remote, they'll pile up, and the sanity checker detects this using some heuristics that are pretty good -- 1000 unused files, or 10% of disk used by unused files, or more disk wasted by unused files than is left free. Once it detects this, it pops up an alert in the webapp, with a button to take action. TODO: Webapp UI to configure this, and also the ability to launch an immediate cleanup of all unused files. This commit was sponsored by Simon Michael.
2014-01-23 02:48:56 +00:00
import Utility.HumanTime
support gpg.program When gpg.program is configured, it's used to get the command to run for gpg. Useful on systems that have only a gpg2 command or want to use it instead of the gpg command.
2015-09-09 22:06:49 +00:00
import Utility.Gpg (GpgCmd, mkGpgCmd)
support annex.shared-sop-command for encryption=shared This works well, and it interoperates with gpg in my testing (although some SOP commands might choose to use a profile that does not so caveat emptor). Note that for creating the Cipher, gpg --gen-random is still used. SOP does not have an eqivilant, and as long as the user has gpg around, which seems likely, it doesn't matter that it uses gpg here, it's not being used for encryption. That seemed better than implementing a second way to get high quality entropy, at least for now. The need for the sop command to run in an empty directory has each call to encrypt and decrypt creating a new temporary directory. That is some unncessary overhead, though probably swamped by the overhead of running the sop command. This could be improved in the future by passing an already empty directory to them, or a sufficiently empty directory (.git/annex/tmp would probably suffice). Sponsored-by: Brett Eisenberg on Patreon
2024-01-12 17:29:34 +00:00
import Utility.StatelessOpenPGP (SOPCmd(..), SOPProfile(..))
pid locking configuration and abstraction layer for git-annex (not actually used anywhere yet)
2015-11-12 21:47:31 +00:00
import Utility.ThreadScheduler (Seconds(..))
limit url downloads to whitelisted schemes Security fix! Allowing any schemes, particularly file: and possibly others like scp: allowed file exfiltration by anyone who had write access to the git repository, since they could add an annexed file using such an url, or using an url that redirected to such an url, and wait for the victim to get it into their repository and send them a copy. * Added annex.security.allowed-url-schemes setting, which defaults to only allowing http and https URLs. Note especially that file:/ is no longer enabled by default. * Removed annex.web-download-command, since its interface does not allow supporting annex.security.allowed-url-schemes across redirects. If you used this setting, you may want to instead use annex.web-options to pass options to curl. With annex.web-download-command removed, nearly all url accesses in git-annex are made via Utility.Url via http-client or curl. http-client only supports http and https, so no problem there. (Disabling one and not the other is not implemented.) Used curl --proto to limit the allowed url schemes. Note that this will cause git annex fsck --from web to mark files using a disallowed url scheme as not being present in the web. That seems acceptable; fsck --from web also does that when a web server is not available. youtube-dl already disabled file: itself (probably for similar reasons). The scheme check was also added to youtube-dl urls for completeness, although that check won't catch any redirects it might follow. But youtube-dl goes off and does its own thing with other protocols anyway, so that's fine. Special remotes that support other domain-specific url schemes are not affected by this change. In the bittorrent remote, aria2c can still download magnet: links. The download of the .torrent file is otherwise now limited by annex.security.allowed-url-schemes. This does not address any external special remotes that might download an url themselves. Current thinking is all external special remotes will need to be audited for this problem, although many of them will use http libraries that only support http and not curl's menagarie. The related problem of accessing private localhost and LAN urls is not addressed by this commit. This commit was sponsored by Brett Eisenberg on Patreon.
2018-06-15 20:52:24 +00:00
import Utility.Url (Scheme, mkScheme)
webapp: Added --port option, and annex.port config The getSocket comment that mentioned using ":port" in the hostname seems to have been incorrect or be out of date. After all, the bug report came when the user first tried doing that, and it didn't work. Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
2024-01-25 18:08:36 +00:00
import Network.Socket (PortNumber)
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
import Control.Concurrent.STM
limit url downloads to whitelisted schemes Security fix! Allowing any schemes, particularly file: and possibly others like scp: allowed file exfiltration by anyone who had write access to the git repository, since they could add an annexed file using such an url, or using an url that redirected to such an url, and wait for the victim to get it into their repository and send them a copy. * Added annex.security.allowed-url-schemes setting, which defaults to only allowing http and https URLs. Note especially that file:/ is no longer enabled by default. * Removed annex.web-download-command, since its interface does not allow supporting annex.security.allowed-url-schemes across redirects. If you used this setting, you may want to instead use annex.web-options to pass options to curl. With annex.web-download-command removed, nearly all url accesses in git-annex are made via Utility.Url via http-client or curl. http-client only supports http and https, so no problem there. (Disabling one and not the other is not implemented.) Used curl --proto to limit the allowed url schemes. Note that this will cause git annex fsck --from web to mark files using a disallowed url scheme as not being present in the web. That seems acceptable; fsck --from web also does that when a web server is not available. youtube-dl already disabled file: itself (probably for similar reasons). The scheme check was also added to youtube-dl urls for completeness, although that check won't catch any redirects it might follow. But youtube-dl goes off and does its own thing with other protocols anyway, so that's fine. Special remotes that support other domain-specific url schemes are not affected by this change. In the bittorrent remote, aria2c can still download magnet: links. The download of the .torrent file is otherwise now limited by annex.security.allowed-url-schemes. This does not address any external special remotes that might download an url themselves. Current thinking is all external special remotes will need to be audited for this problem, although many of them will use http libraries that only support http and not curl's menagarie. The related problem of accessing private localhost and LAN urls is not addressed by this commit. This commit was sponsored by Brett Eisenberg on Patreon.
2018-06-15 20:52:24 +00:00
import qualified Data.Set as S
implement annex.privateremote and remote.name.private configs The slightly unusual parsing in Types.GitConfig avoids the need to look at the remote list to get configs of remotes. annexPrivateRepos combines all the configs, and will only be calculated once, so it's nice and fast. privateUUIDsKnown and regardingPrivateUUID now need to read from the annex mvar, so are not entirely free. But that overhead can be optimised away, as seen in getJournalFileStale. The other call sites didn't seem worth optimising to save a single MVar access. The feature should have impreceptable speed overhead when not being used.
2021-04-23 18:21:57 +00:00
import qualified Data.Map as M
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
import qualified Data.ByteString as B
use a subdirectory of annex.dbdir This allows annex.dbdir to be set globally or always set to the same value when needed. Each repository uses a subdirectory of it. Sponsored-by: Dartmouth College's Datalad project
2022-08-12 17:18:15 +00:00
import qualified System.FilePath.ByteString as P
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
-- | A configurable value, that may not be fully determined yet because
-- the global git config has not yet been loaded.
rename Configurable for clarity
2020-09-16 15:16:48 +00:00
data GlobalConfigurable a
git-annex config annex.largefiles annex.largefiles can be configured by git-annex config, to more easily set a default that will also be used by clones, without needing to shoehorn the expression into the gitattributes file. The git config and gitattributes override that. Whenever something is added to git-annex config, we have to consider what happens if a user puts a purposfully bad value in there. Or, if a new git-annex adds some new value that an old git-annex can't parse. In this case, a global annex.largefiles that can't be parsed currently makes an error be thrown. That might not be ideal, but the gitattribute behaves the same, and is almost equally repo-global. Performance notes: git-annex add and addurl construct a matcher once and uses it for every file, so the added time penalty for reading the global config log is minor. If the gitattributes annex.largefiles were deprecated, git-annex add would get around 2% faster (excluding hashing), because looking that up for each file is not fast. So this new way of setting it is progress toward speeding up add. git-annex smudge does need to load the log every time. As well as checking the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids both overheads.
2019-12-20 16:12:31 +00:00
= HasGitConfig a
-- ^ The git config has a value.
| HasGlobalConfig a
-- ^ The global config has a value (and the git config does not).
annex.autocommit can be configured via git-annex config ... to control the default behavior in all clones of a repository. This includes a new Configurable data type, so the GitConfig type indicates which values can be configured this way. The implementation should be quite efficient; the config log is only read once, and only when a Configurable value has not already been set by git-config. Indeed, it would be nice in the future to extend this, so that git-config is itself only read on demand. Some commands may not need to look at the git configuration at all. This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 17:40:14 +00:00
| DefaultConfig a
-- ^ A default value is known, but not all config sources
-- have been read yet.
New annex.synccontent config setting .. which can be set to true to make git annex sync default to --content. This may become the default at some point in the future. As well as being configuable by git config, it can be configured by git-annex config to control the default behavior in all clones of a repository. Had to add a separate --no-content switch to we can tell if it's been explicitly set, and should override annex.synccontent. If --content was the default, this complication would not be necessary. This commit was sponsored by Jake Vosloo on Patreon.
2017-02-03 18:31:17 +00:00
deriving (Show)
annex.autocommit can be configured via git-annex config ... to control the default behavior in all clones of a repository. This includes a new Configurable data type, so the GitConfig type indicates which values can be configured this way. The implementation should be quite efficient; the config log is only read once, and only when a Configurable value has not already been set by git-config. Indeed, it would be nice in the future to extend this, so that git-config is itself only read on demand. Some commands may not need to look at the git configuration at all. This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 17:40:14 +00:00
git-annex config annex.largefiles annex.largefiles can be configured by git-annex config, to more easily set a default that will also be used by clones, without needing to shoehorn the expression into the gitattributes file. The git config and gitattributes override that. Whenever something is added to git-annex config, we have to consider what happens if a user puts a purposfully bad value in there. Or, if a new git-annex adds some new value that an old git-annex can't parse. In this case, a global annex.largefiles that can't be parsed currently makes an error be thrown. That might not be ideal, but the gitattribute behaves the same, and is almost equally repo-global. Performance notes: git-annex add and addurl construct a matcher once and uses it for every file, so the added time penalty for reading the global config log is minor. If the gitattributes annex.largefiles were deprecated, git-annex add would get around 2% faster (excluding hashing), because looking that up for each file is not fast. So this new way of setting it is progress toward speeding up add. git-annex smudge does need to load the log every time. As well as checking the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids both overheads.
2019-12-20 16:12:31 +00:00
data ConfigSource = FromGitConfig | FromGlobalConfig
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
{- Main git-annex settings. Each setting corresponds to a git-config key
- such as annex.foo -}
data GitConfig = GitConfig
v7 Install new git hooks in this version. This does beg the question of what to do if git later gets eg a post-smudge hook, that could run git-annex smudge --update. I think the thing to do in that case would be to make git-annex smudge --update install the new hooks. That way, as the user uses git-annex, the hook would be created pretty quickly and without needing any extra syscalls except for when git-annex smudge --update is called. I considered doing something like that for installation of the post-checkout and post-merge hooks, which would have avoided the need for v7. But the only place it was cheap to do it would be in git-annex smudge which could cheaply notice that smudge.log didn't exist yet and so know the hooks needed to be installed. But since smudge used to populate pointer files, it would be quite surprising if a single git checkout/merge failed to update the work tree, and so that idea didn't work out. The other reason for v7 is psychological -- users don't need to worry about whether they might be running an old version of git-annex that doesn't support their v7 repository very well. And bug reports about "v6" have gotten a bit of a bad association in my head since they often hit one of the known limitations and didn't realize it was experimental. newtyped RepoVersion Int to avoid needing 2 comparisons in versionSupportsUnlockedPointers etc. Also it's just nicer. This commit was sponsored by John Pellman on Patreon.
2018-10-25 21:23:53 +00:00
{ annexVersion :: Maybe RepoVersion
optimise getUUID This avoids a Map lookup each time it's called, instead the GitConfig field lazily looks it up once and then caches.
2016-01-20 20:55:06 +00:00
, annexUUID :: UUID
numcopies cleanup, part 2 This includes several bug fixes.
2014-01-21 21:08:49 +00:00
, annexNumCopies :: Maybe NumCopies
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
, annexDiskReserve :: Integer
, annexDirect :: Bool
annex.backend is the new name for what was annex.backends It takes a single key-value backend, rather than the unncessary and confusing list. The old option still works if set. Simplified some old old code too. This commit was sponsored by Thomas Hochstein on Patreon.
2017-05-09 19:04:07 +00:00
, annexBackend :: Maybe String
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
, annexQueueSize :: Maybe Int
, annexBloomCapacity :: Maybe Int
, annexBloomAccuracy :: Maybe Int
, annexSshCaching :: Maybe Bool
, annexAlwaysCommit :: Bool
add annex.alwayscompact Added annex.alwayscompact setting which can be unset to speed up writes to the git-annex branch in some cases. Sponsored-by: Dartmouth College's DANDI project
2022-07-18 19:50:36 +00:00
, annexAlwaysCompact :: Bool
added annex.commitmessage Added annex.commitmessage config that can specify a commit message for the git-annex branch instead of the usual "update". This commit was supported by the NSF-funded DataLad project.
2018-08-02 18:06:06 +00:00
, annexCommitMessage :: Maybe String
added annex.commitmessage-command config Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
2024-02-12 18:34:50 +00:00
, annexCommitMessageCommand :: Maybe String
annex.merge-annex-branches Added annex.merge-annex-branches config setting which can be used to disable automatic merge of git-annex branches. I wonder if git-annex merge/sync/assistant should disable this setting? Not sure yet, so have not done so. May be that users will not set it in git config, but pass it via -c to commands that need it. Checking the config setting adds a very small overhead, but it's only checked once per command so should be insignificant. This commit was supported by the NSF-funded DataLad project.
2018-02-22 18:25:32 +00:00
, annexMergeAnnexBranches :: Bool
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
, annexDelayAdd :: Maybe Int
, annexHttpHeaders :: [String]
, annexHttpHeadersCommand :: Maybe String
rename Configurable for clarity
2020-09-16 15:16:48 +00:00
, annexAutoCommit :: GlobalConfigurable Bool
, annexResolveMerge :: GlobalConfigurable Bool
git-annex pull and push Split out two new commands, git-annex pull and git-annex push. Those plus a git commit are equivilant to git-annex sync. In a sense, git-annex sync conflates 3 things, and it would have been better to have push and pull from the beginning and not sync. Although note that git-annex sync --content is faster than a pull followed by a push, because it only has to walk the tree once, look at preferred content once, etc. So there is some value in git-annex sync in speed, as well as user convenience. And it would be hard to split out pull and push from sync, as far as the implementaton goes. The implementation inside sync was easy, just adjust SyncOptions so it does the right thing. Note that the new commands default to syncing content, unless annex.synccontent is explicitly set to false. I'd like sync to also do that, but that's a hard transition to make. As a start to that transition, I added a note to git-annex-sync.mdwn that it may start to do so in a future version of git-annex. But a real transition would necessarily involve displaying warnings when sync is used without --content, and time. Sponsored-by: Kevin Mueller on Patreon
2023-05-16 20:37:30 +00:00
, annexSyncContent :: GlobalConfigurable (Maybe Bool)
rename Configurable for clarity
2020-09-16 15:16:48 +00:00
, annexSyncOnlyAnnex :: GlobalConfigurable Bool
distributed migration for pull and sync --content pull, sync: When operating on content, automatically hard link objects that have been migrated. Added annex.syncmigrations config that can be set to false to prevent pull and sync from migrating object content. I think that true is a good default for this config, because it avoids users having to re-download migrated content or learning about migration. But, some users will surely not like it, whether because it does take some time (especially for the first git-annex branch scan when there is a long history), or because they want to deal with it manually, or because their filesystem doesn't support hard links and they don't want it to copy objects. Sponsored-by: k0ld on Patreon
2023-12-08 18:18:18 +00:00
, annexSyncMigrations :: Bool
annex.debug can now be set to enable debug logging by default. The webapp's debugging check box does this.
2013-06-18 00:41:17 +00:00
, annexDebug :: Bool
implement fastDebug Most of the changes here involve global option parsing: GlobalSetter changed so it can both run an Annex action to set state, but can also change the AnnexRead value, which is immutable once the Annex monad is running. That allowed a debugselector value to be added to AnnexRead, seeded from the git config. The --debugfilter option's GlobalSetter then updates the AnnexRead. This improved GlobalSetter can later be used to move more stuff to AnnexRead. Things that don't involve a git config will be easier to move, and probably a *lot* of things can be moved eventually. fastDebug, while implemented, is not used anywhere yet. But it should be fast..
2021-04-06 19:14:00 +00:00
, annexDebugFilter :: Maybe String
add another setting to GitConfig
2013-01-27 13:33:19 +00:00
, annexWebOptions :: [String]
youtube-dl working Including resuming and cleanup of incomplete downloads. Still todo: --fast, --relaxed, importfeed, disk reserve checking, quvi code cleanup. This commit was sponsored by Anthony DeRobertis on Patreon.
2017-11-29 19:49:05 +00:00
, annexYoutubeDlOptions :: [String]
Added annex.youtube-dl-command config This can be used to run some forks of youtube-dl. Sponsored-by: Brett Eisenberg on Patreon
2021-08-27 13:44:23 +00:00
, annexYoutubeDlCommand :: Maybe String
Added bittorrent special remote addurl behavior change: When downloading an url ending in .torrent, it will download files from bittorrent, instead of the old behavior of adding the torrent file to the repository. Added Recommends on aria2 and bittornado | bittorrent. This commit was sponsored by Asbjørn Sloth Tønnesen.
2014-12-17 03:22:46 +00:00
, annexAriaTorrentOptions :: [String]
crippled filesystem support, probing and initial support git annex init probes for crippled filesystems, and sets direct mode, as well as `annex.crippledfilesystem`. Avoid manipulating permissions of files on crippled filesystems. That would likely cause an exception to be thrown. Very basic support in Command.Add for cripped filesystems; avoids the lock down entirely since doing it needs both permissions and hard links. Will make this better soon.
2013-02-14 18:10:36 +00:00
, annexCrippledFileSystem :: Bool
rename Configurable for clarity
2020-09-16 15:16:48 +00:00
, annexLargeFiles :: GlobalConfigurable (Maybe String)
, annexDotFiles :: GlobalConfigurable Bool
annex.gitaddtoannex configuration Added annex.gitaddtoannex configuration. Setting it to false prevents git add from usually adding files to the annex. (Unless the file was annexed before, or a renamed annexed file is detected.) Currently left at true; some users are encouraging it be set to false.
2019-10-23 19:20:00 +00:00
, annexGitAddToAnnex :: Bool
annex.addsmallfiles: New option controlling what is done when adding files not matching annex.largefiles.
2016-01-28 18:04:32 +00:00
, annexAddSmallFiles :: Bool
webapp: remind user when using repositories that lack consistency checks When starting up the assistant, it'll remind about the current repository, if it doesn't have checks. And when a removable drive is plugged in, it will remind if a repository on it lacks checks. Since that might be annoying, the reminders can be turned off. This commit was sponsored by Nedialko Andreev.
2013-10-29 20:48:06 +00:00
, annexFsckNudge :: Bool
annex.autoupgrade setting
2013-11-22 20:04:20 +00:00
, annexAutoUpgrade :: AutoUpgrade
allow annex.expireunused to be set to false, as well as to a duration
2014-01-23 03:10:51 +00:00
, annexExpireUnused :: Maybe (Maybe Duration)
Added annex.freezecontent-command and annex.thawcontent-command configs Freeze first sets the file perms, and then runs freezecontent-command. Thaw runs thawcontent-command before restoring file permissions. This is in case the freeze command prevents changing file perms, as eg setting a file immutable does. Also, changing file perms tends to mess up previously set ACLs. git-annex init's probe for crippled filesystem uses them, so if file perms don't work, but freezecontent-command manages to prevent write to a file, it won't treat the filesystem as crippled. When the the filesystem has been probed as crippled, the hooks are not used, because there seems to be no point then; git-annex won't be relying on locking annex objects down. Also, this avoids them being run when the file perms have not been changed, in case they somehow rely on git-annex's setting of the file perms in order to work. Sponsored-by: Dartmouth College's Datalad project
2021-06-21 18:40:20 +00:00
, annexFreezeContentCommand :: Maybe String
, annexThawContentCommand :: Maybe String
added annex.secure-erase-command config option.
2014-01-24 16:58:52 +00:00
, annexSecureEraseCommand :: Maybe String
annex.genmetadata can be set to make git-annex automatically set metadata (year and month) when adding files
2014-02-23 04:08:29 +00:00
, annexGenMetaData :: Bool
annex.listen can be configured, instead of using --listen
2014-03-01 04:31:17 +00:00
, annexListen :: Maybe String
webapp: Added --port option, and annex.port config The getSocket comment that mentioned using ":port" in the hostname seems to have been incorrect or be out of date. After all, the bug report came when the user first tried doing that, and it didn't work. Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
2024-01-25 18:08:36 +00:00
, annexPort :: Maybe PortNumber
annex.startupscan can be set to false to disable the assistant's startup scan.
2014-03-05 21:44:14 +00:00
, annexStartupScan :: Bool
New annex.hardlink setting. Closes: #758593 * New annex.hardlink setting. Closes: #758593 * init: Automatically detect when a repository was cloned with --shared, and set annex.hardlink=true, as well as marking the repository as untrusted. Had to reorganize Logs.Trust a bit to avoid a cycle between it and Annex.Init.
2014-09-05 17:44:09 +00:00
, annexHardLink :: Bool
annex.thin Decided it's too scary to make v6 unlocked files have 1 copy by default, but that should be available to those who need it. This is consistent with git-annex not dropping unused content without --force, etc. * Added annex.thin setting, which makes unlocked files in v6 repositories be hard linked to their content, instead of a copy. This saves disk space but means any modification of an unlocked file will lose the local (and possibly only) copy of the old version. * Enable annex.thin by default on upgrade from direct mode to v6, since direct mode made the same tradeoff. * fix: Adjusts unlocked files as configured by annex.thin.
2015-12-27 19:59:59 +00:00
, annexThin :: Bool
support gpg.program When gpg.program is configured, it's used to get the command to run for gpg. Useful on systems that have only a gpg2 command or want to use it instead of the gpg command.
2015-09-09 22:06:49 +00:00
, annexDifferences :: Differences
, annexUsedRefSpec :: Maybe RefSpec
Do verification of checksums of annex objects downloaded from remotes. * When annex objects are received into git repositories, their checksums are verified then too. * To get the old, faster, behavior of not verifying checksums, set annex.verify=false, or remote.<name>.annex-verify=false. * setkey, rekey: These commands also now verify that the provided file matches the key, unless annex.verify=false. * reinject: Already verified content; this can now be disabled by setting annex.verify=false. recvkey and reinject already did verification, so removed now duplicate code from them. fsck still does its own verification, which is ok since it does not use getViaTmp, so verification doesn't happen twice when using fsck --from.
2015-10-01 19:54:37 +00:00
, annexVerify :: Bool
pid locking configuration and abstraction layer for git-annex (not actually used anywhere yet)
2015-11-12 21:47:31 +00:00
, annexPidLock :: Bool
, annexPidLockTimeout :: Seconds
add annex.dbdir (WIP) WIP: This is mostly complete, but there is a problem: createDirectoryUnder throws an error when annex.dbdir is set to outside the git repo. annex.dbdir is a workaround for filesystems where sqlite does not work, due to eg, the filesystem not properly supporting locking. It's intended to be set before initializing the repository. Changing it in an existing repository can be done, but would be the same as making a new repository and moving all the annexed objects into it. While the databases get recreated from the git-annex branch in that situation, any information that is in the databases but not stored in the branch gets lost. It may be that no information ever gets stored in the databases that cannot be reconstructed from the branch, but I have not verified that. Sponsored-by: Dartmouth College's Datalad project
2022-08-11 20:57:44 +00:00
, annexDbDir :: Maybe RawFilePath
rename Configurable for clarity
2020-09-16 15:16:48 +00:00
, annexAddUnlocked :: GlobalConfigurable (Maybe String)
annex.securehashesonly Cryptographically secure hashes can be forced to be used in a repository, by setting annex.securehashesonly. This does not prevent the git repository from containing files with insecure hashes, but it does prevent the content of such files from being pulled into .git/annex/objects from another repository. We want to make sure that at no point does git-annex accept content into .git/annex/objects that is hashed with an insecure key. Here's how it was done: * .git/annex/objects/xx/yy/KEY/ is kept frozen, so nothing can be written to it normally * So every place that writes content must call, thawContent or modifyContent. We can audit for these, and be sure we've considered all cases. * The main functions are moveAnnex, and linkToAnnex; these were made to check annex.securehashesonly, and are the main security boundary for annex.securehashesonly. * Most other calls to modifyContent deal with other files in the KEY directory (inode cache etc). The other ones that mess with the content are: - Annex.Direct.toDirectGen, in which content already in the annex directory is moved to the direct mode file, so not relevant. - fix and lock, which don't add new content - Command.ReKey.linkKey, which manually unlocks it to make a copy. * All other calls to thawContent appear safe. Made moveAnnex return a Bool, so checked all callsites and made them deal with a failure in appropriate ways. linkToAnnex simply returns LinkAnnexFailed; all callsites already deal with it failing in appropriate ways. This commit was sponsored by Riku Voipio.
2017-02-27 17:01:32 +00:00
, annexSecureHashesOnly :: Bool
add retry configuration (not used yet)
2018-03-24 14:37:25 +00:00
, annexRetry :: Maybe Integer
Exposed annex.forward-retry git config One reason is, 5 is an arbitrary number so ought to be configurable. The real reason though, is I wanted to make the man page explain when forward retry can override annex.retry, and having a config made the man page easier to write.
2020-09-04 19:16:40 +00:00
, annexForwardRetry :: Maybe Integer
add retry configuration (not used yet)
2018-03-24 14:37:25 +00:00
, annexRetryDelay :: Maybe Seconds
limit url downloads to whitelisted schemes Security fix! Allowing any schemes, particularly file: and possibly others like scp: allowed file exfiltration by anyone who had write access to the git repository, since they could add an annexed file using such an url, or using an url that redirected to such an url, and wait for the victim to get it into their repository and send them a copy. * Added annex.security.allowed-url-schemes setting, which defaults to only allowing http and https URLs. Note especially that file:/ is no longer enabled by default. * Removed annex.web-download-command, since its interface does not allow supporting annex.security.allowed-url-schemes across redirects. If you used this setting, you may want to instead use annex.web-options to pass options to curl. With annex.web-download-command removed, nearly all url accesses in git-annex are made via Utility.Url via http-client or curl. http-client only supports http and https, so no problem there. (Disabling one and not the other is not implemented.) Used curl --proto to limit the allowed url schemes. Note that this will cause git annex fsck --from web to mark files using a disallowed url scheme as not being present in the web. That seems acceptable; fsck --from web also does that when a web server is not available. youtube-dl already disabled file: itself (probably for similar reasons). The scheme check was also added to youtube-dl urls for completeness, although that check won't catch any redirects it might follow. But youtube-dl goes off and does its own thing with other protocols anyway, so that's fine. Special remotes that support other domain-specific url schemes are not affected by this change. In the bittorrent remote, aria2c can still download magnet: links. The download of the .torrent file is otherwise now limited by annex.security.allowed-url-schemes. This does not address any external special remotes that might download an url themselves. Current thinking is all external special remotes will need to be audited for this problem, although many of them will use http libraries that only support http and not curl's menagarie. The related problem of accessing private localhost and LAN urls is not addressed by this commit. This commit was sponsored by Brett Eisenberg on Patreon.
2018-06-15 20:52:24 +00:00
, annexAllowedUrlSchemes :: S.Set Scheme
rename annex.security.allowed-http-addresses Renamed annex.security.allowed-http-addresses to annex.security.allowed-ip-addresses because it is not really specific to the http protocol, also limiting eg, git-annex's use of ftp and via youtube-dl, several other protocols. The old name for the config will still work. If both old and new name are set, the new name will win.
2019-05-30 16:43:40 +00:00
, annexAllowedIPAddresses :: String
enforce retrievalSecurityPolicy Leveraged the existing verification code by making it also check the retrievalSecurityPolicy. Also, prevented getViaTmp from running the download action at all when the retrievalSecurityPolicy is going to prevent verifying and so storing it. Added annex.security.allow-unverified-downloads. A per-remote version would be nice to have too, but would need more plumbing, so KISS. (Bill the Cat reference not too over the top I hope. The point is to make this something the user reads the documentation for before using.) A few calls to verifyKeyContent and getViaTmp, that don't involve downloads from remotes, have RetrievalAllKeysSecure hard-coded. It was also hard-coded for P2P.Annex and Command.RecvKey, to match the values of the corresponding remotes. A few things use retrieveKeyFile/retrieveKeyFileCheap without going through getViaTmp. * Command.Fsck when downloading content from a remote to verify it. That content does not get into the annex, so this is ok. * Command.AddUrl when using a remote to download an url; this is new content being added, so this is ok. This commit was sponsored by Fernando Jimenez on Patreon.
2018-06-21 17:34:11 +00:00
, annexAllowUnverifiedDownloads :: Bool
annex.maxextensionlength Added annex.maxextensionlength for use cases where extensions longer than 4 characters are needed. This commit was sponsored by Henrik Riomar on Patreon.
2018-09-24 16:07:46 +00:00
, annexMaxExtensionLength :: Maybe Int
annex.maxextensions configuration Controls how many filename extensions to preserve. Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
2024-04-18 18:23:05 +00:00
, annexMaxExtensions :: Maybe Int
annex.jobs Added annex.jobs setting, which is like using the -J option. Of course, -J overrides annex.jobs. This commit was sponsored by Trenton Cronholm on Patreon.
2018-10-04 16:47:27 +00:00
, annexJobs :: Concurrency
annex.cachecreds: New config to allow disabling of credentials caching for special remotes. Note that it does not prevent storing p2p access tokens or multicast encryption keys, since those are not cached; the previous commit established the distinction. How well this works depends on how often getRemoteCredPair is called and how expensive it is. In some cases setting this will result in an annoying number of gpg password prompts and/or slowdowns due to reading creds from the git-annex branch and decrypting, which could be improved by calling getRemoteCredPair less often. This commit was sponsored by Ilya Shlyakhter on Patreon.
2018-12-04 18:16:56 +00:00
, annexCacheCreds :: Bool
Added annex.autoupgraderepository configuration Can be set to false to prevent any automatic repository upgrades. Also, removed direct mode specific upgrade code in Annex.Init, and made needsUpgrade always include the name/path of the repo, so if there's a problem it's clear what repo has the problem. And, made needsUpgrade catch any exceptions that might occur during the upgrade, so it can display a more useful error message than just the exception.
2019-09-01 17:29:55 +00:00
, annexAutoUpgradeRepository :: Bool
refactor Better to not have a single function module, and better to have a more specific type than Bool. This commit was sponsored by Jack Hill on Patreon
2019-11-11 22:20:35 +00:00
, annexCommitMode :: CommitMode
annex.skipunknown with transition plan Added annex.skipunknown git config, that can be set to false to change the behavior of commands like `git annex get foo*`, to not skip over files/dirs that are not checked into git and are explicitly listed in the command line. Significant complexity was needed to handle git-annex add, which uses some git ls-files calls, but needs to not use --error-unmatch because of course the files are not known to git. annex.skipunknown is planned to change to default to false in a git-annex release in early 2022. There's a todo for that.
2020-05-28 19:55:17 +00:00
, annexSkipUnknown :: Bool
annex.adjustedbranchrefresh Added annex.adjustedbranchrefresh git config to update adjusted branches set up by git-annex adjust --unlock-present/--hide-missing. Note, in a few cases, I was not able to make the adjusted branch be updated in calls to moveAnnex, because information about what file corresponds to a key is not available. They are: * If two files point to one file, then eg, `git annex get foo` will update the branch to unlock foo, but will not unlock bar, because it does not know about it. Might be fixable by making `git annex get bar` do something besides skipping bar? * git-annex-shell recvkey likewise (so sends over ssh from old versions of git-annex) * git-annex setkey * git-annex transferkey if the user does not use --file * git-annex multicast sends keys with no associated file info Doing a single full refresh at the end, after any incremental refresh, will deal with those edge cases.
2020-11-16 18:09:55 +00:00
, annexAdjustedBranchRefresh :: Integer
New annex.supportunlocked config Can beet to false to avoid some expensive things needed to support unlocked files. See my comment for why this only controls what init sets up, and not other behavior. I didn't bother with making the v5 upgrade code path look at this, though it easily could, because the docs say to run git-annex init after setting it to make it take effect.
2021-03-23 18:04:34 +00:00
, annexSupportUnlocked :: Bool
start to support core.symlinks=false Utility functions to handle no symlink mode, and converted Annex.Content to use them; still many other places to convert.
2013-02-15 20:02:35 +00:00
, coreSymlinks :: Bool
better memoize core.sharedrepository handling It was memoized, but that was not used consistently. Move it to Types.GitConfig so it will auto-memoize.
2015-05-19 19:04:24 +00:00
, coreSharedRepository :: SharedRepository
git style filename quoting controlled by core.quotePath This is by no means complete, but escaping filenames in actionItemDesc does cover most commands. Note that for ActionItemBranchFilePath, the value is branch:file, and I choose to only quote the file part (if necessary). I considered quoting the whole thing. But, branch names cannot contain control characters, and while they can contain unicode, git coes not quote unicode when displaying branch names. So, it would be surprising for git-annex to quote unicode in a branch name. The find command is the most obvious command that still needs to be dealt with. There are probably other places that filenames also get displayed, eg embedded in error messages. Some other commands use ActionItemOther with a filename, I think that ActionItemOther should either be pre-sanitized, or should explicitly not be used for filenames, so that needs more work. When --json is used, unicode does not get escaped, but control characters were already escaped in json. (Key escaping may turn out to be needed, but I'm ignoring that for now.) Sponsored-by: unqueued on Patreon
2023-04-08 18:20:02 +00:00
, coreQuotePath :: QuotePath
post-recive hook to make updateInstead work in direct mode and adjusted branches * Added post-recieve hook, which makes updateInstead work with direct mode and adjusted branches. * init: Set up the post-receive hook. This commit was sponsored by Fernando Jimenez on Patreon.
2017-02-17 18:04:43 +00:00
, receiveDenyCurrentBranch :: DenyCurrentBranch
git-annex-shell: Added support for operating inside gcrypt repositories. * Note that the layout of gcrypt repositories has changed, and if you created one you must manually upgrade it. See http://git-annex.branchable.com/upgrades/gcrypt/
2013-09-24 21:25:47 +00:00
, gcryptId :: Maybe String
support gpg.program When gpg.program is configured, it's used to get the command to run for gpg. Useful on systems that have only a gpg2 command or want to use it instead of the gpg command.
2015-09-09 22:06:49 +00:00
, gpgCmd :: GpgCmd
sync, assistant: When merge.directoryRenames is not set, default it it to "false" Works better with automatic merge conflict resolution than git's ususual default of "conflict". This is not done when automatic merge conflict resolution is disabled. This commit was sponsored by Mark Reidenbach on Patreon.
2020-09-07 17:50:58 +00:00
, mergeDirectoryRenames :: Maybe String
implement annex.privateremote and remote.name.private configs The slightly unusual parsing in Types.GitConfig avoids the need to look at the remote list to get configs of remotes. annexPrivateRepos combines all the configs, and will only be calculated once, so it's nice and fast. privateUUIDsKnown and regardingPrivateUUID now need to read from the annex mvar, so are not entirely free. But that overhead can be optimised away, as seen in getJournalFileStale. The other call sites didn't seem worth optimising to save a single MVar access. The feature should have impreceptable speed overhead when not being used.
2021-04-23 18:21:57 +00:00
, annexPrivateRepos :: S.Set UUID
Added annex.adviceNoSshCaching config. Sponsored-by: Brock Spratlen on Patreon
2021-05-27 16:37:39 +00:00
, annexAdviceNoSshCaching :: Bool
add directory to views for files that lack specified metadata * view: New field?=glob and ?tag syntax that includes a directory "_" in the view for files that do not have the specified metadata set. * Added annex.viewunsetdirectory git config to change the name of the "_" directory in a view. When in a view using the new syntax, old git-annex will fail to parse the view log. It errors with "Not in a view.", which is not ideal. But that only affects view commands. annex.viewunsetdirectory is included in the View for a couple of reasons. One is to avoid needing to warn the user that it should not be changed when in a view, since that would confuse git-annex. Another reason is that it helped with plumbing the value through to some pure functions. annex.viewunsetdirectory is actually mangled the same as any other view directory. So if it's configured to something like "N/A", there won't be multiple levels of directories, which would also confuse git-annex. Sponsored-By: Jack Hill on Patreon
2023-02-07 20:28:46 +00:00
, annexViewUnsetDirectory :: ViewUnset
add git-annex updatecluster command Seems to work fine, making the right changes to the git-annex branch.
2024-06-14 18:21:50 +00:00
, annexClusters :: M.Map RemoteName ClusterUUID
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
}
git-annex config annex.largefiles annex.largefiles can be configured by git-annex config, to more easily set a default that will also be used by clones, without needing to shoehorn the expression into the gitattributes file. The git config and gitattributes override that. Whenever something is added to git-annex config, we have to consider what happens if a user puts a purposfully bad value in there. Or, if a new git-annex adds some new value that an old git-annex can't parse. In this case, a global annex.largefiles that can't be parsed currently makes an error be thrown. That might not be ideal, but the gitattribute behaves the same, and is almost equally repo-global. Performance notes: git-annex add and addurl construct a matcher once and uses it for every file, so the added time penalty for reading the global config log is minor. If the gitattributes annex.largefiles were deprecated, git-annex add would get around 2% faster (excluding hashing), because looking that up for each file is not fast. So this new way of setting it is progress toward speeding up add. git-annex smudge does need to load the log every time. As well as checking the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids both overheads.
2019-12-20 16:12:31 +00:00
extractGitConfig :: ConfigSource -> Git.Repo -> GitConfig
extractGitConfig configsource r = GitConfig
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
{ annexVersion = RepoVersion <$> getmayberead (annexConfig "version")
implement annex.privateremote and remote.name.private configs The slightly unusual parsing in Types.GitConfig avoids the need to look at the remote list to get configs of remotes. annexPrivateRepos combines all the configs, and will only be calculated once, so it's nice and fast. privateUUIDsKnown and regardingPrivateUUID now need to read from the annex mvar, so are not entirely free. But that overhead can be optimised away, as seen in getJournalFileStale. The other call sites didn't seem worth optimising to save a single MVar access. The feature should have impreceptable speed overhead when not being used.
2021-04-23 18:21:57 +00:00
, annexUUID = hereuuid
prevent numcopies or mincopies being configured to 0 Ignore annex.numcopies set to 0 in gitattributes or git config, or by git-annex numcopies or by --numcopies, since that configuration would make git-annex easily lose data. Same for mincopies. This is a continuation of the work to make data only be able to be lost when --force is used. It earlier led to the --trust option being disabled, and similar reasoning applies here. Most numcopies configs had docs that strongly discouraged setting it to 0 anyway. And I can't imagine a use case for setting to 0. Not that there might not be one, but it's just so far from the intended use case of git-annex, of managing and storing your data, that it does not seem like it makes sense to cater to such a hypothetical use case, where any git-annex drop can lose your data at any time. Using a smart constructor makes sure every place avoids 0. Note that this does mean that NumCopies is for the configured desired values, and not the actual existing number of copies, which of course can be 0. The name configuredNumCopies is used to make that clear. Sponsored-by: Brock Spratlen on Patreon
2022-03-28 19:19:52 +00:00
, annexNumCopies = configuredNumCopies
<$> getmayberead (annexConfig "numcopies")
annex.diskreserve default increased from 1 mb to 100 mb It's hard to know what's a good default for this. But 1 mb seems way too small, because it's very easy for a git pull or some similar operation that we don't think of as using much space to use up 1 mb of space. Most people would want to free up some space if a filesystem only had 100 mb free. But on a small VPS, it's probably not uncommon to have only 1 gb free. So 1 gb is too large for annex.diskreserve. While old 1 gb USB keys are around, it's unlikely that anyone is relying on them to shuttle annex data around; it would be worth anyone's time to upgrade to a 32 gb or larger cheap modern USB key ($5). Sponsored-by: Kevin Mueller on Patreon
2022-09-21 19:00:13 +00:00
, annexDiskReserve = fromMaybe (onemegabyte * 100) $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
readSize dataUnits =<< getmaybe (annexConfig "diskreserve")
, annexDirect = getbool (annexConfig "direct") False
annex.backend is the new name for what was annex.backends It takes a single key-value backend, rather than the unncessary and confusing list. The old option still works if set. Simplified some old old code too. This commit was sponsored by Thomas Hochstein on Patreon.
2017-05-09 19:04:07 +00:00
, annexBackend = maybe
-- annex.backends is the old name of the option, still used
-- when annex.backend is not set.
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
(headMaybe $ getwords (annexConfig "backends"))
annex.backend is the new name for what was annex.backends It takes a single key-value backend, rather than the unncessary and confusing list. The old option still works if set. Simplified some old old code too. This commit was sponsored by Thomas Hochstein on Patreon.
2017-05-09 19:04:07 +00:00
Just
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
(getmaybe (annexConfig "backend"))
, annexQueueSize = getmayberead (annexConfig "queuesize")
, annexBloomCapacity = getmayberead (annexConfig "bloomcapacity")
, annexBloomAccuracy = getmayberead (annexConfig "bloomaccuracy")
, annexSshCaching = getmaybebool (annexConfig "sshcaching")
, annexAlwaysCommit = getbool (annexConfig "alwayscommit") True
add annex.alwayscompact Added annex.alwayscompact setting which can be unset to speed up writes to the git-annex branch in some cases. Sponsored-by: Dartmouth College's DANDI project
2022-07-18 19:50:36 +00:00
, annexAlwaysCompact = getbool (annexConfig "alwayscompact") True
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
, annexCommitMessage = getmaybe (annexConfig "commitmessage")
added annex.commitmessage-command config Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
2024-02-12 18:34:50 +00:00
, annexCommitMessageCommand = getmaybe (annexConfig "commitmessage-command")
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
, annexMergeAnnexBranches = getbool (annexConfig "merge-annex-branches") True
, annexDelayAdd = getmayberead (annexConfig "delayadd")
, annexHttpHeaders = getlist (annexConfig "http-headers")
, annexHttpHeadersCommand = getmaybe (annexConfig "http-headers-command")
annex.autocommit can be configured via git-annex config ... to control the default behavior in all clones of a repository. This includes a new Configurable data type, so the GitConfig type indicates which values can be configured this way. The implementation should be quite efficient; the config log is only read once, and only when a Configurable value has not already been set by git-config. Indeed, it would be nice in the future to extend this, so that git-config is itself only read on demand. Some commands may not need to look at the git configuration at all. This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 17:40:14 +00:00
, annexAutoCommit = configurable True $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
getmaybebool (annexConfig "autocommit")
configuration to disable automatic merge conflict resolution * Added annex.resolvemerge configuration, which can be set to false to disable the usual automatic merge conflict resolution done by git-annex sync and the assistant. * sync: Added --no-resolvemerge option. Note that disabling merge conflict resolution is probably not a good idea in a direct mode repo or adjusted branch. Since updates to both are done outside the usual work tree, if it fails the tree is not left in a conflicted state, and it would be hard to manually resolve the conflict. Still, made annex.resolvemerge be supported in those cases for consistency. This commit was sponsored by Riku Voipio.
2017-06-01 16:46:36 +00:00
, annexResolveMerge = configurable True $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
getmaybebool (annexConfig "resolvemerge")
git-annex pull and push Split out two new commands, git-annex pull and git-annex push. Those plus a git commit are equivilant to git-annex sync. In a sense, git-annex sync conflates 3 things, and it would have been better to have push and pull from the beginning and not sync. Although note that git-annex sync --content is faster than a pull followed by a push, because it only has to walk the tree once, look at preferred content once, etc. So there is some value in git-annex sync in speed, as well as user convenience. And it would be hard to split out pull and push from sync, as far as the implementaton goes. The implementation inside sync was easy, just adjust SyncOptions so it does the right thing. Note that the new commands default to syncing content, unless annex.synccontent is explicitly set to false. I'd like sync to also do that, but that's a hard transition to make. As a start to that transition, I added a note to git-annex-sync.mdwn that it may start to do so in a future version of git-annex. But a real transition would necessarily involve displaying warnings when sync is used without --content, and time. Sponsored-by: Kevin Mueller on Patreon
2023-05-16 20:37:30 +00:00
, annexSyncContent = configurablemaybe $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
getmaybebool (annexConfig "synccontent")
sync --only-annex and annex.synconlyannex * Added sync --only-annex, which syncs the git-annex branch and annexed content but leaves managing the other git branches up to you. * Added annex.synconlyannex git config setting, which can also be set with git-annex config to configure sync in all clones of the repo. Use case is then the user has their own git workflow, and wants to use git-annex without disrupting that, so they sync --only-annex to get the git-annex stuff in sync in addition to their usual git workflow. When annex.synconlyannex is set, --not-only-annex can be used to override it. It's not entirely clear what --only-annex --commit or --only-annex --push should do, and I left that combination not documented because I don't know if I might want to change the current behavior, which is that such options do not override the --only-annex. My gut feeling is that there is no good reasons to use such combinations; if you want to use your own git workflow, you'll be doing your own committing and pulling and pushing. A subtle question is, how should import/export special remotes be handled? Importing updates their remote tracking branch and merges it into master. If --only-annex prevented that git branch stuff, then it would prevent exporting to the special remote, in the case where it has changes that were not imported yet, because there would be a unresolved conflict. I decided that it's best to treat the fact that there's a remote tracking branch for import/export as an implementation detail in this case. The more important thing is that an import/export special remote is entirely annexed content, and so it makes a lot of sense that --only-annex will still sync with it.
2020-02-17 19:19:58 +00:00
, annexSyncOnlyAnnex = configurable False $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
getmaybebool (annexConfig "synconlyannex")
distributed migration for pull and sync --content pull, sync: When operating on content, automatically hard link objects that have been migrated. Added annex.syncmigrations config that can be set to false to prevent pull and sync from migrating object content. I think that true is a good default for this config, because it avoids users having to re-download migrated content or learning about migration. But, some users will surely not like it, whether because it does take some time (especially for the first git-annex branch scan when there is a long history), or because they want to deal with it manually, or because their filesystem doesn't support hard links and they don't want it to copy objects. Sponsored-by: k0ld on Patreon
2023-12-08 18:18:18 +00:00
, annexSyncMigrations = getbool (annexConfig "syncmigrations") True
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
, annexDebug = getbool (annexConfig "debug") False
implement fastDebug Most of the changes here involve global option parsing: GlobalSetter changed so it can both run an Annex action to set state, but can also change the AnnexRead value, which is immutable once the Annex monad is running. That allowed a debugselector value to be added to AnnexRead, seeded from the git config. The --debugfilter option's GlobalSetter then updates the AnnexRead. This improved GlobalSetter can later be used to move more stuff to AnnexRead. Things that don't involve a git config will be easier to move, and probably a *lot* of things can be moved eventually. fastDebug, while implemented, is not used anywhere yet. But it should be fast..
2021-04-06 19:14:00 +00:00
, annexDebugFilter = getmaybe (annexConfig "debugfilter")
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
, annexWebOptions = getwords (annexConfig "web-options")
, annexYoutubeDlOptions = getwords (annexConfig "youtube-dl-options")
Added annex.youtube-dl-command config This can be used to run some forks of youtube-dl. Sponsored-by: Brett Eisenberg on Patreon
2021-08-27 13:44:23 +00:00
, annexYoutubeDlCommand = getmaybe (annexConfig "youtube-dl-command")
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
, annexAriaTorrentOptions = getwords (annexConfig "aria-torrent-options")
, annexCrippledFileSystem = getbool (annexConfig "crippledfilesystem") False
git-annex config annex.largefiles annex.largefiles can be configured by git-annex config, to more easily set a default that will also be used by clones, without needing to shoehorn the expression into the gitattributes file. The git config and gitattributes override that. Whenever something is added to git-annex config, we have to consider what happens if a user puts a purposfully bad value in there. Or, if a new git-annex adds some new value that an old git-annex can't parse. In this case, a global annex.largefiles that can't be parsed currently makes an error be thrown. That might not be ideal, but the gitattribute behaves the same, and is almost equally repo-global. Performance notes: git-annex add and addurl construct a matcher once and uses it for every file, so the added time penalty for reading the global config log is minor. If the gitattributes annex.largefiles were deprecated, git-annex add would get around 2% faster (excluding hashing), because looking that up for each file is not fast. So this new way of setting it is progress toward speeding up add. git-annex smudge does need to load the log every time. As well as checking the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids both overheads.
2019-12-20 16:12:31 +00:00
, annexLargeFiles = configurable Nothing $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
fmap Just $ getmaybe (annexConfig "largefiles")
, annexDotFiles = configurable False $
getmaybebool (annexConfig "dotfiles")
, annexGitAddToAnnex = getbool (annexConfig "gitaddtoannex") True
, annexAddSmallFiles = getbool (annexConfig "addsmallfiles") True
, annexFsckNudge = getbool (annexConfig "fscknudge") True
, annexAutoUpgrade = toAutoUpgrade $
getmaybe (annexConfig "autoupgrade")
avoid using MonadFail in ParseDuration There's no instance for Either String, so that makes it not as useful as it could be, so instead just return an Either String.
2020-08-15 19:53:35 +00:00
, annexExpireUnused = either (const Nothing) Just . parseDuration
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
<$> getmaybe (annexConfig "expireunused")
Added annex.freezecontent-command and annex.thawcontent-command configs Freeze first sets the file perms, and then runs freezecontent-command. Thaw runs thawcontent-command before restoring file permissions. This is in case the freeze command prevents changing file perms, as eg setting a file immutable does. Also, changing file perms tends to mess up previously set ACLs. git-annex init's probe for crippled filesystem uses them, so if file perms don't work, but freezecontent-command manages to prevent write to a file, it won't treat the filesystem as crippled. When the the filesystem has been probed as crippled, the hooks are not used, because there seems to be no point then; git-annex won't be relying on locking annex objects down. Also, this avoids them being run when the file perms have not been changed, in case they somehow rely on git-annex's setting of the file perms in order to work. Sponsored-by: Dartmouth College's Datalad project
2021-06-21 18:40:20 +00:00
, annexFreezeContentCommand = getmaybe (annexConfig "freezecontent-command")
, annexThawContentCommand = getmaybe (annexConfig "thawcontent-command")
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
, annexSecureEraseCommand = getmaybe (annexConfig "secure-erase-command")
, annexGenMetaData = getbool (annexConfig "genmetadata") False
, annexListen = getmaybe (annexConfig "listen")
webapp: Added --port option, and annex.port config The getSocket comment that mentioned using ":port" in the hostname seems to have been incorrect or be out of date. After all, the bug report came when the user first tried doing that, and it didn't work. Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
2024-01-25 18:08:36 +00:00
, annexPort = getmayberead (annexConfig "port")
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
, annexStartupScan = getbool (annexConfig "startupscan") True
, annexHardLink = getbool (annexConfig "hardlink") False
, annexThin = getbool (annexConfig "thin") False
Repository tuning parameters can now be passed when initializing a repository for the first time. * init: Repository tuning parameters can now be passed when initializing a repository for the first time. For details, see http://git-annex.branchable.com/tuning/ * merge: Refuse to merge changes from a git-annex branch of a repo that has been tuned in incompatable ways.
2015-01-27 21:38:06 +00:00
, annexDifferences = getDifferences r
add annex.used-refspec
2015-05-14 19:44:08 +00:00
, annexUsedRefSpec = either (const Nothing) Just . parseRefSpec
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
=<< getmaybe (annexConfig "used-refspec")
, annexVerify = getbool (annexConfig "verify") True
, annexPidLock = getbool (annexConfig "pidlock") False
pid locking configuration and abstraction layer for git-annex (not actually used anywhere yet)
2015-11-12 21:47:31 +00:00
, annexPidLockTimeout = Seconds $ fromMaybe 300 $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
getmayberead (annexConfig "pidlocktimeout")
use a subdirectory of annex.dbdir This allows annex.dbdir to be set globally or always set to the same value when needed. Each repository uses a subdirectory of it. Sponsored-by: Dartmouth College's Datalad project
2022-08-12 17:18:15 +00:00
, annexDbDir = (\d -> toRawFilePath d P.</> fromUUID hereuuid)
<$> getmaybe (annexConfig "dbdir")
annex.addunlocked expressions * annex.addunlocked can be set to an expression with the same format used by annex.largefiles, in case you want to default to unlocking some files but not others. * annex.addunlocked can be configured by git-annex config. Added a git-annex-matching-expression man page, broken out from tips/largefiles. A tricky consequence of this is that git-annex add --relaxed honors annex.addunlocked, but an expression might want to know the size or content of an url, which it's not going to download. I decided it was better not to fail, and just dummy up some plausible data in that case. Performance impact should be negligible. The global config is already loaded for annex.largefiles. The expression only has to be parsed once, and in the simple true/false case, it should not do any additional work matching it.
2019-12-20 19:01:34 +00:00
, annexAddUnlocked = configurable Nothing $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
fmap Just $ getmaybe (annexConfig "addunlocked")
, annexSecureHashesOnly = getbool (annexConfig "securehashesonly") False
, annexRetry = getmayberead (annexConfig "retry")
Exposed annex.forward-retry git config One reason is, 5 is an arbitrary number so ought to be configurable. The real reason though, is I wanted to make the man page explain when forward retry can override annex.retry, and having a config made the man page easier to write.
2020-09-04 19:16:40 +00:00
, annexForwardRetry = getmayberead (annexConfig "forward-retry")
add retry configuration (not used yet)
2018-03-24 14:37:25 +00:00
, annexRetryDelay = Seconds
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
<$> getmayberead (annexConfig "retrydelay")
limit url downloads to whitelisted schemes Security fix! Allowing any schemes, particularly file: and possibly others like scp: allowed file exfiltration by anyone who had write access to the git repository, since they could add an annexed file using such an url, or using an url that redirected to such an url, and wait for the victim to get it into their repository and send them a copy. * Added annex.security.allowed-url-schemes setting, which defaults to only allowing http and https URLs. Note especially that file:/ is no longer enabled by default. * Removed annex.web-download-command, since its interface does not allow supporting annex.security.allowed-url-schemes across redirects. If you used this setting, you may want to instead use annex.web-options to pass options to curl. With annex.web-download-command removed, nearly all url accesses in git-annex are made via Utility.Url via http-client or curl. http-client only supports http and https, so no problem there. (Disabling one and not the other is not implemented.) Used curl --proto to limit the allowed url schemes. Note that this will cause git annex fsck --from web to mark files using a disallowed url scheme as not being present in the web. That seems acceptable; fsck --from web also does that when a web server is not available. youtube-dl already disabled file: itself (probably for similar reasons). The scheme check was also added to youtube-dl urls for completeness, although that check won't catch any redirects it might follow. But youtube-dl goes off and does its own thing with other protocols anyway, so that's fine. Special remotes that support other domain-specific url schemes are not affected by this change. In the bittorrent remote, aria2c can still download magnet: links. The download of the .torrent file is otherwise now limited by annex.security.allowed-url-schemes. This does not address any external special remotes that might download an url themselves. Current thinking is all external special remotes will need to be audited for this problem, although many of them will use http libraries that only support http and not curl's menagarie. The related problem of accessing private localhost and LAN urls is not addressed by this commit. This commit was sponsored by Brett Eisenberg on Patreon.
2018-06-15 20:52:24 +00:00
, annexAllowedUrlSchemes = S.fromList $ map mkScheme $
allow ftp urls by default They're no worse than http certianly. And, the backport of these security fixes has to deal with wget, which supports http https and ftp and has no way to turn off individual schemes, so this will make that easier.
2018-06-18 19:36:12 +00:00
maybe ["http", "https", "ftp"] words $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
getmaybe (annexConfig "security.allowed-url-schemes")
rename annex.security.allowed-http-addresses Renamed annex.security.allowed-http-addresses to annex.security.allowed-ip-addresses because it is not really specific to the http protocol, also limiting eg, git-annex's use of ftp and via youtube-dl, several other protocols. The old name for the config will still work. If both old and new name are set, the new name will win.
2019-05-30 16:43:40 +00:00
, annexAllowedIPAddresses = fromMaybe "" $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
getmaybe (annexConfig "security.allowed-ip-addresses")
rename annex.security.allowed-http-addresses Renamed annex.security.allowed-http-addresses to annex.security.allowed-ip-addresses because it is not really specific to the http protocol, also limiting eg, git-annex's use of ftp and via youtube-dl, several other protocols. The old name for the config will still work. If both old and new name are set, the new name will win.
2019-05-30 16:43:40 +00:00
<|>
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
getmaybe (annexConfig "security.allowed-http-addresses") -- old name
enforce retrievalSecurityPolicy Leveraged the existing verification code by making it also check the retrievalSecurityPolicy. Also, prevented getViaTmp from running the download action at all when the retrievalSecurityPolicy is going to prevent verifying and so storing it. Added annex.security.allow-unverified-downloads. A per-remote version would be nice to have too, but would need more plumbing, so KISS. (Bill the Cat reference not too over the top I hope. The point is to make this something the user reads the documentation for before using.) A few calls to verifyKeyContent and getViaTmp, that don't involve downloads from remotes, have RetrievalAllKeysSecure hard-coded. It was also hard-coded for P2P.Annex and Command.RecvKey, to match the values of the corresponding remotes. A few things use retrieveKeyFile/retrieveKeyFileCheap without going through getViaTmp. * Command.Fsck when downloading content from a remote to verify it. That content does not get into the annex, so this is ok. * Command.AddUrl when using a remote to download an url; this is new content being added, so this is ok. This commit was sponsored by Fernando Jimenez on Patreon.
2018-06-21 17:34:11 +00:00
, annexAllowUnverifiedDownloads = (== Just "ACKTHPPT") $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
getmaybe (annexConfig "security.allow-unverified-downloads")
, annexMaxExtensionLength = getmayberead (annexConfig "maxextensionlength")
annex.maxextensions configuration Controls how many filename extensions to preserve. Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
2024-04-18 18:23:05 +00:00
, annexMaxExtensions = getmayberead (annexConfig "maxextensions")
annex.jobs=cpus etc Added the ability to run one job per CPU (core), by setting annex.jobs=cpus, or using option --jobs=cpus or -Jcpus. Built with future expansion in mind, including not defaulting matching on Concurrency so more constructors can later be added, and using "cpu" instead of "0".
2019-05-10 17:24:31 +00:00
, annexJobs = fromMaybe NonConcurrent $
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
parseConcurrency =<< getmaybe (annexConfig "jobs")
, annexCacheCreds = getbool (annexConfig "cachecreds") True
, annexAutoUpgradeRepository = getbool (annexConfig "autoupgraderepository") True
, annexCommitMode = if getbool (annexConfig "allowsign") False
refactor Better to not have a single function module, and better to have a more specific type than Bool. This commit was sponsored by Jack Hill on Patreon
2019-11-11 22:20:35 +00:00
then ManualCommit
else AutomaticCommit
complete annex.skipunknown transition annex.skipunknown now defaults to false, so commands like `git annex get foo*` will not silently skip over files/dirs that are not checked into git. Sponsored-by: Brock Spratlen on Patreon
2022-02-18 17:18:05 +00:00
, annexSkipUnknown = getbool (annexConfig "skipunknown") False
annex.adjustedbranchrefresh Added annex.adjustedbranchrefresh git config to update adjusted branches set up by git-annex adjust --unlock-present/--hide-missing. Note, in a few cases, I was not able to make the adjusted branch be updated in calls to moveAnnex, because information about what file corresponds to a key is not available. They are: * If two files point to one file, then eg, `git annex get foo` will update the branch to unlock foo, but will not unlock bar, because it does not know about it. Might be fixable by making `git annex get bar` do something besides skipping bar? * git-annex-shell recvkey likewise (so sends over ssh from old versions of git-annex) * git-annex setkey * git-annex transferkey if the user does not use --file * git-annex multicast sends keys with no associated file info Doing a single full refresh at the end, after any incremental refresh, will deal with those edge cases.
2020-11-16 18:09:55 +00:00
, annexAdjustedBranchRefresh = fromMaybe
-- parse as bool if it's not a number
(if getbool "adjustedbranchrefresh" False then 1 else 0)
(getmayberead (annexConfig "adjustedbranchrefresh"))
New annex.supportunlocked config Can beet to false to avoid some expensive things needed to support unlocked files. See my comment for why this only controls what init sets up, and not other behavior. I didn't bother with making the v5 upgrade code path look at this, though it easily could, because the docs say to run git-annex init after setting it to make it take effect.
2021-03-23 18:04:34 +00:00
, annexSupportUnlocked = getbool (annexConfig "supportunlocked") True
support gpg.program When gpg.program is configured, it's used to get the command to run for gpg. Useful on systems that have only a gpg2 command or want to use it instead of the gpg command.
2015-09-09 22:06:49 +00:00
, coreSymlinks = getbool "core.symlinks" True
, coreSharedRepository = getSharedRepository r
git style filename quoting controlled by core.quotePath This is by no means complete, but escaping filenames in actionItemDesc does cover most commands. Note that for ActionItemBranchFilePath, the value is branch:file, and I choose to only quote the file part (if necessary). I considered quoting the whole thing. But, branch names cannot contain control characters, and while they can contain unicode, git coes not quote unicode when displaying branch names. So, it would be surprising for git-annex to quote unicode in a branch name. The find command is the most obvious command that still needs to be dealt with. There are probably other places that filenames also get displayed, eg embedded in error messages. Some other commands use ActionItemOther with a filename, I think that ActionItemOther should either be pre-sanitized, or should explicitly not be used for filenames, so that needs more work. When --json is used, unicode does not get escaped, but control characters were already escaped in json. (Key escaping may turn out to be needed, but I'm ignoring that for now.) Sponsored-by: unqueued on Patreon
2023-04-08 18:20:02 +00:00
, coreQuotePath = QuotePath (getbool "core.quotepath" True)
post-recive hook to make updateInstead work in direct mode and adjusted branches * Added post-recieve hook, which makes updateInstead work with direct mode and adjusted branches. * init: Set up the post-receive hook. This commit was sponsored by Fernando Jimenez on Patreon.
2017-02-17 18:04:43 +00:00
, receiveDenyCurrentBranch = getDenyCurrentBranch r
support gpg.program When gpg.program is configured, it's used to get the command to run for gpg. Useful on systems that have only a gpg2 command or want to use it instead of the gpg command.
2015-09-09 22:06:49 +00:00
, gcryptId = getmaybe "core.gcrypt-id"
, gpgCmd = mkGpgCmd (getmaybe "gpg.program")
sync, assistant: When merge.directoryRenames is not set, default it it to "false" Works better with automatic merge conflict resolution than git's ususual default of "conflict". This is not done when automatic merge conflict resolution is disabled. This commit was sponsored by Mark Reidenbach on Patreon.
2020-09-07 17:50:58 +00:00
, mergeDirectoryRenames = getmaybe "directoryrenames"
implement annex.privateremote and remote.name.private configs The slightly unusual parsing in Types.GitConfig avoids the need to look at the remote list to get configs of remotes. annexPrivateRepos combines all the configs, and will only be calculated once, so it's nice and fast. privateUUIDsKnown and regardingPrivateUUID now need to read from the annex mvar, so are not entirely free. But that overhead can be optimised away, as seen in getJournalFileStale. The other call sites didn't seem worth optimising to save a single MVar access. The feature should have impreceptable speed overhead when not being used.
2021-04-23 18:21:57 +00:00
, annexPrivateRepos = S.fromList $ concat
[ if getbool (annexConfig "private") False
then [hereuuid]
else []
, let get (k, v)
| Git.Config.isTrueFalse' v /= Just True = Nothing
| isRemoteKey (remoteAnnexConfigEnd "private") k = do
remotename <- remoteKeyToRemoteName k
toUUID <$> Git.Config.getMaybe
(remoteAnnexConfig remotename "uuid") r
| otherwise = Nothing
in mapMaybe get (M.toList (Git.config r))
]
Fix annex.adviceNoSshCaching having no effect git will always return option names in lowercase
2022-09-30 14:56:17 +00:00
, annexAdviceNoSshCaching = getbool (annexConfig "advicenosshcaching") True
add directory to views for files that lack specified metadata * view: New field?=glob and ?tag syntax that includes a directory "_" in the view for files that do not have the specified metadata set. * Added annex.viewunsetdirectory git config to change the name of the "_" directory in a view. When in a view using the new syntax, old git-annex will fail to parse the view log. It errors with "Not in a view.", which is not ideal. But that only affects view commands. annex.viewunsetdirectory is included in the View for a couple of reasons. One is to avoid needing to warn the user that it should not be changed when in a view, since that would confuse git-annex. Another reason is that it helped with plumbing the value through to some pure functions. annex.viewunsetdirectory is actually mangled the same as any other view directory. So if it's configured to something like "N/A", there won't be multiple levels of directories, which would also confuse git-annex. Sponsored-By: Jack Hill on Patreon
2023-02-07 20:28:46 +00:00
, annexViewUnsetDirectory = ViewUnset $ fromMaybe "_" $
getmaybe (annexConfig "viewunsetdirectory")
add git configs for clusters
2024-06-14 15:42:32 +00:00
, annexClusters =
M.mapMaybe (mkClusterUUID . toUUID) $
add git-annex updatecluster command Seems to work fine, making the right changes to the git-annex branch.
2024-06-14 18:21:50 +00:00
M.mapKeys removeclusterprefix $
M.filterWithKey isclusternamekey (config r)
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
}
where
import Data.Default in Common
2015-01-28 20:11:28 +00:00
getbool k d = fromMaybe d $ getmaybebool k
annex.addunlocked expressions * annex.addunlocked can be set to an expression with the same format used by annex.largefiles, in case you want to default to unlocking some files but not others. * annex.addunlocked can be configured by git-annex config. Added a git-annex-matching-expression man page, broken out from tips/largefiles. A tricky consequence of this is that git-annex add --relaxed honors annex.addunlocked, but an expression might want to know the size or content of an url, which it's not going to download. I decided it was better not to fail, and just dummy up some plausible data in that case. Performance impact should be negligible. The global config is already loaded for annex.largefiles. The expression only has to be parsed once, and in the simple true/false case, it should not do any additional work matching it.
2019-12-20 19:01:34 +00:00
getmaybebool k = Git.Config.isTrueFalse' =<< getmaybe' k
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
getmayberead k = readish =<< getmaybe k
all commands building except for assistant also, changed ConfigValue to a newtype, and moved it into Git.Config.
2019-12-05 18:36:43 +00:00
getmaybe = fmap fromConfigValue . getmaybe'
use ByteString for git config The parser and looking up config keys in the map should both be faster due to using ByteString. I had hoped this would speed up startup time, but any improvement to that was too small to measure. Seems worth keeping though. Note that the parser breaks up the ByteString, but a config map ends up pointing to the config as read, which is retained in memory until every value from it is no longer used. This can change memory usage patterns marginally, but won't affect git-annex.
2019-11-27 20:54:11 +00:00
getmaybe' k = Git.Config.getMaybe k r
all commands building except for assistant also, changed ConfigValue to a newtype, and moved it into Git.Config.
2019-12-05 18:36:43 +00:00
getlist k = map fromConfigValue $ Git.Config.getList k r
add another setting to GitConfig
2013-01-27 13:33:19 +00:00
getwords k = fromMaybe [] $ words <$> getmaybe k
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
annex.autocommit can be configured via git-annex config ... to control the default behavior in all clones of a repository. This includes a new Configurable data type, so the GitConfig type indicates which values can be configured this way. The implementation should be quite efficient; the config log is only read once, and only when a Configurable value has not already been set by git-config. Indeed, it would be nice in the future to extend this, so that git-config is itself only read on demand. Some commands may not need to look at the git configuration at all. This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 17:40:14 +00:00
configurable d Nothing = DefaultConfig d
git-annex config annex.largefiles annex.largefiles can be configured by git-annex config, to more easily set a default that will also be used by clones, without needing to shoehorn the expression into the gitattributes file. The git config and gitattributes override that. Whenever something is added to git-annex config, we have to consider what happens if a user puts a purposfully bad value in there. Or, if a new git-annex adds some new value that an old git-annex can't parse. In this case, a global annex.largefiles that can't be parsed currently makes an error be thrown. That might not be ideal, but the gitattribute behaves the same, and is almost equally repo-global. Performance notes: git-annex add and addurl construct a matcher once and uses it for every file, so the added time penalty for reading the global config log is minor. If the gitattributes annex.largefiles were deprecated, git-annex add would get around 2% faster (excluding hashing), because looking that up for each file is not fast. So this new way of setting it is progress toward speeding up add. git-annex smudge does need to load the log every time. As well as checking the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids both overheads.
2019-12-20 16:12:31 +00:00
configurable _ (Just v) = case configsource of
FromGitConfig -> HasGitConfig v
FromGlobalConfig -> HasGlobalConfig v
git-annex pull and push Split out two new commands, git-annex pull and git-annex push. Those plus a git commit are equivilant to git-annex sync. In a sense, git-annex sync conflates 3 things, and it would have been better to have push and pull from the beginning and not sync. Although note that git-annex sync --content is faster than a pull followed by a push, because it only has to walk the tree once, look at preferred content once, etc. So there is some value in git-annex sync in speed, as well as user convenience. And it would be hard to split out pull and push from sync, as far as the implementaton goes. The implementation inside sync was easy, just adjust SyncOptions so it does the right thing. Note that the new commands default to syncing content, unless annex.synccontent is explicitly set to false. I'd like sync to also do that, but that's a hard transition to make. As a start to that transition, I added a note to git-annex-sync.mdwn that it may start to do so in a future version of git-annex. But a real transition would necessarily involve displaying warnings when sync is used without --content, and time. Sponsored-by: Kevin Mueller on Patreon
2023-05-16 20:37:30 +00:00
configurablemaybe Nothing = DefaultConfig Nothing
configurablemaybe (Just v) = case configsource of
FromGitConfig -> HasGitConfig (Just v)
FromGlobalConfig -> HasGlobalConfig (Just v)
annex.autocommit can be configured via git-annex config ... to control the default behavior in all clones of a repository. This includes a new Configurable data type, so the GitConfig type indicates which values can be configured this way. The implementation should be quite efficient; the config log is only read once, and only when a Configurable value has not already been set by git-config. Indeed, it would be nice in the future to extend this, so that git-config is itself only read on demand. Some commands may not need to look at the git configuration at all. This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 17:40:14 +00:00
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
onemegabyte = 1000000
implement annex.privateremote and remote.name.private configs The slightly unusual parsing in Types.GitConfig avoids the need to look at the remote list to get configs of remotes. annexPrivateRepos combines all the configs, and will only be calculated once, so it's nice and fast. privateUUIDsKnown and regardingPrivateUUID now need to read from the annex mvar, so are not entirely free. But that overhead can be optimised away, as seen in getJournalFileStale. The other call sites didn't seem worth optimising to save a single MVar access. The feature should have impreceptable speed overhead when not being used.
2021-04-23 18:21:57 +00:00
hereuuid = maybe NoUUID toUUID $ getmaybe (annexConfig "uuid")
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
add git configs for clusters
2024-06-14 15:42:32 +00:00
clusterprefix = annexConfigPrefix <> "cluster."
add git-annex updatecluster command Seems to work fine, making the right changes to the git-annex branch.
2024-06-14 18:21:50 +00:00
isclusternamekey k _ = clusterprefix `B.isPrefixOf` (fromConfigKey' k)
&& isLegalName (removeclusterprefix k)
removeclusterprefix k = drop (B.length clusterprefix) (fromConfigKey k)
add git configs for clusters
2024-06-14 15:42:32 +00:00
annex.autocommit can be configured via git-annex config ... to control the default behavior in all clones of a repository. This includes a new Configurable data type, so the GitConfig type indicates which values can be configured this way. The implementation should be quite efficient; the config log is only read once, and only when a Configurable value has not already been set by git-config. Indeed, it would be nice in the future to extend this, so that git-config is itself only read on demand. Some commands may not need to look at the git configuration at all. This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 17:40:14 +00:00
{- Merge a GitConfig that comes from git-config with one containing
- repository-global defaults. -}
mergeGitConfig :: GitConfig -> GitConfig -> GitConfig
mergeGitConfig gitconfig repoglobals = gitconfig
{ annexAutoCommit = merge annexAutoCommit
New annex.synccontent config setting .. which can be set to true to make git annex sync default to --content. This may become the default at some point in the future. As well as being configuable by git config, it can be configured by git-annex config to control the default behavior in all clones of a repository. Had to add a separate --no-content switch to we can tell if it's been explicitly set, and should override annex.synccontent. If --content was the default, this complication would not be necessary. This commit was sponsored by Jake Vosloo on Patreon.
2017-02-03 18:31:17 +00:00
, annexSyncContent = merge annexSyncContent
sync --only-annex and annex.synconlyannex * Added sync --only-annex, which syncs the git-annex branch and annexed content but leaves managing the other git branches up to you. * Added annex.synconlyannex git config setting, which can also be set with git-annex config to configure sync in all clones of the repo. Use case is then the user has their own git workflow, and wants to use git-annex without disrupting that, so they sync --only-annex to get the git-annex stuff in sync in addition to their usual git workflow. When annex.synconlyannex is set, --not-only-annex can be used to override it. It's not entirely clear what --only-annex --commit or --only-annex --push should do, and I left that combination not documented because I don't know if I might want to change the current behavior, which is that such options do not override the --only-annex. My gut feeling is that there is no good reasons to use such combinations; if you want to use your own git workflow, you'll be doing your own committing and pulling and pushing. A subtle question is, how should import/export special remotes be handled? Importing updates their remote tracking branch and merges it into master. If --only-annex prevented that git branch stuff, then it would prevent exporting to the special remote, in the case where it has changes that were not imported yet, because there would be a unresolved conflict. I decided that it's best to treat the fact that there's a remote tracking branch for import/export as an implementation detail in this case. The more important thing is that an import/export special remote is entirely annexed content, and so it makes a lot of sense that --only-annex will still sync with it.
2020-02-17 19:19:58 +00:00
, annexSyncOnlyAnnex = merge annexSyncOnlyAnnex
fixed an oversight that had always prevented annex.resolvemerge from being honored, when it was configured by git-annex config forgot to add it to the merge function
2019-12-20 14:55:23 +00:00
, annexResolveMerge = merge annexResolveMerge
git-annex config annex.largefiles annex.largefiles can be configured by git-annex config, to more easily set a default that will also be used by clones, without needing to shoehorn the expression into the gitattributes file. The git config and gitattributes override that. Whenever something is added to git-annex config, we have to consider what happens if a user puts a purposfully bad value in there. Or, if a new git-annex adds some new value that an old git-annex can't parse. In this case, a global annex.largefiles that can't be parsed currently makes an error be thrown. That might not be ideal, but the gitattribute behaves the same, and is almost equally repo-global. Performance notes: git-annex add and addurl construct a matcher once and uses it for every file, so the added time penalty for reading the global config log is minor. If the gitattributes annex.largefiles were deprecated, git-annex add would get around 2% faster (excluding hashing), because looking that up for each file is not fast. So this new way of setting it is progress toward speeding up add. git-annex smudge does need to load the log every time. As well as checking the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids both overheads.
2019-12-20 16:12:31 +00:00
, annexLargeFiles = merge annexLargeFiles
annex.dotfiles The git add behavior changes could be avoided if it turns out to be really annoying, but then it would need to behave the old way when annex.dotfiles=false and the new way when annex.dotfiles=true. I'd rather not have the config option result in such divergent behavior as `git annex add .` skipping a dotfile (old) vs adding to annex (new). Note that the assistant always adds dotfiles to the annex. This is surprising, but not new behavior. Might be worth making it also honor annex.dotfiles, but I wonder if perhaps some user somewhere uses it and keeps large files in a directory that happens to begin with a dot. Since dotfiles and dotdirs are a unix culture thing, and the assistant users may not be part of that culture, it seems best to keep its current behavior for now.
2019-12-26 20:24:40 +00:00
, annexDotFiles = merge annexDotFiles
annex.addunlocked expressions * annex.addunlocked can be set to an expression with the same format used by annex.largefiles, in case you want to default to unlocking some files but not others. * annex.addunlocked can be configured by git-annex config. Added a git-annex-matching-expression man page, broken out from tips/largefiles. A tricky consequence of this is that git-annex add --relaxed honors annex.addunlocked, but an expression might want to know the size or content of an url, which it's not going to download. I decided it was better not to fail, and just dummy up some plausible data in that case. Performance impact should be negligible. The global config is already loaded for annex.largefiles. The expression only has to be parsed once, and in the simple true/false case, it should not do any additional work matching it.
2019-12-20 19:01:34 +00:00
, annexAddUnlocked = merge annexAddUnlocked
annex.autocommit can be configured via git-annex config ... to control the default behavior in all clones of a repository. This includes a new Configurable data type, so the GitConfig type indicates which values can be configured this way. The implementation should be quite efficient; the config log is only read once, and only when a Configurable value has not already been set by git-config. Indeed, it would be nice in the future to extend this, so that git-config is itself only read on demand. Some commands may not need to look at the git configuration at all. This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 17:40:14 +00:00
}
where
merge f = case f gitconfig of
git-annex config annex.largefiles annex.largefiles can be configured by git-annex config, to more easily set a default that will also be used by clones, without needing to shoehorn the expression into the gitattributes file. The git config and gitattributes override that. Whenever something is added to git-annex config, we have to consider what happens if a user puts a purposfully bad value in there. Or, if a new git-annex adds some new value that an old git-annex can't parse. In this case, a global annex.largefiles that can't be parsed currently makes an error be thrown. That might not be ideal, but the gitattribute behaves the same, and is almost equally repo-global. Performance notes: git-annex add and addurl construct a matcher once and uses it for every file, so the added time penalty for reading the global config log is minor. If the gitattributes annex.largefiles were deprecated, git-annex add would get around 2% faster (excluding hashing), because looking that up for each file is not fast. So this new way of setting it is progress toward speeding up add. git-annex smudge does need to load the log every time. As well as checking the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids both overheads.
2019-12-20 16:12:31 +00:00
HasGitConfig v -> HasGitConfig v
annex.autocommit can be configured via git-annex config ... to control the default behavior in all clones of a repository. This includes a new Configurable data type, so the GitConfig type indicates which values can be configured this way. The implementation should be quite efficient; the config log is only read once, and only when a Configurable value has not already been set by git-config. Indeed, it would be nice in the future to extend this, so that git-config is itself only read on demand. Some commands may not need to look at the git configuration at all. This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 17:40:14 +00:00
DefaultConfig d -> case f repoglobals of
git-annex config annex.largefiles annex.largefiles can be configured by git-annex config, to more easily set a default that will also be used by clones, without needing to shoehorn the expression into the gitattributes file. The git config and gitattributes override that. Whenever something is added to git-annex config, we have to consider what happens if a user puts a purposfully bad value in there. Or, if a new git-annex adds some new value that an old git-annex can't parse. In this case, a global annex.largefiles that can't be parsed currently makes an error be thrown. That might not be ideal, but the gitattribute behaves the same, and is almost equally repo-global. Performance notes: git-annex add and addurl construct a matcher once and uses it for every file, so the added time penalty for reading the global config log is minor. If the gitattributes annex.largefiles were deprecated, git-annex add would get around 2% faster (excluding hashing), because looking that up for each file is not fast. So this new way of setting it is progress toward speeding up add. git-annex smudge does need to load the log every time. As well as checking the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids both overheads.
2019-12-20 16:12:31 +00:00
HasGlobalConfig v -> HasGlobalConfig v
_ -> HasGitConfig d
HasGlobalConfig v -> HasGlobalConfig v
annex.autocommit can be configured via git-annex config ... to control the default behavior in all clones of a repository. This includes a new Configurable data type, so the GitConfig type indicates which values can be configured this way. The implementation should be quite efficient; the config log is only read once, and only when a Configurable value has not already been set by git-config. Indeed, it would be nice in the future to extend this, so that git-config is itself only read on demand. Some commands may not need to look at the git configuration at all. This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 17:40:14 +00:00
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
{- Configs that can be set repository-global. -}
globalConfigs :: [ConfigKey]
globalConfigs =
git-annex-config: Allow setting annex.securehashesonly Which has otherwise been supported since 2019, but was missing from the list of allowed repo-global configs. Reordered the list to match the order in the git-annex-config man page, to make them easy to cross-compare. This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2021-04-26 17:50:37 +00:00
[ annexConfig "largefiles"
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
, annexConfig "dotfiles"
, annexConfig "addunlocked"
git-annex-config: Allow setting annex.securehashesonly Which has otherwise been supported since 2019, but was missing from the list of allowed repo-global configs. Reordered the list to match the order in the git-annex-config man page, to make them easy to cross-compare. This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2021-04-26 17:50:37 +00:00
, annexConfig "autocommit"
, annexConfig "resolvemerge"
, annexConfig "synccontent"
, annexConfig "synconlyannex"
, annexConfig "securehashesonly"
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
]
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
{- Per-remote git-annex settings. Each setting corresponds to a git-config
- key such as <remote>.annex-foo, or if that is not set, a default from
removal of the rest of remoteGitConfig In keyUrls, the GitConfig is used only by annexLocations to support configured Differences. Since such configurations affect all clones of a repository, the local repo's GitConfig must have the same information as the remote's GitConfig would have. So, used getGitConfig to get the local GitConfig, which is cached and so available cheaply. That actually fixed a bug noone had ever noticed: keyUrls is used for remotes accessed over http. The full git config of such a remote is normally not available, so the remoteGitConfig that keyUrls used would not have the necessary information in it. In copyFromRemoteCheap', it uses gitAnnexLocation, which does need the GitConfig of the remote repo itself in order to check if it's crippled, supports symlinks, etc. So, made the State include that GitConfig, cached. The use of gitAnnexLocation is within a (not $ Git.repoIsUrl repo) guard, so it's local, and so its git config will always be read and available. (Note that gitAnnexLocation in turn calls annexLocations, so the Differences config it uses in this case comes from the remote repo's GitConfig and not from the local repo's GitConfig. As explained above this is ok since they must have the same value.) Not very happy with this mess of different GitConfigs not type-safe and some read only sometimes etc. Very hairy. Think I got it this change right. Test suite passes.. This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
- annex.foo.
-
- Note that this is from the perspective of the local repository,
- it is not influenced in any way by the contents of the remote
- repository's git config.
-}
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
data RemoteGitConfig = RemoteGitConfig
use DynamicConfig to handle cost-command This commit was sponsored by Jake Vosloo on Patreon.
2017-08-17 18:04:29 +00:00
{ remoteAnnexCost :: DynamicConfig (Maybe Cost)
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
, remoteAnnexIgnore :: DynamicConfig Bool
, remoteAnnexSync :: DynamicConfig Bool
Added remote.<name>.annex-push and remote.<name>.annex-pull The former can be useful to make remotes that don't get fully synced with local changes, which comes up in a lot of situations. The latter was mostly added for symmetry, but could be useful (though less likely to be). Implementing `remote.<name>.annex-pull` was a bit tricky, as there's no one place where git-annex pulls/fetches from remotes. I audited all instances of "fetch" and "pull". A few cases were left not checking this config: * Git.Repair can try to pull missing refs from a remote, and if the local repo is corrupted, that seems a reasonable thing to do even though the config would normally prevent it. * Assistant.WebApp.Gpg and Remote.Gcrypt and Remote.Git do fetches as part of the setup process of a remote. The config would probably not be set then, and having the setup fail seems worse than honoring it if it is already set. I have not prevented all the code that does a "merge" from merging branches from remotes with remote.<name>.annex-pull=false. That could perhaps be done, but it would need a way to map from branch name to remote name, and the way refspecs work makes that hard to get really correct. So if the user fetches manually, the git-annex branch will get merged, for example. Anther way of looking at/justifying this is that the setting is called "annex-pull", not "annex-merge". This commit was supported by the NSF-funded DataLad project.
2017-04-05 17:04:02 +00:00
, remoteAnnexPull :: Bool
, remoteAnnexPush :: Bool
Remotes can now be made read-only, by setting remote.<name>.annex-readonly
2014-01-02 17:12:32 +00:00
, remoteAnnexReadOnly :: Bool
Do verification of checksums of annex objects downloaded from remotes. * When annex objects are received into git repositories, their checksums are verified then too. * To get the old, faster, behavior of not verifying checksums, set annex.verify=false, or remote.<name>.annex-verify=false. * setkey, rekey: These commands also now verify that the provided file matches the key, unless annex.verify=false. * reinject: Already verified content; this can now be disabled by setting annex.verify=false. recvkey and reinject already did verification, so removed now duplicate code from them. fsck still does its own verification, which is ok since it does not use getViaTmp, so verification doesn't happen twice when using fsck --from.
2015-10-01 19:54:37 +00:00
, remoteAnnexVerify :: Bool
add remote.<name>.annex-checkuuid Added remote.<name>.annex-checkuuid config, which can be set to false to disable the default checking of the uuid of remotes that point to directories. This can be useful to avoid unncessary drive spin-ups and automounting. Note that the UUID check is still done before writing to the repository, to avoid writing to the wrong repository if it got relocated. Check is also done before checkPresent to avoid getting confused about what is in which repo. This is effectively the same as the use of git-annex-shell with a uuid to check that the remote repository is the expected one. Did not bother with the check for retrieveKeyFile because it doesn't matter if the wrong repo is used then. This commit was sponsored by Trenton Cronholm on Patreon.
2018-01-10 18:21:18 +00:00
, remoteAnnexCheckUUID :: Bool
export: Deprecated the --tracking option. Instead, users can configure remote.<name>.annex-tracking-branch themselves.
2019-02-23 19:48:25 +00:00
, remoteAnnexTrackingBranch :: Maybe Git.Ref
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
, remoteAnnexTrustLevel :: Maybe String
, remoteAnnexStartCommand :: Maybe String
, remoteAnnexStopCommand :: Maybe String
cache remotes via annex-speculate-present Added remote.name.annex-speculate-present config that can be used to make cache remotes. Implemented it in Remote.keyPossibilities, which is used by the get/move/copy/mirror commands, and nothing else. This way, things like whereis will not show content that's speculatively present. The assistant and sync --content were not using Remote.keyPossibilities, and were changed to use it. The efficiency hit should be small; Remote.keyPossibilities is only used before transferring a file, which is the expensive operation. And, it's only doing one lookup of the remoteList and a very cheap filter over it. Note that, git-annex still updates the location log when copying content to a remote with annex-speculate-present set. In this case, the location tracking will indicate that content is present in the remote. This may not be wanted for caches, or may not be a real problem for them. TBD. This commit was supported by the NSF-funded DataLad project.
2018-08-01 18:22:52 +00:00
, remoteAnnexSpeculatePresent :: Bool
Optimise non-bare http remotes; no longer does a 404 to the wrong url every time before trying the right url. Needs annex-bare to be set to false, which is done when initially probing the uuid of a http remote.
2014-01-26 17:03:25 +00:00
, remoteAnnexBare :: Maybe Bool
add retry configuration (not used yet)
2018-03-24 14:37:25 +00:00
, remoteAnnexRetry :: Maybe Integer
Exposed annex.forward-retry git config One reason is, 5 is an arbitrary number so ought to be configurable. The real reason though, is I wanted to make the man page explain when forward retry can override annex.retry, and having a config made the man page easier to write.
2020-09-04 19:16:40 +00:00
, remoteAnnexForwardRetry :: Maybe Integer
add retry configuration (not used yet)
2018-03-24 14:37:25 +00:00
, remoteAnnexRetryDelay :: Maybe Seconds
stall detection is working New config annex.stalldetection, remote.name.annex-stalldetection, which can be used to deal with remotes that stall during transfers, or are sometimes too slow to want to use. This commit was sponsored by Luke Shumaker on Patreon.
2020-12-08 19:22:18 +00:00
, remoteAnnexStallDetection :: Maybe StallDetection
add directional stalldetection and bwlimit configs Sponsored-by: Dartmouth College's DANDI project
2024-01-19 19:14:26 +00:00
, remoteAnnexStallDetectionUpload :: Maybe StallDetection
, remoteAnnexStallDetectionDownload :: Maybe StallDetection
bwlimit Added annex.bwlimit and remote.name.annex-bwlimit config that works for git remotes and many but not all special remotes. This nearly works, at least for a git remote on the same disk. With it set to 100kb/1s, the meter displays an actual bandwidth of 128 kb/s, with occasional spikes to 160 kb/s. So it needs to delay just a bit longer... I'm unsure why. However, at the beginning a lot of data flows before it determines the right bandwidth limit. A granularity of less than 1s would probably improve that. And, I don't know yet if it makes sense to have it be 100ks/1s rather than 100kb/s. Is there a situation where the user would want a larger granularity? Does granulatity need to be configurable at all? I only used that format for the config really in order to reuse an existing parser. This can't support for external special remotes, or for ones that themselves shell out to an external command. (Well, it could, but it would involve pausing and resuming the child process tree, which seems very hard to implement and very strange besides.) There could also be some built-in special remotes that it still doesn't work for, due to them not having a progress meter whose displays blocks the bandwidth using thread. But I don't think there are actually any that run a separate thread for downloads than the thread that displays the progress meter. Sponsored-by: Graham Spencer on Patreon
2021-09-21 20:58:02 +00:00
, remoteAnnexBwLimit :: Maybe BwRate
add directional stalldetection and bwlimit configs Sponsored-by: Dartmouth College's DANDI project
2024-01-19 19:14:26 +00:00
, remoteAnnexBwLimitUpload :: Maybe BwRate
, remoteAnnexBwLimitDownload :: Maybe BwRate
remote.name.annex-security-allow-unverified-downloads Added remote.name.annex-security-allow-unverified-downloads, a per-remote setting for annex.security.allow-unverified-downloads. This commit was sponsored by Brock Spratlen on Patreon.
2018-09-25 19:32:50 +00:00
, remoteAnnexAllowUnverifiedDownloads :: Bool
add remote.<name>.annex-config-uuid for use by sameas remotes
2019-10-10 16:12:22 +00:00
, remoteAnnexConfigUUID :: Maybe UUID
git-remote-annex: brought back max-git-bundles config An incremental push that gets converted to a full push due to this config results in the inManifest having just one bundle in it, and the outManifest listing every other bundle. So it actually takes up more space on the special remote. But, it speeds up clone and fetch to not have to download a long series of bundles for incremental pushes.
2024-05-28 17:26:21 +00:00
, remoteAnnexMaxGitBundles :: Int
max-git-bundles config for git-remote-annex
2024-05-14 18:23:40 +00:00
, remoteAnnexAllowEncryptedGitRepo :: Bool
Added updateproxy command and remote.name.annex-proxy configuration So far this only records proxy information on the git-annex branch.
2024-06-04 18:50:38 +00:00
, remoteAnnexProxy :: Bool
add git-annex updatecluster command Seems to work fine, making the right changes to the git-annex branch.
2024-06-14 18:21:50 +00:00
, remoteAnnexClusterNode :: Maybe [RemoteName]
sync with special remotes with an annex:: url Check explicitly for an annex:: url, not just any url. While no built-in special remotes set an url, except ones that can be synced with, it seems possible that some external special remote sets an url for its own use, but did not expect it to be used by git-annex sync et al. The assistant also syncs with them.
2024-05-24 18:52:22 +00:00
, remoteUrl :: Maybe String
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
Use lower case hash directories for storing files on crippled filesystems, same as is already done for bare repositories. * since this is a crippled filesystem anyway, git-annex doesn't use symlinks on it * so there's no reason to use the mixed case hash directories that we're stuck using to avoid breaking everyone's symlinks to the content * so we can do what is already done for all bare repos, and make non-bare repos on crippled filesystems use the all-lower case hash directories * which are, happily, all 3 letters long, so they cannot conflict with mixed case hash directories * so I was able to 100% fix this and even resuming `git annex add` in the test case will recover and it will all just work.
2013-04-04 19:46:33 +00:00
{- These settings are specific to particular types of remotes
- including special remotes. -}
execute remote.<name>.annex-shell on remote, if set It is useful to be able to specify an alternative git-annex-shell program to execute on the remote, e.g., to run a version not on the PATH. Use remote.<name>.annex-shell if specified, instead of the default "git-annex-shell" i.e., first so-named executable on the PATH.
2014-05-16 11:34:43 +00:00
, remoteAnnexShell :: Maybe String
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
, remoteAnnexSshOptions :: [String]
, remoteAnnexRsyncOptions :: [String]
Added ways to configure rsync options to be used only when uploading or downloading from a remote. Useful to eg limit upload bandwidth.
2014-02-02 20:06:34 +00:00
, remoteAnnexRsyncUploadOptions :: [String]
, remoteAnnexRsyncDownloadOptions :: [String]
Allow rsync to use other remote shells. Introduced a new per-remote option 'annex-rsync-transport' to specify the remote shell that it to be used with rsync. In case the value is 'ssh', connections are cached unless 'sshcaching' is unset.
2013-04-13 22:10:49 +00:00
, remoteAnnexRsyncTransport :: [String]
GnuPG options for symmetric encryption.
2013-03-11 01:33:13 +00:00
, remoteAnnexGnupgOptions :: [String]
Added annex.gnupg-decrypt-options and remote.<name>.annex-gnupg-decrypt-options, which are passed to gpg when it's decrypting data. The naming is unofrtunately not consistent, but the gnupg-options were only used for encrypting, and it's too late to change that. It would be nice to have a third setting that is always passed to gnupg, but ~/.gnupg/options can be used to specify such global options when really needed.
2016-05-10 17:03:56 +00:00
, remoteAnnexGnupgDecryptOptions :: [String]
more groundwork for StatelessOpenPGP no behavior changes
2024-01-12 16:27:58 +00:00
, remoteAnnexSharedSOPCommand :: Maybe SOPCmd
support annex.shared-sop-command for encryption=shared This works well, and it interoperates with gpg in my testing (although some SOP commands might choose to use a profile that does not so caveat emptor). Note that for creating the Cipher, gpg --gen-random is still used. SOP does not have an eqivilant, and as long as the user has gpg around, which seems likely, it doesn't matter that it uses gpg here, it's not being used for encryption. That seemed better than implementing a second way to get high quality entropy, at least for now. The need for the sop command to run in an empty directory has each call to encrypt and decrypt creating a new temporary directory. That is some unncessary overhead, though probably swamped by the overhead of running the sop command. This could be improved in the future by passing an already empty directory to them, or a sufficiently empty directory (.git/annex/tmp would probably suffice). Sponsored-by: Brett Eisenberg on Patreon
2024-01-12 17:29:34 +00:00
, remoteAnnexSharedSOPProfile :: Maybe SOPProfile
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
, remoteAnnexRsyncUrl :: Maybe String
, remoteAnnexBupRepo :: Maybe String
start borg special remote Compiles, but unusable so far.
2020-12-18 20:03:51 +00:00
, remoteAnnexBorgRepo :: Maybe String
Added tahoe special remote. Known problems: 1. Tries to tahoe start when daemon is already running. 2. If multiple tahoe remotes are set up on the same computer, they will have the same node.url configured by default, and this confuses tahoe commands. This commit was sponsored by LeastAuthority.com
2014-01-08 20:14:37 +00:00
, remoteAnnexTahoe :: Maybe FilePath
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
, remoteAnnexBupSplitOptions :: [String]
, remoteAnnexDirectory :: Maybe FilePath
Added adb special remote which allows exporting files to Android devices. git annex testremote passes. exportree not implemented yet, although the documentation talks about it, since it will be the main way this remote will be used. The adb push/pull progress is displayed for now; it would be better to consume it and use it to update the git-annex progress bar. This commit was sponsored by andrea rota.
2018-03-27 16:41:57 +00:00
, remoteAnnexAndroidDirectory :: Maybe FilePath
, remoteAnnexAndroidSerial :: Maybe String
git-annex-shell: Added support for operating inside gcrypt repositories. * Note that the layout of gcrypt repositories has changed, and if you created one you must manually upgrade it. See http://git-annex.branchable.com/upgrades/gcrypt/
2013-09-24 21:25:47 +00:00
, remoteAnnexGCrypt :: Maybe String
skeleton git-lfs special remote This is a special remote and a git remote at the same time; git can pull and push to it and git-annex can use it as a special remote. Remote.Git has to check if it's configured as a git-lfs special remote and sets it up as one if so. Object methods not implemented yet.
2019-08-01 19:11:45 +00:00
, remoteAnnexGitLFS :: Bool
ddar special remote
2014-05-15 18:44:00 +00:00
, remoteAnnexDdarRepo :: Maybe String
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
, remoteAnnexHookType :: Maybe String
external special remotes mostly implemented (untested) This has not been tested at all. It compiles! The only known missing things are support for encryption, and for get/set of special remote configuration, and of key state. (The latter needs separate work to add a new per-key log file to store that state.) Only thing I don't much like is that initremote needs to be passed both type=external and externaltype=foo. It would be better to have just type=foo Most of this is quite straightforward code, that largely wrote itself given the types. The only tricky parts were: * Need to lock the remote when using it to eg make a request, because in theory git-annex could have multiple threads that each try to use a remote at the same time. I don't think that git-annex ever does that currently, but better safe than sorry. * Rather than starting up every external special remote program when git-annex starts, they are started only on demand, when first used. This will avoid slowdown, especially when running fast git-annex query commands. Once started, they keep running until git-annex stops, currently, which may not be ideal, but it's hard to know a better time to stop them. * Bit of a chicken and egg problem with caching the cost of the remote, because setting annex-cost in the git config needs the remote to already be set up. Managed to finesse that. This commit was sponsored by Lukas Anzinger.
2013-12-26 22:23:13 +00:00
, remoteAnnexExternalType :: Maybe String
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
}
removal of the rest of remoteGitConfig In keyUrls, the GitConfig is used only by annexLocations to support configured Differences. Since such configurations affect all clones of a repository, the local repo's GitConfig must have the same information as the remote's GitConfig would have. So, used getGitConfig to get the local GitConfig, which is cached and so available cheaply. That actually fixed a bug noone had ever noticed: keyUrls is used for remotes accessed over http. The full git config of such a remote is normally not available, so the remoteGitConfig that keyUrls used would not have the necessary information in it. In copyFromRemoteCheap', it uses gitAnnexLocation, which does need the GitConfig of the remote repo itself in order to check if it's crippled, supports symlinks, etc. So, made the State include that GitConfig, cached. The use of gitAnnexLocation is within a (not $ Git.repoIsUrl repo) guard, so it's local, and so its git config will always be read and available. (Note that gitAnnexLocation in turn calls annexLocations, so the Differences config it uses in this case comes from the remote repo's GitConfig and not from the local repo's GitConfig. As explained above this is ok since they must have the same value.) Not very happy with this mess of different GitConfigs not type-safe and some read only sometimes etc. Very hairy. Think I got it this change right. Test suite passes.. This commit was sponsored by Ethan Aubin.
2018-06-05 18:23:34 +00:00
{- The Git.Repo is the local repository, which has the remote with the
- given RemoteName. -}
extractRemoteGitConfig :: Git.Repo -> RemoteName -> STM RemoteGitConfig
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
extractRemoteGitConfig r remotename = do
use DynamicConfig to handle cost-command This commit was sponsored by Jake Vosloo on Patreon.
2017-08-17 18:04:29 +00:00
annexcost <- mkDynamicConfig readCommandRunner
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
(notempty $ getmaybe CostCommandField)
(getmayberead CostField)
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
annexignore <- mkDynamicConfig unsuccessfullCommandRunner
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
(notempty $ getmaybe IgnoreCommandField)
(getbool IgnoreField False)
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
annexsync <- mkDynamicConfig successfullCommandRunner
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
(notempty $ getmaybe SyncCommandField)
(getbool SyncField True)
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
return $ RemoteGitConfig
use DynamicConfig to handle cost-command This commit was sponsored by Jake Vosloo on Patreon.
2017-08-17 18:04:29 +00:00
{ remoteAnnexCost = annexcost
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
, remoteAnnexIgnore = annexignore
, remoteAnnexSync = annexsync
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
, remoteAnnexPull = getbool PullField True
, remoteAnnexPush = getbool PushField True
, remoteAnnexReadOnly = getbool ReadOnlyField False
, remoteAnnexCheckUUID = getbool CheckUUIDField True
, remoteAnnexVerify = getbool VerifyField True
ByteString Ref continued Several nice speed wins I think. At 340/633 files converted.
2020-04-07 17:27:11 +00:00
, remoteAnnexTrackingBranch = Git.Ref . encodeBS <$>
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
( notempty (getmaybe TrackingBranchField)
<|> notempty (getmaybe ExportTrackingField) -- old name
export: Deprecated the --tracking option. Instead, users can configure remote.<name>.annex-tracking-branch themselves.
2019-02-23 19:48:25 +00:00
)
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
, remoteAnnexTrustLevel = notempty $ getmaybe TrustLevelField
, remoteAnnexStartCommand = notempty $ getmaybe StartCommandField
, remoteAnnexStopCommand = notempty $ getmaybe StopCommandField
prevent using git-remote-annex with unsuitable special remote configs I hope to support importtree=yes eventually, but it does not currently work. Added remote.<name>.allow-encrypted-gitrepo that needs to be set to allow using it with encrypted git repos. Note that even encryption=pubkey uses a cipher stored in the git repo to encrypt the keys stored in the remote. While it would be possible to not encrypt the GITBUNDLE and GITMANIFEST keys, and then allow using encryption=pubkey, it doesn't currently work, and that would be a complication that I doubt is worth it.
2024-05-14 17:52:20 +00:00
, remoteAnnexSpeculatePresent =
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
getbool SpeculatePresentField False
, remoteAnnexBare = getmaybebool BareField
, remoteAnnexRetry = getmayberead RetryField
, remoteAnnexForwardRetry = getmayberead ForwardRetryField
add retry configuration (not used yet)
2018-03-24 14:37:25 +00:00
, remoteAnnexRetryDelay = Seconds
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
<$> getmayberead RetryDelayField
stall detection is working New config annex.stalldetection, remote.name.annex-stalldetection, which can be used to deal with remotes that stall during transfers, or are sometimes too slow to want to use. This commit was sponsored by Luke Shumaker on Patreon.
2020-12-08 19:22:18 +00:00
, remoteAnnexStallDetection =
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
readStallDetection =<< getmaybe StallDetectionField
add directional stalldetection and bwlimit configs Sponsored-by: Dartmouth College's DANDI project
2024-01-19 19:14:26 +00:00
, remoteAnnexStallDetectionUpload =
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
readStallDetection =<< getmaybe StallDetectionUploadField
add directional stalldetection and bwlimit configs Sponsored-by: Dartmouth College's DANDI project
2024-01-19 19:14:26 +00:00
, remoteAnnexStallDetectionDownload =
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
readStallDetection =<< getmaybe StallDetectionDownloadField
add directional stalldetection and bwlimit configs Sponsored-by: Dartmouth College's DANDI project
2024-01-19 19:14:26 +00:00
, remoteAnnexBwLimit =
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
readBwRatePerSecond =<< getmaybe BWLimitField
add directional stalldetection and bwlimit configs Sponsored-by: Dartmouth College's DANDI project
2024-01-19 19:14:26 +00:00
, remoteAnnexBwLimitUpload =
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
readBwRatePerSecond =<< getmaybe BWLimitUploadField
add directional stalldetection and bwlimit configs Sponsored-by: Dartmouth College's DANDI project
2024-01-19 19:14:26 +00:00
, remoteAnnexBwLimitDownload =
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
readBwRatePerSecond =<< getmaybe BWLimitDownloadField
remote.name.annex-security-allow-unverified-downloads Added remote.name.annex-security-allow-unverified-downloads, a per-remote setting for annex.security.allow-unverified-downloads. This commit was sponsored by Brock Spratlen on Patreon.
2018-09-25 19:32:50 +00:00
, remoteAnnexAllowUnverifiedDownloads = (== Just "ACKTHPPT") $
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
getmaybe SecurityAllowUnverifiedDownloadsField
, remoteAnnexConfigUUID = toUUID <$> getmaybe ConfigUUIDField
git-remote-annex: brought back max-git-bundles config An incremental push that gets converted to a full push due to this config results in the inManifest having just one bundle in it, and the outManifest listing every other bundle. So it actually takes up more space on the special remote. But, it speeds up clone and fetch to not have to download a long series of bundles for incremental pushes.
2024-05-28 17:26:21 +00:00
, remoteAnnexMaxGitBundles =
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
fromMaybe 100 (getmayberead MaxGitBundlesField)
reorder constructors for consistency
2024-06-04 17:11:27 +00:00
, remoteAnnexAllowEncryptedGitRepo =
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
getbool AllowEncryptedGitRepoField False
, remoteAnnexProxy = getbool ProxyField False
add git-annex updatecluster command Seems to work fine, making the right changes to the git-annex branch.
2024-06-14 18:21:50 +00:00
, remoteAnnexClusterNode =
(filter isLegalName . words)
<$> getmaybe ClusterNodeField
reorder constructors for consistency
2024-06-04 17:11:27 +00:00
, remoteUrl =
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
case Git.Config.getMaybe (remoteConfig remotename (remoteGitConfigKey UrlField)) r of
reorder constructors for consistency
2024-06-04 17:11:27 +00:00
Just (ConfigValue b)
| B.null b -> Nothing
| otherwise -> Just (decodeBS b)
_ -> Nothing
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
, remoteAnnexShell = getmaybe ShellField
, remoteAnnexSshOptions = getoptions SshOptionsField
, remoteAnnexRsyncOptions = getoptions RsyncOptionsField
, remoteAnnexRsyncDownloadOptions = getoptions RsyncDownloadOptionsField
, remoteAnnexRsyncUploadOptions = getoptions RsyncUploadOptionsField
, remoteAnnexRsyncTransport = getoptions RsyncTransportField
, remoteAnnexGnupgOptions = getoptions GnupgOptionsField
, remoteAnnexGnupgDecryptOptions = getoptions GnupgDecryptOptionsField
more groundwork for StatelessOpenPGP no behavior changes
2024-01-12 16:27:58 +00:00
, remoteAnnexSharedSOPCommand = SOPCmd <$>
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
notempty (getmaybe SharedSOPCommandField)
support annex.shared-sop-command for encryption=shared This works well, and it interoperates with gpg in my testing (although some SOP commands might choose to use a profile that does not so caveat emptor). Note that for creating the Cipher, gpg --gen-random is still used. SOP does not have an eqivilant, and as long as the user has gpg around, which seems likely, it doesn't matter that it uses gpg here, it's not being used for encryption. That seemed better than implementing a second way to get high quality entropy, at least for now. The need for the sop command to run in an empty directory has each call to encrypt and decrypt creating a new temporary directory. That is some unncessary overhead, though probably swamped by the overhead of running the sop command. This could be improved in the future by passing an already empty directory to them, or a sufficiently empty directory (.git/annex/tmp would probably suffice). Sponsored-by: Brett Eisenberg on Patreon
2024-01-12 17:29:34 +00:00
, remoteAnnexSharedSOPProfile = SOPProfile <$>
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
notempty (getmaybe SharedSOPProfileField)
, remoteAnnexRsyncUrl = notempty $ getmaybe RsyncUrlField
, remoteAnnexBupRepo = getmaybe BupRepoField
, remoteAnnexBorgRepo = getmaybe BorgRepoField
, remoteAnnexTahoe = getmaybe TahoeField
, remoteAnnexBupSplitOptions = getoptions BupSplitOptionsField
, remoteAnnexDirectory = notempty $ getmaybe DirectoryField
, remoteAnnexAndroidDirectory = notempty $ getmaybe AndroidDirectoryField
, remoteAnnexAndroidSerial = notempty $ getmaybe AndroidSerialField
, remoteAnnexGCrypt = notempty $ getmaybe GCryptField
, remoteAnnexGitLFS = getbool GitLFSField False
, remoteAnnexDdarRepo = getmaybe DdarRepoField
, remoteAnnexHookType = notempty $ getmaybe HookTypeField
, remoteAnnexExternalType = notempty $ getmaybe ExternalTypeField
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
}
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
where
import Data.Default in Common
2015-01-28 20:11:28 +00:00
getbool k d = fromMaybe d $ getmaybebool k
annex.addunlocked expressions * annex.addunlocked can be set to an expression with the same format used by annex.largefiles, in case you want to default to unlocking some files but not others. * annex.addunlocked can be configured by git-annex config. Added a git-annex-matching-expression man page, broken out from tips/largefiles. A tricky consequence of this is that git-annex add --relaxed honors annex.addunlocked, but an expression might want to know the size or content of an url, which it's not going to download. I decided it was better not to fail, and just dummy up some plausible data in that case. Performance impact should be negligible. The global config is already loaded for annex.largefiles. The expression only has to be parsed once, and in the simple true/false case, it should not do any additional work matching it.
2019-12-20 19:01:34 +00:00
getmaybebool k = Git.Config.isTrueFalse' =<< getmaybe' k
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
getmayberead k = readish =<< getmaybe k
all commands building except for assistant also, changed ConfigValue to a newtype, and moved it into Git.Config.
2019-12-05 18:36:43 +00:00
getmaybe = fmap fromConfigValue . getmaybe'
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
getmaybe' :: RemoteGitConfigField -> Maybe ConfigValue
getmaybe' f =
let k = remoteGitConfigKey f
in Git.Config.getMaybe (remoteAnnexConfig remotename k) r
<|> Git.Config.getMaybe (annexConfig k) r
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
getoptions k = fromMaybe [] $ words <$> getmaybe k
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
data RemoteGitConfigField
= CostField
| CostCommandField
| IgnoreField
| IgnoreCommandField
| SyncField
| SyncCommandField
| PullField
| PushField
| ReadOnlyField
| CheckUUIDField
| VerifyField
| TrackingBranchField
| ExportTrackingField
| TrustLevelField
| StartCommandField
| StopCommandField
| SpeculatePresentField
| BareField
| RetryField
| ForwardRetryField
| RetryDelayField
| StallDetectionField
| StallDetectionUploadField
| StallDetectionDownloadField
| BWLimitField
| BWLimitUploadField
| BWLimitDownloadField
| ConfigUUIDField
| SecurityAllowUnverifiedDownloadsField
| MaxGitBundlesField
| AllowEncryptedGitRepoField
| ProxyField
add git configs for clusters
2024-06-14 15:42:32 +00:00
| ClusterNodeField
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
| UrlField
| ShellField
| SshOptionsField
| RsyncOptionsField
| RsyncDownloadOptionsField
| RsyncUploadOptionsField
| RsyncTransportField
| GnupgOptionsField
| GnupgDecryptOptionsField
| SharedSOPCommandField
| SharedSOPProfileField
| RsyncUrlField
| BupRepoField
| BorgRepoField
| TahoeField
| BupSplitOptionsField
| DirectoryField
| AndroidDirectoryField
| AndroidSerialField
| GCryptField
| GitLFSField
| DdarRepoField
| HookTypeField
| ExternalTypeField
deriving (Enum, Bounded)
remoteGitConfigField :: RemoteGitConfigField -> (UnqualifiedConfigKey, ProxyInherited)
remoteGitConfigField = \case
-- Hard to know the true cost of accessing eg a slow special
-- remote via the proxy. The cost of the proxy is the best guess
-- so do inherit it.
CostField -> inherited "cost"
CostCommandField -> inherited "cost-command"
IgnoreField -> inherited "ignore"
IgnoreCommandField -> inherited "ignore-command"
SyncField -> inherited "sync"
SyncCommandField -> inherited "sync-command"
PullField -> inherited "pull"
PushField -> inherited "push"
ReadOnlyField -> inherited "readonly"
CheckUUIDField -> uninherited "checkuuid"
VerifyField -> inherited "verify"
TrackingBranchField -> uninherited "tracking-branch"
ExportTrackingField -> uninherited "export-tracking"
TrustLevelField -> uninherited "trustlevel"
StartCommandField -> uninherited "start-command"
StopCommandField -> uninherited "stop-command"
SpeculatePresentField -> inherited "speculate-present"
inherit remote.name.annex-bare Since a proxied remote uses the proxy's git repo, this makes sense. Although I don't think this config is ever used when accessing a remote via git-annex-shell.
2024-06-12 15:53:28 +00:00
BareField -> inherited "bare"
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
RetryField -> inherited "retry"
ForwardRetryField -> inherited "forward-retry"
RetryDelayField -> inherited "retrydelay"
StallDetectionField -> inherited "stalldetection"
StallDetectionUploadField -> inherited "stalldetection-upload"
StallDetectionDownloadField -> inherited "stalldetection-download"
BWLimitField -> inherited "bwlimit"
BWLimitUploadField -> inherited "bwlimit-upload"
BWLimitDownloadField -> inherited "bwlimit-upload"
ConfigUUIDField -> uninherited "config-uuid"
SecurityAllowUnverifiedDownloadsField -> inherited "security-allow-unverified-downloads"
MaxGitBundlesField -> inherited "max-git-bundles"
AllowEncryptedGitRepoField -> inherited "allow-encrypted-gitrepo"
-- Allow proxy chains.
ProxyField -> inherited "proxy"
add git configs for clusters
2024-06-14 15:42:32 +00:00
ClusterNodeField -> uninherited "cluster-node"
remote git config inheritance for proxied remotes When there is a proxy remote, remotes that it proxies need to be constructed with the right subset of the remote git-config settings. Obviously, the url is the same, and the uuid is different. Added proxyInheritedFields that lists all the fields that should be inherited. These will be copied into the proxied remote when instantiating it. There were a lot of decisions here, made without certainty in some cases. May need to revisit them. The RemoteGitConfigField type was added to make sure that every config used in extractRemoteGitConfig gets considered for proxy inheritance, including new ones that get added going forward. And to avoid needing to write the field string more than once.
2024-06-06 19:17:59 +00:00
UrlField -> inherited "url"
ShellField -> inherited "shell"
SshOptionsField -> inherited "ssh-options"
RsyncOptionsField -> inherited "rsync-options"
RsyncDownloadOptionsField -> inherited "rsync-download-options"
RsyncUploadOptionsField -> inherited "rsync-upload-options"
RsyncTransportField -> inherited "rsync-transport"
GnupgOptionsField -> inherited "gnupg-options"
GnupgDecryptOptionsField -> inherited "gnupg-decrypt-options"
SharedSOPCommandField -> inherited "shared-sop-command"
SharedSOPProfileField -> inherited "shared-sop-profile"
RsyncUrlField -> uninherited "rsyncurl"
BupRepoField -> uninherited "buprepo"
BorgRepoField -> uninherited "borgrepo"
TahoeField -> uninherited "tahoe"
BupSplitOptionsField -> uninherited "bup-split-options"
DirectoryField -> uninherited "directory"
AndroidDirectoryField -> uninherited "androiddirectory"
AndroidSerialField -> uninherited "androidserial"
GCryptField -> uninherited "gcrypt"
GitLFSField -> uninherited "git-lfs"
DdarRepoField -> uninherited "ddarrepo"
HookTypeField -> uninherited "hooktype"
ExternalTypeField -> uninherited "externaltype"
where
inherited f = (f, ProxyInherited True)
uninherited f = (f, ProxyInherited False)
newtype ProxyInherited = ProxyInherited Bool
-- All remote config fields that are inherited from a proxy.
proxyInheritedFields :: [UnqualifiedConfigKey]
proxyInheritedFields =
map fst $
filter (\(_, ProxyInherited p) -> p) $
map remoteGitConfigField [minBound..maxBound]
remoteGitConfigKey :: RemoteGitConfigField -> UnqualifiedConfigKey
remoteGitConfigKey = fst . remoteGitConfigField
type based git config handling for remotes Still a couple of places that use git config ad-hoc, but this is most of it done.
2013-01-01 17:52:47 +00:00
notempty :: Maybe String -> Maybe String
notempty Nothing = Nothing
notempty (Just "") = Nothing
notempty (Just s) = Just s
add annex-ignore-command and annex-sync-command configs Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 16:26:14 +00:00
dummyRemoteGitConfig :: IO RemoteGitConfig
dummyRemoteGitConfig = atomically $
extractRemoteGitConfig Git.Construct.fromUnknown "dummy"
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
type UnqualifiedConfigKey = B.ByteString
add git configs for clusters
2024-06-14 15:42:32 +00:00
annexConfigPrefix :: B.ByteString
annexConfigPrefix = "annex."
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
{- A global annex setting in git config. -}
annexConfig :: UnqualifiedConfigKey -> ConfigKey
add git configs for clusters
2024-06-14 15:42:32 +00:00
annexConfig key = ConfigKey (annexConfigPrefix <> key)
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
class RemoteNameable r where
getRemoteName :: r -> RemoteName
instance RemoteNameable Git.Repo where
getRemoteName r = fromMaybe "" (Git.remoteName r)
instance RemoteNameable RemoteName where
getRemoteName = id
{- A per-remote annex setting in git config. -}
remoteAnnexConfig :: RemoteNameable r => r -> UnqualifiedConfigKey -> ConfigKey
implement annex.privateremote and remote.name.private configs The slightly unusual parsing in Types.GitConfig avoids the need to look at the remote list to get configs of remotes. annexPrivateRepos combines all the configs, and will only be calculated once, so it's nice and fast. privateUUIDsKnown and regardingPrivateUUID now need to read from the annex mvar, so are not entirely free. But that overhead can be optimised away, as seen in getJournalFileStale. The other call sites didn't seem worth optimising to save a single MVar access. The feature should have impreceptable speed overhead when not being used.
2021-04-23 18:21:57 +00:00
remoteAnnexConfig r = remoteConfig r . remoteAnnexConfigEnd
remoteAnnexConfigEnd :: UnqualifiedConfigKey -> UnqualifiedConfigKey
remoteAnnexConfigEnd key = "annex-" <> key
git-annex config: guard against non-repo-global configs git-annex config: Only allow configs be set that are ones git-annex actually supports reading from repo-global config, to avoid confused users trying to set other configs with this.
2020-03-02 19:50:40 +00:00
{- A per-remote setting in git config. -}
remoteConfig :: RemoteNameable r => r -> UnqualifiedConfigKey -> ConfigKey
remoteConfig r key = ConfigKey $
simplify and speed up Utility.FileSystemEncoding This eliminates the distinction between decodeBS and decodeBS', encodeBS and encodeBS', etc. The old implementation truncated at NUL, and the primed versions had to do extra work to avoid that problem. The new implementation does not truncate at NUL, and is also a lot faster. (Benchmarked at 2x faster for decodeBS and 3x for encodeBS; more for the primed versions.) Note that filepath-bytestring 1.4.2.1.8 contains the same optimisation, and upgrading to it will speed up to/fromRawFilePath. AFAIK, nothing relied on the old behavior of truncating at NUL. Some code used the faster versions in places where I was sure there would not be a NUL. So this change is unlikely to break anything. Also, moved s2w8 and w82s out of the module, as they do not involve filesystem encoding really. Sponsored-by: Shae Erisson on Patreon
2021-08-11 00:45:02 +00:00
"remote." <> encodeBS (getRemoteName r) <> "." <> key
Reference in a new issue Copy permalink