mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Logs/Remote.hs

43 lines
969 B
Haskell
Raw Normal View History

factor out RemoteLog
2011-07-06 00:16:57 +00:00
{- git-annex remote log
-
sameas RemoteConfig inheritance I found a way to avoid inheritance complicating anything outside of Logs.Remote. It seems fine to require all inherited values to be inherited and not set in the sameas remote's config. Since inherited values will be used for stuff like encryption and perhaps chunking, which control the actual content stored on the remote, it seems likely that there will not be any reason to need them to vary between two remotes that access the same underlying data store. The newer version of containers is free; the minimum ghc version is bundled with a newer version than that.
2019-10-10 19:46:12 +00:00
- Copyright 2011-2019 Joey Hess <id@joeyh.name>
factor out RemoteLog
2011-07-06 00:16:57 +00:00
-
update licenses from GPL to AGPL This does not change the overall license of the git-annex program, which was already AGPL due to a number of sources files being AGPL already. Legally speaking, I'm adding a new license under which these files are now available; I already released their current contents under the GPL license. Now they're dual licensed GPL and AGPL. However, I intend for all my future changes to these files to only be released under the AGPL license, and I won't be tracking the dual licensing status, so I'm simply changing the license statement to say it's AGPL. (In some cases, others wrote parts of the code of a file and released it under the GPL; but in all cases I have contributed a significant portion of the code in each file and it's that code that is getting the AGPL license; the GPL license of other contributors allows combining with AGPL code.)
2019-03-13 19:48:14 +00:00
- Licensed under the GNU AGPL version 3 or higher.
factor out RemoteLog
2011-07-06 00:16:57 +00:00
-}
reorganize log modules no code changes
2011-10-15 20:21:08 +00:00
module Logs.Remote (
add ConfigMonitor thread Monitors git-annex branch for changes, which are noticed by the Merger thread whenever the branch ref is changed (either due to an incoming push, or a local change), and refreshes cached config values for modified config files. Rate limited to run no more often than once per minute. This is important because frequent git-annex branch changes happen when files are being added, or transferred, etc. A primary use case is that, when preferred content changes are made, and get pushed to remotes, the remotes start honoring those settings. Other use cases include propigating repository description and trust changes to remotes, and learning when a remote has added a new special remote, so the webapp can present the GUI to enable that special remote locally. Also added a uuid.log cache. All other config files already had caches.
2012-10-20 20:37:06 +00:00
remoteLog,
factor out RemoteLog
2011-07-06 00:16:57 +00:00
readRemoteLog,
configSet,
keyValToConfig,
configToKeyVal,
another quickcheck
2012-12-20 04:02:33 +00:00
showConfig,
factor out RemoteLog
2011-07-06 00:16:57 +00:00
fix use of hifalutin terminology
2015-11-16 18:37:31 +00:00
prop_isomorphic_configEscape,
another quickcheck
2012-12-20 04:02:33 +00:00
prop_parse_show_Config,
factor out RemoteLog
2011-07-06 00:16:57 +00:00
) where
remove 163 lines of code without changing anything except imports
2016-01-20 20:36:33 +00:00
import Annex.Common
rename
2011-10-04 04:40:47 +00:00
import qualified Annex.Branch
factor out RemoteLog
2011-07-06 00:16:57 +00:00
import Types.Remote
refactor git-annex branch log filename code into central location Having one module that knows about all the filenames used on the branch allows working back from an arbitrary filename to enough information about it to implement dropping dead remotes and doing other log file compacting as part of a forget transition.
2013-08-29 22:51:22 +00:00
import Logs
reorganize log modules no code changes
2011-10-15 20:21:08 +00:00
import Logs.UUIDBased
forget state of sameas remotes during DropDead transitions It would have been a lot less round-about to just make git annex dead also add the uuids of sameas remotes to the trust.log as dead. But, that would fail in the case where there's an unmerged other clone that has a sameas remote that the current repo does not know about. Then it would not get marked as dead. Handling it at transition time avoids that scenario. Note that the generation of trustmap' in dropDead should only happen once, due to the partial application.
2019-10-14 19:38:07 +00:00
import Logs.Remote.Pure
sameas RemoteConfig inheritance I found a way to avoid inheritance complicating anything outside of Logs.Remote. It seems fine to require all inherited values to be inherited and not set in the sameas remote's config. Since inherited values will be used for stuff like encryption and perhaps chunking, which control the actual content stored on the remote, it seems likely that there will not be any reason to need them to vary between two remotes that access the same underlying data store. The newer version of containers is free; the minimum ghc version is bundled with a newer version than that.
2019-10-10 19:46:12 +00:00
import Annex.SpecialRemote.Config
factor out RemoteLog
2011-07-06 00:16:57 +00:00
Added GIT_ANNEX_VECTOR_CLOCK environment variable Can be used to override the default timestamps used in log files in the git-annex branch. This is a dangerous environment variable; use with caution. Note that this only affects writing to the logs on the git-annex branch. It is not used for metadata in git commits (other env vars can be set for that). There are many other places where timestamps are still used, that don't get committed to git, but do touch disk. Including regular timestamps of files, and timestamps embedded in some files in .git/annex/, including the last fsck timestamp and timestamps in transfer log files. A good way to find such things in git-annex is to get for getPOSIXTime and getCurrentTime, although some of the results are of course false positives that never hit disk (unless git-annex gets swapped out..) So this commit does NOT necessarily make git-annex comply with some HIPPA privacy regulations; it's up to the user to determine if they can use it in a way compliant with such regulations. Benchmarking: It takes 0.00114 milliseconds to call getEnv "GIT_ANNEX_VECTOR_CLOCK" when that env var is not set. So, 100 thousand log files can be written with an added overhead of only 0.114 seconds. That should be by far swamped by the actual overhead of writing the log files and making the commit containing them. This commit was supported by the NSF-funded DataLad project.
2017-08-14 17:55:38 +00:00
import qualified Data.Map as M
factor out RemoteLog
2011-07-06 00:16:57 +00:00
{- Adds or updates a remote's config in the log. -}
configSet :: UUID -> RemoteConfig -> Annex ()
Added GIT_ANNEX_VECTOR_CLOCK environment variable Can be used to override the default timestamps used in log files in the git-annex branch. This is a dangerous environment variable; use with caution. Note that this only affects writing to the logs on the git-annex branch. It is not used for metadata in git commits (other env vars can be set for that). There are many other places where timestamps are still used, that don't get committed to git, but do touch disk. Including regular timestamps of files, and timestamps embedded in some files in .git/annex/, including the last fsck timestamp and timestamps in transfer log files. A good way to find such things in git-annex is to get for getPOSIXTime and getCurrentTime, although some of the results are of course false positives that never hit disk (unless git-annex gets swapped out..) So this commit does NOT necessarily make git-annex comply with some HIPPA privacy regulations; it's up to the user to determine if they can use it in a way compliant with such regulations. Benchmarking: It takes 0.00114 milliseconds to call getEnv "GIT_ANNEX_VECTOR_CLOCK" when that env var is not set. So, 100 thousand log files can be written with an added overhead of only 0.114 seconds. That should be by far swamped by the actual overhead of writing the log files and making the commit containing them. This commit was supported by the NSF-funded DataLad project.
2017-08-14 17:55:38 +00:00
configSet u cfg = do
c <- liftIO currentVectorClock
add timestamps to remote.log
2011-10-06 20:07:51 +00:00
Annex.Branch.change remoteLog $
make DropDead transition minimize remote.log for dead sameas remotes All that needs to be retained in remote.log is the sameas-uuid. The rest of the config is eliminated. This doesn't save enough space to bother with, but it prevents anything sensitive in the config of the dead sameas remote from lingering around. Note that minimizesameasdead does not update the VectorClock when changing the log line. That's normally a no-no, but in this case, it makes each DropDead result in the exact same file contents, and vector clocks are not needed because the transition breaks the history chain.
2019-10-15 15:33:33 +00:00
buildRemoteConfigLog
sameas RemoteConfig inheritance I found a way to avoid inheritance complicating anything outside of Logs.Remote. It seems fine to require all inherited values to be inherited and not set in the sameas remote's config. Since inherited values will be used for stuff like encryption and perhaps chunking, which control the actual content stored on the remote, it seems likely that there will not be any reason to need them to vary between two remotes that access the same underlying data store. The newer version of containers is free; the minimum ghc version is bundled with a newer version than that.
2019-10-10 19:46:12 +00:00
. changeLog c u (removeSameasInherited cfg)
make DropDead transition minimize remote.log for dead sameas remotes All that needs to be retained in remote.log is the sameas-uuid. The rest of the config is eliminated. This doesn't save enough space to bother with, but it prevents anything sensitive in the config of the dead sameas remote from lingering around. Note that minimizesameasdead does not update the VectorClock when changing the log line. That's normally a no-no, but in this case, it makes each DropDead result in the exact same file contents, and vector clocks are not needed because the transition breaks the history chain.
2019-10-15 15:33:33 +00:00
. parseRemoteConfigLog
factor out RemoteLog
2011-07-06 00:16:57 +00:00
{- Map of remotes by uuid containing key/value config maps. -}
readRemoteLog :: Annex (M.Map UUID RemoteConfig)
forget state of sameas remotes during DropDead transitions It would have been a lot less round-about to just make git annex dead also add the uuids of sameas remotes to the trust.log as dead. But, that would fail in the case where there's an unmerged other clone that has a sameas remote that the current repo does not know about. Then it would not get marked as dead. Handling it at transition time avoids that scenario. Note that the generation of trustmap' in dropDead should only happen once, due to the partial application.
2019-10-14 19:38:07 +00:00
readRemoteLog = calcRemoteConfigMap
convert old uuid-based log parsers to attoparsec This preserves the workaround for the old bug that caused NoUUID items to be stored in the log, prefixing log lines with " ". It's now handled implicitly, by using takeWhile1 (/= ' ') to get the uuid. There is a behavior change from the old parser, which split the value into words and then recombined it. That meant that "foo bar" and "foo\tbar" came out as "foo bar". That behavior was not documented, and seems surprising; it meant that after a git-annex describe here "foo bar", you wouldn't get that same string back out when git-annex displayed repo descriptions. Otoh, some other parsers relied on the old behavior, and the attoparsec rewrites had to deal with the issue themselves... For group.log, there are some edge cases around the user providing a group name with a leading or trailing space. The old parser would ignore such excess whitespace. The new parser does too, because the alternative is to refuse to parse something like " group1 group2 " due to excess whitespace, which would be even more confusing behavior. The only git-annex branch log file that is not converted to attoparsec and bytestring-builder now is transitions.log.
2019-01-10 18:39:36 +00:00
<$> Annex.Branch.get remoteLog
Reference in a new issue Copy permalink