2011-10-06 19:23:26 +00:00
|
|
|
{- git-annex uuid-based logs
|
|
|
|
-
|
2014-07-24 20:23:36 +00:00
|
|
|
- This is used to store information about UUIDs in a way that can
|
2011-10-06 19:23:26 +00:00
|
|
|
- be union merged.
|
|
|
|
-
|
2019-02-21 17:43:21 +00:00
|
|
|
- The old format looks like: "UUID[ INFO[ timestamp=foo]]"
|
2011-10-06 19:23:26 +00:00
|
|
|
- The timestamp is last for backwards compatability reasons,
|
2019-02-21 17:43:21 +00:00
|
|
|
- and may not be present on very old log lines.
|
add remote state logs
This allows a remote to store a piece of arbitrary state associated with a
key. This is needed to support Tahoe, where the file-cap is calculated from
the data stored in it, and used to retrieve a key later. Glacier also would
be much improved by using this.
GETSTATE and SETSTATE are added to the external special remote protocol.
Note that the state is left as-is even when a key is removed from a remote.
It's up to the remote to decide when it wants to clear the state.
The remote state log, $KEY.log.rmt, is a UUID-based log. However,
rather than using the old UUID-based log format, I created a new variant
of that format. The new varient is more space efficient (since it lacks the
"timestamp=" hack, and easier to parse (and the parser doesn't mess with
whitespace in the value), and avoids compatability cruft in the old one.
This seemed worth cleaning up for these new files, since there could be a
lot of them, while before UUID-based logs were only used for a few log
files at the top of the git-annex branch. The transition code has also
been updated to handle these new UUID-based logs.
This commit was sponsored by Daniel Hofer.
2014-01-03 20:35:57 +00:00
|
|
|
-
|
|
|
|
- New uuid based logs instead use the form: "timestamp UUID INFO"
|
2011-10-06 19:23:26 +00:00
|
|
|
-
|
2019-01-09 17:06:37 +00:00
|
|
|
- Copyright 2011-2019 Joey Hess <id@joeyh.name>
|
2011-10-06 19:23:26 +00:00
|
|
|
-
|
2019-03-13 19:48:14 +00:00
|
|
|
- Licensed under the GNU AGPL version 3 or higher.
|
2011-10-06 19:23:26 +00:00
|
|
|
-}
|
|
|
|
|
convert old uuid-based log parsers to attoparsec
This preserves the workaround for the old bug that caused NoUUID items
to be stored in the log, prefixing log lines with " ". It's now handled
implicitly, by using takeWhile1 (/= ' ') to get the uuid.
There is a behavior change from the old parser, which split the value
into words and then recombined it. That meant that "foo bar" and "foo\tbar"
came out as "foo bar". That behavior was not documented, and seems
surprising; it meant that after a git-annex describe here "foo bar",
you wouldn't get that same string back out when git-annex displayed repo
descriptions.
Otoh, some other parsers relied on the old behavior, and the attoparsec
rewrites had to deal with the issue themselves...
For group.log, there are some edge cases around the user providing a
group name with a leading or trailing space. The old parser would ignore
such excess whitespace. The new parser does too, because the alternative
is to refuse to parse something like " group1 group2 " due to excess
whitespace, which would be even more confusing behavior.
The only git-annex branch log file that is not converted to attoparsec
and bytestring-builder now is transitions.log.
2019-01-10 18:39:36 +00:00
|
|
|
{-# LANGUAGE OverloadedStrings, TupleSections #-}
|
2019-01-09 18:00:35 +00:00
|
|
|
|
2011-10-15 20:21:08 +00:00
|
|
|
module Logs.UUIDBased (
|
2011-10-06 19:23:26 +00:00
|
|
|
Log,
|
|
|
|
LogEntry(..),
|
2017-08-14 17:55:38 +00:00
|
|
|
VectorClock,
|
|
|
|
currentVectorClock,
|
2019-02-21 17:43:21 +00:00
|
|
|
parseLogOld,
|
add remote state logs
This allows a remote to store a piece of arbitrary state associated with a
key. This is needed to support Tahoe, where the file-cap is calculated from
the data stored in it, and used to retrieve a key later. Glacier also would
be much improved by using this.
GETSTATE and SETSTATE are added to the external special remote protocol.
Note that the state is left as-is even when a key is removed from a remote.
It's up to the remote to decide when it wants to clear the state.
The remote state log, $KEY.log.rmt, is a UUID-based log. However,
rather than using the old UUID-based log format, I created a new variant
of that format. The new varient is more space efficient (since it lacks the
"timestamp=" hack, and easier to parse (and the parser doesn't mess with
whitespace in the value), and avoids compatability cruft in the old one.
This seemed worth cleaning up for these new files, since there could be a
lot of them, while before UUID-based logs were only used for a few log
files at the top of the git-annex branch. The transition code has also
been updated to handle these new UUID-based logs.
This commit was sponsored by Daniel Hofer.
2014-01-03 20:35:57 +00:00
|
|
|
parseLogNew,
|
2019-02-21 17:43:21 +00:00
|
|
|
parseLogOldWithUUID,
|
|
|
|
buildLogOld,
|
2019-01-09 17:06:37 +00:00
|
|
|
buildLogNew,
|
2011-10-06 19:23:26 +00:00
|
|
|
changeLog,
|
|
|
|
addLog,
|
|
|
|
simpleMap,
|
|
|
|
) where
|
|
|
|
|
|
|
|
import qualified Data.Map as M
|
|
|
|
|
|
|
|
import Common
|
|
|
|
import Types.UUID
|
2017-08-14 17:55:38 +00:00
|
|
|
import Annex.VectorClock
|
2014-03-15 17:44:31 +00:00
|
|
|
import Logs.MapLog
|
2016-05-27 15:45:13 +00:00
|
|
|
import Logs.Line
|
2011-10-06 19:23:26 +00:00
|
|
|
|
convert old uuid-based log parsers to attoparsec
This preserves the workaround for the old bug that caused NoUUID items
to be stored in the log, prefixing log lines with " ". It's now handled
implicitly, by using takeWhile1 (/= ' ') to get the uuid.
There is a behavior change from the old parser, which split the value
into words and then recombined it. That meant that "foo bar" and "foo\tbar"
came out as "foo bar". That behavior was not documented, and seems
surprising; it meant that after a git-annex describe here "foo bar",
you wouldn't get that same string back out when git-annex displayed repo
descriptions.
Otoh, some other parsers relied on the old behavior, and the attoparsec
rewrites had to deal with the issue themselves...
For group.log, there are some edge cases around the user providing a
group name with a leading or trailing space. The old parser would ignore
such excess whitespace. The new parser does too, because the alternative
is to refuse to parse something like " group1 group2 " due to excess
whitespace, which would be even more confusing behavior.
The only git-annex branch log file that is not converted to attoparsec
and bytestring-builder now is transitions.log.
2019-01-10 18:39:36 +00:00
|
|
|
import qualified Data.ByteString as S
|
2019-01-10 17:23:42 +00:00
|
|
|
import qualified Data.ByteString.Lazy as L
|
2021-03-24 16:11:50 +00:00
|
|
|
import qualified Data.Attoparsec.ByteString as A
|
|
|
|
import qualified Data.Attoparsec.ByteString.Lazy as AL
|
convert old uuid-based log parsers to attoparsec
This preserves the workaround for the old bug that caused NoUUID items
to be stored in the log, prefixing log lines with " ". It's now handled
implicitly, by using takeWhile1 (/= ' ') to get the uuid.
There is a behavior change from the old parser, which split the value
into words and then recombined it. That meant that "foo bar" and "foo\tbar"
came out as "foo bar". That behavior was not documented, and seems
surprising; it meant that after a git-annex describe here "foo bar",
you wouldn't get that same string back out when git-annex displayed repo
descriptions.
Otoh, some other parsers relied on the old behavior, and the attoparsec
rewrites had to deal with the issue themselves...
For group.log, there are some edge cases around the user providing a
group name with a leading or trailing space. The old parser would ignore
such excess whitespace. The new parser does too, because the alternative
is to refuse to parse something like " group1 group2 " due to excess
whitespace, which would be even more confusing behavior.
The only git-annex branch log file that is not converted to attoparsec
and bytestring-builder now is transitions.log.
2019-01-10 18:39:36 +00:00
|
|
|
import qualified Data.Attoparsec.ByteString.Char8 as A8
|
2019-01-09 17:06:37 +00:00
|
|
|
import Data.ByteString.Builder
|
convert old uuid-based log parsers to attoparsec
This preserves the workaround for the old bug that caused NoUUID items
to be stored in the log, prefixing log lines with " ". It's now handled
implicitly, by using takeWhile1 (/= ' ') to get the uuid.
There is a behavior change from the old parser, which split the value
into words and then recombined it. That meant that "foo bar" and "foo\tbar"
came out as "foo bar". That behavior was not documented, and seems
surprising; it meant that after a git-annex describe here "foo bar",
you wouldn't get that same string back out when git-annex displayed repo
descriptions.
Otoh, some other parsers relied on the old behavior, and the attoparsec
rewrites had to deal with the issue themselves...
For group.log, there are some edge cases around the user providing a
group name with a leading or trailing space. The old parser would ignore
such excess whitespace. The new parser does too, because the alternative
is to refuse to parse something like " group1 group2 " due to excess
whitespace, which would be even more confusing behavior.
The only git-annex branch log file that is not converted to attoparsec
and bytestring-builder now is transitions.log.
2019-01-10 18:39:36 +00:00
|
|
|
import qualified Data.DList as D
|
2019-01-09 17:06:37 +00:00
|
|
|
|
2014-03-15 17:44:31 +00:00
|
|
|
type Log v = MapLog UUID v
|
2011-10-06 19:23:26 +00:00
|
|
|
|
2019-02-21 17:43:21 +00:00
|
|
|
buildLogOld :: (v -> Builder) -> Log v -> Builder
|
|
|
|
buildLogOld builder = mconcat . map genline . M.toList
|
2012-11-11 04:51:07 +00:00
|
|
|
where
|
2019-01-09 18:00:35 +00:00
|
|
|
genline (u, LogEntry c@(VectorClock {}) v) =
|
|
|
|
buildUUID u <> sp <> builder v <> sp <>
|
|
|
|
byteString "timestamp=" <> buildVectorClock c <> nl
|
|
|
|
genline (u, LogEntry Unknown v) =
|
|
|
|
buildUUID u <> sp <> builder v <> nl
|
|
|
|
sp = charUtf8 ' '
|
|
|
|
nl = charUtf8 '\n'
|
2011-10-06 19:23:26 +00:00
|
|
|
|
2019-02-21 17:43:21 +00:00
|
|
|
parseLogOld :: A.Parser a -> L.ByteString -> Log a
|
|
|
|
parseLogOld = parseLogOldWithUUID . const
|
2012-10-10 17:52:24 +00:00
|
|
|
|
2019-02-21 17:43:21 +00:00
|
|
|
parseLogOldWithUUID :: (UUID -> A.Parser a) -> L.ByteString -> Log a
|
2021-03-24 16:11:50 +00:00
|
|
|
parseLogOldWithUUID parser = fromMaybe M.empty . AL.maybeResult
|
|
|
|
. AL.parse (logParserOld parser)
|
convert old uuid-based log parsers to attoparsec
This preserves the workaround for the old bug that caused NoUUID items
to be stored in the log, prefixing log lines with " ". It's now handled
implicitly, by using takeWhile1 (/= ' ') to get the uuid.
There is a behavior change from the old parser, which split the value
into words and then recombined it. That meant that "foo bar" and "foo\tbar"
came out as "foo bar". That behavior was not documented, and seems
surprising; it meant that after a git-annex describe here "foo bar",
you wouldn't get that same string back out when git-annex displayed repo
descriptions.
Otoh, some other parsers relied on the old behavior, and the attoparsec
rewrites had to deal with the issue themselves...
For group.log, there are some edge cases around the user providing a
group name with a leading or trailing space. The old parser would ignore
such excess whitespace. The new parser does too, because the alternative
is to refuse to parse something like " group1 group2 " due to excess
whitespace, which would be even more confusing behavior.
The only git-annex branch log file that is not converted to attoparsec
and bytestring-builder now is transitions.log.
2019-01-10 18:39:36 +00:00
|
|
|
|
2019-02-21 17:43:21 +00:00
|
|
|
logParserOld :: (UUID -> A.Parser a) -> A.Parser (Log a)
|
|
|
|
logParserOld parser = M.fromListWith best <$> parseLogLines go
|
2012-11-11 04:51:07 +00:00
|
|
|
where
|
convert old uuid-based log parsers to attoparsec
This preserves the workaround for the old bug that caused NoUUID items
to be stored in the log, prefixing log lines with " ". It's now handled
implicitly, by using takeWhile1 (/= ' ') to get the uuid.
There is a behavior change from the old parser, which split the value
into words and then recombined it. That meant that "foo bar" and "foo\tbar"
came out as "foo bar". That behavior was not documented, and seems
surprising; it meant that after a git-annex describe here "foo bar",
you wouldn't get that same string back out when git-annex displayed repo
descriptions.
Otoh, some other parsers relied on the old behavior, and the attoparsec
rewrites had to deal with the issue themselves...
For group.log, there are some edge cases around the user providing a
group name with a leading or trailing space. The old parser would ignore
such excess whitespace. The new parser does too, because the alternative
is to refuse to parse something like " group1 group2 " due to excess
whitespace, which would be even more confusing behavior.
The only git-annex branch log file that is not converted to attoparsec
and bytestring-builder now is transitions.log.
2019-01-10 18:39:36 +00:00
|
|
|
go = do
|
|
|
|
u <- toUUID <$> A8.takeWhile1 (/= ' ')
|
|
|
|
(dl, ts) <- accumval D.empty
|
|
|
|
v <- either fail return $ A.parseOnly (parser u <* A.endOfInput)
|
|
|
|
(S.intercalate " " $ D.toList dl)
|
|
|
|
return (u, LogEntry ts v)
|
|
|
|
accumval dl =
|
|
|
|
((dl,) <$> parsetimestamp)
|
|
|
|
<|> (A8.char ' ' *> (A8.takeWhile (/= ' ')) >>= accumval . D.snoc dl)
|
|
|
|
parsetimestamp =
|
|
|
|
(A8.string " timestamp=" *> vectorClockParser <* A.endOfInput)
|
|
|
|
<|> (const Unknown <$> A.endOfInput)
|
2011-10-06 19:23:26 +00:00
|
|
|
|
2019-01-09 17:06:37 +00:00
|
|
|
buildLogNew :: (v -> Builder) -> Log v -> Builder
|
2019-01-09 18:00:35 +00:00
|
|
|
buildLogNew = buildMapLog buildUUID
|
2011-10-06 19:23:26 +00:00
|
|
|
|
2019-01-10 17:23:42 +00:00
|
|
|
parseLogNew :: A.Parser v -> L.ByteString -> Log v
|
|
|
|
parseLogNew = parseMapLog (toUUID <$> A.takeByteString)
|
2011-10-06 19:23:26 +00:00
|
|
|
|
2017-08-14 17:55:38 +00:00
|
|
|
changeLog :: VectorClock -> UUID -> v -> Log v -> Log v
|
2014-03-15 17:44:31 +00:00
|
|
|
changeLog = changeMapLog
|
2011-10-06 19:23:26 +00:00
|
|
|
|
2014-03-15 17:44:31 +00:00
|
|
|
addLog :: UUID -> LogEntry v -> Log v -> Log v
|
|
|
|
addLog = addMapLog
|