mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Logs/Chunk/Pure.hs

63 lines
1.6 KiB
Haskell
Raw Normal View History

implement chunk logs Slightly tricky as they are not normal UUIDBased logs, but are instead maps from (uuid, chunksize) to chunkcount. This commit was sponsored by Frank Thomas.
2014-07-24 20:23:36 +00:00
{- Chunk logs, pure operations.
-
converting more log files to use Builder Probably not any particular speedup in this, since most of these logs are not written to often. Possibly chunk log writing is sped up, but writes to chunk logs are interleaved with expensive data transfers to remotes, so unlikely to be a noticiable speedup.
2019-01-09 17:06:37 +00:00
- Copyright 2014, 2019 Joey Hess <id@joeyh.name>
implement chunk logs Slightly tricky as they are not normal UUIDBased logs, but are instead maps from (uuid, chunksize) to chunkcount. This commit was sponsored by Frank Thomas.
2014-07-24 20:23:36 +00:00
-
update licenses from GPL to AGPL This does not change the overall license of the git-annex program, which was already AGPL due to a number of sources files being AGPL already. Legally speaking, I'm adding a new license under which these files are now available; I already released their current contents under the GPL license. Now they're dual licensed GPL and AGPL. However, I intend for all my future changes to these files to only be released under the AGPL license, and I won't be tracking the dual licensing status, so I'm simply changing the license statement to say it's AGPL. (In some cases, others wrote parts of the code of a file and released it under the GPL; but in all cases I have contributed a significant portion of the code in each file and it's that code that is getting the AGPL license; the GPL license of other contributors allows combining with AGPL code.)
2019-03-13 19:48:14 +00:00
- Licensed under the GNU AGPL version 3 or higher.
implement chunk logs Slightly tricky as they are not normal UUIDBased logs, but are instead maps from (uuid, chunksize) to chunkcount. This commit was sponsored by Frank Thomas.
2014-07-24 20:23:36 +00:00
-}
improve a bit
2014-07-24 21:18:14 +00:00
module Logs.Chunk.Pure
add ChunkMethod type and make Logs.Chunk use it, rather than assuming fixed size chunks (so eg, rolling hash chunks can be supported later) If a newer git-annex starts logging something else in the chunk log, it won't be used by this version, but it will be preserved when updating the log.
2014-07-28 17:19:08 +00:00
( ChunkMethod(..)
, ChunkSize
improve a bit
2014-07-24 21:18:14 +00:00
, ChunkCount
, ChunkLog
, parseLog
converting more log files to use Builder Probably not any particular speedup in this, since most of these logs are not written to often. Possibly chunk log writing is sped up, but writes to chunk logs are interleaved with expensive data transfers to remotes, so unlikely to be a noticiable speedup.
2019-01-09 17:06:37 +00:00
, buildLog
improve a bit
2014-07-24 21:18:14 +00:00
) where
implement chunk logs Slightly tricky as they are not normal UUIDBased logs, but are instead maps from (uuid, chunksize) to chunkcount. This commit was sponsored by Frank Thomas.
2014-07-24 20:23:36 +00:00
remove 163 lines of code without changing anything except imports
2016-01-20 20:36:33 +00:00
import Annex.Common
implement chunk logs Slightly tricky as they are not normal UUIDBased logs, but are instead maps from (uuid, chunksize) to chunkcount. This commit was sponsored by Frank Thomas.
2014-07-24 20:23:36 +00:00
import Logs.MapLog
import Data.Int
converting more log files to use Builder Probably not any particular speedup in this, since most of these logs are not written to often. Possibly chunk log writing is sped up, but writes to chunk logs are interleaved with expensive data transfers to remotes, so unlikely to be a noticiable speedup.
2019-01-09 17:06:37 +00:00
import qualified Data.ByteString as S
attoparsec parsers for all new-format uuid-based logs There should be some speed gains here, especially for chunk and remote state logs, which are queried once per key. Now only old-format uuid-based logs still need to be converted to attoparsec.
2019-01-10 17:23:42 +00:00
import qualified Data.ByteString.Lazy as L
import qualified Data.Attoparsec.ByteString.Lazy as A
import qualified Data.Attoparsec.ByteString.Char8 as A8
converting more log files to use Builder Probably not any particular speedup in this, since most of these logs are not written to often. Possibly chunk log writing is sped up, but writes to chunk logs are interleaved with expensive data transfers to remotes, so unlikely to be a noticiable speedup.
2019-01-09 17:06:37 +00:00
import Data.ByteString.Builder
add ChunkMethod type and make Logs.Chunk use it, rather than assuming fixed size chunks (so eg, rolling hash chunks can be supported later) If a newer git-annex starts logging something else in the chunk log, it won't be used by this version, but it will be preserved when updating the log.
2014-07-28 17:19:08 +00:00
-- Currently chunks are all fixed size, but other chunking methods
-- may be added.
converting more log files to use Builder Probably not any particular speedup in this, since most of these logs are not written to often. Possibly chunk log writing is sped up, but writes to chunk logs are interleaved with expensive data transfers to remotes, so unlikely to be a noticiable speedup.
2019-01-09 17:06:37 +00:00
data ChunkMethod = FixedSizeChunks ChunkSize | UnknownChunks S.ByteString
deriving Show
2014-08-01 20:30:23 +00:00
deriving (Ord, Eq, Show)
add ChunkMethod type and make Logs.Chunk use it, rather than assuming fixed size chunks (so eg, rolling hash chunks can be supported later) If a newer git-annex starts logging something else in the chunk log, it won't be used by this version, but it will be preserved when updating the log.
2014-07-28 17:19:08 +00:00
implement chunk logs Slightly tricky as they are not normal UUIDBased logs, but are instead maps from (uuid, chunksize) to chunkcount. This commit was sponsored by Frank Thomas.
2014-07-24 20:23:36 +00:00
type ChunkSize = Int64
add ChunkMethod type and make Logs.Chunk use it, rather than assuming fixed size chunks (so eg, rolling hash chunks can be supported later) If a newer git-annex starts logging something else in the chunk log, it won't be used by this version, but it will be preserved when updating the log.
2014-07-28 17:19:08 +00:00
-- 0 when chunks are no longer present
implement chunk logs Slightly tricky as they are not normal UUIDBased logs, but are instead maps from (uuid, chunksize) to chunkcount. This commit was sponsored by Frank Thomas.
2014-07-24 20:23:36 +00:00
type ChunkCount = Integer
add ChunkMethod type and make Logs.Chunk use it, rather than assuming fixed size chunks (so eg, rolling hash chunks can be supported later) If a newer git-annex starts logging something else in the chunk log, it won't be used by this version, but it will be preserved when updating the log.
2014-07-28 17:19:08 +00:00
type ChunkLog = MapLog (UUID, ChunkMethod) ChunkCount
converting more log files to use Builder Probably not any particular speedup in this, since most of these logs are not written to often. Possibly chunk log writing is sped up, but writes to chunk logs are interleaved with expensive data transfers to remotes, so unlikely to be a noticiable speedup.
2019-01-09 17:06:37 +00:00
buildChunkMethod :: ChunkMethod -> Builder
buildChunkMethod (FixedSizeChunks sz) = int64Dec sz
buildChunkMethod (UnknownChunks s) = byteString s
implement chunk logs Slightly tricky as they are not normal UUIDBased logs, but are instead maps from (uuid, chunksize) to chunkcount. This commit was sponsored by Frank Thomas.
2014-07-24 20:23:36 +00:00
attoparsec parsers for all new-format uuid-based logs There should be some speed gains here, especially for chunk and remote state logs, which are queried once per key. Now only old-format uuid-based logs still need to be converted to attoparsec.
2019-01-10 17:23:42 +00:00
chunkMethodParser :: A.Parser ChunkMethod
chunkMethodParser =
(FixedSizeChunks <$> A8.decimal) <|> (UnknownChunks <$> A.takeByteString)
implement chunk logs Slightly tricky as they are not normal UUIDBased logs, but are instead maps from (uuid, chunksize) to chunkcount. This commit was sponsored by Frank Thomas.
2014-07-24 20:23:36 +00:00
converting more log files to use Builder Probably not any particular speedup in this, since most of these logs are not written to often. Possibly chunk log writing is sped up, but writes to chunk logs are interleaved with expensive data transfers to remotes, so unlikely to be a noticiable speedup.
2019-01-09 17:06:37 +00:00
buildLog :: ChunkLog -> Builder
buildLog = buildMapLog fieldbuilder valuebuilder
implement chunk logs Slightly tricky as they are not normal UUIDBased logs, but are instead maps from (uuid, chunksize) to chunkcount. This commit was sponsored by Frank Thomas.
2014-07-24 20:23:36 +00:00
where
convert all per-uuid log files to use Builder Mostly didn't push the ByteStrings down very deep, but all of these log files are not written to frequently at all, so slight remaining innefficiency doesn't matter. In Logs.UUID, removed the fixBadUUID code that cleaned up after a bug in git-annex versions 3.20111105-3.20111110. In the unlikely event that a repo was last touched by that ancient git-annex version, the descriptions of remotes would appear missing when used with this version of git-annex. That is such minor breakage, and so unlikely to still be a problem for any repos, that it was not worth forward-porting that code to ByteString.
2019-01-09 18:00:35 +00:00
fieldbuilder (u, m) = buildUUID u <> sep <> buildChunkMethod m
converting more log files to use Builder Probably not any particular speedup in this, since most of these logs are not written to often. Possibly chunk log writing is sped up, but writes to chunk logs are interleaved with expensive data transfers to remotes, so unlikely to be a noticiable speedup.
2019-01-09 17:06:37 +00:00
valuebuilder = integerDec
sep = charUtf8 ':'
attoparsec parsers for all new-format uuid-based logs There should be some speed gains here, especially for chunk and remote state logs, which are queried once per key. Now only old-format uuid-based logs still need to be converted to attoparsec.
2019-01-10 17:23:42 +00:00
parseLog :: L.ByteString -> ChunkLog
parseLog = parseMapLog fieldparser valueparser
where
fieldparser = (,)
<$> (toUUID <$> A8.takeTill (== ':'))
<* A8.char ':'
<*> chunkMethodParser
<* A.endOfInput
valueparser = A8.decimal
Reference in a new issue Copy permalink