mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Types/Key.hs

364 lines
13 KiB
Haskell
Raw Normal View History

a new Key data type with metadata
2011-03-15 21:47:29 +00:00
{- git-annex Key data type
-
support VURL backend Not yet implemented is recording hashes on download from web and verifying hashes. addurl --verifiable option added with -V short option because I expect a lot of people will want to use this. It seems likely that --verifiable will become the default eventually, and possibly rather soon. While old git-annex versions don't support VURL, that doesn't prevent using them with keys that use VURL. Of course, they won't verify the content on transfer, and fsck will warn that it doesn't know about VURL. So there's not much problem with starting to use VURL even when interoperating with old versions. Sponsored-by: Joshua Antonishen on Patreon
2024-02-29 17:26:06 +00:00
- Copyright 2011-2024 Joey Hess <id@joeyh.name>
a new Key data type with metadata
2011-03-15 21:47:29 +00:00
-
update licenses from GPL to AGPL This does not change the overall license of the git-annex program, which was already AGPL due to a number of sources files being AGPL already. Legally speaking, I'm adding a new license under which these files are now available; I already released their current contents under the GPL license. Now they're dual licensed GPL and AGPL. However, I intend for all my future changes to these files to only be released under the AGPL license, and I won't be tracking the dual licensing status, so I'm simply changing the license statement to say it's AGPL. (In some cases, others wrote parts of the code of a file and released it under the GPL; but in all cases I have contributed a significant portion of the code in each file and it's that code that is getting the AGPL license; the GPL license of other contributors allows combining with AGPL code.)
2019-03-13 19:48:14 +00:00
- Licensed under the GNU AGPL version 3 or higher.
a new Key data type with metadata
2011-03-15 21:47:29 +00:00
-}
benchmark: Add --databases to benchmark sqlite databases Rescued from commit 11d6e2e260d70ba99e35464c19c2b2772ce9efaa which removed db benchmarks in favor of benchmarking arbitrary git-annex commands. Which is nice and general, but microbenchmarks are useful too.
2019-10-29 19:16:15 +00:00
{-# LANGUAGE OverloadedStrings, DeriveGeneric #-}
fix build with cryptonite-0.20 Some blake hash varieties were not yet available in that version. Rather than tracking exact details of what cryptonite supported when, disable blake unless using a current cryptonite.
2018-03-15 15:16:00 +00:00
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
module Types.Key (
KeyData(..),
Key,
fromKey,
new protocol for transferkeys, with message serialization Necessarily threw out the old protocol, so if an old git-annex assistant is running, and starts a transferkeys from the new git-annex, it would fail. But, that seems unlikely; the assistant starts up transferkeys processes and then keeps them running. Still, may need to test that scenario. The new protocol is simple read/show and looks like this: TransferRequest Download (Right "origin") (Key {keyName = "f8f8766a836fb6120abf4d5328ce8761404e437529e997aaa0363bdd4fecd7bb", keyVariety = SHA2Key (HashSize 256) (HasExt True), keySize = Just 30, keyMtime = Nothing, keyChunkSize = Nothing, keyChunkNum = Nothing}) (AssociatedFile (Just "foo")) TransferOutput (ProgressMeter (Just 30) (MeterState {meterBytesProcessed = BytesProcessed 0, meterTimeStamp = 1.6070268727892535e9}) (MeterState {meterBytesProcessed = BytesProcessed 30, meterTimeStamp = 1.6070268728043e9})) TransferOutput (OutputMessage "(checksum...) ") TransferResult True Granted, this is not optimally fast, but it seems good enough, and is probably nearly as fast as the old protocol anyhow. emitSerializedOutput for ProgressMeter is not yet implemented. It needs to somehow start or update a progress meter. There may need to be a new message that allocates a progress meter, and then have ProgressMeter update it. This commit was sponsored by Ethan Aubin
2020-12-03 20:21:20 +00:00
keyData,
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
mkKey,
alterKey,
isKeyPrefix,
splitKeyNameExtension,
keyParser,
keySerialization,
AssociatedFile(..),
KeyVariety(..),
HasExt(..),
HashSize(..),
hasExt,
sameExceptExt,
formatKeyVariety,
parseKeyVariety,
) where
a new Key data type with metadata
2011-03-15 21:47:29 +00:00
use ByteString in Key for speed This is an easy win for parseKeyVariety: benchmarking old/parseKeyVariety time 1.515 μs (1.512 μs .. 1.517 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.515 μs (1.513 μs .. 1.517 μs) std dev 6.417 ns (4.992 ns .. 8.113 ns) benchmarking new/parseKeyVariety time 54.97 ns (54.70 ns .. 55.40 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 55.42 ns (55.05 ns .. 56.03 ns) std dev 1.562 ns (969.5 ps .. 2.442 ns) variance introduced by outliers: 44% (moderately inflated) For formatKeyVariety, using a Builder is marginally worse than building a String... (This is with criterion evaluating fully to nf not whnf) benchmarking old/formatKeyVariety time 434.3 ns (428.0 ns .. 440.4 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 430.6 ns (428.2 ns .. 433.9 ns) std dev 9.166 ns (6.932 ns .. 11.94 ns) variance introduced by outliers: 27% (moderately inflated) benchmarking Builder/formatKeyVariety time 526.5 ns (524.7 ns .. 528.8 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 526.1 ns (524.9 ns .. 528.5 ns) std dev 5.687 ns (3.762 ns .. 8.000 ns) Manually building the ByteString was better, but still slightly slower than String, due to innefficient need to S.pack . show the HashSize: benchmarking formatKeyVariety time 459.5 ns (455.8 ns .. 463.2 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 459.9 ns (457.4 ns .. 466.6 ns) std dev 11.65 ns (6.860 ns .. 21.41 ns) variance introduced by outliers: 35% (moderately inflated) So I cheated and made parseKeyVariety cache the original ByteString, for formatKeyVariety to use instead of re-building it. Final benchmark: benchmarking new/formatKeyVariety time 50.64 ns (50.57 ns .. 50.73 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 51.05 ns (50.60 ns .. 52.71 ns) std dev 2.790 ns (259.6 ps .. 5.916 ns) variance introduced by outliers: 75% (severely inflated) benchmarking new/parseKeyVariety time 71.88 ns (71.54 ns .. 72.24 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 71.97 ns (71.69 ns .. 72.47 ns) std dev 1.249 ns (910.7 ps .. 1.791 ns) variance introduced by outliers: 22% (moderately inflated)
2019-01-11 17:55:00 +00:00
import qualified Data.ByteString as S
convert Key to ShortByteString This adds the overhead of a copy when serializing and deserializing keys. I have not benchmarked much, but runtimes seem barely changed at all by that. When a lot of keys are in memory, it improves memory use. And, it prevents keys sometimes getting PINNED in memory and failing to GC, which is a problem ByteString has sometimes. In particular, git-annex sync from a borg special remote had that problem and this improved its memory use by a large amount. Sponsored-by: Shae Erisson on Patreon
2021-10-06 00:20:08 +00:00
import qualified Data.ByteString.Short as S (ShortByteString, toShort, fromShort)
use ByteString in Key for speed This is an easy win for parseKeyVariety: benchmarking old/parseKeyVariety time 1.515 μs (1.512 μs .. 1.517 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.515 μs (1.513 μs .. 1.517 μs) std dev 6.417 ns (4.992 ns .. 8.113 ns) benchmarking new/parseKeyVariety time 54.97 ns (54.70 ns .. 55.40 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 55.42 ns (55.05 ns .. 56.03 ns) std dev 1.562 ns (969.5 ps .. 2.442 ns) variance introduced by outliers: 44% (moderately inflated) For formatKeyVariety, using a Builder is marginally worse than building a String... (This is with criterion evaluating fully to nf not whnf) benchmarking old/formatKeyVariety time 434.3 ns (428.0 ns .. 440.4 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 430.6 ns (428.2 ns .. 433.9 ns) std dev 9.166 ns (6.932 ns .. 11.94 ns) variance introduced by outliers: 27% (moderately inflated) benchmarking Builder/formatKeyVariety time 526.5 ns (524.7 ns .. 528.8 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 526.1 ns (524.9 ns .. 528.5 ns) std dev 5.687 ns (3.762 ns .. 8.000 ns) Manually building the ByteString was better, but still slightly slower than String, due to innefficient need to S.pack . show the HashSize: benchmarking formatKeyVariety time 459.5 ns (455.8 ns .. 463.2 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 459.9 ns (457.4 ns .. 466.6 ns) std dev 11.65 ns (6.860 ns .. 21.41 ns) variance introduced by outliers: 35% (moderately inflated) So I cheated and made parseKeyVariety cache the original ByteString, for formatKeyVariety to use instead of re-building it. Final benchmark: benchmarking new/formatKeyVariety time 50.64 ns (50.57 ns .. 50.73 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 51.05 ns (50.60 ns .. 52.71 ns) std dev 2.790 ns (259.6 ps .. 5.916 ns) variance introduced by outliers: 75% (severely inflated) benchmarking new/parseKeyVariety time 71.88 ns (71.54 ns .. 72.24 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 71.97 ns (71.69 ns .. 72.47 ns) std dev 1.249 ns (910.7 ps .. 1.791 ns) variance introduced by outliers: 22% (moderately inflated)
2019-01-11 17:55:00 +00:00
import qualified Data.ByteString.Char8 as S8
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
import qualified Data.ByteString.Lazy as L
import Data.ByteString.Builder
import Data.ByteString.Builder.Extra
import qualified Data.Attoparsec.ByteString as A
import qualified Data.Attoparsec.ByteString.Char8 as A8
wip RawFilePath 2x git-annex find speedup Finally builds (oh the agoncy of making it build), but still very unmergable, only Command.Find is included and lots of stuff is badly hacked to make it compile. Benchmarking vs master, this git-annex find is significantly faster! Specifically: num files old new speedup 48500 4.77 3.73 28% 12500 1.36 1.02 66% 20 0.075 0.074 0% (so startup time is unchanged) That's without really finishing the optimization. Things still to do: * Eliminate all the fromRawFilePath, toRawFilePath, encodeBS, decodeBS conversions. * Use versions of IO actions like getFileStatus that take a RawFilePath. * Eliminate some Data.ByteString.Lazy.toStrict, which is a slow copy. * Use ByteString for parsing git config to speed up startup. It's likely several of those will speed up git-annex find further. And other commands will certianly benefit even more.
2019-11-26 19:27:22 +00:00
import Utility.FileSystemEncoding
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
import Data.List
external backends wip It's able to start them up, the only thing not implemented is generating and verifying keys. And, the key translation for HasExt.
2020-07-29 19:23:18 +00:00
import Data.Char
first pass at using new keys It compiles. It sorta works. Several subcommands are FIXME marked and broken, because things that used to accept separate --backend and --key params need to be changed to accept just a --key that encodes all the key info, now that there is metadata in keys.
2011-03-16 01:34:13 +00:00
import System.Posix.Types
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
import Foreign.C.Types
fix build with old ghc
2019-01-18 17:59:29 +00:00
import Data.Monoid
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
import Control.Applicative
benchmark: Add --databases to benchmark sqlite databases Rescued from commit 11d6e2e260d70ba99e35464c19c2b2772ce9efaa which removed db benchmarks in favor of benchmarking arbitrary git-annex commands. Which is nice and general, but microbenchmarks are useful too.
2019-10-29 19:16:15 +00:00
import GHC.Generics
import Control.DeepSeq
fix build with old ghc
2019-01-18 17:59:29 +00:00
import Prelude
factor out common imports no code changes
2011-10-04 02:24:57 +00:00
moved AssociatedFile definition
2013-07-04 06:36:02 +00:00
{- A Key has a unique name, which is derived from a particular backend,
first pass at using new keys It compiles. It sorta works. Several subcommands are FIXME marked and broken, because things that used to accept separate --backend and --key params need to be changed to accept just a --key that encodes all the key info, now that there is metadata in keys.
2011-03-16 01:34:13 +00:00
- and may contain other optional metadata. -}
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
data KeyData = Key
convert Key to ShortByteString This adds the overhead of a copy when serializing and deserializing keys. I have not benchmarked much, but runtimes seem barely changed at all by that. When a lot of keys are in memory, it improves memory use. And, it prevents keys sometimes getting PINNED in memory and failing to GC, which is a problem ByteString has sometimes. In particular, git-annex sync from a borg special remote had that problem and this improved its memory use by a large amount. Sponsored-by: Shae Erisson on Patreon
2021-10-06 00:20:08 +00:00
{ keyName :: S.ShortByteString
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
, keyVariety :: KeyVariety
layout
2013-07-04 06:45:46 +00:00
, keySize :: Maybe Integer
, keyMtime :: Maybe EpochTime
add chunk metadata to Key Added new fields for chunk number, and chunk size. These will not appear in normal keys ever, but will be used for chunked data stored on special remotes. This commit was sponsored by Jouni K Seppanen.
2014-07-24 17:36:23 +00:00
, keyChunkSize :: Maybe Integer
, keyChunkNum :: Maybe Integer
benchmark: Add --databases to benchmark sqlite databases Rescued from commit 11d6e2e260d70ba99e35464c19c2b2772ce9efaa which removed db benchmarks in favor of benchmarking arbitrary git-annex commands. Which is nice and general, but microbenchmarks are useful too.
2019-10-29 19:16:15 +00:00
} deriving (Eq, Ord, Read, Show, Generic)
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
instance NFData KeyData
run codespell throughout fixing typos automagically === Do not change lines below === { "chain": [], "cmd": "codespell -w", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^
2024-04-06 13:50:58 +00:00
{- Caching the serialization of a key is an optimization.
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
-
- This constructor is not exported, and all smart constructors maintain
- the serialization.
-}
data Key = MkKey
{ keyData :: KeyData
convert Key to ShortByteString This adds the overhead of a copy when serializing and deserializing keys. I have not benchmarked much, but runtimes seem barely changed at all by that. When a lot of keys are in memory, it improves memory use. And, it prevents keys sometimes getting PINNED in memory and failing to GC, which is a problem ByteString has sometimes. In particular, git-annex sync from a borg special remote had that problem and this improved its memory use by a large amount. Sponsored-by: Shae Erisson on Patreon
2021-10-06 00:20:08 +00:00
, keySerialization :: S.ShortByteString
git-annex sim command is working Had to add Read instances to Key and NumCopies and some other similar types. I only expect to use those in serializing a sim. Of course, this risks that implementation changes break reading old data. For a sim, that would not be a big problem.
2024-09-12 20:07:44 +00:00
} deriving (Show, Read, Generic)
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
instance Eq Key where
Typo fix unncessary -> unnecessary. Detected while reading recent CHANGELOG entry but then decided to apply to entire codebase and docs since why not?
2022-08-19 21:45:04 +00:00
-- comparing the serialization would be unnecessary work
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
a == b = keyData a == keyData b
instance Ord Key where
compare a b = compare (keyData a) (keyData b)
benchmark: Add --databases to benchmark sqlite databases Rescued from commit 11d6e2e260d70ba99e35464c19c2b2772ce9efaa which removed db benchmarks in favor of benchmarking arbitrary git-annex commands. Which is nice and general, but microbenchmarks are useful too.
2019-10-29 19:16:15 +00:00
instance NFData Key
a new Key data type with metadata
2011-03-15 21:47:29 +00:00
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
{- Access a field of data from the KeyData. -}
{-# INLINE fromKey #-}
fromKey :: (KeyData -> a) -> Key -> a
fromKey f = f . keyData
{- Smart constructor for a Key. The provided KeyData has all values empty. -}
mkKey :: (KeyData -> KeyData) -> Key
mkKey f =
let d = f stub
in MkKey d (mkKeySerialization d)
where
stub = Key
{ keyName = mempty
, keyVariety = OtherKey mempty
, keySize = Nothing
, keyMtime = Nothing
, keyChunkSize = Nothing
, keyChunkNum = Nothing
}
{- Alter a Key's data. -}
alterKey :: Key -> (KeyData -> KeyData) -> Key
alterKey k f =
let d = f (keyData k)
in MkKey d (mkKeySerialization d)
-- Checks if a string looks like at least the start of a key.
isKeyPrefix :: String -> Bool
isKeyPrefix s = [fieldSep, fieldSep] `isInfixOf` s
fieldSep :: Char
fieldSep = '-'
convert Key to ShortByteString This adds the overhead of a copy when serializing and deserializing keys. I have not benchmarked much, but runtimes seem barely changed at all by that. When a lot of keys are in memory, it improves memory use. And, it prevents keys sometimes getting PINNED in memory and failing to GC, which is a problem ByteString has sometimes. In particular, git-annex sync from a borg special remote had that problem and this improved its memory use by a large amount. Sponsored-by: Shae Erisson on Patreon
2021-10-06 00:20:08 +00:00
mkKeySerialization :: KeyData -> S.ShortByteString
mkKeySerialization = S.toShort . L.toStrict
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
. toLazyByteStringWith (safeStrategy 128 smallChunkSize) L.empty
. buildKeyData
{- Builds a ByteString from a KeyData.
-
- The name field is always shown last, separated by doubled fieldSeps,
- and is the only field allowed to contain the fieldSep.
-}
buildKeyData :: KeyData -> Builder
buildKeyData k = byteString (formatKeyVariety (keyVariety k))
<> 's' ?: (integerDec <$> keySize k)
<> 'm' ?: (integerDec . (\(CTime t) -> fromIntegral t) <$> keyMtime k)
<> 'S' ?: (integerDec <$> keyChunkSize k)
<> 'C' ?: (integerDec <$> keyChunkNum k)
minor optimisation
2021-10-06 01:26:11 +00:00
<> sepbefore (sepbefore (shortByteString (keyName k)))
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
where
sepbefore s = char7 fieldSep <> s
c ?: (Just b) = sepbefore (char7 c <> b)
_ ?: Nothing = mempty
add equivilant key log for VURL keys When downloading a VURL from the web, make sure that the equivilant key log is populated. Unfortunately, this does not hash the content while it's being downloaded from the web. There is not an interface in Backend currently for incrementally hash generation, only for incremental verification of an existing hash. So this might add a noticiable delay, and it has to show a "(checksum...") message. This could stand to be improved. But, that separate hashing step only has to happen on the first download of new content from the web. Once the hash is known, the VURL key can have its hash verified incrementally while downloading except when the content in the web has changed. (Doesn't happen yet because verifyKeyContentIncrementally is not implemented yet for VURL keys.) Note that the equivilant key log file is formatted as a presence log. This adds a tiny bit of overhead (eg "1 ") per line over just listing the urls. The reason I chose to use that format is it seems possible that there will need to be a way to remove an equivilant key at some point in the future. I don't know why that would be necessary, but it seemed wise to allow for the possibility. Downloads of VURL keys from other special remotes that claim urls, like bittorrent for example, does not popilate the equivilant key log. So for now, no checksum verification will be done for those. Sponsored-by: Nicholas Golder-Manning on Patreon
2024-02-29 19:41:57 +00:00
{- This is a strict parser for security reasons; in addition to keyName,
- a key can contain only 4 fields, which all consist only of numbers.
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
- Any key containing other fields, or non-numeric data will fail
- to parse.
-
add equivilant key log for VURL keys When downloading a VURL from the web, make sure that the equivilant key log is populated. Unfortunately, this does not hash the content while it's being downloaded from the web. There is not an interface in Backend currently for incrementally hash generation, only for incremental verification of an existing hash. So this might add a noticiable delay, and it has to show a "(checksum...") message. This could stand to be improved. But, that separate hashing step only has to happen on the first download of new content from the web. Once the hash is known, the VURL key can have its hash verified incrementally while downloading except when the content in the web has changed. (Doesn't happen yet because verifyKeyContentIncrementally is not implemented yet for VURL keys.) Note that the equivilant key log file is formatted as a presence log. This adds a tiny bit of overhead (eg "1 ") per line over just listing the urls. The reason I chose to use that format is it seems possible that there will need to be a way to remove an equivilant key at some point in the future. I don't know why that would be necessary, but it seemed wise to allow for the possibility. Downloads of VURL keys from other special remotes that claim urls, like bittorrent for example, does not popilate the equivilant key log. So for now, no checksum verification will be done for those. Sponsored-by: Nicholas Golder-Manning on Patreon
2024-02-29 19:41:57 +00:00
- If a key contained other non-numeric fields, they could be used to
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
- embed data used in a SHA1 collision attack, which would be a
- problem since the keys are committed to git.
-}
keyParser :: A.Parser Key
keyParser = do
-- key variety cannot be empty
v <- (parseKeyVariety <$> A8.takeWhile1 (/= fieldSep))
s <- parsesize
m <- parsemtime
cs <- parsechunksize
cn <- parsechunknum
_ <- A8.char fieldSep
_ <- A8.char fieldSep
n <- A.takeByteString
if validKeyName v n
then
let d = Key
convert Key to ShortByteString This adds the overhead of a copy when serializing and deserializing keys. I have not benchmarked much, but runtimes seem barely changed at all by that. When a lot of keys are in memory, it improves memory use. And, it prevents keys sometimes getting PINNED in memory and failing to GC, which is a problem ByteString has sometimes. In particular, git-annex sync from a borg special remote had that problem and this improved its memory use by a large amount. Sponsored-by: Shae Erisson on Patreon
2021-10-06 00:20:08 +00:00
{ keyName = S.toShort n
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
, keyVariety = v
, keySize = s
, keyMtime = m
, keyChunkSize = cs
, keyChunkNum = cn
}
in pure $ MkKey d (mkKeySerialization d)
else fail "invalid keyName"
where
parseopt p = (Just <$> (A8.char fieldSep *> p)) <|> pure Nothing
parsesize = parseopt $ A8.char 's' *> A8.decimal
parsemtime = parseopt $ CTime <$> (A8.char 'm' *> A8.decimal)
parsechunksize = parseopt $ A8.char 'S' *> A8.decimal
parsechunknum = parseopt $ A8.char 'C' *> A8.decimal
{- Limits the length of the extension in the keyName to mitigate against
- SHA1 collision attacks.
-
- In such an attack, the extension of the key could be made to contain
- the collision generation data, with the result that a signed git commit
- including such keys would not be secure.
-
- The maximum extension length ever generated for such a key was 8
- characters, but they may be unicode which could use up to 4 bytes each,
- so 32 bytes. 64 bytes is used here to give a little future wiggle-room.
- The SHA1 common-prefix attack needs 128 bytes of data.
-}
validKeyName :: KeyVariety -> S.ByteString -> Bool
validKeyName kv name
| hasExt kv =
let ext = snd $ splitKeyNameExtension' name
in S.length ext <= 64
| otherwise = True
{- This splits any extension out of the keyName, returning the
- keyName minus extension, and the extension (including leading dot).
-}
splitKeyNameExtension :: Key -> (S.ByteString, S.ByteString)
convert Key to ShortByteString This adds the overhead of a copy when serializing and deserializing keys. I have not benchmarked much, but runtimes seem barely changed at all by that. When a lot of keys are in memory, it improves memory use. And, it prevents keys sometimes getting PINNED in memory and failing to GC, which is a problem ByteString has sometimes. In particular, git-annex sync from a borg special remote had that problem and this improved its memory use by a large amount. Sponsored-by: Shae Erisson on Patreon
2021-10-06 00:20:08 +00:00
splitKeyNameExtension = splitKeyNameExtension' . S.fromShort . keyName . keyData
cache the serialization of a Key This will speed up the common case where a Key is deserialized from disk, but is then serialized to build eg, the path to the annex object. Previously attempted in 4536c93bb2ecf114ab711beac33fa358facd6985 and reverted in 96aba8eff7597898f53bfbab5865ff30a927d355. The problems mentioned in the latter commit are addressed now: Read/Show of KeyData is backwards-compatible with Read/Show of Key from before this change, so Types.Distribution will keep working. The Eq instance is fixed. Also, Key has smart constructors, avoiding needing to remember to update the cached serialization. Used git-annex benchmark: find is 7% faster whereis is 3% faster get when all files are already present is 5% faster Generally, the benchmarks are running 0.1 seconds faster per 2000 files, on a ram disk in my laptop.
2019-11-22 20:24:04 +00:00
splitKeyNameExtension' :: S.ByteString -> (S.ByteString, S.ByteString)
splitKeyNameExtension' keyname = S8.span (/= '.') keyname
moved AssociatedFile definition
2013-07-04 06:36:02 +00:00
{- A filename may be associated with a Key. -}
wip RawFilePath 2x git-annex find speedup Finally builds (oh the agoncy of making it build), but still very unmergable, only Command.Find is included and lots of stuff is badly hacked to make it compile. Benchmarking vs master, this git-annex find is significantly faster! Specifically: num files old new speedup 48500 4.77 3.73 28% 12500 1.36 1.02 66% 20 0.075 0.074 0% (so startup time is unchanged) That's without really finishing the optimization. Things still to do: * Eliminate all the fromRawFilePath, toRawFilePath, encodeBS, decodeBS conversions. * Use versions of IO actions like getFileStatus that take a RawFilePath. * Eliminate some Data.ByteString.Lazy.toStrict, which is a slow copy. * Use ByteString for parsing git config to speed up startup. It's likely several of those will speed up git-annex find further. And other commands will certianly benefit even more.
2019-11-26 19:27:22 +00:00
newtype AssociatedFile = AssociatedFile (Maybe RawFilePath)
new protocol for transferkeys, with message serialization Necessarily threw out the old protocol, so if an old git-annex assistant is running, and starts a transferkeys from the new git-annex, it would fail. But, that seems unlikely; the assistant starts up transferkeys processes and then keeps them running. Still, may need to test that scenario. The new protocol is simple read/show and looks like this: TransferRequest Download (Right "origin") (Key {keyName = "f8f8766a836fb6120abf4d5328ce8761404e437529e997aaa0363bdd4fecd7bb", keyVariety = SHA2Key (HashSize 256) (HasExt True), keySize = Just 30, keyMtime = Nothing, keyChunkSize = Nothing, keyChunkNum = Nothing}) (AssociatedFile (Just "foo")) TransferOutput (ProgressMeter (Just 30) (MeterState {meterBytesProcessed = BytesProcessed 0, meterTimeStamp = 1.6070268727892535e9}) (MeterState {meterBytesProcessed = BytesProcessed 30, meterTimeStamp = 1.6070268728043e9})) TransferOutput (OutputMessage "(checksum...) ") TransferResult True Granted, this is not optimally fast, but it seems good enough, and is probably nearly as fast as the old protocol anyhow. emitSerializedOutput for ProgressMeter is not yet implemented. It needs to somehow start or update a progress meter. There may need to be a new message that allocates a progress meter, and then have ProgressMeter update it. This commit was sponsored by Ethan Aubin
2020-12-03 20:21:20 +00:00
deriving (Show, Read, Eq, Ord)
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
{- There are several different varieties of keys. -}
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
data KeyVariety
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
= SHA2Key HashSize HasExt
| SHA3Key HashSize HasExt
| SKEINKey HashSize HasExt
| Blake2bKey HashSize HasExt
Add BLAKE2BP512 and BLAKE2BP512E backends using a blake2 variant optimised for 4-way CPUs This had been deferred because the Debian package of cryptonite, and possibly other builds, was broken for blake2bp, but I've confirmed #892855 is fixed. This commit was sponsored by Brett Eisenberg on Patreon.
2019-07-05 19:29:00 +00:00
| Blake2bpKey HashSize HasExt
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
| Blake2sKey HashSize HasExt
| Blake2spKey HashSize HasExt
| SHA1Key HasExt
| MD5Key HasExt
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
| WORMKey
| URLKey
support VURL backend Not yet implemented is recording hashes on download from web and verifying hashes. addurl --verifiable option added with -V short option because I expect a lot of people will want to use this. It seems likely that --verifiable will become the default eventually, and possibly rather soon. While old git-annex versions don't support VURL, that doesn't prevent using them with keys that use VURL. Of course, they won't verify the content on transfer, and fsck will warn that it doesn't know about VURL. So there's not much problem with starting to use VURL even when interoperating with old versions. Sponsored-by: Joshua Antonishen on Patreon
2024-02-29 17:26:06 +00:00
| VURLKey
add Backend.GitRemoteAnnex Making GITBUNDLE be in the backend list allows those keys to be hashed to verify, both when git-remote-annex downloads them, and by other transfers and by git fsck. GITMANIFEST is not in the backend list, because those keys will never be stored in .git/annex/objects and can't be verified in any case. This does mean that git-annex version will include GITBUNDLE in the list of backends. Also documented these in backends.mdwn Sponsored-by: Kevin Mueller on Patreon
2024-05-07 17:42:12 +00:00
| GitBundleKey
git-annex unused --from remote skips its git-remote-annex keys This turns out to only be necessary is edge cases. Most of the time, git-annex unused --from remote doesn't see git-remote-annex keys at all, because it does not record a location log for them. On the other hand, git-annex unused does find them, since it does not rely on the location log. And that's good because they're a local cache that the user should be able to drop. If, however, the user ran git-annex unused and then git-annex move --unused --to remote, the keys would have a location log for that remote. Then git-annex unused --from remote would see them, and would consider them unused. Even when they are present on the special remote they belong to. And that risks losing data if they drop the keys from the special remote, but didn't expect it would delete git branches they had pushed to it. So, make git-annex unused --from skip git-remote-annex keys whose uuid is the same as the remote.
2024-05-14 19:12:07 +00:00
| GitManifestKey
external backends wip It's able to start them up, the only thing not implemented is generating and verifying keys. And, the key translation for HasExt.
2020-07-29 19:23:18 +00:00
-- A key that is handled by some external backend.
| ExternalKey S.ByteString HasExt
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
-- Some repositories may contain keys of other varieties,
-- which can still be processed to some extent.
use ByteString in Key for speed This is an easy win for parseKeyVariety: benchmarking old/parseKeyVariety time 1.515 μs (1.512 μs .. 1.517 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.515 μs (1.513 μs .. 1.517 μs) std dev 6.417 ns (4.992 ns .. 8.113 ns) benchmarking new/parseKeyVariety time 54.97 ns (54.70 ns .. 55.40 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 55.42 ns (55.05 ns .. 56.03 ns) std dev 1.562 ns (969.5 ps .. 2.442 ns) variance introduced by outliers: 44% (moderately inflated) For formatKeyVariety, using a Builder is marginally worse than building a String... (This is with criterion evaluating fully to nf not whnf) benchmarking old/formatKeyVariety time 434.3 ns (428.0 ns .. 440.4 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 430.6 ns (428.2 ns .. 433.9 ns) std dev 9.166 ns (6.932 ns .. 11.94 ns) variance introduced by outliers: 27% (moderately inflated) benchmarking Builder/formatKeyVariety time 526.5 ns (524.7 ns .. 528.8 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 526.1 ns (524.9 ns .. 528.5 ns) std dev 5.687 ns (3.762 ns .. 8.000 ns) Manually building the ByteString was better, but still slightly slower than String, due to innefficient need to S.pack . show the HashSize: benchmarking formatKeyVariety time 459.5 ns (455.8 ns .. 463.2 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 459.9 ns (457.4 ns .. 466.6 ns) std dev 11.65 ns (6.860 ns .. 21.41 ns) variance introduced by outliers: 35% (moderately inflated) So I cheated and made parseKeyVariety cache the original ByteString, for formatKeyVariety to use instead of re-building it. Final benchmark: benchmarking new/formatKeyVariety time 50.64 ns (50.57 ns .. 50.73 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 51.05 ns (50.60 ns .. 52.71 ns) std dev 2.790 ns (259.6 ps .. 5.916 ns) variance introduced by outliers: 75% (severely inflated) benchmarking new/parseKeyVariety time 71.88 ns (71.54 ns .. 72.24 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 71.97 ns (71.69 ns .. 72.47 ns) std dev 1.249 ns (910.7 ps .. 1.791 ns) variance introduced by outliers: 22% (moderately inflated)
2019-01-11 17:55:00 +00:00
| OtherKey S.ByteString
benchmark: Add --databases to benchmark sqlite databases Rescued from commit 11d6e2e260d70ba99e35464c19c2b2772ce9efaa which removed db benchmarks in favor of benchmarking arbitrary git-annex commands. Which is nice and general, but microbenchmarks are useful too.
2019-10-29 19:16:15 +00:00
deriving (Eq, Ord, Read, Show, Generic)
instance NFData KeyVariety
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
{- Some varieties of keys may contain an extension at the end of the
- keyName -}
newtype HasExt = HasExt Bool
benchmark: Add --databases to benchmark sqlite databases Rescued from commit 11d6e2e260d70ba99e35464c19c2b2772ce9efaa which removed db benchmarks in favor of benchmarking arbitrary git-annex commands. Which is nice and general, but microbenchmarks are useful too.
2019-10-29 19:16:15 +00:00
deriving (Eq, Ord, Read, Show, Generic)
instance NFData HasExt
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
newtype HashSize = HashSize Int
benchmark: Add --databases to benchmark sqlite databases Rescued from commit 11d6e2e260d70ba99e35464c19c2b2772ce9efaa which removed db benchmarks in favor of benchmarking arbitrary git-annex commands. Which is nice and general, but microbenchmarks are useful too.
2019-10-29 19:16:15 +00:00
deriving (Eq, Ord, Read, Show, Generic)
instance NFData HashSize
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
hasExt :: KeyVariety -> Bool
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
hasExt (SHA2Key _ (HasExt b)) = b
hasExt (SHA3Key _ (HasExt b)) = b
hasExt (SKEINKey _ (HasExt b)) = b
hasExt (Blake2bKey _ (HasExt b)) = b
Add BLAKE2BP512 and BLAKE2BP512E backends using a blake2 variant optimised for 4-way CPUs This had been deferred because the Debian package of cryptonite, and possibly other builds, was broken for blake2bp, but I've confirmed #892855 is fixed. This commit was sponsored by Brett Eisenberg on Patreon.
2019-07-05 19:29:00 +00:00
hasExt (Blake2bpKey _ (HasExt b)) = b
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
hasExt (Blake2sKey _ (HasExt b)) = b
hasExt (Blake2spKey _ (HasExt b)) = b
hasExt (SHA1Key (HasExt b)) = b
hasExt (MD5Key (HasExt b)) = b
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
hasExt WORMKey = False
hasExt URLKey = False
support VURL backend Not yet implemented is recording hashes on download from web and verifying hashes. addurl --verifiable option added with -V short option because I expect a lot of people will want to use this. It seems likely that --verifiable will become the default eventually, and possibly rather soon. While old git-annex versions don't support VURL, that doesn't prevent using them with keys that use VURL. Of course, they won't verify the content on transfer, and fsck will warn that it doesn't know about VURL. So there's not much problem with starting to use VURL even when interoperating with old versions. Sponsored-by: Joshua Antonishen on Patreon
2024-02-29 17:26:06 +00:00
hasExt VURLKey = False
add Backend.GitRemoteAnnex Making GITBUNDLE be in the backend list allows those keys to be hashed to verify, both when git-remote-annex downloads them, and by other transfers and by git fsck. GITMANIFEST is not in the backend list, because those keys will never be stored in .git/annex/objects and can't be verified in any case. This does mean that git-annex version will include GITBUNDLE in the list of backends. Also documented these in backends.mdwn Sponsored-by: Kevin Mueller on Patreon
2024-05-07 17:42:12 +00:00
hasExt GitBundleKey = False
git-annex unused --from remote skips its git-remote-annex keys This turns out to only be necessary is edge cases. Most of the time, git-annex unused --from remote doesn't see git-remote-annex keys at all, because it does not record a location log for them. On the other hand, git-annex unused does find them, since it does not rely on the location log. And that's good because they're a local cache that the user should be able to drop. If, however, the user ran git-annex unused and then git-annex move --unused --to remote, the keys would have a location log for that remote. Then git-annex unused --from remote would see them, and would consider them unused. Even when they are present on the special remote they belong to. And that risks losing data if they drop the keys from the special remote, but didn't expect it would delete git branches they had pushed to it. So, make git-annex unused --from skip git-remote-annex keys whose uuid is the same as the remote.
2024-05-14 19:12:07 +00:00
hasExt GitManifestKey = False
external backends wip It's able to start them up, the only thing not implemented is generating and verifying keys. And, the key translation for HasExt.
2020-07-29 19:23:18 +00:00
hasExt (ExternalKey _ (HasExt b)) = b
use ByteString in Key for speed This is an easy win for parseKeyVariety: benchmarking old/parseKeyVariety time 1.515 μs (1.512 μs .. 1.517 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.515 μs (1.513 μs .. 1.517 μs) std dev 6.417 ns (4.992 ns .. 8.113 ns) benchmarking new/parseKeyVariety time 54.97 ns (54.70 ns .. 55.40 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 55.42 ns (55.05 ns .. 56.03 ns) std dev 1.562 ns (969.5 ps .. 2.442 ns) variance introduced by outliers: 44% (moderately inflated) For formatKeyVariety, using a Builder is marginally worse than building a String... (This is with criterion evaluating fully to nf not whnf) benchmarking old/formatKeyVariety time 434.3 ns (428.0 ns .. 440.4 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 430.6 ns (428.2 ns .. 433.9 ns) std dev 9.166 ns (6.932 ns .. 11.94 ns) variance introduced by outliers: 27% (moderately inflated) benchmarking Builder/formatKeyVariety time 526.5 ns (524.7 ns .. 528.8 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 526.1 ns (524.9 ns .. 528.5 ns) std dev 5.687 ns (3.762 ns .. 8.000 ns) Manually building the ByteString was better, but still slightly slower than String, due to innefficient need to S.pack . show the HashSize: benchmarking formatKeyVariety time 459.5 ns (455.8 ns .. 463.2 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 459.9 ns (457.4 ns .. 466.6 ns) std dev 11.65 ns (6.860 ns .. 21.41 ns) variance introduced by outliers: 35% (moderately inflated) So I cheated and made parseKeyVariety cache the original ByteString, for formatKeyVariety to use instead of re-building it. Final benchmark: benchmarking new/formatKeyVariety time 50.64 ns (50.57 ns .. 50.73 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 51.05 ns (50.60 ns .. 52.71 ns) std dev 2.790 ns (259.6 ps .. 5.916 ns) variance introduced by outliers: 75% (severely inflated) benchmarking new/parseKeyVariety time 71.88 ns (71.54 ns .. 72.24 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 71.97 ns (71.69 ns .. 72.47 ns) std dev 1.249 ns (910.7 ps .. 1.791 ns) variance introduced by outliers: 22% (moderately inflated)
2019-01-11 17:55:00 +00:00
hasExt (OtherKey s) = (snd <$> S8.unsnoc s) == Just 'E'
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
sameExceptExt :: KeyVariety -> KeyVariety -> Bool
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
sameExceptExt (SHA2Key sz1 _) (SHA2Key sz2 _) = sz1 == sz2
sameExceptExt (SHA3Key sz1 _) (SHA3Key sz2 _) = sz1 == sz2
sameExceptExt (SKEINKey sz1 _) (SKEINKey sz2 _) = sz1 == sz2
sameExceptExt (Blake2bKey sz1 _) (Blake2bKey sz2 _) = sz1 == sz2
Add BLAKE2BP512 and BLAKE2BP512E backends using a blake2 variant optimised for 4-way CPUs This had been deferred because the Debian package of cryptonite, and possibly other builds, was broken for blake2bp, but I've confirmed #892855 is fixed. This commit was sponsored by Brett Eisenberg on Patreon.
2019-07-05 19:29:00 +00:00
sameExceptExt (Blake2bpKey sz1 _) (Blake2bpKey sz2 _) = sz1 == sz2
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
sameExceptExt (Blake2sKey sz1 _) (Blake2sKey sz2 _) = sz1 == sz2
sameExceptExt (Blake2spKey sz1 _) (Blake2spKey sz2 _) = sz1 == sz2
sameExceptExt (SHA1Key _) (SHA1Key _) = True
sameExceptExt (MD5Key _) (MD5Key _) = True
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
sameExceptExt _ _ = False
use ByteString in Key for speed This is an easy win for parseKeyVariety: benchmarking old/parseKeyVariety time 1.515 μs (1.512 μs .. 1.517 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.515 μs (1.513 μs .. 1.517 μs) std dev 6.417 ns (4.992 ns .. 8.113 ns) benchmarking new/parseKeyVariety time 54.97 ns (54.70 ns .. 55.40 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 55.42 ns (55.05 ns .. 56.03 ns) std dev 1.562 ns (969.5 ps .. 2.442 ns) variance introduced by outliers: 44% (moderately inflated) For formatKeyVariety, using a Builder is marginally worse than building a String... (This is with criterion evaluating fully to nf not whnf) benchmarking old/formatKeyVariety time 434.3 ns (428.0 ns .. 440.4 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 430.6 ns (428.2 ns .. 433.9 ns) std dev 9.166 ns (6.932 ns .. 11.94 ns) variance introduced by outliers: 27% (moderately inflated) benchmarking Builder/formatKeyVariety time 526.5 ns (524.7 ns .. 528.8 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 526.1 ns (524.9 ns .. 528.5 ns) std dev 5.687 ns (3.762 ns .. 8.000 ns) Manually building the ByteString was better, but still slightly slower than String, due to innefficient need to S.pack . show the HashSize: benchmarking formatKeyVariety time 459.5 ns (455.8 ns .. 463.2 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 459.9 ns (457.4 ns .. 466.6 ns) std dev 11.65 ns (6.860 ns .. 21.41 ns) variance introduced by outliers: 35% (moderately inflated) So I cheated and made parseKeyVariety cache the original ByteString, for formatKeyVariety to use instead of re-building it. Final benchmark: benchmarking new/formatKeyVariety time 50.64 ns (50.57 ns .. 50.73 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 51.05 ns (50.60 ns .. 52.71 ns) std dev 2.790 ns (259.6 ps .. 5.916 ns) variance introduced by outliers: 75% (severely inflated) benchmarking new/parseKeyVariety time 71.88 ns (71.54 ns .. 72.24 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 71.97 ns (71.69 ns .. 72.47 ns) std dev 1.249 ns (910.7 ps .. 1.791 ns) variance introduced by outliers: 22% (moderately inflated)
2019-01-11 17:55:00 +00:00
formatKeyVariety :: KeyVariety -> S.ByteString
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
formatKeyVariety v = case v of
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
SHA2Key sz e -> adde e (addsz sz "SHA")
SHA3Key sz e -> adde e (addsz sz "SHA3_")
SKEINKey sz e -> adde e (addsz sz "SKEIN")
Blake2bKey sz e -> adde e (addsz sz "BLAKE2B")
Add BLAKE2BP512 and BLAKE2BP512E backends using a blake2 variant optimised for 4-way CPUs This had been deferred because the Debian package of cryptonite, and possibly other builds, was broken for blake2bp, but I've confirmed #892855 is fixed. This commit was sponsored by Brett Eisenberg on Patreon.
2019-07-05 19:29:00 +00:00
Blake2bpKey sz e -> adde e (addsz sz "BLAKE2BP")
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
Blake2sKey sz e -> adde e (addsz sz "BLAKE2S")
Blake2spKey sz e -> adde e (addsz sz "BLAKE2SP")
SHA1Key e -> adde e "SHA1"
MD5Key e -> adde e "MD5"
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
WORMKey -> "WORM"
URLKey -> "URL"
support VURL backend Not yet implemented is recording hashes on download from web and verifying hashes. addurl --verifiable option added with -V short option because I expect a lot of people will want to use this. It seems likely that --verifiable will become the default eventually, and possibly rather soon. While old git-annex versions don't support VURL, that doesn't prevent using them with keys that use VURL. Of course, they won't verify the content on transfer, and fsck will warn that it doesn't know about VURL. So there's not much problem with starting to use VURL even when interoperating with old versions. Sponsored-by: Joshua Antonishen on Patreon
2024-02-29 17:26:06 +00:00
VURLKey -> "VURL"
add Backend.GitRemoteAnnex Making GITBUNDLE be in the backend list allows those keys to be hashed to verify, both when git-remote-annex downloads them, and by other transfers and by git fsck. GITMANIFEST is not in the backend list, because those keys will never be stored in .git/annex/objects and can't be verified in any case. This does mean that git-annex version will include GITBUNDLE in the list of backends. Also documented these in backends.mdwn Sponsored-by: Kevin Mueller on Patreon
2024-05-07 17:42:12 +00:00
GitBundleKey -> "GITBUNDLE"
git-annex unused --from remote skips its git-remote-annex keys This turns out to only be necessary is edge cases. Most of the time, git-annex unused --from remote doesn't see git-remote-annex keys at all, because it does not record a location log for them. On the other hand, git-annex unused does find them, since it does not rely on the location log. And that's good because they're a local cache that the user should be able to drop. If, however, the user ran git-annex unused and then git-annex move --unused --to remote, the keys would have a location log for that remote. Then git-annex unused --from remote would see them, and would consider them unused. Even when they are present on the special remote they belong to. And that risks losing data if they drop the keys from the special remote, but didn't expect it would delete git branches they had pushed to it. So, make git-annex unused --from skip git-remote-annex keys whose uuid is the same as the remote.
2024-05-14 19:12:07 +00:00
GitManifestKey -> "GITMANIFEST"
external backends wip It's able to start them up, the only thing not implemented is generating and verifying keys. And, the key translation for HasExt.
2020-07-29 19:23:18 +00:00
ExternalKey s e -> adde e ("X" <> s)
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
OtherKey s -> s
where
adde (HasExt False) s = s
use ByteString in Key for speed This is an easy win for parseKeyVariety: benchmarking old/parseKeyVariety time 1.515 μs (1.512 μs .. 1.517 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.515 μs (1.513 μs .. 1.517 μs) std dev 6.417 ns (4.992 ns .. 8.113 ns) benchmarking new/parseKeyVariety time 54.97 ns (54.70 ns .. 55.40 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 55.42 ns (55.05 ns .. 56.03 ns) std dev 1.562 ns (969.5 ps .. 2.442 ns) variance introduced by outliers: 44% (moderately inflated) For formatKeyVariety, using a Builder is marginally worse than building a String... (This is with criterion evaluating fully to nf not whnf) benchmarking old/formatKeyVariety time 434.3 ns (428.0 ns .. 440.4 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 430.6 ns (428.2 ns .. 433.9 ns) std dev 9.166 ns (6.932 ns .. 11.94 ns) variance introduced by outliers: 27% (moderately inflated) benchmarking Builder/formatKeyVariety time 526.5 ns (524.7 ns .. 528.8 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 526.1 ns (524.9 ns .. 528.5 ns) std dev 5.687 ns (3.762 ns .. 8.000 ns) Manually building the ByteString was better, but still slightly slower than String, due to innefficient need to S.pack . show the HashSize: benchmarking formatKeyVariety time 459.5 ns (455.8 ns .. 463.2 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 459.9 ns (457.4 ns .. 466.6 ns) std dev 11.65 ns (6.860 ns .. 21.41 ns) variance introduced by outliers: 35% (moderately inflated) So I cheated and made parseKeyVariety cache the original ByteString, for formatKeyVariety to use instead of re-building it. Final benchmark: benchmarking new/formatKeyVariety time 50.64 ns (50.57 ns .. 50.73 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 51.05 ns (50.60 ns .. 52.71 ns) std dev 2.790 ns (259.6 ps .. 5.916 ns) variance introduced by outliers: 75% (severely inflated) benchmarking new/parseKeyVariety time 71.88 ns (71.54 ns .. 72.24 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 71.97 ns (71.69 ns .. 72.47 ns) std dev 1.249 ns (910.7 ps .. 1.791 ns) variance introduced by outliers: 22% (moderately inflated)
2019-01-11 17:55:00 +00:00
adde (HasExt True) s = s <> "E"
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
addsz (HashSize n) s = s <> case n of
256 -> "256"
512 -> "512"
224 -> "224"
384 -> "384"
160 -> "160"
-- This is relatively slow, which is why the common hash
-- sizes are hardcoded above.
_ -> S8.pack (show n)
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
use ByteString in Key for speed This is an easy win for parseKeyVariety: benchmarking old/parseKeyVariety time 1.515 μs (1.512 μs .. 1.517 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.515 μs (1.513 μs .. 1.517 μs) std dev 6.417 ns (4.992 ns .. 8.113 ns) benchmarking new/parseKeyVariety time 54.97 ns (54.70 ns .. 55.40 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 55.42 ns (55.05 ns .. 56.03 ns) std dev 1.562 ns (969.5 ps .. 2.442 ns) variance introduced by outliers: 44% (moderately inflated) For formatKeyVariety, using a Builder is marginally worse than building a String... (This is with criterion evaluating fully to nf not whnf) benchmarking old/formatKeyVariety time 434.3 ns (428.0 ns .. 440.4 ns) 0.999 R² (0.999 R² .. 1.000 R²) mean 430.6 ns (428.2 ns .. 433.9 ns) std dev 9.166 ns (6.932 ns .. 11.94 ns) variance introduced by outliers: 27% (moderately inflated) benchmarking Builder/formatKeyVariety time 526.5 ns (524.7 ns .. 528.8 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 526.1 ns (524.9 ns .. 528.5 ns) std dev 5.687 ns (3.762 ns .. 8.000 ns) Manually building the ByteString was better, but still slightly slower than String, due to innefficient need to S.pack . show the HashSize: benchmarking formatKeyVariety time 459.5 ns (455.8 ns .. 463.2 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 459.9 ns (457.4 ns .. 466.6 ns) std dev 11.65 ns (6.860 ns .. 21.41 ns) variance introduced by outliers: 35% (moderately inflated) So I cheated and made parseKeyVariety cache the original ByteString, for formatKeyVariety to use instead of re-building it. Final benchmark: benchmarking new/formatKeyVariety time 50.64 ns (50.57 ns .. 50.73 ns) 1.000 R² (0.999 R² .. 1.000 R²) mean 51.05 ns (50.60 ns .. 52.71 ns) std dev 2.790 ns (259.6 ps .. 5.916 ns) variance introduced by outliers: 75% (severely inflated) benchmarking new/parseKeyVariety time 71.88 ns (71.54 ns .. 72.24 ns) 1.000 R² (1.000 R² .. 1.000 R²) mean 71.97 ns (71.69 ns .. 72.47 ns) std dev 1.249 ns (910.7 ps .. 1.791 ns) variance introduced by outliers: 22% (moderately inflated)
2019-01-11 17:55:00 +00:00
parseKeyVariety :: S.ByteString -> KeyVariety
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
parseKeyVariety "SHA256" = SHA2Key (HashSize 256) (HasExt False)
parseKeyVariety "SHA256E" = SHA2Key (HashSize 256) (HasExt True)
parseKeyVariety "SHA512" = SHA2Key (HashSize 512) (HasExt False)
parseKeyVariety "SHA512E" = SHA2Key (HashSize 512) (HasExt True)
parseKeyVariety "SHA224" = SHA2Key (HashSize 224) (HasExt False)
parseKeyVariety "SHA224E" = SHA2Key (HashSize 224) (HasExt True)
parseKeyVariety "SHA384" = SHA2Key (HashSize 384) (HasExt False)
parseKeyVariety "SHA384E" = SHA2Key (HashSize 384) (HasExt True)
parseKeyVariety "SHA3_512" = SHA3Key (HashSize 512) (HasExt False)
parseKeyVariety "SHA3_512E" = SHA3Key (HashSize 512) (HasExt True)
parseKeyVariety "SHA3_384" = SHA3Key (HashSize 384) (HasExt False)
parseKeyVariety "SHA3_384E" = SHA3Key (HashSize 384) (HasExt True)
parseKeyVariety "SHA3_256" = SHA3Key (HashSize 256) (HasExt False)
parseKeyVariety "SHA3_256E" = SHA3Key (HashSize 256) (HasExt True)
parseKeyVariety "SHA3_224" = SHA3Key (HashSize 224) (HasExt False)
parseKeyVariety "SHA3_224E" = SHA3Key (HashSize 224) (HasExt True)
parseKeyVariety "SKEIN512" = SKEINKey (HashSize 512) (HasExt False)
parseKeyVariety "SKEIN512E" = SKEINKey (HashSize 512) (HasExt True)
parseKeyVariety "SKEIN256" = SKEINKey (HashSize 256) (HasExt False)
parseKeyVariety "SKEIN256E" = SKEINKey (HashSize 256) (HasExt True)
parseKeyVariety "BLAKE2B160" = Blake2bKey (HashSize 160) (HasExt False)
parseKeyVariety "BLAKE2B160E" = Blake2bKey (HashSize 160) (HasExt True)
parseKeyVariety "BLAKE2B224" = Blake2bKey (HashSize 224) (HasExt False)
parseKeyVariety "BLAKE2B224E" = Blake2bKey (HashSize 224) (HasExt True)
parseKeyVariety "BLAKE2B256" = Blake2bKey (HashSize 256) (HasExt False)
parseKeyVariety "BLAKE2B256E" = Blake2bKey (HashSize 256) (HasExt True)
parseKeyVariety "BLAKE2B384" = Blake2bKey (HashSize 384) (HasExt False)
parseKeyVariety "BLAKE2B384E" = Blake2bKey (HashSize 384) (HasExt True)
parseKeyVariety "BLAKE2B512" = Blake2bKey (HashSize 512) (HasExt False)
parseKeyVariety "BLAKE2B512E" = Blake2bKey (HashSize 512) (HasExt True)
Add BLAKE2BP512 and BLAKE2BP512E backends using a blake2 variant optimised for 4-way CPUs This had been deferred because the Debian package of cryptonite, and possibly other builds, was broken for blake2bp, but I've confirmed #892855 is fixed. This commit was sponsored by Brett Eisenberg on Patreon.
2019-07-05 19:29:00 +00:00
parseKeyVariety "BLAKE2BP512" = Blake2bpKey (HashSize 512) (HasExt False)
parseKeyVariety "BLAKE2BP512E" = Blake2bpKey (HashSize 512) (HasExt True)
remove recently added cache from KeyVariety Adding that field broke the Read/Show serialization back-compat, and also the Eq and Ord instances were not blinded to it, which broke git annex fsck and probably more. I think that the new approach used in formatKeyVariety will be nearly as fast, but have not benchmarked it.
2019-01-16 20:33:08 +00:00
parseKeyVariety "BLAKE2S160" = Blake2sKey (HashSize 160) (HasExt False)
parseKeyVariety "BLAKE2S160E" = Blake2sKey (HashSize 160) (HasExt True)
parseKeyVariety "BLAKE2S224" = Blake2sKey (HashSize 224) (HasExt False)
parseKeyVariety "BLAKE2S224E" = Blake2sKey (HashSize 224) (HasExt True)
parseKeyVariety "BLAKE2S256" = Blake2sKey (HashSize 256) (HasExt False)
parseKeyVariety "BLAKE2S256E" = Blake2sKey (HashSize 256) (HasExt True)
parseKeyVariety "BLAKE2SP224" = Blake2spKey (HashSize 224) (HasExt False)
parseKeyVariety "BLAKE2SP224E" = Blake2spKey (HashSize 224) (HasExt True)
parseKeyVariety "BLAKE2SP256" = Blake2spKey (HashSize 256) (HasExt False)
parseKeyVariety "BLAKE2SP256E" = Blake2spKey (HashSize 256) (HasExt True)
external backends wip It's able to start them up, the only thing not implemented is generating and verifying keys. And, the key translation for HasExt.
2020-07-29 19:23:18 +00:00
parseKeyVariety "SHA1" = SHA1Key (HasExt False)
parseKeyVariety "SHA1E" = SHA1Key (HasExt True)
parseKeyVariety "MD5" = MD5Key (HasExt False)
parseKeyVariety "MD5E" = MD5Key (HasExt True)
parseKeyVariety "WORM" = WORMKey
parseKeyVariety "URL" = URLKey
support VURL backend Not yet implemented is recording hashes on download from web and verifying hashes. addurl --verifiable option added with -V short option because I expect a lot of people will want to use this. It seems likely that --verifiable will become the default eventually, and possibly rather soon. While old git-annex versions don't support VURL, that doesn't prevent using them with keys that use VURL. Of course, they won't verify the content on transfer, and fsck will warn that it doesn't know about VURL. So there's not much problem with starting to use VURL even when interoperating with old versions. Sponsored-by: Joshua Antonishen on Patreon
2024-02-29 17:26:06 +00:00
parseKeyVariety "VURL" = VURLKey
add Backend.GitRemoteAnnex Making GITBUNDLE be in the backend list allows those keys to be hashed to verify, both when git-remote-annex downloads them, and by other transfers and by git fsck. GITMANIFEST is not in the backend list, because those keys will never be stored in .git/annex/objects and can't be verified in any case. This does mean that git-annex version will include GITBUNDLE in the list of backends. Also documented these in backends.mdwn Sponsored-by: Kevin Mueller on Patreon
2024-05-07 17:42:12 +00:00
parseKeyVariety "GITBUNDLE" = GitBundleKey
external backends wip It's able to start them up, the only thing not implemented is generating and verifying keys. And, the key translation for HasExt.
2020-07-29 19:23:18 +00:00
parseKeyVariety b
| "X" `S.isPrefixOf` b =
let b' = S.tail b
fix bug caught by test suite
2020-07-31 20:11:50 +00:00
in if not (S.null b') && S.last b' == fromIntegral (ord 'E')
external backends wip It's able to start them up, the only thing not implemented is generating and verifying keys. And, the key translation for HasExt.
2020-07-29 19:23:18 +00:00
then ExternalKey (S.init b') (HasExt True)
else ExternalKey b' (HasExt False)
| otherwise = OtherKey b
Reference in a new issue Copy permalink