mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Types/Key.hs

176 lines
7.1 KiB
Haskell
Raw Normal View History

a new Key data type with metadata
2011-03-15 21:47:29 +00:00
{- git-annex Key data type
-
add retrievalSecurityPolicy This will be used to protect against CVE-2018-10859, where an encrypted special remote is fed the wrong encrypted data, and so tricked into decrypting something that the user encrypted with their gpg key and did not store in git-annex. It also protects against CVE-2018-10857, where a remote follows a http redirect to a file:// url or to a local private web server. While that's already been prevented in git-annex's own use of http, external special remotes, hooks, etc use other http implementations and could still be vulnerable. The policy is not yet enforced, this commit only adds the appropriate metadata to remotes. This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2018-06-21 15:35:27 +00:00
- Copyright 2011-2018 Joey Hess <id@joeyh.name>
a new Key data type with metadata
2011-03-15 21:47:29 +00:00
-
- Licensed under the GNU GPL version 3 or higher.
-}
fix build with cryptonite-0.20 Some blake hash varieties were not yet available in that version. Rather than tracking exact details of what cryptonite supported when, disable blake unless using a current cryptonite.
2018-03-15 15:16:00 +00:00
{-# LANGUAGE CPP #-}
factor non-type stuff out of Key
2017-02-24 17:42:30 +00:00
module Types.Key where
a new Key data type with metadata
2011-03-15 21:47:29 +00:00
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
import Utility.PartialPrelude
first pass at using new keys It compiles. It sorta works. Several subcommands are FIXME marked and broken, because things that used to accept separate --backend and --key params need to be changed to accept just a --key that encodes all the key info, now that there is metadata in keys.
2011-03-16 01:34:13 +00:00
import System.Posix.Types
factor out common imports no code changes
2011-10-04 02:24:57 +00:00
moved AssociatedFile definition
2013-07-04 06:36:02 +00:00
{- A Key has a unique name, which is derived from a particular backend,
first pass at using new keys It compiles. It sorta works. Several subcommands are FIXME marked and broken, because things that used to accept separate --backend and --key params need to be changed to accept just a --key that encodes all the key info, now that there is metadata in keys.
2011-03-16 01:34:13 +00:00
- and may contain other optional metadata. -}
layout
2013-07-04 06:45:46 +00:00
data Key = Key
{ keyName :: String
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
, keyVariety :: KeyVariety
layout
2013-07-04 06:45:46 +00:00
, keySize :: Maybe Integer
, keyMtime :: Maybe EpochTime
add chunk metadata to Key Added new fields for chunk number, and chunk size. These will not appear in normal keys ever, but will be used for chunked data stored on special remotes. This commit was sponsored by Jouni K Seppanen.
2014-07-24 17:36:23 +00:00
, keyChunkSize :: Maybe Integer
, keyChunkNum :: Maybe Integer
layout
2013-07-04 06:45:46 +00:00
} deriving (Eq, Ord, Read, Show)
a new Key data type with metadata
2011-03-15 21:47:29 +00:00
moved AssociatedFile definition
2013-07-04 06:36:02 +00:00
{- A filename may be associated with a Key. -}
AssociatedFile newtype To prevent any further mistakes like 301aff34c42d896038c42e0a0bc7b1cf71e0ad22 This commit was sponsored by Francois Marier on Patreon.
2017-03-10 17:12:24 +00:00
newtype AssociatedFile = AssociatedFile (Maybe FilePath)
deriving (Show, Eq, Ord)
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
{- There are several different varieties of keys. -}
data KeyVariety
= SHA2Key HashSize HasExt
| SHA3Key HashSize HasExt
| SKEINKey HashSize HasExt
Added backends for the BLAKE2 family of hashes. There are a lot of different variants and sizes, I suppose we might as well export all the common ones. Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support. Even android has 0.16 or newer. On Debian, Blake2bp_512 is buggy, so I have omitted it for now. http://bugs.debian.org/892855 This commit was sponsored by andrea rota.
2018-03-13 20:15:35 +00:00
| Blake2bKey HashSize HasExt
| Blake2sKey HashSize HasExt
| Blake2spKey HashSize HasExt
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
| SHA1Key HasExt
| MD5Key HasExt
| WORMKey
| URLKey
-- Some repositories may contain keys of other varieties,
-- which can still be processed to some extent.
| OtherKey String
deriving (Eq, Ord, Read, Show)
{- Some varieties of keys may contain an extension at the end of the
- keyName -}
newtype HasExt = HasExt Bool
deriving (Eq, Ord, Read, Show)
newtype HashSize = HashSize Int
deriving (Eq, Ord, Read, Show)
hasExt :: KeyVariety -> Bool
hasExt (SHA2Key _ (HasExt b)) = b
hasExt (SHA3Key _ (HasExt b)) = b
hasExt (SKEINKey _ (HasExt b)) = b
Added backends for the BLAKE2 family of hashes. There are a lot of different variants and sizes, I suppose we might as well export all the common ones. Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support. Even android has 0.16 or newer. On Debian, Blake2bp_512 is buggy, so I have omitted it for now. http://bugs.debian.org/892855 This commit was sponsored by andrea rota.
2018-03-13 20:15:35 +00:00
hasExt (Blake2bKey _ (HasExt b)) = b
hasExt (Blake2sKey _ (HasExt b)) = b
hasExt (Blake2spKey _ (HasExt b)) = b
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
hasExt (SHA1Key (HasExt b)) = b
hasExt (MD5Key (HasExt b)) = b
hasExt WORMKey = False
hasExt URLKey = False
hasExt (OtherKey s) = end s == "E"
sameExceptExt :: KeyVariety -> KeyVariety -> Bool
sameExceptExt (SHA2Key sz1 _) (SHA2Key sz2 _) = sz1 == sz2
sameExceptExt (SHA3Key sz1 _) (SHA3Key sz2 _) = sz1 == sz2
sameExceptExt (SKEINKey sz1 _) (SKEINKey sz2 _) = sz1 == sz2
Added backends for the BLAKE2 family of hashes. There are a lot of different variants and sizes, I suppose we might as well export all the common ones. Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support. Even android has 0.16 or newer. On Debian, Blake2bp_512 is buggy, so I have omitted it for now. http://bugs.debian.org/892855 This commit was sponsored by andrea rota.
2018-03-13 20:15:35 +00:00
sameExceptExt (Blake2bKey sz1 _) (Blake2bKey sz2 _) = sz1 == sz2
sameExceptExt (Blake2sKey sz1 _) (Blake2sKey sz2 _) = sz1 == sz2
sameExceptExt (Blake2spKey sz1 _) (Blake2spKey sz2 _) = sz1 == sz2
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
sameExceptExt (SHA1Key _) (SHA1Key _) = True
sameExceptExt (MD5Key _) (MD5Key _) = True
sameExceptExt _ _ = False
add cryptographicallySecure Note that GPGHMAC keys are not cryptographically secure, because their content has no relation to the name of the key. So, things that use this function to avoid sending keys to a remote will need to special case in support for those keys. If GPGHMAC keys were accepted as cryptographically secure, symlinks using them could be committed to a git repo, and their content would be accepted into the repo, with no guarantee that two repos got the same content, which is what we're aiming to prevent.
2017-02-27 16:54:06 +00:00
{- Is the Key variety cryptographically secure, such that no two differing
- file contents can be mapped to the same Key? -}
cryptographicallySecure :: KeyVariety -> Bool
cryptographicallySecure (SHA2Key _ _) = True
cryptographicallySecure (SHA3Key _ _) = True
cryptographicallySecure (SKEINKey _ _) = True
Added backends for the BLAKE2 family of hashes. There are a lot of different variants and sizes, I suppose we might as well export all the common ones. Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support. Even android has 0.16 or newer. On Debian, Blake2bp_512 is buggy, so I have omitted it for now. http://bugs.debian.org/892855 This commit was sponsored by andrea rota.
2018-03-13 20:15:35 +00:00
cryptographicallySecure (Blake2bKey _ _) = True
cryptographicallySecure (Blake2sKey _ _) = True
cryptographicallySecure (Blake2spKey _ _) = True
add cryptographicallySecure Note that GPGHMAC keys are not cryptographically secure, because their content has no relation to the name of the key. So, things that use this function to avoid sending keys to a remote will need to special case in support for those keys. If GPGHMAC keys were accepted as cryptographically secure, symlinks using them could be committed to a git repo, and their content would be accepted into the repo, with no guarantee that two repos got the same content, which is what we're aiming to prevent.
2017-02-27 16:54:06 +00:00
cryptographicallySecure _ = False
add retrievalSecurityPolicy This will be used to protect against CVE-2018-10859, where an encrypted special remote is fed the wrong encrypted data, and so tricked into decrypting something that the user encrypted with their gpg key and did not store in git-annex. It also protects against CVE-2018-10857, where a remote follows a http redirect to a file:// url or to a local private web server. While that's already been prevented in git-annex's own use of http, external special remotes, hooks, etc use other http implementations and could still be vulnerable. The policy is not yet enforced, this commit only adds the appropriate metadata to remotes. This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2018-06-21 15:35:27 +00:00
{- Is the Key variety backed by a hash, which allows verifying content?
- It does not have to be cryptographically secure against eg birthday
- attacks.
-}
isVerifiable :: KeyVariety -> Bool
isVerifiable (SHA2Key _ _) = True
isVerifiable (SHA3Key _ _) = True
isVerifiable (SKEINKey _ _) = True
isVerifiable (Blake2bKey _ _) = True
isVerifiable (Blake2sKey _ _) = True
isVerifiable (Blake2spKey _ _) = True
isVerifiable (SHA1Key _) = True
isVerifiable (MD5Key _) = True
isVerifiable WORMKey = False
isVerifiable URLKey = False
isVerifiable (OtherKey _) = False
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
formatKeyVariety :: KeyVariety -> String
formatKeyVariety v = case v of
SHA2Key sz e -> adde e (addsz sz "SHA")
SHA3Key sz e -> adde e (addsz sz "SHA3_")
SKEINKey sz e -> adde e (addsz sz "SKEIN")
Added backends for the BLAKE2 family of hashes. There are a lot of different variants and sizes, I suppose we might as well export all the common ones. Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support. Even android has 0.16 or newer. On Debian, Blake2bp_512 is buggy, so I have omitted it for now. http://bugs.debian.org/892855 This commit was sponsored by andrea rota.
2018-03-13 20:15:35 +00:00
Blake2bKey sz e -> adde e (addsz sz "BLAKE2B")
Blake2sKey sz e -> adde e (addsz sz "BLAKE2S")
Blake2spKey sz e -> adde e (addsz sz "BLAKE2SP")
add KeyVariety type Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
2017-02-24 19:16:56 +00:00
SHA1Key e -> adde e "SHA1"
MD5Key e -> adde e "MD5"
WORMKey -> "WORM"
URLKey -> "URL"
OtherKey s -> s
where
adde (HasExt False) s = s
adde (HasExt True) s = s ++ "E"
addsz (HashSize n) s = s ++ show n
parseKeyVariety :: String -> KeyVariety
Added backends for the BLAKE2 family of hashes. There are a lot of different variants and sizes, I suppose we might as well export all the common ones. Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support. Even android has 0.16 or newer. On Debian, Blake2bp_512 is buggy, so I have omitted it for now. http://bugs.debian.org/892855 This commit was sponsored by andrea rota.
2018-03-13 20:15:35 +00:00
parseKeyVariety "SHA256" = SHA2Key (HashSize 256) (HasExt False)
parseKeyVariety "SHA256E" = SHA2Key (HashSize 256) (HasExt True)
parseKeyVariety "SHA512" = SHA2Key (HashSize 512) (HasExt False)
parseKeyVariety "SHA512E" = SHA2Key (HashSize 512) (HasExt True)
parseKeyVariety "SHA224" = SHA2Key (HashSize 224) (HasExt False)
parseKeyVariety "SHA224E" = SHA2Key (HashSize 224) (HasExt True)
parseKeyVariety "SHA384" = SHA2Key (HashSize 384) (HasExt False)
parseKeyVariety "SHA384E" = SHA2Key (HashSize 384) (HasExt True)
parseKeyVariety "SHA3_512" = SHA3Key (HashSize 512) (HasExt False)
parseKeyVariety "SHA3_512E" = SHA3Key (HashSize 512) (HasExt True)
parseKeyVariety "SHA3_384" = SHA3Key (HashSize 384) (HasExt False)
parseKeyVariety "SHA3_384E" = SHA3Key (HashSize 384) (HasExt True)
parseKeyVariety "SHA3_256" = SHA3Key (HashSize 256) (HasExt False)
parseKeyVariety "SHA3_256E" = SHA3Key (HashSize 256) (HasExt True)
parseKeyVariety "SHA3_224" = SHA3Key (HashSize 224) (HasExt False)
parseKeyVariety "SHA3_224E" = SHA3Key (HashSize 224) (HasExt True)
parseKeyVariety "SKEIN512" = SKEINKey (HashSize 512) (HasExt False)
parseKeyVariety "SKEIN512E" = SKEINKey (HashSize 512) (HasExt True)
parseKeyVariety "SKEIN256" = SKEINKey (HashSize 256) (HasExt False)
parseKeyVariety "SKEIN256E" = SKEINKey (HashSize 256) (HasExt True)
fix build with cryptonite-0.20 Some blake hash varieties were not yet available in that version. Rather than tracking exact details of what cryptonite supported when, disable blake unless using a current cryptonite.
2018-03-15 15:16:00 +00:00
#if MIN_VERSION_cryptonite(0,23,0)
Added backends for the BLAKE2 family of hashes. There are a lot of different variants and sizes, I suppose we might as well export all the common ones. Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support. Even android has 0.16 or newer. On Debian, Blake2bp_512 is buggy, so I have omitted it for now. http://bugs.debian.org/892855 This commit was sponsored by andrea rota.
2018-03-13 20:15:35 +00:00
parseKeyVariety "BLAKE2B160" = Blake2bKey (HashSize 160) (HasExt False)
parseKeyVariety "BLAKE2B160E" = Blake2bKey (HashSize 160) (HasExt True)
parseKeyVariety "BLAKE2B224" = Blake2bKey (HashSize 224) (HasExt False)
parseKeyVariety "BLAKE2B224E" = Blake2bKey (HashSize 224) (HasExt True)
parseKeyVariety "BLAKE2B256" = Blake2bKey (HashSize 256) (HasExt False)
parseKeyVariety "BLAKE2B256E" = Blake2bKey (HashSize 256) (HasExt True)
parseKeyVariety "BLAKE2B384" = Blake2bKey (HashSize 384) (HasExt False)
parseKeyVariety "BLAKE2B384E" = Blake2bKey (HashSize 384) (HasExt True)
parseKeyVariety "BLAKE2B512" = Blake2bKey (HashSize 512) (HasExt False)
parseKeyVariety "BLAKE2B512E" = Blake2bKey (HashSize 512) (HasExt True)
parseKeyVariety "BLAKE2S160" = Blake2sKey (HashSize 160) (HasExt False)
parseKeyVariety "BLAKE2S160E" = Blake2sKey (HashSize 160) (HasExt True)
parseKeyVariety "BLAKE2S224" = Blake2sKey (HashSize 224) (HasExt False)
parseKeyVariety "BLAKE2S224E" = Blake2sKey (HashSize 224) (HasExt True)
parseKeyVariety "BLAKE2S256" = Blake2sKey (HashSize 256) (HasExt False)
parseKeyVariety "BLAKE2S256E" = Blake2sKey (HashSize 256) (HasExt True)
parseKeyVariety "BLAKE2SP224" = Blake2spKey (HashSize 224) (HasExt False)
parseKeyVariety "BLAKE2SP224E" = Blake2spKey (HashSize 224) (HasExt True)
parseKeyVariety "BLAKE2SP256" = Blake2spKey (HashSize 256) (HasExt False)
parseKeyVariety "BLAKE2SP256E" = Blake2spKey (HashSize 256) (HasExt True)
fix build with cryptonite-0.20 Some blake hash varieties were not yet available in that version. Rather than tracking exact details of what cryptonite supported when, disable blake unless using a current cryptonite.
2018-03-15 15:16:00 +00:00
#endif
Added backends for the BLAKE2 family of hashes. There are a lot of different variants and sizes, I suppose we might as well export all the common ones. Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support. Even android has 0.16 or newer. On Debian, Blake2bp_512 is buggy, so I have omitted it for now. http://bugs.debian.org/892855 This commit was sponsored by andrea rota.
2018-03-13 20:15:35 +00:00
parseKeyVariety "SHA1" = SHA1Key (HasExt False)
parseKeyVariety "SHA1E" = SHA1Key (HasExt True)
parseKeyVariety "MD5" = MD5Key (HasExt False)
parseKeyVariety "MD5E" = MD5Key (HasExt True)
parseKeyVariety "WORM" = WORMKey
parseKeyVariety "URL" = URLKey
parseKeyVariety s = OtherKey s
Reference in a new issue Copy permalink