improve data types for sqlite

This is a non-backwards compatable change, so not suitable for merging
w/o a annex.version bump and transition code. Not yet tested.

This improves performance of git-annex benchmark --databases
across the board by 10-25%, since eg Key roundtrips as a ByteString.

(serializeKey' produces a lazy ByteString, so there is still a
copy involved in converting it to a strict ByteString. It may be faster
to switch to using bytestring-strict-builder.)

FilePath and Key are both stored as blobs. This avoids mojibake in some
situations. It would be possible to use varchar instead, if persistent
could avoid converting that to Text, but it seems there is no good
way to do so. See doc/todo/sqlite_database_improvements.mdwn

Eliminated some ugly artifacts of using Read/Show serialization;
constructors and quoted strings are no longer stored in sqlite.

Renamed SRef to SSha to reflect that it is only ever a git sha,
not a ref name. Since it is limited to the characters in a sha,
it is not affected by mojibake, so still uses String.
This commit is contained in:
Joey Hess 2019-10-29 12:28:01 -04:00
parent e1b21a0491
commit c35a9047d3
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
9 changed files with 135 additions and 189 deletions

View file

@ -1,6 +1,6 @@
{- Sqlite database used for incremental fsck.
-
- Copyright 2015 Joey Hess <id@joeyh.name>
- Copyright 2015-2019 Joey Hess <id@joeyh.name>
-:
- Licensed under the GNU AGPL version 3 or higher.
-}
@ -39,7 +39,7 @@ data FsckHandle = FsckHandle H.DbQueue UUID
- of the latest incremental fsck pass. -}
share [mkPersist sqlSettings, mkMigrate "migrateFsck"] [persistLowerCase|
Fscked
key SKey
key Key
UniqueKey key
|]
@ -74,10 +74,8 @@ closeDb (FsckHandle h u) = do
addDb :: FsckHandle -> Key -> IO ()
addDb (FsckHandle h _) k = H.queueDb h checkcommit $
void $ insertUnique $ Fscked sk
void $ insertUnique $ Fscked k
where
sk = toSKey k
-- commit queue after 1000 files or 5 minutes, whichever comes first
checkcommit sz lastcommittime
| sz > 1000 = return True
@ -87,9 +85,9 @@ addDb (FsckHandle h _) k = H.queueDb h checkcommit $
{- Doesn't know about keys that were just added with addDb. -}
inDb :: FsckHandle -> Key -> IO Bool
inDb (FsckHandle h _) = H.queryDbQueue h . inDb' . toSKey
inDb (FsckHandle h _) = H.queryDbQueue h . inDb'
inDb' :: SKey -> SqlPersistM Bool
inDb' sk = do
r <- selectList [FsckedKey ==. sk] []
inDb' :: Key -> SqlPersistM Bool
inDb' k = do
r <- selectList [FsckedKey ==. k] []
return $ not $ null r