comment

2021-06-23 17:30:50 -04:00 · 2021-06-23 17:30:50 -04:00 · a3a6e64122
commit a3a6e64122
parent af64c184f3
1 changed files with 102 additions and 0 deletions
--- a/doc/todo/add_option_to_use_sqlite39s_synchronous61OFF/comment_2_e262f441e1b224215e68b5707db1d180._comment
+++ b/doc/todo/add_option_to_use_sqlite39s_synchronous61OFF/comment_2_e262f441e1b224215e68b5707db1d180._comment
@ -0,0 +1,102 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2021-06-23T19:15:55Z"
+ content="""
+Benchmarked with NORMAL:
+
+	joey@darkstar:~/tmp/t>/usr/bin/time ~/git-annex.synchrousNORMAL add 1??? --quiet
+	6.99user 5.09system 0:11.63elapsed 103%CPU (0avgtext+0avgdata 68356maxresident)k
+	143760inputs+40352outputs (819major+404774minor)pagefaults 0swaps
+	joey@darkstar:~/tmp/t>/usr/bin/time ~/git-annex.synchrousNORMAL add 2??? --quiet
+	7.71user 5.15system 0:11.93elapsed 107%CPU (0avgtext+0avgdata 69876maxresident)k
+	11336inputs+42648outputs (9major+414417minor)pagefaults 0swaps
+	joey@darkstar:~/tmp/t>/usr/bin/time ~/git-annex.synchrousNORMAL add 3??? --quiet
+	7.99user 5.16system 0:12.20elapsed 107%CPU (0avgtext+0avgdata 70452maxresident)k
+	11952inputs+44200outputs (8major+415267minor)pagefaults 0swaps
+	joey@darkstar:~/tmp/t>/usr/bin/time ~/git-annex.synchrousNORMAL add 4??? --quiet
+	8.30user 5.25system 0:12.62elapsed 107%CPU (0avgtext+0avgdata 69496maxresident)k
+	17784inputs+45776outputs (9major+416640minor)pagefaults 0swaps
+
+Which is no improvement over git-annex with no pragmas. Actually slower.
+
+	joey@darkstar:~/tmp/t>/usr/bin/time ~/git-annex.orig add 1??? --quiet
+	6.89user 5.36system 0:11.39elapsed 107%CPU (0avgtext+0avgdata 50576maxresident)k
+	47064inputs+40352outputs (5616major+404472minor)pagefaults 0swaps
+	joey@darkstar:~/tmp/u>/usr/bin/time git-annex add 2??? --quiet
+	7.76user 5.09system 0:11.88elapsed 108%CPU (0avgtext+0avgdata 70848maxresident)k
+	12776inputs+42648outputs (9major+414346minor)pagefaults 0swaps
+	joey@darkstar:~/tmp/u>/usr/bin/time git-annex add 3??? --quiet
+	7.90user 5.26system 0:12.14elapsed 108%CPU (0avgtext+0avgdata 71676maxresident)k
+	13824inputs+44200outputs (8major+415258minor)pagefaults 0swaps
+	joey@darkstar:~/tmp/u>/usr/bin/time git-annex add 4??? --quiet
+	8.22user 5.38system 0:12.49elapsed 108%CPU (0avgtext+0avgdata 71652maxresident)k
+	14216inputs+45776outputs (8major+416784minor)pagefaults 0swaps
+
+OFF also benchmarks very close to the same.
+
+	joey@darkstar:~/tmp/v>/usr/bin/time ~/git-annex.synchrousOFF add 1??? --quiet
+	6.85user 5.58system 0:12.01elapsed 103%CPU (0avgtext+0avgdata 71100maxresident)k
+	50080inputs+40352outputs (16major+405312minor)pagefaults 0swaps
+	joey@darkstar:~/tmp/v>/usr/bin/time ~/git-annex.synchrousOFF add 2??? --quiet
+	7.64user 5.31system 0:11.96elapsed 108%CPU (0avgtext+0avgdata 71392maxresident)k
+	12672inputs+42640outputs (8major+414373minor)pagefaults 0swaps
+	joey@darkstar:~/tmp/v>/usr/bin/time ~/git-annex.synchrousOFF add 3??? --quiet
+	8.02user 5.15system 0:12.19elapsed 108%CPU (0avgtext+0avgdata 71556maxresident)k
+	11648inputs+43928outputs (8major+415140minor)pagefaults 0swaps
+	joey@darkstar:~/tmp/v>/usr/bin/time ~/git-annex.synchrousOFF add 4??? --quiet
+	8.24user 5.24system 0:12.41elapsed 108%CPU (0avgtext+0avgdata 71224maxresident)k
+	10952inputs+45304outputs (8major+416560minor)pagefaults 0swaps
+
+One pass did run 0.08s faster, could be due to not syncing but it does
+not seem a significant optimisation, at least not on this SSD.
+
+Should be noted that transactions build up 1000 changes, and that benchmark
+was operating on 1000 files per run, so it probably only wrote one or two
+transactions.
+
+Here's the patch that adds a pragma:
+
+	diff --git a/Database/Handle.hs b/Database/Handle.hs
+	index d7f1822dc..2d66af5e6 100644
+	--- a/Database/Handle.hs
+	+++ b/Database/Handle.hs
+	@@ -1,11 +1,11 @@
+	 {- Persistent sqlite database handles.
+	  -
+	- - Copyright 2015-2019 Joey Hess <id@joeyh.name>
+	+ - Copyright 2015-2021 Joey Hess <id@joeyh.name>
+	  -
+	  - Licensed under the GNU AGPL version 3 or higher.
+	  -}
+	 
+	-{-# LANGUAGE TypeFamilies, FlexibleContexts #-}
+	+{-# LANGUAGE TypeFamilies, FlexibleContexts, OverloadedStrings #-}
+	 
+	 module Database.Handle (
+	 	DbHandle,
+	@@ -34,6 +34,7 @@ import qualified Data.Text as T
+	 import Control.Monad.Trans.Resource (runResourceT)
+	 import Control.Monad.Logger (runNoLoggingT)
+	 import System.IO
+	+import Lens.Micro
+	 
+	 {- A DbHandle is a reference to a worker thread that communicates with
+	  - the database. It has a MVar which Jobs are submitted to. -}
+	@@ -194,10 +195,13 @@ runSqliteRobustly tablename db a = do
+	 	maxretries = 100 :: Int
+	 	
+	 	rethrow msg e = throwIO $ userError $ show e ++ "(" ++ msg ++ ")"
+	-	
+	+
+	+	conninfo = over extraPragmas (const ["PRAGMA synchronous=OFF"]) $
+	+		mkSqliteConnectionInfo db
+	+
+	 	go conn retries = do
+	 		r <- try $ runResourceT $ runNoLoggingT $
+	-			withSqlConnRobustly (wrapConnection conn) $
+	+			withSqlConnRobustly (wrapConnectionInfo conninfo conn) $
+	 				runSqlConn a
+	 		case r of
+	 			Right v -> return v
+"""]]