summary of progress

doc/todo/make_copy_--fast__faster/comment_10_1af4ac0d37c876912678522895c1656b._comment

@@ -0,0 +1,61 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 10"""
+ date="2016-09-29T18:33:33Z"
+ content="""
+* Optimised key2file and file2key. 18% scanning time speedup.
+* Optimised adjustGitEnv. 50% git-annex branch query speedup
+* Optimised parsePOSIXTime. 10% git-annex branch query speedup
+* Tried making catObjectDetails.receive use ByteString for parsing, 
+  but that did not seem to speed it up significantly.
+  So it parsing is already fairly optimal, it's just that a
+  lot of data passes through it when querying the git-annex
+  branch.
+
+After all that, profiling `git-annex find`:
+
+	        Thu Sep 29 16:51 2016 Time and Allocation Profiling Report  (Final)
+	
+	           git-annex.1 +RTS -p -RTS find
+	
+	        total time  =        1.73 secs   (1730 ticks @ 1000 us, 1 processor)
+	        total alloc = 1,812,406,632 bytes  (excludes profiling overheads)
+	
+	COST CENTRE            MODULE                  %time %alloc
+	
+	md5                    Data.Hash.MD5            28.0   37.9
+	catchIO                Utility.Exception        10.2   12.5
+	inAnnex'.checkindirect Annex.Content             9.9    3.7
+	catches                Control.Monad.Catch       8.7    5.7
+	readish                Utility.PartialPrelude    5.7    3.0
+	isAnnexLink            Annex.Link                5.0    8.4
+	keyFile                Annex.Locations           4.2    5.8
+	spanList               Data.List.Utils           4.0    6.3
+	startswith             Data.List.Utils           2.0    1.3
+
+And `git-annex find --not --in web`:
+
+	        Thu Sep 29 16:35 2016 Time and Allocation Profiling Report  (Final)
+	
+	           git-annex +RTS -p -RTS find --not --in web
+	
+	        total time  =        5.24 secs   (5238 ticks @ 1000 us, 1 processor)
+	        total alloc = 3,293,314,472 bytes  (excludes profiling overheads)
+	
+	COST CENTRE               MODULE                      %time %alloc
+	
+	catObjectDetails.receive  Git.CatFile                  12.9    5.5
+	md5                       Data.Hash.MD5                10.6   20.8
+	readish                   Utility.PartialPrelude        7.3    8.2
+	catchIO                   Utility.Exception             6.7    7.3
+	spanList                  Data.List.Utils               4.1    7.4
+	readFileStrictAnyEncoding Utility.Misc                  3.5    1.3
+	catches                   Control.Monad.Catch           3.3    3.2
+
+So, quite a large speedup overall!
+
+This leaves md5 still unoptimised at 10-28% of CPU use. I looked at switching
+it to cryptohash's implementation, but it would require quite a lot of
+bit-banging math to pull the used values out of the ByteString containing
+the md5sum.
+"""]]