comment

Annex/Branch.hs

@@ -439,7 +439,7 @@ createMessage = fromMaybe "branch created" . annexCommitMessage <$> Annex.getGit
 
 {- Stages the journal, and commits staged changes to the branch. -}
 commit :: String -> Annex ()
-commit = whenM (journalDirty gitAnnexJournalDir) . forceCommit
+commit = whenM (journalDirty gitAnnexJournalDir) . tforceCommit
 
 {- Commits the current index to the branch even without any journalled
  - changes. -}

doc/todo/more___34__filesystem_efficient__34___journalling__63___/comment_1_5bbc2d651e4ce0ea5b20f199be5cb717._comment

@@ -0,0 +1,38 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-07-13T17:09:59Z"
+ content="""
+Is this the same machine that is 60x slower than my laptop at `git-annex
+init`?
+
+Is catting a single file from the journal measurably
+slower when the directory has so many files in it vs a small number?
+
+200k files in a directory does not produce a measurable slow down at reading a
+random file for me:
+
+	root@darkstar:/home/joey/tmp/d>ls -1 |wc -l
+	200000
+	root@darkstar:/home/joey/tmp/d>sync; echo 3 > /proc/sys/vm/drop_caches
+	root@darkstar:/home/joey/tmp/d>/usr/bin/time cat 99999
+	99999
+	0.00user 0.00system 0:00.00elapsed 66%CPU (0avgtext+0avgdata 1868maxresident)k
+
+At 2 million files, that cat is still running in 0s here. Writing to a new
+file and overwriting an existing file also are still very fast.
+
+It does take around 3 seconds to list the contents of the directory (37 seconds
+with 2 million files), but the only time git-annex does that is when it commits
+the journal, which is done only once per command.
+
+Committing the journal more frequently to keep the number of files down would
+mean more commits to the git-annex branch. Also committing has overhead, and
+probably more overhead than listing a directory with a lot of files in it has.
+
+What might be worth considering is periodically staging the journal to the
+annex index file without immediately committing it (still making one commit at
+the end). That seems like it would probably be safe to do, although I have not
+fully analized it. But I'm not convinced it would be an optmisation; it would
+mean repeatedly re-writing the index file, which gets expensive.
+"""]]