diff --git a/doc/bugs/significant_performance_regression_impacting_datal/comment_16_65aaf8e15cd15187cd63863634f25091._comment b/doc/bugs/significant_performance_regression_impacting_datal/comment_16_65aaf8e15cd15187cd63863634f25091._comment new file mode 100644 index 0000000000..39e0f92db3 --- /dev/null +++ b/doc/bugs/significant_performance_regression_impacting_datal/comment_16_65aaf8e15cd15187cd63863634f25091._comment @@ -0,0 +1,33 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 16""" + date="2021-06-08T20:56:50Z" + content=""" +This is starting to make some sense. If you're running git-annex add +N times adding M files each time, then each run will now diff the +changes in the index made by the previous run. + +And the first part of diffing the index is generating a tree from all the +files in it, which is to some extent `O(N*M)` (though IDK, git may have +optimisations involving subtrees or such). So the combined N git-annex add +runs come to `O(N*N*M)` + +On linux, `git write-tree` with 100,000 files in the index runs in under 1 +second, so athe `N*M` is not too bad. And then there's the overhead of +git-annex processing the resulting diff, which takes more time but is what +I've been optimising. + +Perhaps on OSX something is making the write-tree significantly slower. +Or something is making it run the command more with fewer files per run. +Although IIRC OSX has close to the same maximum command line length as +linux. + +Or maybe the index diffing is diffing from the wrong start point. +One way I can think of where this would happen is if +it somehow misdetects the index as being locked. + +A --debug trace of the one of the later git-annex add runs in that +test case would probably shed some useful light. + +Yes, --batch should avoid the problem... +"""]] diff --git a/doc/todo/display_when_reconcileStaged_is_taking_a_long_time/comment_2_b713931f34be1f06b52a75789c3b58ee._comment b/doc/todo/display_when_reconcileStaged_is_taking_a_long_time/comment_2_b713931f34be1f06b52a75789c3b58ee._comment new file mode 100644 index 0000000000..bc731d0c7a --- /dev/null +++ b/doc/todo/display_when_reconcileStaged_is_taking_a_long_time/comment_2_b713931f34be1f06b52a75789c3b58ee._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2021-06-08T16:03:13Z" + content=""" +I tried making reconcileStaged display the message itself, this is the +result: + + add foo + 100% 30 B 73 KiB/s 0s(scanning for annexed files...) + ok + +So for that to be done, showSideAction would need to clear the progress +bar display first. Note that the display is ok when concurrent output is +enabled: + + add c (scanning for annexed files...) + ok + +Ok.. Fixed that display glitch, and made reconcileStaged display +the message itself when it's taking a while to run. +"""]]