[[!comment format=mdwn
 username="joey"
 subject="""comment 32"""
 date="2021-06-14T20:26:52Z"
 content="""
Some thoughts leading to a workable plan:

It's easy to detect this edge case because getAssociatedFiles will be
returning a long list of files. So it could detect say 10 files in the list
and start doing something other than the usual, without bothering the usual
case with any extra work.

Git starts to get slow anyway in the 1 million to 10 million file range. So
we can assume less than that many files are being added. And there need to
be a fairly large number of duplicates of a key for speed to become a problem
when adding that key. Around 1000 based on above benchmarks, but 100 would
be safer.

If it's adding 10 million files, there can be at most 10000 keys
that have `>=` 1000 duplicates (10 million / 1000).
No problem to remember 10000 keys; a key is less than 128 bytes long, so
that would take 1250 kb, plus the overhead of the Map. Might as well
remember 12 mb worth of keys, to catch 100 duplicates.

It would be even better to use a bloom filter, which could remember many
more, and I thought I had a way, but the false positive case seems the
wrong way around. If the bloom filter remembers keys that have already had
their associated files populated, then a false positive would prevent doing
that for a key that it's not been done for.

It would make sense to do this not only in populateUnlockedFiles but in
Annex.Content.moveAnnex and Annex.Content.removeAnnex. Although removeAnnex
would need a different bloom filter, since a file might have been populated
and then somehow get removed in the same git-annex call.
"""]]
