Using Sets is the right thing; they have constant size lookup like my
SizeList, and logn insertation, which beats nub to death.
Runs faster than --fast mode did before, and gives accurate counts.
13 seconds total runtime with a warm cache in a repository with 40 thousand
keys.
Do not need to check the location log in this case, can just check inAnnex.
This is both an optimisation and perhaps a correctness measure
(fsck --in . should fsck files even if the location log is damaged.)
filterM is not a good idea if you were streaming in a large list of files.
Fixing this memory leak that I introduced earlier today was a PITA because
to avoid the filterM, it's necessary to do the filtering only after
building up the data structures like BackendFile, and that means each
separate data structure needs it own function to apply the filter,
at least in this naive implementation.
There is also a minor performance regression, when using copy/drop/get/fsck
with a filter, git is now asked to look up attributes for all files,
since that now comes before the filter is applied. This is only a very
minor thing, since getting the attributes is very fast and --exclude was
probably not typically used to speed it up.
find: Rather than only showing files whose contents are present, when used
with --exclude --copies or --in, displays all files that match the
specified conditions.
Note that this is a behavior change for find --exclude! Old behavior
can be gotten with find --in . --exclude=...
Only one place need to filter the list of remotes for ignored remotes:
keyPossibilities. Make the full list available to everything else.
This allows getting rid of the special case handing for --from and --to
to make ignored remotes not be ignored with those options.
I dislike -( and -), but without using a different option parser, can't
easily use bare parens.
--and and --or will become more useful once there are more interesting
limits than --exclude
These were a mistake, they make the type signatures harder to read and
less flexible. The CommandSeek, CommandStart, CommandPerform, and
CommandCleanup types were a good idea, but composing them with the
parameters expected is going too far.
It probably does not make sense to enable auto mode for move. I cannot
think of a situation where it would make sense to try to use it.
A hypothetical auto mode for move would only differ from a normal
move in one case -- when both repositories have a file, move deletes it
from one, and this reduces the number of copies. So an auto mode would
either only let move work in that situation, or avoid removing the file
in that situation, depending on the number of copies. This would be
complex to implement, and is perhaps not a very obvious behavior.
The error is a good thing to have, so users don't expect it to do something
it does not.
get, drop: Added --auto option, which decides whether to get/drop content
as needed to work toward the configured numcopies.
The problem with bundling it up in optimize was that I then found I wanted
to run an optmize that did not drop files, only got them. Considered adding
a --only-get switch to it, but that seemed wrong. Instead, let's make
existing subcommands optionally smarter.
Note that the only actual difference between drop and drop --auto is that
the latter does not even try to drop a file if it knows of not enough
copies, and does not print any error messages about files it was unable to
drop.
It might be nice to make get avoid asking git for attributes when not in
auto mode. For now it always asks for attributes.
