2011-04-07 18:45:10 +00:00
|
|
|
`git-annex unused` has to compare large sets of data
|
|
|
|
|
(all keys with content present in the repository,
|
|
|
|
|
with all keys used by files in the repository), and so
|
|
|
|
|
uses more memory than git-annex typically needs; around
|
2011-11-08 05:27:06 +00:00
|
|
|
50 mb when run in a repository with 80 thousand files.
|
|
|
|
|
|
|
|
|
|
(Used to be 80 mb, but implementation improved.)
|
2011-04-07 18:45:10 +00:00
|
|
|
|
|
|
|
|
I would like to reduce this. One idea is to use a bloom filter.
|
|
|
|
|
For example, construct a bloom filter of all keys used by files in
|
|
|
|
|
the repository. Then for each key with content present, check if it's
|
2011-11-08 05:27:06 +00:00
|
|
|
in the bloom filter. Since there can be false positives, this might
|
2011-04-07 18:45:10 +00:00
|
|
|
miss finding some unused keys. The probability/size of filter
|
|
|
|
|
could be tunable.
|
|
|
|
|
|
|
|
|
|
Another way might be to scan the git log for files that got removed
|
|
|
|
|
or changed what key they pointed to. Correlate with keys with content
|
|
|
|
|
currently present in the repository (possibly using a bloom filter again),
|
|
|
|
|
and that would yield a shortlist of keys that are probably not used.
|
|
|
|
|
Then scan thru all files in the repo to make sure that none point to keys
|
|
|
|
|
on the shortlist.
|
