62 lines
2.7 KiB
Markdown
62 lines
2.7 KiB
Markdown
Support eg `git-annex info --size-history=30d` which would display
|
|
the combined size of all repositories every 30 days throughout the history
|
|
of the git-annex branch. This would allow graphing, analysis, etc of repo
|
|
growth patterns.
|
|
|
|
Also, `git-annex info somerepo --size-history=30d` would display the size
|
|
of only the selected repository.
|
|
|
|
Maybe also a way to get the size of each repository plus total size in a
|
|
single line of output?
|
|
|
|
----
|
|
|
|
Implementation of this is fairly subtle. My abandoned first try just went
|
|
through `git log` and updated counters as the location logs were updated.
|
|
That resulted in bad numbers. (The size went negative eventually in fact!)
|
|
The problem is that the git-annex branch is often updated both locally and
|
|
on a remote, eg when copying a file to a remote. And that results in 2
|
|
changes to the git-annex branch that both record the same data. So it gets
|
|
counted twice by my naive implementation.
|
|
|
|
I think it is not possible for an accumulation based approach to work in
|
|
constant memory and fast. In the worst case, there is a fork of the branch
|
|
that diverges hugely over a long period of time. So that divergence either
|
|
needs to be buffered in memory, or recalculated repeatedly.
|
|
|
|
What I think needs to be done is use `git log --reverse --date-order git-annex`.
|
|
Feed the changed annex log file refs into catObjectStream to get the log
|
|
files. (Or use --patch and parse the diff to extract log file lines,
|
|
might be faster?) Parse the log files, and update a simple data structure:
|
|
|
|
Map Key [UUIDPointer]
|
|
|
|
Where UUIDPointer is a number that points to the UUID in a Map. This
|
|
avoids storing copies of the uuids in the map.
|
|
|
|
This is essentially union merging all forks of the git-annex branch at
|
|
each commit, but far faster and in memory. Since union merging a git-annex
|
|
branch can be done at any point and always results in a consistent view of
|
|
the data, this will be consistent as well.
|
|
|
|
And when updating the data structure, then it can update a counter when
|
|
something changed, and avoid updating it when a redundant log was logged.
|
|
|
|
This approach will use an amount of memory that scales with
|
|
the number of keys and numbers of copies. I mocked it up using my big
|
|
repository. Storing every key in it in such a map, with 64 UUIDPointers
|
|
in the list (many more than the usual number of copies) took 2 gb of
|
|
memory. Which is a lot but also most users have that much if necessary.
|
|
With a more usual 5 copies, memory use was only 0.5 gb. So I think this is
|
|
an acceptable exception to git-annex's desire to use a constant amount of
|
|
memory.
|
|
|
|
(I considered a bloom filter, but a false positive would wreck the
|
|
statistics. An in-memory sqlite db might be more efficient, but probably
|
|
not worth the bother.)
|
|
|
|
[[!tag confirmed]]
|
|
|
|
--[[Joey]]
|
|
|
|
> [[done]]
|