mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Added a comment

Browse source
This commit is contained in:
Lukey 2020-07-01 20:37:13 +00:00 committed by admin
parent afc705f7c8
commit 5a64acf790
1 changed files with 12 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
doc/todo/speed_up_git_annex_sync_--content_--all/comment_6_4e307b9488b207fa24fa6b80de1623ae._comment Normal file
View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="Lukey"
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
subject="comment 6"
date="2020-07-01T20:37:13Z"
content="""
The memory consumption is indeed problematic.
Digging further, I found a even better solution:
\"time (git ls-tree -r git-annex | awk '/SHA256.*.log$/{print $3\" \"$4}' | git cat-file --batch='%(objectname) %(objecttype) %(objectsize) %(rest)' --buffer > /dev/null)\" takes just 7 seconds in my repo.
Note the '%(rest)' in the batch format, awk feeds the objectname and the key (separated by space) to cat-file, which outputs the key in place of '%(rest)'. So a git-annex thread reading from cat-file has all the information (location log and key) readily available. This can be extended to also feed metadata at the same time by making the location log directly followed by metadata.
"""]]