mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Added a comment

Browse source
This commit is contained in:
CandyAngel 2015-06-17 07:51:53 +00:00 • committed by admin
parent a558e91fc7
commit d096cd5c40
1 changed files with 13 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
doc/forum/Handling_a_large_number_of_files/comment_1_33d2e19fbfdf5a0bfb7b4fbc840824a2._comment Normal file
View file

@ -0,0 +1,13 @@
[[!comment format=mdwn
username="CandyAngel"
subject="comment 1"
date="2015-06-17T07:51:53Z"
content="""
I've posted a few tips on this site about dealing with large (file count) repositories.. I should probably collate them into a single post in tips (and I shall go so!).
In terms of directories containing hundreds of thousands of files, my research indicates that the slowness stems from how the filesystem (at least, for ext3) fetches the listing, as it requires multiple \"trips\" to query it (it only returns 32KB of metadata at a time or something like that).
So you can really speed it up by partitioning your files in such a way that a directory listing returns quickly. Personally, I use fpart and 5000 files per directory, but I think you can go up to about 20000.
If you do 'find', and it takes a really long time to *start* printing paths, you have too many files :)
"""]]