From 47a5853da8351470e0fae9a024bdeb61a6119fab Mon Sep 17 00:00:00 2001 From: jgoerzen Date: Sun, 4 Sep 2022 22:31:14 +0000 Subject: [PATCH] Added a comment --- ...omment_11_49585db4fbf9f22bf806ee08d73b7db1._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/tips/Repositories_with_large_number_of_files/comment_11_49585db4fbf9f22bf806ee08d73b7db1._comment diff --git a/doc/tips/Repositories_with_large_number_of_files/comment_11_49585db4fbf9f22bf806ee08d73b7db1._comment b/doc/tips/Repositories_with_large_number_of_files/comment_11_49585db4fbf9f22bf806ee08d73b7db1._comment new file mode 100644 index 0000000000..a6f7af3695 --- /dev/null +++ b/doc/tips/Repositories_with_large_number_of_files/comment_11_49585db4fbf9f22bf806ee08d73b7db1._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="jgoerzen" + avatar="http://cdn.libravatar.org/avatar/090740822c9dcdb39ffe506b890981b4" + subject="comment 11" + date="2022-09-04T22:31:14Z" + content=""" +I'm trying to use a repo consisting of about 150,000 photos/videos. I tried all the tipes here as well as the ones at [[/forum /__34__git_annex_sync__34___synced_after_8_hours]] and the time is still quite poor. I don't know if using the special remote directory with importtree=yes hurts; I don't think it should. The problem seems to be largely CPU-bound and RAM-bound; syncs can use many GB of RAM and a large amount of CPU time (even when there is no evident hashing of source files). --jobs=10 hasn't caused much evident parallelization. Changing the git index type, repacking, etc. rocketed through almost instantly and made no evident change. I'd be very interested in ideas here, because at this rate, a sync that is a no-op has been running for 15 minutes just sitting there after \"list source ok\". I'll let it run and see what it does. + +If it makes a difference, this is an unlocked repo (via git annex adjust --unlocked), not running assistant. There are no directories with excessive numbers of photos. The underlying filesystem is ZFS. +"""]]