From d3fe648919a295ebe470d12997e17c769f1a0e43 Mon Sep 17 00:00:00 2001 From: cznug Date: Fri, 16 May 2025 08:29:33 +0000 Subject: [PATCH] Added a comment --- ..._76087a3733f97655a825c96a2345db36._comment | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 doc/forum/Scalability_Issues/comment_2_76087a3733f97655a825c96a2345db36._comment diff --git a/doc/forum/Scalability_Issues/comment_2_76087a3733f97655a825c96a2345db36._comment b/doc/forum/Scalability_Issues/comment_2_76087a3733f97655a825c96a2345db36._comment new file mode 100644 index 0000000000..57fb6d240a --- /dev/null +++ b/doc/forum/Scalability_Issues/comment_2_76087a3733f97655a825c96a2345db36._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="cznug" + avatar="http://cdn.libravatar.org/avatar/fc76e1657886a3bb6f2905c554d0f80c" + subject="comment 2" + date="2025-05-16T08:29:33Z" + content=""" +Thanks a lot **joey** for your help. + +I gave it another try without setting the metadata and by using v4 index. + +Instead of directly adding all files to the gateway repository, I distributed the files equally across the 16 nodes to make use of their resources. +On each node I added the file portion to a git-annex repository in order to merge them later via the gateway repository. +Adding the files on each node worked very well using the `--jobs=\"cpus\"` flag. + +However, once I tried to merge all 16 repos using `git-annex sync --no-content --allow-unrelated-histories --jobs=\"cpus\"` all of the nodes crashed due to out-of-memory during this step: + +`remote: (merging synced/git-annex bigserver/git-annex into git-annex...)` + +I assume that you are right and that I simply have too many files. + +Unfortunately, I currently cannot spend more time on investigating the issues. +Thanks again for your help. :) +"""]]