mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Added a comment

Browse source
This commit is contained in:
cznug 2025-05-16 08:29:33 +00:00 • committed by admin
parent aa880b4585
commit d3fe648919
1 changed files with 23 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

23
doc/forum/Scalability_Issues/comment_2_76087a3733f97655a825c96a2345db36._comment Normal file
View file

@ -0,0 +1,23 @@
[[!comment format=mdwn
username="cznug"
avatar="http://cdn.libravatar.org/avatar/fc76e1657886a3bb6f2905c554d0f80c"
subject="comment 2"
date="2025-05-16T08:29:33Z"
content="""
Thanks a lot **joey** for your help.
I gave it another try without setting the metadata and by using v4 index.
Instead of directly adding all files to the gateway repository, I distributed the files equally across the 16 nodes to make use of their resources.
On each node I added the file portion to a git-annex repository in order to merge them later via the gateway repository.
Adding the files on each node worked very well using the `--jobs=\"cpus\"` flag.
However, once I tried to merge all 16 repos using `git-annex sync --no-content --allow-unrelated-histories --jobs=\"cpus\"` all of the nodes crashed due to out-of-memory during this step:
`remote: (merging synced/git-annex bigserver/git-annex into git-annex...)`
I assume that you are right and that I simply have too many files.
Unfortunately, I currently cannot spend more time on investigating the issues.
Thanks again for your help. :)
"""]]