diff --git a/doc/forum/Scalability_Issues.mdwn b/doc/forum/Scalability_Issues.mdwn new file mode 100644 index 0000000000..d9f371e184 --- /dev/null +++ b/doc/forum/Scalability_Issues.mdwn @@ -0,0 +1,66 @@ +# Scalability Issues + +I am struggling with very slow operations in my cluster setup and looking for some advice. + +## Setup + +My repository contains **24 million files** with a total of **2.3 terabyte**. +The files are divided into several sub folders so the file count per directory is below 20k. +I am using the git-annex cluster setup with **one gateway** and **16 nodes**. + +The gateway repository stores a copy of all files. +The distribution of the files across the 16 nodes should be random but deterministic. Therefore, I want to use the filehash property (SHA-256) and store all files starting with one of the 16 characters `0-f` on one of the 16 nodes. For example, `node-01` stores all files starting with filehash `0...` and `node-16` stores all files starting with filehash `f...`. + +Since I did not find an immediate preferred content expression for this special distribution method, I decided to add the hash as metadata to every annexed file using a pre-commit hook. +Inside the git-annex config, I define for each node that it only stores files with one of the characters. + +``` +wanted node-01 = metadata=hash=0* +wanted node-02 = metadata=hash=1* +... +wanted node-16 = metadata=hash=f* +``` + +## Issues + +### Adding files +While adding more and more files with git-annex, I noticed that `(recording state in git...)` takes very long and my whole memory (8GB) is used by git-annex. +Therefore, increasing the `annex.queuesize` at the expense of more memory is not possible [ [1] ](https://git-annex.branchable.com/scalability/). +Is there anything else to adjust? Why does it use so much RAM? Is my number of files out of scope for git? + +### Commiting files +Furthermore, I use the following `pre-commit-annex` hook to add the filehash as metadata to the files. +However, it takes almost one second for each file to add the metadata. +I am trying to adjust the script to support batches as well but maybe there is a different way to distribute the files across nodes by the filehash? + +``` +#!/bin/bash + +if git rev-parse --verify HEAD >/dev/null 2>&1; then + against="HEAD" +else + # Initial commit: diff against an empty tree object + against="4b825dc642cb6eb9a060e54bf8d69288fbee4904" +fi + +process() { + file_hash=$(readlink "$file_path" | tail -c 65) + if \[[ -n "$file_hash" ]]; then + git -c annex.alwayscommit=false annex metadata --set "hash=$file_hash" --quiet -- "$file_path" + fi +} + +if \[[ -n "$*" ]]; then + for file_path in "$@"; do + process "$file_path" + done +else + git diff-index -z --name-only --cached $against | sed 's/\x00/\n/g' | while read file_path; do + process "$file_path" + done +fi +``` + +[1] [https://git-annex.branchable.com/scalability/](https://git-annex.branchable.com/scalability/) + +I would be glad for any feedback :)