From 043201c4d8e02e66b564093d33b1b360f08b3485 Mon Sep 17 00:00:00 2001 From: cjmarkie Date: Thu, 27 Mar 2025 20:09:28 +0000 Subject: [PATCH] Propose skipping checksums in git-annex get --fast --- ...sable_checksum_with___96__get_--fast__96__.mdwn | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/todo/Disable_checksum_with___96__get_--fast__96__.mdwn diff --git a/doc/todo/Disable_checksum_with___96__get_--fast__96__.mdwn b/doc/todo/Disable_checksum_with___96__get_--fast__96__.mdwn new file mode 100644 index 0000000000..dd7abb816a --- /dev/null +++ b/doc/todo/Disable_checksum_with___96__get_--fast__96__.mdwn @@ -0,0 +1,14 @@ +In ["--fast" option for git annex get?](https://git-annex.branchable.com/forum/__34__--fast__34___option_for_git_annex_get__63__/) it was indicated that `git annex get --fast` doesn't have any effect. + +In an HPC context, users are frequently expected to use login (or dedicated data transfer) nodes for data transfer, and can get their sessions killed for excessive CPU use. For OpenNeuro, the high bandwidth between many HPC centers and S3 means that checksums can become the bottleneck in data transfer. I would like to be able to recommend something like: + +```console +git annex get -f s3-PUBLIC --fast --all +srun git annex fsck +``` + +Is this feasible? + + +[[!meta author=cjmarkie]] +[[!tag projects/openneuro]]