From a2b2708ab61bfd64fbf5fd8e17d07c72fd4c6213 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Fri, 17 Apr 2020 20:34:55 +0000 Subject: [PATCH] Added a comment: quick follow up --- ...ent_5_0e7b492da14e067c34693b7be02e6864._comment | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__/comment_5_0e7b492da14e067c34693b7be02e6864._comment diff --git a/doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__/comment_5_0e7b492da14e067c34693b7be02e6864._comment b/doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__/comment_5_0e7b492da14e067c34693b7be02e6864._comment new file mode 100644 index 0000000000..c759ea9b36 --- /dev/null +++ b/doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__/comment_5_0e7b492da14e067c34693b7be02e6864._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="quick follow up" + date="2020-04-17T20:34:55Z" + content=""" +> It's not clear to me either how datalad's --jobs interacts with git-annex's -J, does it pass through or do you run multiple git-annex processes? + +ATM we just run a single `annex get` with `-J` option ATM (FWIW -- in `--batch` mode IIRC). Things might change in the future to balance across different submodules. + +> All these -J5 etc values seem a bit high. I doubt that more than -J2 makes a lot of sense given the command stages optimisation, that makes it use 6 threads and balance the work better than it used to. + +I could do some timing later on, but I did see benefits as I could not go over 40-60MBps in a single download process (e.g. from S3) but parallel ones (even as high as 8 or 10) could easily carry that throughput in parallel, thus scaling up quite nicely. If interested -- you could experiment on smaug to which you have access to possibly observe similar effects. +"""]]