From b6b29a62a7acef2393181cffbd5ca9fa838574b5 Mon Sep 17 00:00:00 2001 From: erewhon Date: Thu, 5 Mar 2020 19:53:37 +0000 Subject: [PATCH 1/4] --- doc/forum/Preferred_contents_based_on_presence_in_repo.mdwn | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/forum/Preferred_contents_based_on_presence_in_repo.mdwn diff --git a/doc/forum/Preferred_contents_based_on_presence_in_repo.mdwn b/doc/forum/Preferred_contents_based_on_presence_in_repo.mdwn new file mode 100644 index 0000000000..8436c25fd4 --- /dev/null +++ b/doc/forum/Preferred_contents_based_on_presence_in_repo.mdwn @@ -0,0 +1,5 @@ +For [preferred content expressions](https://git-annex.branchable.com/git-annex-preferred-content/), is there an option analogous to the "--in=repo" [matching option](https://git-annex.branchable.com/git-annex-matching-options/)? + +I was surprised not to find it, but maybe I have overlooked something. + +Many thanks From dda1e8a2c8d999ee1823deb80611c2c69332730a Mon Sep 17 00:00:00 2001 From: kyle Date: Thu, 5 Mar 2020 19:56:50 +0000 Subject: [PATCH 2/4] Added a comment: re: annexing only large dotfiles --- ...ment_2_15de9ba04480697bdf4086cdf91226e4._comment | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 doc/forum/Get_annex.dotfiles__61__true_behavior_without_persistent_configuration__63__/comment_2_15de9ba04480697bdf4086cdf91226e4._comment diff --git a/doc/forum/Get_annex.dotfiles__61__true_behavior_without_persistent_configuration__63__/comment_2_15de9ba04480697bdf4086cdf91226e4._comment b/doc/forum/Get_annex.dotfiles__61__true_behavior_without_persistent_configuration__63__/comment_2_15de9ba04480697bdf4086cdf91226e4._comment new file mode 100644 index 0000000000..71d9aa7e2a --- /dev/null +++ b/doc/forum/Get_annex.dotfiles__61__true_behavior_without_persistent_configuration__63__/comment_2_15de9ba04480697bdf4086cdf91226e4._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="kyle" + avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" + subject="re: annexing only large dotfiles" + date="2020-03-05T19:56:50Z" + content=""" +Thanks for the reply, Ilya. Sorry, I think my mention of `.datalad/` +along with an example that used a top-level dotfile was unnecessarily +confusing. It's not an issue of sending some files dotfiles to annex +and some to git. It's just wanting to avoid setting `annex.dotfiles` +in `.git/config` (or `git-annex:config.log`) of the repos that DataLad +touches. +"""]] From b078bc6f9189259d8e8a2ce5c1bf58c3c2dfd46c Mon Sep 17 00:00:00 2001 From: kyle Date: Thu, 5 Mar 2020 20:26:02 +0000 Subject: [PATCH 3/4] Added a comment: can use copies=groupname:number --- ..._1_2109fe43fccc6576e782a2e765eb6b50._comment | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/forum/Preferred_contents_based_on_presence_in_repo/comment_1_2109fe43fccc6576e782a2e765eb6b50._comment diff --git a/doc/forum/Preferred_contents_based_on_presence_in_repo/comment_1_2109fe43fccc6576e782a2e765eb6b50._comment b/doc/forum/Preferred_contents_based_on_presence_in_repo/comment_1_2109fe43fccc6576e782a2e765eb6b50._comment new file mode 100644 index 0000000000..5746c77d01 --- /dev/null +++ b/doc/forum/Preferred_contents_based_on_presence_in_repo/comment_1_2109fe43fccc6576e782a2e765eb6b50._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="kyle" + avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" + subject="can use copies=groupname:number" + date="2020-03-05T20:26:01Z" + content=""" +You could use the approach mentioned in the `preferred-content` +manpage: + + Preferred content expressions have no equivalent to the `--in` + option, but groups can accomplish similar things. You can add + repositories to groups, and match against the groups in a + preferred content expression. So rather than `--in=usbdrive`, + put all the USB drives into a \"transfer\" group, and use + `copies=transfer:1` + +"""]] From be2e83987033f827113ebecf4fd9b5ebefacd9f4 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Thu, 5 Mar 2020 20:26:48 +0000 Subject: [PATCH 4/4] initial report on "Too many open files" on OSX --- ...usted___40__Too_many_open_files__41__.mdwn | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__.mdwn diff --git a/doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__.mdwn b/doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__.mdwn new file mode 100644 index 0000000000..3e58b2cde4 --- /dev/null +++ b/doc/bugs/get_-J8_on_OSX_leads_to_git-annex__58___git__58___createProcess__58___runInteractiveProcess__58___pipe__58___resource_exhausted___40__Too_many_open_files__41__.mdwn @@ -0,0 +1,52 @@ +### Please describe the problem. + +running `git annex get -J8` on OSX (git annex 8.20200226 installed via brew) + + - distribution: 10.14.6/x86_64 + - encoding: + - default: utf-8 + - filesystem: utf-8 + - locale.prefered: UTF-8 + - max_path_length: 272 + - name: Darwin + - release: 18.7.0 + - type: posix + - version: Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 + + +results in git annex failing to get some files with errors about too many open files: + +[[!format sh """ +get .datalad/metadata/objects/0b/cn-65a581e2e22f95210db4d750cbcd1c.xz (from origin...) (checksum...) ok +get .datalad/metadata/objects/16/cn-020f8a40eed616291d855f13793e6e.xz (from origin...) (checksum...) ok +get .datalad/metadata/objects/0e/ds-2e262d6ab40f1f0fa399e80866f732 (from origin...) (checksum...) ok +get .datalad/metadata/objects/0f/cn-550a7aca2cc94645e63cda96476d40.xz +git-annex: git: createProcess: runInteractiveProcess: pipe: resource exhausted (Too many open files) +failed +get .datalad/metadata/objects/0f/cn-3c0d085acbb8ea3af430f826828b69.xz (from origin...) (checksum...) ok +get .datalad/metadata/objects/1d/ds-8e8fbd9d2834f108c7bd5ddb4294eb (from origin...) (checksum...) ok +get .datalad/metadata/objects/0b/ds-cd8a472d84af1980703baca7ad4910 (from origin...) (checksum...) ok +get .datalad/metadata/objects/1b/ds-9f695168fd28140c49a45523b9ce5f (from origin...) (checksum...) ok +get .datalad/metadata/objects/21/ds-92598325890e1d079e1a0a8e4fd9df +git-annex: git: createProcess: runInteractiveProcess: pipe: resource exhausted (Too many open files) +failed +get .datalad/metadata/objects/0f/ds-550a7aca2cc94645e63cda96476d40 (from origin...) (checksum...) ok +get .datalad/metadata/objects/06/cn-2c3eade47bd2d9052658c6a9d10a57.xz (from origin...) (checksum...) ok +get .datalad/metadata/objects/21/cn-92598325890e1d079e1a0a8e4fd9df.xz (from origin...) (checksum...) ok +get .datalad/metadata/objects/1b/cn-f55a26461f99a8ac1dde6b618e3425.xz (from origin...) (checksum...) +git-annex: .git/annex/othertmp.lck: openFd: resource exhausted (Too many open files) +failed +get .datalad/metadata/objects/26/ds-0ad917bee8d05db1dd27d0ad50c1bb +git-annex: git: createProcess: runInteractiveProcess: pipe: resource exhausted (Too many open files) +"""]] + +Full bundle of logs with a bit older (20191230) and this newer version, and full output of `datalad wtf` is available from [http://www.onerussian.com/tmp/git-annex-J8-logs.tar.gz](http://www.onerussian.com/tmp/git-annex-J8-logs.tar.gz). + +possibly related, but not yet marked done [issue I found is from 6 years ago](https://git-annex.branchable.com/bugs/Resource_exhausted/). + +I have tried to replicate this on some other OSX boxes a few weeks back but I have not reproduced it, so might be specific about this one. FWIW `ulimit` reported "unlimited" + +I will try to get a chance to troubleshoot it more to provide possibly more details whenever I get access to it again. Meanwhile just informing, and would welcome additional instructions on what to look at (besides adding `--debug` and running lsof in a loop in parallel) to troubleshoot it. + +[[!meta author=yoh]] +[[!tag projects/datalad]]