From ed7a0c9956034a3eea34d3f20f38517754f3d6cf Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Mon, 5 Oct 2020 23:11:32 +0000 Subject: [PATCH 1/4] Added a comment: just comments --- ...t_2_7d37b8d79fcb6a0edd99e0bd6f2b0515._comment | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/bugs/some_annex_addurl_--fast_--with-files_--json_--json-error-messages_--batch__do_not_quit/comment_2_7d37b8d79fcb6a0edd99e0bd6f2b0515._comment diff --git a/doc/bugs/some_annex_addurl_--fast_--with-files_--json_--json-error-messages_--batch__do_not_quit/comment_2_7d37b8d79fcb6a0edd99e0bd6f2b0515._comment b/doc/bugs/some_annex_addurl_--fast_--with-files_--json_--json-error-messages_--batch__do_not_quit/comment_2_7d37b8d79fcb6a0edd99e0bd6f2b0515._comment new file mode 100644 index 0000000000..dad9b3c741 --- /dev/null +++ b/doc/bugs/some_annex_addurl_--fast_--with-files_--json_--json-error-messages_--batch__do_not_quit/comment_2_7d37b8d79fcb6a0edd99e0bd6f2b0515._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="just comments" + date="2020-10-05T23:11:29Z" + content=""" +> git cat-file --batch that this one is missing. + +may be that one already exited \"peacefully\"? I believe I was looking at the moment when we already requested closing that `annex addurl --batch`. + +> But git-annex rarely exec's git-annex, and when it does, it's never with the same params as the parent git-annex was run with + +I wonder if may be `htop` (I believe I pasted process tree from it) somehow may be groupped incorrectly and those are \"independent\" hanging `git annex addurl` processes which got \"folded\"... I don't know.. + +Anyways, to hopefully \"shine the light\", I have managed to reproduce (with up to date snapshot of annex, didn't get there with systemwide older git-annex) on smaug using initially stalling datalad-crawler tests (so 100% reproducible): just login as `test` user to `smaug`. There is a screen session with one shell there with python's PDB where we were waiting for batched process to exit but it didn't; then 2nd one shows `ps auxw -H | grep '^test'`. Please `strace` or inspect anything you might find pertinent (IIRC, raising debug level for datalad, or disabling code tests coverage collection, was making the stall going away). FTR the command is `PATH=~/git-annex/20201005-snapshot/:$PATH DATALAD_LOG_LEVEL=WARNING DATALAD_TESTS_TEMP_KEEP=1 python -m nose -s -v --with-cov --cover-package datalad ../datalad_crawler/dbs/tests/test_files.py ../datalad_crawler/nodes/tests/test_annex.py` and it ends up in pdb (python debugger) if batched annex did not exit although we waited. +"""]] From 8035e2073ddaad72295219b255afefeafc1b9fb6 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Mon, 5 Oct 2020 23:17:08 +0000 Subject: [PATCH 2/4] Added a comment --- ...omment_11_9554e07b916b8d7ca1caf72b522fefd4._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/todo/clarify_that_v7_applies_to_all_clones/comment_11_9554e07b916b8d7ca1caf72b522fefd4._comment diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_11_9554e07b916b8d7ca1caf72b522fefd4._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_11_9554e07b916b8d7ca1caf72b522fefd4._comment new file mode 100644 index 0000000000..0dca147893 --- /dev/null +++ b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_11_9554e07b916b8d7ca1caf72b522fefd4._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 11" + date="2020-10-05T23:17:07Z" + content=""" +> .. after the upgrade that would add any, you don't need to worry about them. + +With the datalad pixie dust on top of git-annex, I am never 100% sure ;) I would better worry and do some basic check before proceeding... -- will do later today/tomorrow, God bless BTRFS and its snapshots - I can get a \"sandbox\" clone of the entire filesystem to play with safely. +"""]] From b98d601bfbe2157ee72fb3d40b65be5891b45390 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Tue, 6 Oct 2020 00:04:38 +0000 Subject: [PATCH 3/4] Added a comment: FTR --- ...omment_7_2abc6e368033bc636debbcd83dd2b4ed._comment | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 doc/bugs/standalone_runshell_can_race_and_fail_to_remove___96____126____47__.cache__47__git-annex__47__locales__47____96___dirs/comment_7_2abc6e368033bc636debbcd83dd2b4ed._comment diff --git a/doc/bugs/standalone_runshell_can_race_and_fail_to_remove___96____126____47__.cache__47__git-annex__47__locales__47____96___dirs/comment_7_2abc6e368033bc636debbcd83dd2b4ed._comment b/doc/bugs/standalone_runshell_can_race_and_fail_to_remove___96____126____47__.cache__47__git-annex__47__locales__47____96___dirs/comment_7_2abc6e368033bc636debbcd83dd2b4ed._comment new file mode 100644 index 0000000000..f43d103a66 --- /dev/null +++ b/doc/bugs/standalone_runshell_can_race_and_fail_to_remove___96____126____47__.cache__47__git-annex__47__locales__47____96___dirs/comment_7_2abc6e368033bc636debbcd83dd2b4ed._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="FTR" + date="2020-10-06T00:04:36Z" + content=""" +Thank you Joey! FTR (if I located properly): fixed in [8.20200908-163-gd74d97896](https://git.kitenet.net/index.cgi/git-annex.git/commit/?id=d74d9789688fde1151ece6a751d1c549af87de84) + +If for what \"flavor\" of git-annex build I use -- every day is not like another these days ;) But yeah -- tend to use a year old neurodebian built .deb when I can afford and do not feel adventurous, but I think in these cases was just using a standalone build indeed. + +"""]] From 95d02d6e295d0dc308e1b33e5134e1f5dfbe4701 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Tue, 6 Oct 2020 01:27:01 +0000 Subject: [PATCH 4/4] Added a comment --- ..._300f74178ef17e2e549fd7a04a262428._comment | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 doc/bugs/impossible__40____63____41___to_continuously_re-import_a_directory_while_keeping_original_files_in_place/comment_3_300f74178ef17e2e549fd7a04a262428._comment diff --git a/doc/bugs/impossible__40____63____41___to_continuously_re-import_a_directory_while_keeping_original_files_in_place/comment_3_300f74178ef17e2e549fd7a04a262428._comment b/doc/bugs/impossible__40____63____41___to_continuously_re-import_a_directory_while_keeping_original_files_in_place/comment_3_300f74178ef17e2e549fd7a04a262428._comment new file mode 100644 index 0000000000..76afcc3b47 --- /dev/null +++ b/doc/bugs/impossible__40____63____41___to_continuously_re-import_a_directory_while_keeping_original_files_in_place/comment_3_300f74178ef17e2e549fd7a04a262428._comment @@ -0,0 +1,59 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 3" + date="2020-10-06T01:26:59Z" + content=""" +I think it worked wonderfully + +
+here is my script I have tried + +```shell +#!/bin/bash + +export PS4='> ' +set -x +set -eu +cd \"$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)\" + +mkdir d-in d-repo +echo content >| d-in/file + +function dance() { + git annex import master --from d-in + # but we need to merge it + git merge d-in/master + ls -l + grep -e . * +} + +( +cd d-repo +git init +git annex init +git annex initremote d-in type=directory directory=../d-in exporttree=yes importtree=yes encryption=none + +ls -l ../d-in + +for r in 1 2; do + echo \"Run $r\"; + dance +done + +echo \"more\" >> ../d-in/file +echo \"new\" > ../d-in/newfile +dance + +rm ../d-in/file +dance + +) + +``` +
+ +and it seemed to do the right job! I have not tried to add some `.gitattributes` into that branch it imports into to tell some files to go to git, but I hope it would just work, and if not -- I will come back! feel welcome to close this issue. + +Cheers +"""]]