From 5330e3772f9a2555ddffca0fbb56007580a4b24c Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 13 Nov 2019 22:33:59 +0000 Subject: [PATCH 1/8] Added a comment: catch-all deadlock breaker --- ...comment_6_880c962d5ca77c494c984e7f74725265._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_6_880c962d5ca77c494c984e7f74725265._comment diff --git a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_6_880c962d5ca77c494c984e7f74725265._comment b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_6_880c962d5ca77c494c984e7f74725265._comment new file mode 100644 index 0000000000..f8d18b1cf6 --- /dev/null +++ b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_6_880c962d5ca77c494c984e7f74725265._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="catch-all deadlock breaker" + date="2019-11-13T22:33:59Z" + content=""" +Not sure if feasible, but maybe a [[catch-all deadlock breaker|todo/more_extensive_retries_to_mask_transient_failures]] could be implemented to mask this and other deadlocks? + +The moon landings software [[had something|https://www.ibiblio.org/apollo/hrst/archive/1033.pdf]] [[like this|https://history.nasa.gov/computers/Ch2-6.html]], and it worked [[pretty well|https://www.wsj.com/articles/apollo-11-had-a-hidden-hero-software-11563153001]]... +"""]] From 9f6924356d0dec52ec8d6619711a5421c20f9937 Mon Sep 17 00:00:00 2001 From: "git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f" Date: Thu, 14 Nov 2019 00:38:56 +0000 Subject: [PATCH 2/8] Added a comment --- .../comment_2_ff0dab8dc544f7aa8e4ebfbec8f2a081._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common/comment_2_ff0dab8dc544f7aa8e4ebfbec8f2a081._comment diff --git a/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common/comment_2_ff0dab8dc544f7aa8e4ebfbec8f2a081._comment b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common/comment_2_ff0dab8dc544f7aa8e4ebfbec8f2a081._comment new file mode 100644 index 0000000000..0e4a37c0ec --- /dev/null +++ b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common/comment_2_ff0dab8dc544f7aa8e4ebfbec8f2a081._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f" + nickname="git-annex" + avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4" + subject="comment 2" + date="2019-11-14T00:38:56Z" + content=""" +Thank's Joey - that's what I needed. +"""]] From 25af2da994cede6007d14a8fab54a5225af991c4 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Thu, 14 Nov 2019 03:27:37 +0000 Subject: [PATCH 3/8] initial report of standalone build regression --- ...-g908476a9b-1__126__ndall+1_some_time.mdwn | 75 +++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn diff --git a/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn new file mode 100644 index 0000000000..153a7e2c8d --- /dev/null +++ b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn @@ -0,0 +1,75 @@ +### Please describe the problem. + +There were a few changes introduced since then to Makefile (I will not guess which one broke it) which resulted in git within git-annex-standalone of neurodebian to be unable to clone from https://: + + +[[!format sh """ +$> /usr/lib/git-annex.linux/git clone https://github.com/afni/afni_ci_test_data.git +Cloning into 'afni_ci_test_data'... +fatal: unable to find remote helper for 'https' + +"""]] + +
+diff between list of files in 7.20190819+git60-gcdb679818 and 7.20191017+git2-g7b13db551 package builds shows many git-* missing +[[!format sh """ +lena:/tmp +$> ls 7.2019*/usr/lib/git-annex.linux/exe/ +7.20190819/usr/lib/git-annex.linux/exe/: +cp@ git-diff-index@ git-mktag@ git-sh-i18n--envsubst@ +curl@ git-diff-tree@ git-mktree@ git-shell@ +git@ git-difftool@ git-multi-pack-index@ git-shortlog@ +git-add@ git-fast-export@ git-mv@ git-show@ +git-am@ git-fast-import@ git-name-rev@ git-show-branch@ +git-annex@ git-fetch@ git-notes@ git-show-index@ +git-annex-shell@ git-fetch-pack@ git-pack-objects@ git-show-ref@ +git-annotate@ git-fmt-merge-msg@ git-pack-redundant@ git-stage@ +git-apply@ git-for-each-ref@ git-pack-refs@ git-status@ +git-archive@ git-format-patch@ git-patch-id@ git-stripspace@ +git-bisect--helper@ git-fsck@ git-prune@ git-submodule--helper@ +git-blame@ git-fsck-objects@ git-prune-packed@ git-symbolic-ref@ +git-branch@ git-gc@ git-pull@ git-tag@ +git-bundle@ git-get-tar-commit-id@ git-push@ git-unpack-file@ +git-cat-file@ git-grep@ git-range-diff@ git-unpack-objects@ +git-check-attr@ git-hash-object@ git-read-tree@ git-update-index@ +git-check-ignore@ git-help@ git-rebase@ git-update-ref@ +git-check-mailmap@ git-http-backend@ git-rebase--interactive@ git-update-server-info@ +git-check-ref-format@ git-http-fetch@ git-receive-pack@ git-upload-archive@ +git-checkout@ git-http-push@ git-reflog@ git-upload-pack@ +git-checkout-index@ git-imap-send@ git-remote@ git-var@ +git-cherry@ git-index-pack@ git-remote-ext@ git-verify-commit@ +git-cherry-pick@ git-init@ git-remote-fd@ git-verify-pack@ +git-clean@ git-init-db@ git-remote-ftp@ git-verify-tag@ +git-clone@ git-interpret-trailers@ git-remote-ftps@ git-whatchanged@ +git-column@ git-log@ git-remote-http@ git-worktree@ +git-commit@ git-ls-files@ git-remote-https@ git-write-tree@ +git-commit-graph@ git-ls-remote@ git-remote-testsvn@ localedef@ +git-commit-tree@ git-ls-tree@ git-remote-tor-annex@ lsof@ +git-config@ git-mailinfo@ git-repack@ rsync@ +git-count-objects@ git-mailsplit@ git-replace@ sh@ +git-credential@ git-merge@ git-rerere@ ssh@ +git-credential-cache@ git-merge-base@ git-reset@ ssh-keygen@ +git-credential-cache--daemon@ git-merge-file@ git-rev-list@ tar@ +git-credential-store@ git-merge-index@ git-rev-parse@ uname@ +git-daemon@ git-merge-ours@ git-revert@ xargs@ +git-describe@ git-merge-recursive@ git-rm@ +git-diff@ git-merge-subtree@ git-send-pack@ +git-diff-files@ git-merge-tree@ git-serve@ + +7.20191017/usr/lib/git-annex.linux/exe/: +cp@ git-credential-cache--daemon@ git-http-push@ git-sh-i18n--envsubst@ sh@ +curl@ git-credential-store@ git-imap-send@ git-shell@ ssh@ +git@ git-daemon@ git-receive-pack@ git-upload-pack@ ssh-keygen@ +git-annex@ git-fast-import@ git-remote-http@ localedef@ tar@ +git-annex-shell@ git-http-backend@ git-remote-testsvn@ lsof@ uname@ +git-credential-cache@ git-http-fetch@ git-remote-tor-annex@ rsync@ xargs@ + + +"""]] + +so may be that is related. + +Unfortunately in datalad we had no test testing cloning over https, so I added such integration test in https://github.com/datalad/datalad/pull/3867 to at least detect such regressions in the future before hitting the userland + +[[!meta author=yoh]] +[[!tag projects/datalad]] From 45ad67c6f0b8e5a6ae579e566662447f8f131471 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Thu, 14 Nov 2019 03:28:27 +0000 Subject: [PATCH 4/8] forgotten /details --- ...er_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn index 153a7e2c8d..acdcb73e8d 100644 --- a/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn +++ b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn @@ -67,6 +67,8 @@ git-credential-cache@ git-http-fetch@ git-remote-tor-annex@ rsy """]] +
+ so may be that is related. Unfortunately in datalad we had no test testing cloning over https, so I added such integration test in https://github.com/datalad/datalad/pull/3867 to at least detect such regressions in the future before hitting the userland From 621128cfe596894fe2fe7076fe3b1f99062312fe Mon Sep 17 00:00:00 2001 From: "git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f" Date: Thu, 14 Nov 2019 04:36:07 +0000 Subject: [PATCH 5/8] --- ...nt_and_gitlab_as_a_metadata_only_repo.mdwn | 63 +++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo.mdwn diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo.mdwn b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo.mdwn new file mode 100644 index 0000000000..c7f821b2b0 --- /dev/null +++ b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo.mdwn @@ -0,0 +1,63 @@ +Greetings, + +Two more issues have come up. + +1) webapp (and assistant) don't always seem to inherit ssh-agent keys (I've setup a passwordless key for gitlab). It usually takes multiple cycles of killing off ssh and annex-related daemons before a new instance will not fail auth with gitlab - is there a more solid way of doing this? + +2) When I do get auth, the assistant / webui will notice file changes (i.e moving a small file that doesn't match annex.largefiles, but the changes don't get committed to origin (gitlab - metadata only). The log file is below - thougts? + +[2019-11-13 20:24:33.378362] main: starting assistant version 7.20191106 +[2019-11-13 20:24:33.439501] TransferScanner: Syncing with origin +(scanning...) [2019-11-13 20:24:33.529954] Watcher: Performing startup scan +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' +fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif' + +git cat-file EOF: user error + +fd:39: hFlush: resource vanished (Broken pipe) + +fd:39: hFlush: resource vanished (Broken pipe) +(started...) +[2019-11-13 20:24:34.481957] Committer: Committing changes to git +(recording state in git...) +> GitLab: Disallowed command +ControlSocket .git/annex/ssh/git@gitlab.com already exists, disabling multiplexing +[2019-11-13 20:24:35.42995] Pusher: Syncing with origin +Everything up-to-date +> GitLab: Disallowed command +Everything up-to-date +> GitLab: Disallowed command +> GitLab: Disallowed command +> GitLab: Disallowed command +> GitLab: Disallowed command +> GitLab: Disallowed command + +fd:39: hFlush: resource vanished (Broken pipe) + +fd:39: hFlush: resource vanished (Broken pipe) +[2019-11-13 20:26:09.256217] main: Syncing with origin +Everything up-to-date +> GitLab: Disallowed command + +fd:39: hFlush: resource vanished (Broken pipe) +[2019-11-13 20:27:31.922901] Committer: Committing changes to git +(recording state in git...) +[2019-11-13 20:27:31.956249] Pusher: Syncing with origin +Everything up-to-date + +fd:39: hFlush: resource vanished (Broken pipe) +[2019-11-13 20:28:35.846741] Committer: Committing changes to git +(recording state in git...) +[2019-11-13 20:28:35.87987] Pusher: Syncing with origin +Everything up-to-date +> GitLab: Disallowed command +> GitLab: Disallowed command From 4934edc661e0c02eb895469bbb744b12fd75c746 Mon Sep 17 00:00:00 2001 From: "git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f" Date: Thu, 14 Nov 2019 17:24:12 +0000 Subject: [PATCH 6/8] Added a comment: Sometimes it sorta works --- ...omment_1_b61139816f55dbc4f7eac22207f243c0._comment | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_1_b61139816f55dbc4f7eac22207f243c0._comment diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_1_b61139816f55dbc4f7eac22207f243c0._comment b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_1_b61139816f55dbc4f7eac22207f243c0._comment new file mode 100644 index 0000000000..71e8b6fe65 --- /dev/null +++ b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_1_b61139816f55dbc4f7eac22207f243c0._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f" + nickname="git-annex" + avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4" + subject="Sometimes it sorta works" + date="2019-11-14T17:24:11Z" + content=""" +So, sometimes (often), it says it synched in the webapp, but no changes show up in any branch on gitlab. However, sometimes it works. For example, sometimes after killing everything off (including any ssh sessions) and restarting the webapp, a file I added to the repo locally as a test will show up in the gitlab repo. When I then move that file, the webapp/assistant wakes up, syncs (says it was successful), and the \"delete\" is processed in the gitlab repo (the old file is deleted), but the \"add\" (the file showing up with a new name) doesn't. + + +"""]] From c547b75c20a3fbf123372b38065949eaebb61a61 Mon Sep 17 00:00:00 2001 From: "git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f" Date: Thu, 14 Nov 2019 17:24:29 +0000 Subject: [PATCH 7/8] Added a comment: Sometimes it sorta works --- ...omment_2_a77d982399c91c2513ae74fb346243f0._comment | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment new file mode 100644 index 0000000000..b242899072 --- /dev/null +++ b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f" + nickname="git-annex" + avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4" + subject="Sometimes it sorta works" + date="2019-11-14T17:24:29Z" + content=""" +So, sometimes (often), it says it synched in the webapp, but no changes show up in any branch on gitlab. However, sometimes it works. For example, sometimes after killing everything off (including any ssh sessions) and restarting the webapp, a file I added to the repo locally as a test will show up in the gitlab repo. When I then move that file, the webapp/assistant wakes up, syncs (says it was successful), and the \"delete\" is processed in the gitlab repo (the old file is deleted), but the \"add\" (the file showing up with a new name) doesn't. + + +"""]] From db99ab74fc0c7158fa96210933cfc68a36aa118b Mon Sep 17 00:00:00 2001 From: "git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f" Date: Thu, 14 Nov 2019 17:40:36 +0000 Subject: [PATCH 8/8] removed --- ...omment_2_a77d982399c91c2513ae74fb346243f0._comment | 11 ----------- 1 file changed, 11 deletions(-) delete mode 100644 doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment deleted file mode 100644 index b242899072..0000000000 --- a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f" - nickname="git-annex" - avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4" - subject="Sometimes it sorta works" - date="2019-11-14T17:24:29Z" - content=""" -So, sometimes (often), it says it synched in the webapp, but no changes show up in any branch on gitlab. However, sometimes it works. For example, sometimes after killing everything off (including any ssh sessions) and restarting the webapp, a file I added to the repo locally as a test will show up in the gitlab repo. When I then move that file, the webapp/assistant wakes up, syncs (says it was successful), and the \"delete\" is processed in the gitlab repo (the old file is deleted), but the \"add\" (the file showing up with a new name) doesn't. - - -"""]]