From 79874325b8827c3b3dbad79af7f3265d6d7b6f0c Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Mon, 2 Nov 2020 18:04:20 +0000 Subject: [PATCH 01/14] a plea for more --debug output --- ...-level__41___+_more_debug_information_from_git-annex.mdwn | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex.mdwn diff --git a/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex.mdwn b/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex.mdwn new file mode 100644 index 0000000000..796b4272b6 --- /dev/null +++ b/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex.mdwn @@ -0,0 +1,5 @@ +In the light of the recent debugging of git-annex+datalad special remote tandem, I would have appreciated if there was a way to see reflection of git-annex actions beyond current `--debug`: what is annex intending to do internally (not just reporting which commands to be ran etc). Most likely such higher level of debug information would generally be not needed, so probably not worth "enriching" default `--debug` output, so I wonder if there could be an option or config setting which would set `git-annex` on the path to provide more debug output if requested (e.g. like those on closing stdin/stdout/stderr/stderr-handler), may be it could be a config setting which would allow for `DEFAULT|HEAVY|VERYHEAVY` or alike values. +I hope that I would never need to use that, but it might end up being "days-savior" + +[[!meta author=yoh]] +[[!tag projects/datalad]] From 9252f86b2e9a3ee218867e1ef4572766eb44418c Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Mon, 2 Nov 2020 14:47:08 -0400 Subject: [PATCH 02/14] view: Fix a reversion in 8.20200522 that broke entering or changing views. Commit 2dc7b5186a586acb768744242083ad48d658a538 messed up indentation. This commit was sponsored by Noam Kremen on Patreon. --- Annex/View.hs | 2 +- CHANGELOG | 1 + doc/bugs/view.mdwn | 4 ++++ 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/Annex/View.hs b/Annex/View.hs index 725cce3cb6..04481a4fd9 100644 --- a/Annex/View.hs +++ b/Annex/View.hs @@ -360,7 +360,7 @@ applyView' mkviewedfile getfilemetadata view = do topf <- inRepo (toTopFilePath f) go uh topf sha (toTreeItemType mode) =<< lookupKey f liftIO $ void clean - genViewBranch view + genViewBranch view where genviewedfiles = viewedFiles view mkviewedfile -- enables memoization diff --git a/CHANGELOG b/CHANGELOG index 97b43d9030..7234d4c452 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -27,6 +27,7 @@ git-annex (8.20201008) UNRELEASED; urgency=medium backends. Make sure to relay any remaining stderr from the process, and potentially avoid the process getting a SIGPIPE if it writes to stderr too late. + * view: Fix a reversion in 8.20200522 that broke entering or changing views. -- Joey Hess Thu, 08 Oct 2020 10:48:17 -0400 diff --git a/doc/bugs/view.mdwn b/doc/bugs/view.mdwn index 4dcbf82d58..e1e44370cb 100644 --- a/doc/bugs/view.mdwn +++ b/doc/bugs/view.mdwn @@ -65,3 +65,7 @@ git-annex: view: 1 failed ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) git-annex is an awesome tool! Really appreciate your work and everything else works great! + +> Thank you for reporting, I've fixed this reversion and am also going to +> add something like that to the test suite, which lacked any testing of +> views. [[done]] --[[Joey]] From 40679616edc710ed3d3874ffa88b5119910c72bf Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Mon, 2 Nov 2020 15:01:17 -0400 Subject: [PATCH 03/14] comment --- ..._58e729e2fa58b28d247eeda579bb9329._comment | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_1_58e729e2fa58b28d247eeda579bb9329._comment diff --git a/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_1_58e729e2fa58b28d247eeda579bb9329._comment b/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_1_58e729e2fa58b28d247eeda579bb9329._comment new file mode 100644 index 0000000000..b7846d2372 --- /dev/null +++ b/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_1_58e729e2fa58b28d247eeda579bb9329._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2020-11-02T18:48:44Z" + content=""" +Every call to debugM does add overhead both in time to run it +(checking if debugging is enabled is nonzero overhead and this will add up +if it's done a few million times or whatever) and in time to write +a useful explanation. + +It seems to me that debugging is best added when trying to debug something, +or when there's a readily available value that can be output (such as the +safety proof that is debugged when dropping, or external special remote +messages). + +Trying to add enough useful debugging information to 60,000 lines of code +that any possible thing it might do will have a useful debug trace seems +like a recipe to end up with 100,000 lines of code. It would be easier to +run the code in a debugger and step through it at that point. +"""]] From c2056c1cafce5538b61ac7112771780b92c17dc1 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Mon, 2 Nov 2020 15:06:55 -0400 Subject: [PATCH 04/14] test case for view For reversion fixed in 9252f86b2e9a3ee218867e1ef4572766eb44418c This commit was sponsored by Ethan Aubin --- Test.hs | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/Test.hs b/Test.hs index b9d4a7d130..94b400954f 100644 --- a/Test.hs +++ b/Test.hs @@ -298,6 +298,7 @@ unitTests note = testGroup ("Unit Tests " ++ note) , testCase "export_import_subdir" test_export_import_subdir , testCase "shared clone" test_shared_clone , testCase "log" test_log + , testCase "view" test_view , testCase "magic" test_magic , testCase "import" test_import , testCase "reinject" test_reinject @@ -447,6 +448,14 @@ test_log :: Assertion test_log = intmpclonerepo $ do git_annex "log" [annexedfile] @? "log failed" +test_view :: Assertion +test_view = intmpclonerepo $ do + git_annex "metadata" ["-s", "test=test1", annexedfile] @? "metadata failed" + git_annex "metadata" ["-s", "test=test2", sha1annexedfile] @? "metadata failed" + git_annex "view" ["test=test1"] @? "entering view failed" + checkexists annexedfile + checkdoesnotexist sha1annexedfile + test_magic :: Assertion test_magic = intmpclonerepo $ do #ifdef WITH_MAGICMIME From e9263052210fd05bb59df7e982b3e16619b2ad82 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Mon, 2 Nov 2020 15:21:24 -0400 Subject: [PATCH 05/14] add maxlength=50 to ikiwiki template used for page title entry --- ...t_6_2474b1cc6256afefb3d0530c3a0a24e8._comment | 12 ++++++++++++ doc/templates/blogpost.tmpl | 16 ++++++++++++++++ 2 files changed, 28 insertions(+) create mode 100644 doc/bugs/Error_cloning_repository_on_Windows/comment_6_2474b1cc6256afefb3d0530c3a0a24e8._comment create mode 100644 doc/templates/blogpost.tmpl diff --git a/doc/bugs/Error_cloning_repository_on_Windows/comment_6_2474b1cc6256afefb3d0530c3a0a24e8._comment b/doc/bugs/Error_cloning_repository_on_Windows/comment_6_2474b1cc6256afefb3d0530c3a0a24e8._comment new file mode 100644 index 0000000000..0ef6e775aa --- /dev/null +++ b/doc/bugs/Error_cloning_repository_on_Windows/comment_6_2474b1cc6256afefb3d0530c3a0a24e8._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 6""" + date="2020-11-02T19:17:01Z" + content=""" +Hmm, the form has size=40, but people are pasting entire long error +messages into it. Although special characters also being escaped increases +the length, if it were really limited to around that length, it would still +be short enough. + +Ok, added maxlength=50 to it. +"""]] diff --git a/doc/templates/blogpost.tmpl b/doc/templates/blogpost.tmpl new file mode 100644 index 0000000000..e9722ea255 --- /dev/null +++ b/doc/templates/blogpost.tmpl @@ -0,0 +1,16 @@ +
id="" action="" method="get"> +
+ +RSS + + +Atom + + + + + + + +
+ From 49475eb3d9542ae49c5e89efed4c1854ba67066e Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Mon, 2 Nov 2020 15:25:10 -0400 Subject: [PATCH 06/14] rename 2 bug report titles that are too long due to pasting enormous error messages into title entry So the repo can be cloned on windows. --- ...> __34__git_submodule_foreach_git_annex_init__34___fails.mdwn} | 0 .../comment_1_8a6cd8dc6e7c2cdefd60a07356be1d27._comment | 0 .../comment_3_23f3dac4a22cf7202a0bb849ee499766._comment | 0 .../comment_3_fd3a287ee6e73946889a4dd406fae0fc._comment | 0 .../comment_4_e772639c09c1089700b2c2c170ea458e._comment | 0 .../comment_5_167ade7bcd58354280df31442b586024._comment | 0 ...40__pidlock__41___leads_to_git-annex__58___SQLite3_error.mdwn} | 0 .../comment_1_aaea812a8f2b04d9b17d1ba9c6fad39f._comment | 0 .../comment_2_c55f2d95f7f08e6f6439f36790e5a538._comment | 0 .../comment_3_b49a3df3b13f564e588824910a07bc43._comment | 0 .../comment_4_a9e6e37dbaa1664d825bdbd64de4fbdd._comment | 0 11 files changed, 0 insertions(+), 0 deletions(-) rename doc/bugs/{__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__.mdwn => __34__git_submodule_foreach_git_annex_init__34___fails.mdwn} (100%) rename doc/bugs/{__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__ => __34__git_submodule_foreach_git_annex_init__34___fails}/comment_1_8a6cd8dc6e7c2cdefd60a07356be1d27._comment (100%) rename doc/bugs/{__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__ => __34__git_submodule_foreach_git_annex_init__34___fails}/comment_3_23f3dac4a22cf7202a0bb849ee499766._comment (100%) rename doc/bugs/{__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__ => __34__git_submodule_foreach_git_annex_init__34___fails}/comment_3_fd3a287ee6e73946889a4dd406fae0fc._comment (100%) rename doc/bugs/{__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__ => __34__git_submodule_foreach_git_annex_init__34___fails}/comment_4_e772639c09c1089700b2c2c170ea458e._comment (100%) rename doc/bugs/{__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__ => __34__git_submodule_foreach_git_annex_init__34___fails}/comment_5_167ade7bcd58354280df31442b586024._comment (100%) rename doc/bugs/{crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error.mdwn => crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error.mdwn} (100%) rename doc/bugs/{crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error => crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error}/comment_1_aaea812a8f2b04d9b17d1ba9c6fad39f._comment (100%) rename doc/bugs/{crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error => crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error}/comment_2_c55f2d95f7f08e6f6439f36790e5a538._comment (100%) rename doc/bugs/{crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error => crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error}/comment_3_b49a3df3b13f564e588824910a07bc43._comment (100%) rename doc/bugs/{crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error => crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error}/comment_4_a9e6e37dbaa1664d825bdbd64de4fbdd._comment (100%) diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__.mdwn b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails.mdwn similarity index 100% rename from doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__.mdwn rename to doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails.mdwn diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_1_8a6cd8dc6e7c2cdefd60a07356be1d27._comment b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_1_8a6cd8dc6e7c2cdefd60a07356be1d27._comment similarity index 100% rename from doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_1_8a6cd8dc6e7c2cdefd60a07356be1d27._comment rename to doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_1_8a6cd8dc6e7c2cdefd60a07356be1d27._comment diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_3_23f3dac4a22cf7202a0bb849ee499766._comment b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_3_23f3dac4a22cf7202a0bb849ee499766._comment similarity index 100% rename from doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_3_23f3dac4a22cf7202a0bb849ee499766._comment rename to doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_3_23f3dac4a22cf7202a0bb849ee499766._comment diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_3_fd3a287ee6e73946889a4dd406fae0fc._comment b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_3_fd3a287ee6e73946889a4dd406fae0fc._comment similarity index 100% rename from doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_3_fd3a287ee6e73946889a4dd406fae0fc._comment rename to doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_3_fd3a287ee6e73946889a4dd406fae0fc._comment diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_4_e772639c09c1089700b2c2c170ea458e._comment b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_4_e772639c09c1089700b2c2c170ea458e._comment similarity index 100% rename from doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_4_e772639c09c1089700b2c2c170ea458e._comment rename to doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_4_e772639c09c1089700b2c2c170ea458e._comment diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_5_167ade7bcd58354280df31442b586024._comment b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_5_167ade7bcd58354280df31442b586024._comment similarity index 100% rename from doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_5_167ade7bcd58354280df31442b586024._comment rename to doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_5_167ade7bcd58354280df31442b586024._comment diff --git a/doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error.mdwn b/doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error.mdwn similarity index 100% rename from doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error.mdwn rename to doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error.mdwn diff --git a/doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_1_aaea812a8f2b04d9b17d1ba9c6fad39f._comment b/doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error/comment_1_aaea812a8f2b04d9b17d1ba9c6fad39f._comment similarity index 100% rename from doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_1_aaea812a8f2b04d9b17d1ba9c6fad39f._comment rename to doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error/comment_1_aaea812a8f2b04d9b17d1ba9c6fad39f._comment diff --git a/doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_2_c55f2d95f7f08e6f6439f36790e5a538._comment b/doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error/comment_2_c55f2d95f7f08e6f6439f36790e5a538._comment similarity index 100% rename from doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_2_c55f2d95f7f08e6f6439f36790e5a538._comment rename to doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error/comment_2_c55f2d95f7f08e6f6439f36790e5a538._comment diff --git a/doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_3_b49a3df3b13f564e588824910a07bc43._comment b/doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error/comment_3_b49a3df3b13f564e588824910a07bc43._comment similarity index 100% rename from doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_3_b49a3df3b13f564e588824910a07bc43._comment rename to doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error/comment_3_b49a3df3b13f564e588824910a07bc43._comment diff --git a/doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_4_a9e6e37dbaa1664d825bdbd64de4fbdd._comment b/doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error/comment_4_a9e6e37dbaa1664d825bdbd64de4fbdd._comment similarity index 100% rename from doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_4_a9e6e37dbaa1664d825bdbd64de4fbdd._comment rename to doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_error/comment_4_a9e6e37dbaa1664d825bdbd64de4fbdd._comment From 5b4d5f6d648a22d5053503fd622271f85c2eaf34 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Mon, 2 Nov 2020 20:37:47 +0000 Subject: [PATCH 07/14] Added a comment --- ..._66a32cef4ff6bc93ad04346612f543ec._comment | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_2_66a32cef4ff6bc93ad04346612f543ec._comment diff --git a/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_2_66a32cef4ff6bc93ad04346612f543ec._comment b/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_2_66a32cef4ff6bc93ad04346612f543ec._comment new file mode 100644 index 0000000000..4243193749 --- /dev/null +++ b/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_2_66a32cef4ff6bc93ad04346612f543ec._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 2" + date="2020-11-02T20:37:47Z" + content=""" +> Every call to debugM does add overhead both in time to run it (checking if debugging is enabled is nonzero overhead and this will add up if it's done a few million times or whatever) and in time to write a useful explanation. + +debug-vs-not could be decided at start up once, and debug function could be set to noop if no debugging output (given the level requested) was desired. I really doubt that any reasonable number of noop function calls would add any notable burden on top of operation/interaction with external git commands and remotes. + +> It seems to me that debugging is best added when trying to debug something, or when there's a readily available value that can be output (such as the safety proof that is debugged when dropping, or external special remote messages). + +yes, but then first you need to identify the code where to add it and know how to add it... users (myself included) would not be be able to do so that easily. + +> Trying to add enough useful debugging information to 60,000 lines of code that any possible thing it might do will have a useful debug trace seems like a recipe to end up with 100,000 lines of code + +sure thing should not be added to every code line! But there would be value from knowing some critical \"interface\" points where e.g. hanging or incorrect operation could happen, e.g. if I saw earlier the output of what `trace.patch` you provided today I might have saved myself quite a lot of time, so sounds like a location of value to add some `heavydebug()` logging. + +> It would be easier to run the code in a debugger and step through it at that point. + +might not be that easy especially when relevant git-annex call is sandwiched deep in the hierarchy of datalad/git/git-annex/datalad/git-annex/datalad/git-annex calls through all the special remotes etc. +"""]] From 7f4e51161362c4215374cd36a75f0642eed1800d Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Mon, 2 Nov 2020 21:04:35 +0000 Subject: [PATCH 08/14] Added a comment --- .../comment_16_b4b60bed666355ace286775afe87860d._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/install/Windows/comment_16_b4b60bed666355ace286775afe87860d._comment diff --git a/doc/install/Windows/comment_16_b4b60bed666355ace286775afe87860d._comment b/doc/install/Windows/comment_16_b4b60bed666355ace286775afe87860d._comment new file mode 100644 index 0000000000..0b33a6ce6d --- /dev/null +++ b/doc/install/Windows/comment_16_b4b60bed666355ace286775afe87860d._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 16" + date="2020-11-02T21:04:35Z" + content=""" +FWIW -- there is now CI for git-annex on linux (debian-based), osx, and Windows on https://github.com/datalad/datalad-extensions/ which builds git-annex installer, tests git-annex, and then tests datalad against it in a few scenarios. +ATM it runs on cron, so not for each git-annex commit. We will see if we need/want more frequent builds. Artifacts (re)distribution will eventually be established on http://datasets.datalad.org/datalad/packages/ but we aren't there yet. +"""]] From 2c88c5e25894f65d4774cccb1830d62a4403121c Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Tue, 3 Nov 2020 02:02:41 +0000 Subject: [PATCH 09/14] initial formal report on drop --- ...ms_that___34__content_is_locked__34__.mdwn | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/bugs/Windows__58___drop_claims_that___34__content_is_locked__34__.mdwn diff --git a/doc/bugs/Windows__58___drop_claims_that___34__content_is_locked__34__.mdwn b/doc/bugs/Windows__58___drop_claims_that___34__content_is_locked__34__.mdwn new file mode 100644 index 0000000000..a8c4682cf5 --- /dev/null +++ b/doc/bugs/Windows__58___drop_claims_that___34__content_is_locked__34__.mdwn @@ -0,0 +1,97 @@ +### Please describe the problem. + +Happens as a part of our test [for datalad-archives external special remote](https://github.com/datalad/datalad/blob/master/datalad/customremotes/tests/test_archives.py#L171) we drop/get the file we just added to git-annex from a tarball. + +It works fine on Linux, and from todays fix [8.20201007-144-g7245a9ed5](https://git.kitenet.net/index.cgi/git-annex.git/commit/?id=7245a9ed533c87abb70b2b9d06a5d3d5ea2f23b4) we no longer stall, and proceed to that `drop`. But drop fails claiming that the content is locked. I have checked processes - there is no some other git annex hanging around and .lck even doesn't exist before we call `drop`. + + (datalad-3) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\a\d>dir ..\..\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\ + Volume in drive C has no label. + Volume Serial Number is AC6C-BABE + + Directory of C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc + + 11/02/2020 05:17 PM . + 11/02/2020 05:17 PM .. + 11/02/2020 05:17 PM 3 SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc 1 File(s) 3 bytes + 2 Dir(s) 23,168,851,968 bytes free + + +but then it is created by `drop`: + + (datalad-3) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\a\d>git annex drop --debug -- a.datc + [2020-11-02 17:21:26.8245443] process [5128] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","symbolic-ref","-q","HEAD"] + [2020-11-02 17:21:26.8393936] process [5128] done ExitSuccess + [2020-11-02 17:21:26.8393936] process [1452] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","show-ref","refs/heads/adjusted/dl-test-branch(unlocked)"] + [2020-11-02 17:21:26.8860693] process [1452] done ExitSuccess + [2020-11-02 17:21:26.8860693] process [2004] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","ls-files","--stage","-z","--","a.datc"] + [2020-11-02 17:21:26.9331701] process [3452] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"] + [2020-11-02 17:21:26.9501808] process [4632] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"] + [2020-11-02 17:21:26.9501808] process [2920] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","show-ref","git-annex"] + [2020-11-02 17:21:27.0113712] process [2920] done ExitSuccess + [2020-11-02 17:21:27.0113712] process [976] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] + [2020-11-02 17:21:27.0580403] process [976] done ExitSuccess + [2020-11-02 17:21:27.0580403] process [8] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","log","refs/heads/git-annex..6217b1e92de175374dcb552d2bddee0b785073ce","--pretty=%H","-n1"] + [2020-11-02 17:21:27.1363275] process [8] done ExitSuccess + [2020-11-02 17:21:27.1363275] process [6948] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","hash-object","-w","--stdin-paths","--no-filters"] + [2020-11-02 17:21:27.1675675] process [1728] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch"] + [2020-11-02 17:21:27.1675675] process [4840] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] + [2020-11-02 17:21:27.1834634] process [7832] feed: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","update-index","-z","--index-info"] + [2020-11-02 17:21:27.2146236] process [6244] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","diff-index","--raw","-z","-r","--no-renames","-l0","--cached","refs/heads/git-annex","--"] + [2020-11-02 17:21:27.2769931] process [6244] done ExitSuccess + [2020-11-02 17:21:27.3111859] process [7832] done ExitSuccess + [2020-11-02 17:21:27.3238724] process [5528] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"] + [2020-11-02 17:21:27.3864742] process [8556] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"] + [2020-11-02 17:21:27.4331328] process [1100] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch"] + [2020-11-02 17:21:27.4331328] process [3516] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] + drop a.datc + git-annex: content is locked + failed + [2020-11-02 17:21:27.5114868] process [1100] done ExitSuccess + [2020-11-02 17:21:27.5114868] process [3516] done ExitSuccess + [2020-11-02 17:21:27.5114868] process [8556] done ExitSuccess + [2020-11-02 17:21:27.5275735] process [6948] done ExitSuccess + [2020-11-02 17:21:27.5422938] process [5528] done ExitSuccess + [2020-11-02 17:21:27.5422938] process done ExitSuccess + [2020-11-02 17:21:27.5422938] process done ExitSuccess + [2020-11-02 17:21:27.5422938] process [4632] done ExitSuccess + [2020-11-02 17:21:27.5422938] process [3452] done ExitSuccess + [2020-11-02 17:21:27.5422938] process [2004] done ExitSuccess + [2020-11-02 17:21:27.5422938] process done ExitSuccess + git-annex: drop: 1 failed + + + (datalad-3) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\a\d>dir ..\..\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\ + Volume in drive C has no label. + Volume Serial Number is AC6C-BABE + + Directory of C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc + + 11/02/2020 05:21 PM . + 11/02/2020 05:21 PM .. + 11/02/2020 05:17 PM 3 SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc + 11/02/2020 05:21 PM 0 SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck + 2 File(s) 3 bytes + 2 Dir(s) 23,168,679,936 bytes free + + +so it seems that `drop` somehow steps on its own feet and gets freaked out. + +Here is also some dump from procmon captured during `drop`, and filtered on this lck file name + +``` +5:07:49.2392479 PM git-annex.exe 2144 CreateFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS Desired Access: Generic Read/Write, Disposition: OpenIf, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, Delete, AllocationSize: 0, OpenResult: Opened File System 8228 8664 +5:07:49.2393198 PM git-annex.exe 2144 QueryInformationVolume C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS VolumeCreationTime: 10/22/2020 2:55:22 PM, VolumeSerialNumber: AC6C-BABE, SupportsObjects: True, VolumeLabel: File System 8228 8664 +5:07:49.2393295 PM git-annex.exe 2144 QueryAllInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck BUFFER OVERFLOW CreationTime: 11/2/2020 5:02:41 PM, LastAccessTime: 11/2/2020 5:04:56 PM, LastWriteTime: 11/2/2020 5:04:56 PM, ChangeTime: 11/2/2020 5:04:56 PM, FileAttributes: A, AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x2e000000015258, EaSize: 0, Access: Generic Read/Write, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Word File System 8228 8664 +5:07:49.2393613 PM git-annex.exe 2144 QueryInformationVolume C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS VolumeCreationTime: 10/22/2020 2:55:22 PM, VolumeSerialNumber: AC6C-BABE, SupportsObjects: True, VolumeLabel: File System 8228 8664 +5:07:49.2393740 PM git-annex.exe 2144 QueryAllInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck BUFFER OVERFLOW CreationTime: 11/2/2020 5:02:41 PM, LastAccessTime: 11/2/2020 5:04:56 PM, LastWriteTime: 11/2/2020 5:04:56 PM, ChangeTime: 11/2/2020 5:04:56 PM, FileAttributes: A, AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x2e000000015258, EaSize: 0, Access: Generic Read/Write, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Word File System 8228 8664 +5:07:49.2394246 PM git-annex.exe 2144 QueryStandardInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: False File System 8228 8664 +5:07:49.2394500 PM git-annex.exe 2144 SetEndOfFileInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS EndOfFile: 0 File System 8228 8664 +5:07:49.2395339 PM git-annex.exe 2144 SetAllocationInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS AllocationSize: 0 File System 8228 8664 +5:07:49.2395801 PM git-annex.exe 2144 CloseFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS File System 8228 8664 +``` + +so it seems to happily create that file. + + +[[!meta author=yoh]] +[[!tag projects/datalad]] From 7e24b2587ebeb4b8e7c6a9448220ab90a50fe4ab Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Tue, 3 Nov 2020 02:12:17 +0000 Subject: [PATCH 10/14] Added a comment --- ...t_1_ddf822288da76a16eaaade2da51c5b43._comment | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/bugs/Windows__58___drop_claims_that___34__content_is_locked__34__/comment_1_ddf822288da76a16eaaade2da51c5b43._comment diff --git a/doc/bugs/Windows__58___drop_claims_that___34__content_is_locked__34__/comment_1_ddf822288da76a16eaaade2da51c5b43._comment b/doc/bugs/Windows__58___drop_claims_that___34__content_is_locked__34__/comment_1_ddf822288da76a16eaaade2da51c5b43._comment new file mode 100644 index 0000000000..a739060457 --- /dev/null +++ b/doc/bugs/Windows__58___drop_claims_that___34__content_is_locked__34__/comment_1_ddf822288da76a16eaaade2da51c5b43._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 1" + date="2020-11-03T02:12:16Z" + content=""" +hehehey -- I think I got the reason at least! + +``` +In [4]: len(r'C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4 + ...: fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck') +Out[4]: 257 + +``` +so we exceeded the volume of magical dust windows could swallow without it sticking out somewhere... If I change backend to MD5E we do not observe that issue. +"""]] From 664bec4297f6aacd959569c0c648076784454544 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Mon, 2 Nov 2020 22:34:04 -0400 Subject: [PATCH 11/14] comment --- ..._ea60112be683188e43b27942667ddd77._comment | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_3_ea60112be683188e43b27942667ddd77._comment diff --git a/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_3_ea60112be683188e43b27942667ddd77._comment b/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_3_ea60112be683188e43b27942667ddd77._comment new file mode 100644 index 0000000000..a14cdc39ef --- /dev/null +++ b/doc/todo/--debug-level___40__and__47__or_-c_annex.debug-level__41___+_more_debug_information_from_git-annex/comment_3_ea60112be683188e43b27942667ddd77._comment @@ -0,0 +1,39 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2020-11-03T01:47:26Z" + content=""" +It would be possible to split out of AnnexState a value that is +immutable, and have that part not gated behind a MVar, and so avoid the +MVar lookup overhead, since it would just be part of the Annex Reader +monad. And the debug action in that could just be a noop when it's disabled, +and so be essentially zero overhead. One little problem is that option +parsing (eg for --debug) currently runs in the Annex monad, so too late to +change that immutable value. + +Obviously having debug logging in a place where you just needed debug +logging would have been useful. It does not logically follow that we can +determine such places prior to your needing debug logging, or that we will +need debug logging there later. After all, it seems to have been debugged +now. + +I suppose we can say that anything involving canceling a thread, or closing +a file descriptor connected to a pipe, is fairly delicate and so it makes +sense to wrap and debug log those. (Which were the things needing debug +logging in your case.) But the only information available to display is +"thread N" or "FD N", so it would also need to debug log each thread spawn +and FD open point in order to work out which are interesting. (Or wrap the +values with more information, but that rapidly gets ugly.) + +Anyway, I rarely find myself adding debugM calls when debugging git-annex, +generally just playing with the test case rapidly lets a theory be formed +about what's happening, and then do something to test the theory, and I'm +probably most of the way to a fix or at least understanding the problem. + +The main way the current debug output is useful to me is it often hints at +the proximate location of the problem, because of whatever git command +git-annex ran just before it. It would certianly be useful to have +additional debugM wherever there's a lot of code that does not involve +anything that currently logs.. But I don't know a good way to identify +such stretches of code. +"""]] From 78178d4c339a7b910c8a0090373803e69f0e8474 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Tue, 3 Nov 2020 11:36:48 -0400 Subject: [PATCH 12/14] clean build warning --- Annex/ExternalAddonProcess.hs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Annex/ExternalAddonProcess.hs b/Annex/ExternalAddonProcess.hs index 988ee550eb..8a6c6c32fd 100644 --- a/Annex/ExternalAddonProcess.hs +++ b/Annex/ExternalAddonProcess.hs @@ -67,7 +67,7 @@ startExternalAddonProcess basecmd pid = do `onException` cleanupProcess pall -- This thread will exit after consuming any -- remaining stderr from the process. - wait stderrelay + () <- wait stderrelay hClose herr return $ ExternalAddonProcess { externalSend = hin From 2dabd4cc2df05f77f9a1a17a97fcaf2b50dc282f Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Tue, 3 Nov 2020 11:53:11 -0400 Subject: [PATCH 13/14] releasing package git-annex version 8.20201103 --- CHANGELOG | 26 +++++++++++++------------- git-annex.cabal | 2 +- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index 7234d4c452..acf7da4257 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,11 +1,5 @@ -git-annex (8.20201008) UNRELEASED; urgency=medium +git-annex (8.20201103) upstream; urgency=medium - * Fix bug that prevented linux standalone bundle from working on a fresh - install. - * Fix build on Windows with network-3. - * Fix a memory leak introduced in the last release. - * add, import: Fix a reversion in 7.20191009 that broke handling - of --largerthan and --smallerthan. * move: Improve resuming a move that was interrupted after the object was transferred, in cases where numcopies checks prevented the resumed move from dropping the object from the source repository. @@ -14,7 +8,12 @@ git-annex (8.20201008) UNRELEASED; urgency=medium chunks using the configured chunk size. * Fixed some problems that prevented this command from working: git submodule foreach git annex init - * testremote: Display exceptions when tests fail, to aid debugging. + * Improve shutdown process for external special remotes and external + backends. Make sure to relay any remaining stderr from the process, + and potentially avoid the process getting a SIGPIPE if it writes to + stderr too late. + * Fix a bug that prevented linux standalone bundle from working on a fresh + install. * Windows build changed to one done by the datalad-extensions project using Github actions. * Windows build now includes libmagic, so mimetype and mimeencoding @@ -23,13 +22,14 @@ git-annex (8.20201008) UNRELEASED; urgency=medium on this. * view: Avoid using ':' from metadata when generating a view, because it's a special character on Windows ("c:") - * Improve shutdown process for external special remotes and external - backends. Make sure to relay any remaining stderr from the process, - and potentially avoid the process getting a SIGPIPE if it writes to - stderr too late. + * Fix a memory leak introduced in the last release. + * add, import: Fix a reversion in 7.20191009 that broke handling + of --largerthan and --smallerthan. * view: Fix a reversion in 8.20200522 that broke entering or changing views. + * Fix build on Windows with network-3. + * testremote: Display exceptions when tests fail, to aid debugging. - -- Joey Hess Thu, 08 Oct 2020 10:48:17 -0400 + -- Joey Hess Tue, 03 Nov 2020 11:40:56 -0400 git-annex (8.20201007) upstream; urgency=medium diff --git a/git-annex.cabal b/git-annex.cabal index 605833f1fe..19e6f80e31 100644 --- a/git-annex.cabal +++ b/git-annex.cabal @@ -1,5 +1,5 @@ Name: git-annex -Version: 8.20201007 +Version: 8.20201103 Cabal-Version: >= 1.10 License: AGPL-3 Maintainer: Joey Hess From 0cf77eb41c5d3ec3225c6809d7ae11422a631b4e Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Tue, 3 Nov 2020 11:53:33 -0400 Subject: [PATCH 14/14] add news item for git-annex 8.20201103 --- doc/news/version_8.20200720.1.mdwn | 3 --- doc/news/version_8.20200720.mdwn | 40 ------------------------------ doc/news/version_8.20201103.mdwn | 33 ++++++++++++++++++++++++ 3 files changed, 33 insertions(+), 43 deletions(-) delete mode 100644 doc/news/version_8.20200720.1.mdwn delete mode 100644 doc/news/version_8.20200720.mdwn create mode 100644 doc/news/version_8.20201103.mdwn diff --git a/doc/news/version_8.20200720.1.mdwn b/doc/news/version_8.20200720.1.mdwn deleted file mode 100644 index 11a01b6aa7..0000000000 --- a/doc/news/version_8.20200720.1.mdwn +++ /dev/null @@ -1,3 +0,0 @@ -git-annex 8.20200720.1 released with [[!toggle text="these changes"]] -[[!toggleable text=""" - * Fix a bug in find --batch in the previous version."""]] \ No newline at end of file diff --git a/doc/news/version_8.20200720.mdwn b/doc/news/version_8.20200720.mdwn deleted file mode 100644 index bb53c5a451..0000000000 --- a/doc/news/version_8.20200720.mdwn +++ /dev/null @@ -1,40 +0,0 @@ -git-annex 8.20200720 released with [[!toggle text="these changes"]] -[[!toggleable text=""" - * import: Added --no-content option, which avoids downloading files - from a special remote. Currently only supported by the directory - special remote. - * Honor annex.largefiles when importing a tree from a special remote. - (Except for when --no-content is used.) - * Fix a deadlock that could occur after git-annex got an unlocked - file, causing the command to hang indefinitely. Known to happen on - vfat filesystems, possibly others. - * Build with the http-client-restricted and git-lfs libraries when - available, otherwise use the vendored copy as before. - * testremote: Fix over-allocation of resources and bad caching, - including starting up a large number of external special remote processes. - (Regression introduced in version 8.20200501) - * test: Fix some test cases that assumed git's default branch name. - * importfeed: Added some additional --template variables: - itempubyear, itempubmonth, itempubday, itempubhour, - itempubminute, itempubsecond. - * Made several special remotes support locking content on them, - which allows dropping from other special remotes in some situations - where it was not possible before. Supported special remotes: - S3 (with versioning=yes), git-lfs, tahoe - * Fix reversion that broke passing annex.* and remote.*.annex-* - git configs with -c. (Since version 8.20200330.) - * Bring back git-annex branch read cache. This speeds up some operations, - eg git-annex sync --content --all gets 20% faster. - * Fix a recently introduced bug that could cause a "fork: resource exhausted" - after getting several thousand files. - * Sped up the --all option by 2x to 16x by using git cat-file --buffer. - Thanks to Lukey for finding this optimisation. - * Sped up seeking for annexed files to operate on by a factor of nearly 2x. - * Sped up sync --content by 2x and other commands like fsck --fast and - whereis by around 50%, by using git cat-file --buffer. - * importfeed: Made checking known urls step around 15% faster. - * fsck: Detect if WORM keys contain a carriage return, and recommend - upgrading the key. (git-annex could have maybe created such keys back - in 2013). - * When on an adjust --hide-missing branch, fix handling of files that - have been deleted but the deletion is not yet staged."""]] \ No newline at end of file diff --git a/doc/news/version_8.20201103.mdwn b/doc/news/version_8.20201103.mdwn new file mode 100644 index 0000000000..8ca899cee9 --- /dev/null +++ b/doc/news/version_8.20201103.mdwn @@ -0,0 +1,33 @@ +git-annex 8.20201103 released with [[!toggle text="these changes"]] +[[!toggleable text=""" +git-annex (8.20201103) upstream; urgency=medium +. + * move: Improve resuming a move that was interrupted after the object + was transferred, in cases where numcopies checks prevented the resumed + move from dropping the object from the source repository. + * When a special remote has chunking enabled, but no chunk sizes are + recorded (or the recorded ones are not found), speculatively try + chunks using the configured chunk size. + * Fixed some problems that prevented this command from working: + git submodule foreach git annex init + * Improve shutdown process for external special remotes and external + backends. Make sure to relay any remaining stderr from the process, + and potentially avoid the process getting a SIGPIPE if it writes to + stderr too late. + * Fix a bug that prevented linux standalone bundle from working on a fresh + install. + * Windows build changed to one done by the datalad-extensions project + using Github actions. + * Windows build now includes libmagic, so mimetype and mimeencoding + will work. + Thanks to John Thorvald Wodder II and Yaroslav Halchenko for their work + on this. + * view: Avoid using ':' from metadata when generating a view, because + it's a special character on Windows ("c:") + * Fix a memory leak introduced in the last release. + * add, import: Fix a reversion in 7.20191009 that broke handling + of --largerthan and --smallerthan. + * view: Fix a reversion in 8.20200522 that broke entering or changing views. + * Fix build on Windows with network-3. + * testremote: Display exceptions when tests fail, to aid debugging. +"""]] \ No newline at end of file