From 6d59508f0269c8842642ad1c5dce2620e0b53191 Mon Sep 17 00:00:00 2001 From: jkniiv Date: Tue, 23 Nov 2021 21:46:43 +0000 Subject: [PATCH 01/10] Added a comment --- .../comment_4_50aa0ea762d09990f8a8f633e56aa205._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/bugs/git_2.34__58___some_conflict_resolution_unit_tests_fail/comment_4_50aa0ea762d09990f8a8f633e56aa205._comment diff --git a/doc/bugs/git_2.34__58___some_conflict_resolution_unit_tests_fail/comment_4_50aa0ea762d09990f8a8f633e56aa205._comment b/doc/bugs/git_2.34__58___some_conflict_resolution_unit_tests_fail/comment_4_50aa0ea762d09990f8a8f633e56aa205._comment new file mode 100644 index 0000000000..9d841b0b28 --- /dev/null +++ b/doc/bugs/git_2.34__58___some_conflict_resolution_unit_tests_fail/comment_4_50aa0ea762d09990f8a8f633e56aa205._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="jkniiv" + avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d" + subject="comment 4" + date="2021-11-23T21:46:43Z" + content=""" +Thanks Joey for investigating this issue thoroughly. Also, my thanks to kyle for expanding my initial report +with Linux-related details I didn't have time to prepare. +"""]] From 65494a9f81845bfdc706ac45d14795f4825a6587 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Tue, 23 Nov 2021 23:12:47 +0000 Subject: [PATCH 02/10] Added a comment --- .../comment_2_da46fc7a6bbdae0c3b44d241bc527b55._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/tests___34__exit_code_124__34___with_HOME_and_PWD_on_NFS/comment_2_da46fc7a6bbdae0c3b44d241bc527b55._comment diff --git a/doc/bugs/tests___34__exit_code_124__34___with_HOME_and_PWD_on_NFS/comment_2_da46fc7a6bbdae0c3b44d241bc527b55._comment b/doc/bugs/tests___34__exit_code_124__34___with_HOME_and_PWD_on_NFS/comment_2_da46fc7a6bbdae0c3b44d241bc527b55._comment new file mode 100644 index 0000000000..0f35fd7057 --- /dev/null +++ b/doc/bugs/tests___34__exit_code_124__34___with_HOME_and_PWD_on_NFS/comment_2_da46fc7a6bbdae0c3b44d241bc527b55._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 2" + date="2021-11-23T23:12:47Z" + content=""" +I see that. But the entire run was not prohibitively long - just 30m 21s... previous succesful though ran total 22m 14s (see eg [here](https://github.com/datalad/git-annex/runs/4271323847?check_suite_focus=true)) so may be test started to stall? +"""]] From 574938c2348a35491264a06302dd3e3890d35f92 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Tue, 23 Nov 2021 23:22:36 +0000 Subject: [PATCH 03/10] Added a comment --- .../comment_3_82b8570c9496ee1b111dbd1ea53e823c._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/tests___34__exit_code_124__34___with_HOME_and_PWD_on_NFS/comment_3_82b8570c9496ee1b111dbd1ea53e823c._comment diff --git a/doc/bugs/tests___34__exit_code_124__34___with_HOME_and_PWD_on_NFS/comment_3_82b8570c9496ee1b111dbd1ea53e823c._comment b/doc/bugs/tests___34__exit_code_124__34___with_HOME_and_PWD_on_NFS/comment_3_82b8570c9496ee1b111dbd1ea53e823c._comment new file mode 100644 index 0000000000..b2ad0e8ffb --- /dev/null +++ b/doc/bugs/tests___34__exit_code_124__34___with_HOME_and_PWD_on_NFS/comment_3_82b8570c9496ee1b111dbd1ea53e823c._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 3" + date="2021-11-23T23:22:36Z" + content=""" +oh that [successful log](https://github.com/datalad/git-annex/runs/4271323847?check_suite_focus=true) doesn't even have any line with `migrate`, so some new test stalling on NFS HOME/CWD? +"""]] From e83c07427a9966bbe4d1e2bb3bcee118ff310971 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Wed, 24 Nov 2021 14:13:20 +0000 Subject: [PATCH 04/10] --- ...__58___pidlock__58___getFileStatus__58___does_not_exist.mdwn | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn index 36b5dbbab6..917748b90b 100644 --- a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn +++ b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn @@ -15,3 +15,5 @@ conda linux nodep (standalone) build 8.20211012-geb95ed486 and then with another +[[!meta author=yoh]] +[[!tag projects/datalad]] From 5aca043296a3e3f0fcdc3717836a78d10359e2f8 Mon Sep 17 00:00:00 2001 From: "dev@c1c358f0d3c8563701193b66791eb1bc57a25ac9" Date: Wed, 24 Nov 2021 21:01:41 +0000 Subject: [PATCH 05/10] --- ..._47__Glacier.hs_build_error_GHC_9.0.1.mdwn | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 doc/bugs/Remote__47__Glacier.hs_build_error_GHC_9.0.1.mdwn diff --git a/doc/bugs/Remote__47__Glacier.hs_build_error_GHC_9.0.1.mdwn b/doc/bugs/Remote__47__Glacier.hs_build_error_GHC_9.0.1.mdwn new file mode 100644 index 0000000000..758defb8cf --- /dev/null +++ b/doc/bugs/Remote__47__Glacier.hs_build_error_GHC_9.0.1.mdwn @@ -0,0 +1,52 @@ +### Please describe the problem. +Build error 1 during build process of git-annex on alpine 5.15 using GHC 9.0.1. + +### What steps will reproduce the problem? +Build git-annex on alpine 5.15 using following APKBUILD: +https://github.com/ayakael/aports/blob/testing/git-annex/testing/git-annex/APKBUILD + +### What version of git-annex are you using? On what operating system? +git-annex: 20211123 +os: alpine linux 5.15 +ghc: 9.0.1 +cabal: 3.6.2.0 + +### Please provide any additional information below. +I made an attempt of fixing by reverting the first fix attempt done on line 181 by commit 2739adc. It gave the same error, but it is most likely related. I know very little of haskell, thus could not push the debugging further. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log +[411 of 670] Compiling Remote.Glacier ( Remote/Glacier.hs, /var/build/aports/testing/git-annex/src/git-annex-8.20211123/dist-newstyle/build/x86_64-linux/ghc-9.0.1/git-annex-8.20211123/build/git-annex/git-annex-tmp/Remote/Glacier.o, /var/build/aports/testing/git-annex/src/git-annex-8.20211123/dist-newstyle/build/x86_64-linux/ghc-9.0.1/git-annex-8.20211123/build/git-annex/git-annex-tmp/Remote/Glacier.dyn_o ) + +Remote/Glacier.hs:179:12: error: + • Couldn't match type: Key + -> MeterUpdate + -> Maybe Utility.Hash.IncrementalVerifier + -> (ContentSource -> Annex a0) + -> Annex a0 + with: forall a. + Key + -> MeterUpdate + -> Maybe Utility.Hash.IncrementalVerifier + -> (ContentSource -> Annex a) + -> Annex a + Expected: Remote -> Retriever + Actual: Remote + -> Key + -> MeterUpdate + -> Maybe Utility.Hash.IncrementalVerifier + -> (ContentSource -> Annex a0) + -> Annex a0 + • In the expression: byteRetriever . retrieve' + In an equation for ‘retrieve’: retrieve = byteRetriever . retrieve' + | +179 | retrieve = byteRetriever . retrieve' + | ^^^^^^^^^^^^^^^^^^^^^^^^^ +make: *** [Makefile:58: git-annex] Error 1 +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) +Oh absolutely. This version compiles fine under alpine 5.14, which uses GHC 8.8.4. I've been using this software for many years without major issues, and it's a linchpin of my backup infrastructure. Thanks for the all the good work! + From fe143b2cb96b01adbb75039c3ff7c7ce685cacea Mon Sep 17 00:00:00 2001 From: mih Date: Thu, 25 Nov 2021 07:34:49 +0000 Subject: [PATCH 06/10] Added a comment: Translates to Windows! --- ...omment_18_2e8e840cabbc547ef758568bb2779f9a._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_18_2e8e840cabbc547ef758568bb2779f9a._comment diff --git a/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_18_2e8e840cabbc547ef758568bb2779f9a._comment b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_18_2e8e840cabbc547ef758568bb2779f9a._comment new file mode 100644 index 0000000000..22fbac94a1 --- /dev/null +++ b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_18_2e8e840cabbc547ef758568bb2779f9a._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="mih" + avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd" + subject="Translates to Windows!" + date="2021-11-25T07:34:48Z" + content=""" +Sorry for being silent for so long. I just got a contemporary machine with windows, such that hardware age should no longer be a concern for any performance comparison. + +I did not yet find the time to re-assess this issue in full, but I tried the new filter-process setting with a simple `datalad create` (this only adds a few tiny files, but nevertheless took long, and was the original motivation for this issue). Enabling the new setting reduces the runtime by 25% (from 4.5s to 3.5s on average). +"""]] From 0b2314ffe0499097f1357309a8a0c012cac8e341 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9mi?= Date: Thu, 25 Nov 2021 10:15:49 +0000 Subject: [PATCH 07/10] bug on export tree remote. --- ...ry_to_rexeport_already_exported_files.mdwn | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 doc/bugs/git_annex_try_to_rexeport_already_exported_files.mdwn diff --git a/doc/bugs/git_annex_try_to_rexeport_already_exported_files.mdwn b/doc/bugs/git_annex_try_to_rexeport_already_exported_files.mdwn new file mode 100644 index 0000000000..b2527070fe --- /dev/null +++ b/doc/bugs/git_annex_try_to_rexeport_already_exported_files.mdwn @@ -0,0 +1,29 @@ +### Please describe the problem. +I've a type=directory remote, with exporttree=yes on a USB disk. when running `git annex --content` git annex list a long list of `export cibox-usb toto ok` lines for file it already exported before. I’ve check some of those file, and their content are correct. + +Those line hide real new export, and I wonder if it doesn’t slow the process. + +There seem to be no way to tell git-annex that those file are already ok, and it should try again to export them. +` + +### What steps will reproduce the problem? + +Just running `git annex --content name-of-the-remote` + + +### What version of git-annex are you using? On what operating system? +git annex from Debian : + + git-annex version: 8.20211011 + build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV + dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0 + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* + remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external + operating system: linux x86_64 + supported repository versions: 8 + upgrade supported from repository versions: 0 1 2 3 4 5 6 7 + local repository version: + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) +It’s only a bug on what git-annex display. It does the asked work: the file are correctly exported where I look for them. + From 120a94bcb8401cfb5aa1eb4b2e3a421c304c85ab Mon Sep 17 00:00:00 2001 From: mih Date: Thu, 25 Nov 2021 13:09:11 +0000 Subject: [PATCH 08/10] Added a comment: More statistics --- ...nt_19_c89853eca5c16e9dfa4a4acc982ab526._comment | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_19_c89853eca5c16e9dfa4a4acc982ab526._comment diff --git a/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_19_c89853eca5c16e9dfa4a4acc982ab526._comment b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_19_c89853eca5c16e9dfa4a4acc982ab526._comment new file mode 100644 index 0000000000..ffbf19ebf9 --- /dev/null +++ b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_19_c89853eca5c16e9dfa4a4acc982ab526._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="mih" + avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd" + subject="More statistics" + date="2021-11-25T13:09:11Z" + content=""" +I looked into the global affect of this switch on a large and versatile set of use cases in the form of the DataLad test suite: https://github.com/datalad/datalad/pull/6245 + +It is worth keeping in mind that there are only small-size files involved! + +The benefit is somewhere between noticeable and remarkable. An overall runtime reduction of 16% with benefits ranging from 5% to 32% depending on the tested functionality. + +This will need a bit of further investigation to drill down on the reason for this large variability, but given that the sign is always in the right direction this is really great! Thx much! +"""]] From 0f48796532926134a8bc5b737dd65f9018ad00b8 Mon Sep 17 00:00:00 2001 From: Atemu Date: Thu, 25 Nov 2021 19:23:46 +0000 Subject: [PATCH 09/10] Added a comment --- ...omment_20_f12f7f14ef09fa765c78dbeef809da7d._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/bugs/__34__error__58___invalid_object__34____44___after_add__59___cannot_commit/comment_20_f12f7f14ef09fa765c78dbeef809da7d._comment diff --git a/doc/bugs/__34__error__58___invalid_object__34____44___after_add__59___cannot_commit/comment_20_f12f7f14ef09fa765c78dbeef809da7d._comment b/doc/bugs/__34__error__58___invalid_object__34____44___after_add__59___cannot_commit/comment_20_f12f7f14ef09fa765c78dbeef809da7d._comment new file mode 100644 index 0000000000..2a44d60f9b --- /dev/null +++ b/doc/bugs/__34__error__58___invalid_object__34____44___after_add__59___cannot_commit/comment_20_f12f7f14ef09fa765c78dbeef809da7d._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="Atemu" + avatar="http://cdn.libravatar.org/avatar/d1f0f4275931c552403f4c6707bead7a" + subject="comment 20" + date="2021-11-25T19:23:46Z" + content=""" +Also ran across this today. All I did was a simple copy of a few files to a remote using a filter. I can provide the index and lck files if needed. + + git-annex version: 8.20211011 +"""]] From 02e3756bd732ce7e65346d33c1faed0ce6e661c3 Mon Sep 17 00:00:00 2001 From: mih Date: Fri, 26 Nov 2021 14:23:34 +0000 Subject: [PATCH 10/10] Added a comment: Even more impact on real systems --- .../comment_20_281ded4caa0dd407431b205ce6746c05._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_20_281ded4caa0dd407431b205ce6746c05._comment diff --git a/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_20_281ded4caa0dd407431b205ce6746c05._comment b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_20_281ded4caa0dd407431b205ce6746c05._comment new file mode 100644 index 0000000000..39230325a6 --- /dev/null +++ b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_20_281ded4caa0dd407431b205ce6746c05._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="mih" + avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd" + subject="Even more impact on real systems" + date="2021-11-26T14:23:34Z" + content=""" +It seems the impact on real systems is even more sever: https://github.com/datalad/datalad/pull/6245#issuecomment-979992551 -- factor 2-3 faster on that particular machine. Maybe consumer systems do some kind of expensive process startup inspection (antivirus?) that bring a swarm of child processes to a crawl. +"""]]