From 65f9623f028d94ceb25fb16a2ac4df926965835c Mon Sep 17 00:00:00 2001 From: "lykos@d125a37d89b1cfac20829f12911656c40cb70018" Date: Wed, 18 Sep 2019 19:42:48 +0000 Subject: [PATCH 01/11] --- .../git_status_extremely_slow_with_v7.mdwn | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 doc/bugs/git_status_extremely_slow_with_v7.mdwn diff --git a/doc/bugs/git_status_extremely_slow_with_v7.mdwn b/doc/bugs/git_status_extremely_slow_with_v7.mdwn new file mode 100644 index 0000000000..11bdcb661e --- /dev/null +++ b/doc/bugs/git_status_extremely_slow_with_v7.mdwn @@ -0,0 +1,26 @@ +### Please describe the problem. +I just upgraded one of my repositories to v7. Now, git status takes ages. The "Refresh index" count increments every second, so it will take ~12 hours to complete. With such slow operations I really fear the moment when v7 becomes default. + +### What steps will reproduce the problem? + +[[!format sh """ +$ git status +Refresh index: 0% (49/43974) +"""]] + +### What version of git-annex are you using? On what operating system? + +[[!format sh """ +$ git annex version +git-annex version: 7.20190819-g972fd11f4e +build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite +dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.3 feed-1.2.0.0 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.10.5 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external +operating system: linux x86_64 +supported repository versions: 5 7 +upgrade supported from repository versions: 0 1 2 3 4 5 6 +local repository version: 7 + +"""]] + From db376fe17f73f769cf21882776b84a7911f17042 Mon Sep 17 00:00:00 2001 From: "lykos@d125a37d89b1cfac20829f12911656c40cb70018" Date: Wed, 18 Sep 2019 20:01:35 +0000 Subject: [PATCH 02/11] --- doc/bugs/git_status_extremely_slow_with_v7.mdwn | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/bugs/git_status_extremely_slow_with_v7.mdwn b/doc/bugs/git_status_extremely_slow_with_v7.mdwn index 11bdcb661e..1fe94b14d9 100644 --- a/doc/bugs/git_status_extremely_slow_with_v7.mdwn +++ b/doc/bugs/git_status_extremely_slow_with_v7.mdwn @@ -1,6 +1,8 @@ ### Please describe the problem. I just upgraded one of my repositories to v7. Now, git status takes ages. The "Refresh index" count increments every second, so it will take ~12 hours to complete. With such slow operations I really fear the moment when v7 becomes default. +I've got ~1000 unlocked files. + ### What steps will reproduce the problem? [[!format sh """ From 2c6bb780a874140a5ac46e7c9f930679e0c1ee9d Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 18 Sep 2019 20:16:26 +0000 Subject: [PATCH 03/11] Added a comment --- .../comment_2_5421641e133cdadd0be5587b882ed3e5._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/dbus_and_magicmime_build_flags/comment_2_5421641e133cdadd0be5587b882ed3e5._comment diff --git a/doc/forum/dbus_and_magicmime_build_flags/comment_2_5421641e133cdadd0be5587b882ed3e5._comment b/doc/forum/dbus_and_magicmime_build_flags/comment_2_5421641e133cdadd0be5587b882ed3e5._comment new file mode 100644 index 0000000000..eaaecee7c6 --- /dev/null +++ b/doc/forum/dbus_and_magicmime_build_flags/comment_2_5421641e133cdadd0be5587b882ed3e5._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="comment 2" + date="2019-09-18T20:16:26Z" + content=""" +That's what I thought, just checking. The [[conda install|install/conda]] builds with these flags on. +"""]] From 6842443c5ba836b378a86c9583a46d9ac98a169f Mon Sep 17 00:00:00 2001 From: "lykos@d125a37d89b1cfac20829f12911656c40cb70018" Date: Wed, 18 Sep 2019 20:20:34 +0000 Subject: [PATCH 04/11] Added a comment --- .../comment_1_53a56019a17d3d644b1ea282f5228750._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/bugs/git_status_extremely_slow_with_v7/comment_1_53a56019a17d3d644b1ea282f5228750._comment diff --git a/doc/bugs/git_status_extremely_slow_with_v7/comment_1_53a56019a17d3d644b1ea282f5228750._comment b/doc/bugs/git_status_extremely_slow_with_v7/comment_1_53a56019a17d3d644b1ea282f5228750._comment new file mode 100644 index 0000000000..2468931182 --- /dev/null +++ b/doc/bugs/git_status_extremely_slow_with_v7/comment_1_53a56019a17d3d644b1ea282f5228750._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="lykos@d125a37d89b1cfac20829f12911656c40cb70018" + nickname="lykos" + avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2" + subject="comment 1" + date="2019-09-18T20:20:33Z" + content=""" +OK so after processing the unlocked files, it got a lot faster. Will it be like that on every new clone and with every new unlocked file? +"""]] From b33a2f3ffff1275bc96396580512b724b830292c Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 18 Sep 2019 20:25:43 +0000 Subject: [PATCH 05/11] added small bug report about message printed by git-annex-version --- doc/bugs/__34__supported_repository_versions__34___printout.mdwn | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/bugs/__34__supported_repository_versions__34___printout.mdwn diff --git a/doc/bugs/__34__supported_repository_versions__34___printout.mdwn b/doc/bugs/__34__supported_repository_versions__34___printout.mdwn new file mode 100644 index 0000000000..8b8e524789 --- /dev/null +++ b/doc/bugs/__34__supported_repository_versions__34___printout.mdwn @@ -0,0 +1 @@ +`git-annex-version` prints "supported repository versions: 5 7" for git-annex 7.20190819 . But, if I understand correctly, version 5 repos aren't really supported anymore, except to upgrade from? From fdd8920bd07d87c3e6a5ce048d9ca74fd52e5b8c Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 18 Sep 2019 20:33:04 +0000 Subject: [PATCH 06/11] Added a comment --- .../comment_2_aa709c1ee602ae3211f36cba68c155c4._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/git-annex-watch_clarification/comment_2_aa709c1ee602ae3211f36cba68c155c4._comment diff --git a/doc/forum/git-annex-watch_clarification/comment_2_aa709c1ee602ae3211f36cba68c155c4._comment b/doc/forum/git-annex-watch_clarification/comment_2_aa709c1ee602ae3211f36cba68c155c4._comment new file mode 100644 index 0000000000..64991533e6 --- /dev/null +++ b/doc/forum/git-annex-watch_clarification/comment_2_aa709c1ee602ae3211f36cba68c155c4._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="comment 2" + date="2019-09-18T20:33:04Z" + content=""" +\"The assistant handles copying files to other remotes, so use it instead of watch if you want that\" -- but the assistant is a GUI, right? My use case has to run in batch mode. I just run [[`git-annex-sync`|git-annex-sync]] periodically. I assume it's ok to both do that and have [[`git-annex-watch`|git-annex-watch]] running in the background? +"""]] From 225e0ae1055c74ae9d868b801c48f40b3fa02dec Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 18 Sep 2019 20:40:54 +0000 Subject: [PATCH 07/11] removed --- doc/bugs/__34__supported_repository_versions__34___printout.mdwn | 1 - 1 file changed, 1 deletion(-) delete mode 100644 doc/bugs/__34__supported_repository_versions__34___printout.mdwn diff --git a/doc/bugs/__34__supported_repository_versions__34___printout.mdwn b/doc/bugs/__34__supported_repository_versions__34___printout.mdwn deleted file mode 100644 index 8b8e524789..0000000000 --- a/doc/bugs/__34__supported_repository_versions__34___printout.mdwn +++ /dev/null @@ -1 +0,0 @@ -`git-annex-version` prints "supported repository versions: 5 7" for git-annex 7.20190819 . But, if I understand correctly, version 5 repos aren't really supported anymore, except to upgrade from? From 177b3f9dce498a3ece505bcebcce5717b0c00cbb Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 18 Sep 2019 20:59:15 +0000 Subject: [PATCH 08/11] added suggestion to split off clean and smudge filter into separate lightweight executables --- doc/todo/split_off_clean__47__smudge_filter__63__.mdwn | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/todo/split_off_clean__47__smudge_filter__63__.mdwn diff --git a/doc/todo/split_off_clean__47__smudge_filter__63__.mdwn b/doc/todo/split_off_clean__47__smudge_filter__63__.mdwn new file mode 100644 index 0000000000..161cdb577a --- /dev/null +++ b/doc/todo/split_off_clean__47__smudge_filter__63__.mdwn @@ -0,0 +1 @@ +If running the clean/smudge filter once per file is a [[bottleneck|forum/Adding_files_to_git__58___Very_long___34__recording_state_in_git__34___phase]], might it speed things up to split them off into something more lightweight than the full git-annex binary? From 1fc9cae9fe52202e632435a19469ad2e6342d37c Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 18 Sep 2019 21:30:46 +0000 Subject: [PATCH 09/11] Added a comment --- .../comment_5_1a11280fbed406677e30cc5e149c55a4._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/todo/symlinks_for_not-present_unlocked_files/comment_5_1a11280fbed406677e30cc5e149c55a4._comment diff --git a/doc/todo/symlinks_for_not-present_unlocked_files/comment_5_1a11280fbed406677e30cc5e149c55a4._comment b/doc/todo/symlinks_for_not-present_unlocked_files/comment_5_1a11280fbed406677e30cc5e149c55a4._comment new file mode 100644 index 0000000000..0f91982202 --- /dev/null +++ b/doc/todo/symlinks_for_not-present_unlocked_files/comment_5_1a11280fbed406677e30cc5e149c55a4._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="comment 5" + date="2019-09-18T21:30:46Z" + content=""" +One more option is to add the paths for missing unlocked files to .git/info/exclude . +"""]] From 31a218a6a3a128bd59413fd7794c80a9a8f19ff1 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 18 Sep 2019 22:55:06 +0000 Subject: [PATCH 10/11] Added a comment: git-annex REST API --- .../comment_1_e365d5374ca46114149c30d4dbe60148._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/todo/split_off_clean__47__smudge_filter__63__/comment_1_e365d5374ca46114149c30d4dbe60148._comment diff --git a/doc/todo/split_off_clean__47__smudge_filter__63__/comment_1_e365d5374ca46114149c30d4dbe60148._comment b/doc/todo/split_off_clean__47__smudge_filter__63__/comment_1_e365d5374ca46114149c30d4dbe60148._comment new file mode 100644 index 0000000000..c47b3a5029 --- /dev/null +++ b/doc/todo/split_off_clean__47__smudge_filter__63__/comment_1_e365d5374ca46114149c30d4dbe60148._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="git-annex REST API" + date="2019-09-18T22:55:06Z" + content=""" +Even better would be if git-annex had a \"server\" mode where one git-annex process runs and exposes a REST API for invoking git-annex commands (cf. [[todo/universal_batch_mode]]; something like [this](https://cromwell.readthedocs.io/en/stable/api/RESTAPI/)). Then the clean and smudge filters could be lightweight scripts that just call curl. +"""]] From d1ec2a1faa7da968ca357cfaedbaf94506fc34a5 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Thu, 19 Sep 2019 01:47:10 +0000 Subject: [PATCH 11/11] added todo for finding unlocked files --- doc/todo/find_unlocked_files.mdwn | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/todo/find_unlocked_files.mdwn diff --git a/doc/todo/find_unlocked_files.mdwn b/doc/todo/find_unlocked_files.mdwn new file mode 100644 index 0000000000..0f00804b7a --- /dev/null +++ b/doc/todo/find_unlocked_files.mdwn @@ -0,0 +1 @@ +As suggested at [[forum/Find_unlocked__47__locked_files]], can the file matching conditions be extended to be able to find unlocked files?