From 62fad04cce9221889bc61c4d267e1e6f49d90700 Mon Sep 17 00:00:00 2001 From: falsifian Date: Sat, 12 Sep 2020 03:36:03 +0000 Subject: [PATCH 01/13] Added a comment: Can I actually add a requirement like "at least two copies stored offsite" without specifying a particular repo? --- ...mment_3_c1255ef37ef7b0161c9855a3ef4785a2._comment | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/required_content/comment_3_c1255ef37ef7b0161c9855a3ef4785a2._comment diff --git a/doc/required_content/comment_3_c1255ef37ef7b0161c9855a3ef4785a2._comment b/doc/required_content/comment_3_c1255ef37ef7b0161c9855a3ef4785a2._comment new file mode 100644 index 0000000000..c4be030cb1 --- /dev/null +++ b/doc/required_content/comment_3_c1255ef37ef7b0161c9855a3ef4785a2._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="falsifian" + avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae" + subject="Can I actually add a requirement like "at least two copies stored offsite" without specifying a particular repo?" + date="2020-09-12T03:36:03Z" + content=""" +I'd like to be able to say things like \"the data must be stored on at least one live server and on at least two offline backups\", where \"live server\" and \"offline backup\" are categories I put my remotes into via something like git annex vicfg. + +When I read the first paragraph of this page, I thought that's what it was describing, and was excited to try out that feature. Reading the documentation more carefully, though, it looks like git-annex doesn't actually do this --- the \"required content\" feature requires me to explicitly choose which repos to use for which content. Is that right? + +(For example: I'd like to be able to put the remotes \"hd0\", \"hd1\", \"hd2\" into a category \"offline_disks\" and tell git-annex that everything needs to be stored on at least one remote in the \"offline_disks\" category, in addition to other requirements.) +"""]] From 7691d758bc07a115cf0c5c25051b6332b31c46da Mon Sep 17 00:00:00 2001 From: falsifian Date: Sat, 12 Sep 2020 03:44:52 +0000 Subject: [PATCH 02/13] Added a comment --- .../comment_4_92661adad39dbca09623d882903f60b9._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/required_content/comment_4_92661adad39dbca09623d882903f60b9._comment diff --git a/doc/required_content/comment_4_92661adad39dbca09623d882903f60b9._comment b/doc/required_content/comment_4_92661adad39dbca09623d882903f60b9._comment new file mode 100644 index 0000000000..fe5ff2b5b5 --- /dev/null +++ b/doc/required_content/comment_4_92661adad39dbca09623d882903f60b9._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="falsifian" + avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae" + subject="comment 4" + date="2020-09-12T03:44:49Z" + content=""" +Oops, never mind. I guess I can achieve this with the copies=groupname:number syntax. +"""]] From 320f918e9cb74a31d14a6ddaab51b289baa05cd2 Mon Sep 17 00:00:00 2001 From: falsifian Date: Sat, 12 Sep 2020 03:51:52 +0000 Subject: [PATCH 03/13] removed --- ...mment_3_c1255ef37ef7b0161c9855a3ef4785a2._comment | 12 ------------ 1 file changed, 12 deletions(-) delete mode 100644 doc/required_content/comment_3_c1255ef37ef7b0161c9855a3ef4785a2._comment diff --git a/doc/required_content/comment_3_c1255ef37ef7b0161c9855a3ef4785a2._comment b/doc/required_content/comment_3_c1255ef37ef7b0161c9855a3ef4785a2._comment deleted file mode 100644 index c4be030cb1..0000000000 --- a/doc/required_content/comment_3_c1255ef37ef7b0161c9855a3ef4785a2._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="falsifian" - avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae" - subject="Can I actually add a requirement like "at least two copies stored offsite" without specifying a particular repo?" - date="2020-09-12T03:36:03Z" - content=""" -I'd like to be able to say things like \"the data must be stored on at least one live server and on at least two offline backups\", where \"live server\" and \"offline backup\" are categories I put my remotes into via something like git annex vicfg. - -When I read the first paragraph of this page, I thought that's what it was describing, and was excited to try out that feature. Reading the documentation more carefully, though, it looks like git-annex doesn't actually do this --- the \"required content\" feature requires me to explicitly choose which repos to use for which content. Is that right? - -(For example: I'd like to be able to put the remotes \"hd0\", \"hd1\", \"hd2\" into a category \"offline_disks\" and tell git-annex that everything needs to be stored on at least one remote in the \"offline_disks\" category, in addition to other requirements.) -"""]] From c3cd22877a25225771fdf06fac5ccb839c382c9f Mon Sep 17 00:00:00 2001 From: falsifian Date: Sat, 12 Sep 2020 03:52:07 +0000 Subject: [PATCH 04/13] removed --- .../comment_4_92661adad39dbca09623d882903f60b9._comment | 8 -------- 1 file changed, 8 deletions(-) delete mode 100644 doc/required_content/comment_4_92661adad39dbca09623d882903f60b9._comment diff --git a/doc/required_content/comment_4_92661adad39dbca09623d882903f60b9._comment b/doc/required_content/comment_4_92661adad39dbca09623d882903f60b9._comment deleted file mode 100644 index fe5ff2b5b5..0000000000 --- a/doc/required_content/comment_4_92661adad39dbca09623d882903f60b9._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="falsifian" - avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae" - subject="comment 4" - date="2020-09-12T03:44:49Z" - content=""" -Oops, never mind. I guess I can achieve this with the copies=groupname:number syntax. -"""]] From 6308af9cf522eae4a29abc029d6228f9b2888f50 Mon Sep 17 00:00:00 2001 From: falsifian Date: Sat, 12 Sep 2020 15:14:00 +0000 Subject: [PATCH 05/13] --- doc/forum/Can_I_remove_a_groupwanted_expression__63__.mdwn | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 doc/forum/Can_I_remove_a_groupwanted_expression__63__.mdwn diff --git a/doc/forum/Can_I_remove_a_groupwanted_expression__63__.mdwn b/doc/forum/Can_I_remove_a_groupwanted_expression__63__.mdwn new file mode 100644 index 0000000000..cbf3f327a6 --- /dev/null +++ b/doc/forum/Can_I_remove_a_groupwanted_expression__63__.mdwn @@ -0,0 +1,7 @@ +I associated a groupwanted expression with a group I called "common", but I don't want it any more. Can I remove it, so the group doesn't have any groupwanted expression? + +I tried to remove it with `git annex groupwanted common ''`, but when I run `git annex vicfg`, I just end up with `groupwanted common =`. Similarly, if I run `git annex vicfg` and delete that line, it seems to have no effect. + +I'm assuming (but haven't verified) that means I won't be able to use `groupwanted` properly with repos that are in `common` and also in a group that has a real expression associated with it. + +How can I make it so `common` no longer has an associated expression? From be59e433b4fa7e07799c359ee46831109402bbc0 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Sat, 12 Sep 2020 15:20:21 +0000 Subject: [PATCH 06/13] Added a comment: updating the standalone distribution to 8.20200908 --- ...ment_13_121ae7a2803975e3bdcb5f9df0c4af3f._comment | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/forum/standalone_tarballs_for_specific_versions/comment_13_121ae7a2803975e3bdcb5f9df0c4af3f._comment diff --git a/doc/forum/standalone_tarballs_for_specific_versions/comment_13_121ae7a2803975e3bdcb5f9df0c4af3f._comment b/doc/forum/standalone_tarballs_for_specific_versions/comment_13_121ae7a2803975e3bdcb5f9df0c4af3f._comment new file mode 100644 index 0000000000..b496f647ed --- /dev/null +++ b/doc/forum/standalone_tarballs_for_specific_versions/comment_13_121ae7a2803975e3bdcb5f9df0c4af3f._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="updating the standalone distribution to 8.20200908" + date="2020-09-12T15:20:18Z" + content=""" +Building the standalone distribution on conda-forge doesn't seem possible, unfortunately, since the build environment there is CentOS not Debian. + +@joey, when you get a chance, could you update the [https://downloads.kitenet.net/git-annex/linux/current/](https://downloads.kitenet.net/git-annex/linux/current/) and [http://archive.org/download/git-annex-builds](http://archive.org/download/git-annex-builds) version to 8.20200908 ? [Current version there](http://archive.org/download/git-annex-builds/SHA256E-s55287727--50bd5f4673fe85859b0a2c1cc4b68c93ace4ba3fca1b4eeeedeb0f3f243b9bad.tar.gz) is 8.20200815 , which is not an official release. +Thanks! + +"""]] From 9fb8397936dd15d217844b981e148f2ac9731b4e Mon Sep 17 00:00:00 2001 From: falsifian Date: Sat, 12 Sep 2020 15:29:42 +0000 Subject: [PATCH 07/13] Added a comment --- .../comment_1_458f1ce11cabaa01706461596340912b._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_1_458f1ce11cabaa01706461596340912b._comment diff --git a/doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_1_458f1ce11cabaa01706461596340912b._comment b/doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_1_458f1ce11cabaa01706461596340912b._comment new file mode 100644 index 0000000000..676add3b35 --- /dev/null +++ b/doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_1_458f1ce11cabaa01706461596340912b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="falsifian" + avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae" + subject="comment 1" + date="2020-09-12T15:29:38Z" + content=""" +Update: I've verified that the empty groupwanted expression indeed does indeed block the groupwanted expression from another group. So, other than making sure I never use the `common` group again, or some manual repository surgery, I'm not sure how to recover from this situation. +"""]] From 37abb540e9ef07c31841fd10db79288909115fca Mon Sep 17 00:00:00 2001 From: falsifian Date: Sat, 12 Sep 2020 15:30:06 +0000 Subject: [PATCH 08/13] Added a comment --- .../comment_2_23d18b52d87468ffadf0a3e04bf8052d._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_2_23d18b52d87468ffadf0a3e04bf8052d._comment diff --git a/doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_2_23d18b52d87468ffadf0a3e04bf8052d._comment b/doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_2_23d18b52d87468ffadf0a3e04bf8052d._comment new file mode 100644 index 0000000000..c1c622e132 --- /dev/null +++ b/doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_2_23d18b52d87468ffadf0a3e04bf8052d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="falsifian" + avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae" + subject="comment 2" + date="2020-09-12T15:30:05Z" + content=""" +Update: I've verified that the empty groupwanted expression indeed does indeed block the groupwanted expression from another group. So, other than making sure I never use the `common` group again, or some manual repository surgery, I'm not sure how to recover from this situation. +"""]] From 2cedadf82ad63f4dff58e3391ede93064e325c01 Mon Sep 17 00:00:00 2001 From: falsifian Date: Sat, 12 Sep 2020 15:30:38 +0000 Subject: [PATCH 09/13] removed --- .../comment_1_458f1ce11cabaa01706461596340912b._comment | 8 -------- 1 file changed, 8 deletions(-) delete mode 100644 doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_1_458f1ce11cabaa01706461596340912b._comment diff --git a/doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_1_458f1ce11cabaa01706461596340912b._comment b/doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_1_458f1ce11cabaa01706461596340912b._comment deleted file mode 100644 index 676add3b35..0000000000 --- a/doc/forum/Can_I_remove_a_groupwanted_expression__63__/comment_1_458f1ce11cabaa01706461596340912b._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="falsifian" - avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae" - subject="comment 1" - date="2020-09-12T15:29:38Z" - content=""" -Update: I've verified that the empty groupwanted expression indeed does indeed block the groupwanted expression from another group. So, other than making sure I never use the `common` group again, or some manual repository surgery, I'm not sure how to recover from this situation. -"""]] From 1eac1c26bf0c7390c38fa071e25957ffcd5cb793 Mon Sep 17 00:00:00 2001 From: falsifian Date: Sat, 12 Sep 2020 15:43:49 +0000 Subject: [PATCH 10/13] --- ...ying_the_groupwanted_expression_does..mdwn | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 doc/forum/Setting_required_to___34__groupwanted__34___doesn__39__t_work__44___but_copying_the_groupwanted_expression_does..mdwn diff --git a/doc/forum/Setting_required_to___34__groupwanted__34___doesn__39__t_work__44___but_copying_the_groupwanted_expression_does..mdwn b/doc/forum/Setting_required_to___34__groupwanted__34___doesn__39__t_work__44___but_copying_the_groupwanted_expression_does..mdwn new file mode 100644 index 0000000000..5e40d15fa1 --- /dev/null +++ b/doc/forum/Setting_required_to___34__groupwanted__34___doesn__39__t_work__44___but_copying_the_groupwanted_expression_does..mdwn @@ -0,0 +1,26 @@ +I'm trying to give all my repositories the same requirements by putting them in a group "`common_requirements`" with a `groupwanted` expression. However, it looks like the requirement is not actually being applied: + + $ git annex required . + groupwanted + $ git annex group . + common_requirements + $ git annex groupwanted common_requirements + include=dump/angel-openbsd/* and not copies=not_angel-openbsd:3 + $ git annex fsck --quiet + +The `fsck` should be complaining that some files are not present. If I set the `required` expression directly instead of using `groupwanted`, I get the behaviour I expected: + + $ git annex fsck --quiet + ** Required content dump/angel-openbsd/2020-09-12T01:20:52Z_level_0/_.bzip2 is missing from these repositories: + 91c090a3-a8fc-4540-83ec-57493967c708 -- em [here] + ** Required content dump/angel-openbsd/2020-09-12T01:20:52Z_level_0/_home.bzip2 is missing from these repositories: + 91c090a3-a8fc-4540-83ec-57493967c708 -- em [here] + ... (some output omitted) + git-annex: fsck: 10 failed + +Any idea what could cause `git annex required groupwanted` to behave differently from just copying the groupwanted expression in this case? Note that the repo is not in any other groups (see output of `git annex group .` above). I tried to reproduce this with some fresh toy repositories but wasn't able to --- in my controlled setting, `git annex required groupwanted` did what I wanted. + +A couple notes about the context here: + +* `dump/angel-openbsd` contains dumps of my host "`angel-openbsd`", and I want them to be stored in at last two repos other than `angel-openbsd` (no point in storing backups on the same computer). This repo should really be in the `not_angel-openbsd` group but I took it out of that group to make sure the issue isn't multiple conflicting `groupwanted` expressions. +* I would really like to have a global "requirements" setting that generalizes `numcopies`. My `common_requirements` group is a hack to try to achieve this, but I think I will run into trouble if I want to actually use git-annex's `groupwanted` functionality for anything else. From d2427daa5c84d769a904eb60b1a1f7563e285e35 Mon Sep 17 00:00:00 2001 From: falsifian Date: Sat, 12 Sep 2020 15:50:18 +0000 Subject: [PATCH 11/13] clarify first fsck call gave no output --- ..._work__44___but_copying_the_groupwanted_expression_does..mdwn | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/forum/Setting_required_to___34__groupwanted__34___doesn__39__t_work__44___but_copying_the_groupwanted_expression_does..mdwn b/doc/forum/Setting_required_to___34__groupwanted__34___doesn__39__t_work__44___but_copying_the_groupwanted_expression_does..mdwn index 5e40d15fa1..c885ae7db1 100644 --- a/doc/forum/Setting_required_to___34__groupwanted__34___doesn__39__t_work__44___but_copying_the_groupwanted_expression_does..mdwn +++ b/doc/forum/Setting_required_to___34__groupwanted__34___doesn__39__t_work__44___but_copying_the_groupwanted_expression_does..mdwn @@ -7,6 +7,7 @@ I'm trying to give all my repositories the same requirements by putting them in $ git annex groupwanted common_requirements include=dump/angel-openbsd/* and not copies=not_angel-openbsd:3 $ git annex fsck --quiet + [no output] The `fsck` should be complaining that some files are not present. If I set the `required` expression directly instead of using `groupwanted`, I get the behaviour I expected: From 9f91f83b75cb3526079379eb2c4a467558089311 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Sat, 12 Sep 2020 19:07:58 +0000 Subject: [PATCH 12/13] Added a comment: standalone build version oddity --- .../comment_14_5183f3d940ef116b07f88bac25cb97a8._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/standalone_tarballs_for_specific_versions/comment_14_5183f3d940ef116b07f88bac25cb97a8._comment diff --git a/doc/forum/standalone_tarballs_for_specific_versions/comment_14_5183f3d940ef116b07f88bac25cb97a8._comment b/doc/forum/standalone_tarballs_for_specific_versions/comment_14_5183f3d940ef116b07f88bac25cb97a8._comment new file mode 100644 index 0000000000..6a26d0f56a --- /dev/null +++ b/doc/forum/standalone_tarballs_for_specific_versions/comment_14_5183f3d940ef116b07f88bac25cb97a8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="standalone build version oddity" + date="2020-09-12T19:07:55Z" + content=""" +Oddly, the mtime of the files at [https://downloads.kitenet.net/git-annex/linux/current](https://downloads.kitenet.net/git-annex/linux/current) is 2020-09-08, which does correspond to the latest git-annex release; but when run, it prints \"git-annex version: 8.20200815-g6ea511beb\". +"""]] From b6d3c8a0af008460b8fa1314c717cb8d0ba5f0de Mon Sep 17 00:00:00 2001 From: kyotov Date: Sun, 13 Sep 2020 16:40:53 +0000 Subject: [PATCH 13/13] --- doc/forum/is_windows_really_slow__63__.mdwn | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/forum/is_windows_really_slow__63__.mdwn diff --git a/doc/forum/is_windows_really_slow__63__.mdwn b/doc/forum/is_windows_really_slow__63__.mdwn new file mode 100644 index 0000000000..18e74fb21e --- /dev/null +++ b/doc/forum/is_windows_really_slow__63__.mdwn @@ -0,0 +1,16 @@ +I am trying annex for the first time. +I am trying it on windows (installed the beta). +My understanding from reading the docs is that it uses the repo in fully unlocked mode because windows does not support symlinks (or so git / git-annex believes). + +I currently have about 55K files and 220GB... mostly pictures. + +I have sidecar XMP files (text) next to each jpeg. +When I git add the XMP files, it is very very slow (about 1s per file). +When I do git status after adding them it takes minutes (refreshing index...). + +I have a pretty powerful machine: +Processor Intel(R) Core(TM) i9-7940X CPU @ 3.10GHz, 3096 Mhz, 14 Core(s), 28 Logical Processor(s) +128GB RAM +2TB NVMe SSD (~3GB/s symmetric transfer rate). + +My gut feel is that it is because all the jpgs remain in the worktree as opposed to symlinked, but wanted to hear an expert opinion?