From 8db097e0819f386c1bfce7974010c4013d5dc812 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 5 Feb 2020 16:39:40 +0000 Subject: [PATCH 1/6] Added a comment: aborting stuck operations so they can be retried --- .../comment_4_418a9cbc38142ec1b2d08fd617a3e4d4._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/todo/more_extensive_retries_to_mask_transient_failures/comment_4_418a9cbc38142ec1b2d08fd617a3e4d4._comment diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_4_418a9cbc38142ec1b2d08fd617a3e4d4._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_4_418a9cbc38142ec1b2d08fd617a3e4d4._comment new file mode 100644 index 0000000000..8141141765 --- /dev/null +++ b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_4_418a9cbc38142ec1b2d08fd617a3e4d4._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="aborting stuck operations so they can be retried" + date="2020-02-05T16:39:36Z" + content=""" +\"The only way to guarantee such an abort is to kill the whole git-annex process and let the signal reap its children\" -- then maybe the initial `git-annex` command can be made a wrapper that starts a separate `git-annex` process to do the actual work, monitors its progress, and kills/reaps/restarts it if it gets stuck? Or `-Jn` could work by starting up several separate git-annex processes, [[each handling a subset of files|parallel_possibilities/#comment-304240ba804513291c1a996b8eb3fd1c]], and the original process could kill/reap/restart any sub-process that gets stuck. This of course presumes idempotent operations. +"""]] From 28dc7d01decba408bb3db911a58f94842a279ae8 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 5 Feb 2020 16:59:44 +0000 Subject: [PATCH 2/6] Added a comment: retries due to locked index file --- .../comment_5_69b132f465851421acb7e5edf009995d._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/todo/more_extensive_retries_to_mask_transient_failures/comment_5_69b132f465851421acb7e5edf009995d._comment diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_5_69b132f465851421acb7e5edf009995d._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_5_69b132f465851421acb7e5edf009995d._comment new file mode 100644 index 0000000000..e764bdd56b --- /dev/null +++ b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_5_69b132f465851421acb7e5edf009995d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="retries due to locked index file" + date="2020-02-05T16:59:40Z" + content=""" +\"A locked git index file does not prevent git-annex from making transfers\" -- by \"mask transient failures\" I meant all types of failures, not just transfers. So e.g. if concurrent operations fail due to contention for the index file lock, retries (after increasing, randomized intervals) could mask the failure. This would help especially for writing scripts/tools on top of git-annex. Logically, some operations -- like `git-annex-add` -- should never fail, and being able to assume that makes scripting easier. +"""]] From 5c57440482264dd8911bb5835f9b3f14a696e686 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 5 Feb 2020 22:19:30 +0000 Subject: [PATCH 3/6] Added a comment: example of where retries could help --- .../comment_6_5a7771a6169caa90fd91bc6ea7b4fe3d._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/todo/more_extensive_retries_to_mask_transient_failures/comment_6_5a7771a6169caa90fd91bc6ea7b4fe3d._comment diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_6_5a7771a6169caa90fd91bc6ea7b4fe3d._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_6_5a7771a6169caa90fd91bc6ea7b4fe3d._comment new file mode 100644 index 0000000000..51b9a693b3 --- /dev/null +++ b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_6_5a7771a6169caa90fd91bc6ea7b4fe3d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="example of where retries could help" + date="2020-02-05T22:19:26Z" + content=""" +As one example, I just had a `git-annex-copy` command fail twice with `git-annex: thread blocked indefinitely in an STM transaction`, then have the same command succeed (or at least get much further -- still running) on the third try. I can write my own wrappers to mask such errors, but a built-in implementation seems generally useful and would know better which failures are likely transient. +"""]] From 227206c089f8ec45465c92e9c907e65b03ccad17 Mon Sep 17 00:00:00 2001 From: "lykos@d125a37d89b1cfac20829f12911656c40cb70018" Date: Thu, 6 Feb 2020 06:56:41 +0000 Subject: [PATCH 4/6] --- ...-encrypt_when_key_is_already_in_.git__47__annex__47__tmp.mdwn | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/todo/Don__39__t_re-encrypt_when_key_is_already_in_.git__47__annex__47__tmp.mdwn diff --git a/doc/todo/Don__39__t_re-encrypt_when_key_is_already_in_.git__47__annex__47__tmp.mdwn b/doc/todo/Don__39__t_re-encrypt_when_key_is_already_in_.git__47__annex__47__tmp.mdwn new file mode 100644 index 0000000000..6a461cffff --- /dev/null +++ b/doc/todo/Don__39__t_re-encrypt_when_key_is_already_in_.git__47__annex__47__tmp.mdwn @@ -0,0 +1 @@ +I've implemented true resumable upload in git-annex-remote-googledrive which means that uploads can, just as downloads, be resumed at any point, even within one chunk. However, it currently does not work with encrypted files (or chunks) due to the non-deterministic nature of GPG. In order to make this feature useable on encrypted files, I propose to not overwrite encrypted files which are already present inside the `tmp` directory. From 9918952a5b0a3bfa955e431236437ca51fc2bf67 Mon Sep 17 00:00:00 2001 From: "t+gitannex@1d62779e8b54f30a854739f61542a6885167b01f" Date: Thu, 6 Feb 2020 11:07:35 +0000 Subject: [PATCH 5/6] Added a comment --- .../comment_6_e0bfde0d53042ce8d310f356f88c610b._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/bugs/git_keeps_refreshing_index/comment_6_e0bfde0d53042ce8d310f356f88c610b._comment diff --git a/doc/bugs/git_keeps_refreshing_index/comment_6_e0bfde0d53042ce8d310f356f88c610b._comment b/doc/bugs/git_keeps_refreshing_index/comment_6_e0bfde0d53042ce8d310f356f88c610b._comment new file mode 100644 index 0000000000..b561288e73 --- /dev/null +++ b/doc/bugs/git_keeps_refreshing_index/comment_6_e0bfde0d53042ce8d310f356f88c610b._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="t+gitannex@1d62779e8b54f30a854739f61542a6885167b01f" + nickname="t+gitannex" + avatar="http://cdn.libravatar.org/avatar/87c7f62c00e4a744aa500423e421120f" + subject="comment 6" + date="2020-02-06T11:07:34Z" + content=""" +I'm able to reproduce this with git annex 7.20191230 and git 2.25.0 on Arch Linux, but I've had it on OSX in the past as well. The annex uses a v7 repository. I don't need to do anything besides unlocking some files and running git status. Unlocking 10 files, git status takes 3s and with 85 files it takes 20s, so it seems to scale linearly with the no of files. Happy to share more details about the repository if it's useful. +"""]] From 83fb0314606306592e1d126ddfcb9f2e41255997 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Thu, 6 Feb 2020 21:00:57 +0000 Subject: [PATCH 6/6] Added a comment: potential security issues? --- ...comment_7_1c0a975893c63c14b3f6e17712b5191c._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/todo/alternate_keys_for_same_content/comment_7_1c0a975893c63c14b3f6e17712b5191c._comment diff --git a/doc/todo/alternate_keys_for_same_content/comment_7_1c0a975893c63c14b3f6e17712b5191c._comment b/doc/todo/alternate_keys_for_same_content/comment_7_1c0a975893c63c14b3f6e17712b5191c._comment new file mode 100644 index 0000000000..80f61be55d --- /dev/null +++ b/doc/todo/alternate_keys_for_same_content/comment_7_1c0a975893c63c14b3f6e17712b5191c._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="potential security issues?" + date="2020-02-06T21:00:55Z" + content=""" +I wonder if storing checksums in a general-purpose mutable metadata field may cause security issues. Someone could use the [[`git-annex-metadata`|git-annex-metadata]] command to overwrite the checksum. It should be stored in a read-only field written only by `git-annex` itself, like the `field-lastchanged` metadata already is. + +Of course, if someone is able to write the [[git-annex branch|internals#The_git-annex_branch]] directly, or get the user to pull merges to it, they could alter the checksum stored there. Maybe, only trust stored checksums if `merge.verifySignatures=true`? +"""]]