From e0022e6505437a4ed70498e83d85c3795950bb62 Mon Sep 17 00:00:00 2001 From: CandyAngel Date: Mon, 7 Oct 2019 08:30:53 +0000 Subject: [PATCH 1/7] Added a comment --- ..._3ff54cd13085a9fc2d8f9953691da8d2._comment | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 doc/forum/lets_discuss_git_add_behavior/comment_6_3ff54cd13085a9fc2d8f9953691da8d2._comment diff --git a/doc/forum/lets_discuss_git_add_behavior/comment_6_3ff54cd13085a9fc2d8f9953691da8d2._comment b/doc/forum/lets_discuss_git_add_behavior/comment_6_3ff54cd13085a9fc2d8f9953691da8d2._comment new file mode 100644 index 0000000000..9f23dbdf88 --- /dev/null +++ b/doc/forum/lets_discuss_git_add_behavior/comment_6_3ff54cd13085a9fc2d8f9953691da8d2._comment @@ -0,0 +1,36 @@ +[[!comment format=mdwn + username="CandyAngel" + avatar="http://cdn.libravatar.org/avatar/15c0aade8bec5bf004f939dd73cf9ed8" + subject="comment 6" + date="2019-10-07T08:30:52Z" + content=""" +If you want to add the file to git, use `git add`. +If you want to add the file to git-annex, use `git annex add`. +Simples! + +There isn't any other behaviour which is a more obvious default. + +> Suppose you have an unlocked file in your repo, and you rename it (not using git move), and then git add it. Oops, now you've added to git a large file that you wanted to be annexed + +If you wanted it to be annexed, you should have `git annex add`'d it! git-annex doesn't (and can't) know that the user wanted something different from the totally valid command they issued. + +> you would surely hope that the annexed ones stay annexed and don't get committed directly to git + +If the modified file changes its match state from largefiles (e.g. crossing a filesize threshold), it would still change state between annexed/non-annexed, wouldn't it? + +> keeping track of which files are supposed to be in the annex and which in git is very failure prone +> And it needs to default to adding files to the annex, otherwise the above two cases can cause problems. + +Not only is it failure prone, the only thing that knows which is wanted is.. the user. The decision to usurp git and the user creates the first 2 problem cases. If you go with the expectation that the user will issue the correct commands for what they want to happen (fair, considering only the user knows), the first two cases are obviously not problems. + +> If git add does something the user doesn't want + +Why would it? It just adds files to git, right? + +> Recovery [..] from [adding file to git] can be arbitrarily compilicated, including needing to fix problems in clones on other people's computers. + +And this can still totally happen if largefiles is not set correctly for what the user wants. + + +Sure, you can set up git-annex to do magic to make your workflow easier or more seamless. Key words there being \"*set up*\". It shouldn't be doing such magic by default. +"""]] From 3d677c0990dbfd6a27dae8ff0f1fe6c5bd2afa8f Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Mon, 7 Oct 2019 18:08:44 +0000 Subject: [PATCH 2/7] added suggestion for more fine-grained config of when files are added as locked vs unlocked --- doc/todo/addunlocked_config_setting.mdwn | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/todo/addunlocked_config_setting.mdwn diff --git a/doc/todo/addunlocked_config_setting.mdwn b/doc/todo/addunlocked_config_setting.mdwn new file mode 100644 index 0000000000..fd9269b1cc --- /dev/null +++ b/doc/todo/addunlocked_config_setting.mdwn @@ -0,0 +1,5 @@ +Can the `annex.addunlocked` be extended to have the same syntax as `annex.largefiles`? Also, can there be separate settings affecting `git add` and `git annex add`, e.g. `annex.git-add.addunlocked` and `annex.git-annex-add.addunlocked`, with both defaulting to the value of `annex.addunlocked` if not set? + +Basically, I want a reliable way to prevent inadvertently adding files as annexed unlocked files. + +Related: [[forum/lets_discuss_git_add_behavior]] From daa1fa72effc9201eec37671b2662b7db1eb3732 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Mon, 7 Oct 2019 18:20:21 +0000 Subject: [PATCH 3/7] Added a comment --- ...comment_7_a228667f09285bb466076fb40cc20edb._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/forum/lets_discuss_git_add_behavior/comment_7_a228667f09285bb466076fb40cc20edb._comment diff --git a/doc/forum/lets_discuss_git_add_behavior/comment_7_a228667f09285bb466076fb40cc20edb._comment b/doc/forum/lets_discuss_git_add_behavior/comment_7_a228667f09285bb466076fb40cc20edb._comment new file mode 100644 index 0000000000..7d7ceb2d66 --- /dev/null +++ b/doc/forum/lets_discuss_git_add_behavior/comment_7_a228667f09285bb466076fb40cc20edb._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="comment 7" + date="2019-10-07T18:20:21Z" + content=""" +Related: [[todo/addunlocked_config_setting]]. + +Re: this thread, I also think preserving `git add` default behavior (adding to git) is better. I'm not sure it should *always* add to git. The whole point of v7 was, as I understand it, to make it possible to use normal git workflow (`git add`; `git commit`; [make changes]; `git add`; `git commit`) with large files without thinking about it. Existing scripts that just call `git add` and are unaware of `git-annex-add` would still work. So it makes sense to let `git add` add to annex *when explicitly configured*. In my use case, I'd like to configure it so that any files it adds to the annex are added as locked by default. +"""]] From 22a6fcaa24da2bc8f3435de2ed0fa4be61e40d62 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Mon, 7 Oct 2019 18:57:20 +0000 Subject: [PATCH 4/7] added report of repeated git index refresh --- doc/bugs/git_keeps_refreshing_index.mdwn | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/bugs/git_keeps_refreshing_index.mdwn diff --git a/doc/bugs/git_keeps_refreshing_index.mdwn b/doc/bugs/git_keeps_refreshing_index.mdwn new file mode 100644 index 0000000000..cd1f0e165d --- /dev/null +++ b/doc/bugs/git_keeps_refreshing_index.mdwn @@ -0,0 +1 @@ +Since upgrading to git-annex 7.20190912, when doing `git status` I keep getting the message `Refresh index:`, and there is a delay until it turns into something like `Refresh index: 100% (601422/601422), done.`. I don't recall this happening with earlier `git-annex` versions. Have others seen this? (Not a "bug" in terms of correctness, but posting here as it affects usability and speed.) From a728b78ca2dd6ebf0b50581f8a06faa21aa76a83 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Tue, 8 Oct 2019 05:13:23 +0000 Subject: [PATCH 5/7] added todo for operating on files touched by commits in a range --- doc/todo/operate_on_files_affected_by_a_commit_range.mdwn | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/todo/operate_on_files_affected_by_a_commit_range.mdwn diff --git a/doc/todo/operate_on_files_affected_by_a_commit_range.mdwn b/doc/todo/operate_on_files_affected_by_a_commit_range.mdwn new file mode 100644 index 0000000000..74047fcf90 --- /dev/null +++ b/doc/todo/operate_on_files_affected_by_a_commit_range.mdwn @@ -0,0 +1 @@ +Sometimes you want to operate on files touched by commits in a range, e.g. to `git-annex-copy` files added in the last 10 commits to an S3 special remote. Could the option be added, to commands that take a path to operate on, to give a commit range, with the meaning "operate on files changed by these commits"? From d2369894f021ad445e0714896775730c86ad8355 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Tue, 8 Oct 2019 06:43:33 +0000 Subject: [PATCH 6/7] Added a comment: automaticallly sync content on git push/pull --- .../comment_1_4e813660a7743ecc3378b87adb0cd0f8._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/todo/option_to_block_git_from_pushing_references_to_not-yet-saved_contents/comment_1_4e813660a7743ecc3378b87adb0cd0f8._comment diff --git a/doc/todo/option_to_block_git_from_pushing_references_to_not-yet-saved_contents/comment_1_4e813660a7743ecc3378b87adb0cd0f8._comment b/doc/todo/option_to_block_git_from_pushing_references_to_not-yet-saved_contents/comment_1_4e813660a7743ecc3378b87adb0cd0f8._comment new file mode 100644 index 0000000000..3a2658b6da --- /dev/null +++ b/doc/todo/option_to_block_git_from_pushing_references_to_not-yet-saved_contents/comment_1_4e813660a7743ecc3378b87adb0cd0f8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="automaticallly sync content on git push/pull" + date="2019-10-08T06:43:33Z" + content=""" +Even better would be an option to automatically sync copy the content referenced by any pushed commits, to a specified remote; and/or, to automatically `git-annex-get` content referenced by any pulled commits. Then can use git-annex like git without needing to remember to `git-annex-sync`. [[todo/operate_on_files_affected_by_a_commit_range]] could be useful in implementing the hooks. +"""]] From 2e8abdae5afe059a07e26cabf7486617453db3d3 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Tue, 8 Oct 2019 16:01:52 +0000 Subject: [PATCH 7/7] Added a comment: automatic retries if index is locked --- ...1_938d2c43a9a1b8432b54bcecaa14c20a._comment | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 doc/todo/more_extensive_retries_to_mask_transient_failures/comment_1_938d2c43a9a1b8432b54bcecaa14c20a._comment diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_1_938d2c43a9a1b8432b54bcecaa14c20a._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_1_938d2c43a9a1b8432b54bcecaa14c20a._comment new file mode 100644 index 0000000000..2863a81d0f --- /dev/null +++ b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_1_938d2c43a9a1b8432b54bcecaa14c20a._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="automatic retries if index is locked" + date="2019-10-08T16:01:52Z" + content=""" +As a concrete example, if the `index.lock` file exists and has relatively recent mtime and a git process is running, it would help if git-annex could be configured to retry, up to a given number of times with increasing delays between retries, the operation that failed because the index is locked. + +Also, from the log +[[!format sh \"\"\" +add metadata_orig.json ok +(recording state in git...) +fatal: Unable to create '/ssd/crogrun_191008_043145__8684__/viral-ngs-benchmarks/.git/index.lock': File exists. +\"\"\"]] + +it looks like the index.lock conflict is due to writing the [[git-annex branch|internals/#The_git-annex_branch]]? I thought git-annex used a separate index for that? + +"""]]