Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2019-12-18 13:48:48 -04:00
commit 83a3b1bec4
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
6 changed files with 164 additions and 0 deletions

View file

@ -0,0 +1,76 @@
In datalad test builds with git-annex 7.20191114+git43-ge29663773, one
of the new test failures is due to an unexpectedly dirty repository
([related datalad issue][0]). The dirty status comes from a file that
was tracked in Git switching over to an annex pointer file. Here's a
script that distills enough of the test to trigger the failure on my
end.
[[!format sh """
#!/bin/sh
set -eu
assert_clean () {
if test -n "$(git status --porcelain)"
then
printf "\n\nUnexpectedly dirty:\n" >&2
git status >&2
git diff >&2
exit 1
fi
}
cd "$(mktemp -d --tmpdir gx-pointer-dirty-XXXXXXX)"
git init && git annex init
printf content-git >file-git
git -c annex.largefiles=nothing annex add -- file-git
git commit -m'file-git added'
assert_clean
printf content-annex >file-annex
git -c annex.largefiles=anything annex add -- file-annex
git commit -m'file-annex annexed'
assert_clean
"""]]
On Travis as well as my local machine, the failure is intermittent,
but seems to happen much more often than not. In the failing case,
the last assert_clean call shows:
```
Unexpectedly dirty:
On branch master
Changes not staged for commit:
modified: file-git
no changes added to commit
diff --git a/file-git b/file-git
index d1c416a..b41ca32 100644
--- a/file-git
+++ b/file-git
@@ -1 +1 @@
-content-git
\ No newline at end of file
+/annex/objects/SHA256E-s11--726732d25826965592478fcc7c145d5a10fa1aa70c49fe3a4f847174b6d8889c
```
I see the failure with git-annex built from the latest master
b962471c2 (2019-12-12). Bisecting against the git-annex repo (with a
commit being marked "bad" if there was a failure within ten runs of the
above script), points to ec08b66bd (shouldAnnex: check isInodeKnown,
2019-10-23) as the first bad commit. Just looking at the topic of
the commit, that result seems plausible to me.
### Other details
My git version 2.24.1 and locally I'm building git-annex through guix.
On the failing Travis run, git-annex 7.20191114+git43-ge29663773 came
from neurodebian, and the git version was 2.24.0.
Hopefully the script above is sufficient to trigger the issue on your end.
Thanks for having a look.
[0]: https://github.com/datalad/datalad/issues/3890
[[!meta author=kyle]]

View file

@ -0,0 +1,51 @@
[[!comment format=mdwn
username="lykos@d125a37d89b1cfac20829f12911656c40cb70018"
nickname="lykos"
avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2"
subject="comment 3"
date="2019-12-12T21:11:58Z"
content="""
Apparently it happens when the remote returns PREPARE-FAILURE (in this case due to insufficient internet connection). So we get back to
[PREPARE-LOCAL](http://git-annex.branchable.com/design/external_special_remote_protocol/#comment-2eec51c9f774f577b8634e9cdc86cde3) and [external_remote_querying_transition](https://git-annex.branchable.com/todo/external_remote_querying_transition).
I'm going to change git-annex-remote-googledrive to defer network connection to when when it's needed. But as external remotes are encouraged to establish network connections in PREPARE, something should be done on git-annex's side, too.
% git annex addurl --batch --with-files --debug
[url] [filename]
[2019-12-12 21:41:14.540686936] read: git [\"--git-dir=../../../.git\",\"--work-tree=../../..\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
[2019-12-12 21:41:14.546632041] process done ExitSuccess
[2019-12-12 21:41:14.546795605] read: git [\"--git-dir=../../../.git\",\"--work-tree=../../..\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2019-12-12 21:41:14.551391222] process done ExitSuccess
[2019-12-12 21:41:14.551593051] read: git [\"--git-dir=../../../.git\",\"--work-tree=../../..\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..802429a871b754d61047457aa9936bde404e3172\",\"--pretty=%H\",\"-n1\"]
[2019-12-12 21:41:14.553261192] process done ExitSuccess
[2019-12-12 21:41:14.553318623] read: git [\"--git-dir=../../../.git\",\"--work-tree=../../..\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..0e8f6eb6cc45d9bb11f42fcf8590b3473e6f2623\",\"--pretty=%H\",\"-n1\"]
[2019-12-12 21:41:14.555647103] process done ExitSuccess
[2019-12-12 21:41:14.555710805] read: git [\"--git-dir=../../../.git\",\"--work-tree=../../..\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..7271aaaa110b0f84b5e730673c66b176789ddcaf\",\"--pretty=%H\",\"-n1\"]
[2019-12-12 21:41:14.557235619] process done ExitSuccess
[2019-12-12 21:41:14.557298991] read: git [\"--git-dir=../../../.git\",\"--work-tree=../../..\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..b9112bcb862041cf96f00e12c6dd33dc31fda5fa\",\"--pretty=%H\",\"-n1\"]
[2019-12-12 21:41:14.560071044] process done ExitSuccess
[2019-12-12 21:41:14.560126818] read: git [\"--git-dir=../../../.git\",\"--work-tree=../../..\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..1598b00a78067edc438652f401498a25635cd5a9\",\"--pretty=%H\",\"-n1\"]
[2019-12-12 21:41:14.562966332] process done ExitSuccess
[2019-12-12 21:41:14.563218447] chat: git [\"--git-dir=../../../.git\",\"--work-tree=../../..\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
[2019-12-12 21:41:14.563535554] chat: git [\"--git-dir=../../../.git\",\"--work-tree=../../..\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
[2019-12-12 21:41:14.56562967] read: git [\"config\",\"--null\",\"--list\"]
[2019-12-12 21:41:14.566788099] process done ExitSuccess
[2019-12-12 21:41:14.567203332] read: git [\"config\",\"--null\",\"--list\"]
[2019-12-12 21:41:14.568286923] read: git [\"config\",\"--null\",\"--list\"]
[2019-12-12 21:41:14.569077497] chat: /home/silvio/.local/bin/git-annex-remote-googledrive []
[2019-12-12 21:41:14.944193013] git-annex-remote-googledrive[1] --> VERSION 1
[2019-12-12 21:41:14.944312387] git-annex-remote-googledrive[1] <-- EXTENSIONS INFO
[2019-12-12 21:41:14.944486597] git-annex-remote-googledrive[1] --> EXTENSIONS
[2019-12-12 21:41:14.944557495] git-annex-remote-googledrive[1] <-- EXPORTSUPPORTED
[2019-12-12 21:41:14.94467435] git-annex-remote-googledrive[1] --> EXPORTSUPPORTED-SUCCESS
[2019-12-12 21:41:14.944906493] chat: /home/silvio/.local/bin/git-annex-remote-googledrive []
[2019-12-12 21:41:15.333971094] git-annex-remote-googledrive[1] --> VERSION 1
[2019-12-12 21:41:15.334086558] git-annex-remote-googledrive[1] <-- EXTENSIONS INFO
[2019-12-12 21:41:15.334273359] git-annex-remote-googledrive[1] --> EXTENSIONS
[2019-12-12 21:41:15.334334558] git-annex-remote-googledrive[1] <-- PREPARE
[...]
[2019-12-12 21:41:43.372727829] git-annex-remote-googledrive[1] --> PREPARE-FAILURE ('Failed to connect with Google. Please check your internet connection.', ServerNotFoundError('Unable to find the server at www.googleapis.com'))
git-annex: ('Failed to connect with Google. Please check your internet connection.', ServerNotFoundError('Unable to find the server at www.googleapis.com'))
"""]]

View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="mike@4e7a118bd37129091199ae0fb28184999c5f4725"
nickname="mike"
avatar="http://cdn.libravatar.org/avatar/9751260f8a13fc27ad837eaf66fac5d7"
subject="repodata is outdated"
date="2019-12-18T10:06:08Z"
content="""
The repodata under https://downloads.kitenet.net/git-annex/linux/current/rpms/repodata/ is outdated (from September), latest RPM in https://downloads.kitenet.net/git-annex/linux/current/rpms/ is from November. I (think) I had an issue using git-annex-shell using the 201909 version, so this is somewhat relevant.
"""]]

View file

@ -0,0 +1,15 @@
[[!comment format=mdwn
username="https://christian.amsuess.com/chrysn"
nickname="chrysn"
avatar="http://christian.amsuess.com/avatar/c6c0d57d63ac88f3541522c4b21198c3c7169a665a2f2d733b4f78670322ffdc"
subject="Would be useful"
date="2019-12-17T09:08:08Z"
content="""
A `git annex cat` would be useful for the very web server purpose you describe (WIP at https://gitlab.com/chrysn/annex-to-web, though I'm not sure it's going anywhere).
Unlike `git annex inprogress` that I (will) use for a workaround, this could take a `--skip` argument that usually just seeks into the file.
If the data is served from a remote that allows seeking access (eg. IPFS),
then that access could be priorized and that part downloaded first.
(Implementing this would require another tmp pool for sparse files as they couldn't go with the `git annex inprogress` files for there is the expectation that those would grow to completion,
but anyway this would be an entry point for such a feature if it is ever added).
"""]]

View file

@ -0,0 +1,8 @@
Unlike `whereis` and other subcommands, `inprogress` does not offer a `--key` argument to select files by key rather than checked-out name,
making it unusable in bare repositories.
Please consider adding a `--key` option there, which would display the single incomplete file corresponding to the key if one is in progress.
My use case is serving git-annexed files to the web from a bare repository (<https://gitlab.com/chrysn/annex-to-web>, see also [[todo/git-annex-cat]]), which would be especially useful with gitolite repositories as they are by design bare, and on devices where checkouts are cumbersome (cf. [[forum/Dealing_with_crippled_Android_file_system]]).
A workaround is running `git annex inprogress --all | grep $KEY`, but that's probably relying on an implementation detail that could be changed at any time (though it probably won't as to avoid race conditions as in `tail -f $(git annex inprogress file-thats-almost.done)`).

View file

@ -0,0 +1,5 @@
I want to add some dotfiles in the root of my repository to git-annex as unlocked annexed files. So I edited `.git/info/attributes` to remove the line `.* !filter`, such that it only contains the line `* filter=annex`. This seems to be working fine.
I was thinking that it might make sense to have a `git annex config` option to tell git-annex not to add the `.* !filter` line to `.git/info/attributes` when initialising other clones of this repo. In the meantime, I've worked around it using a `post_checkout` hook in my `~/.mrconfig` which edits `.git/info/attributes`.
--spwhitton