move old fixed datalad/dandi/repronim bugs to the project pages
This is to cut down on the number of files in bugs/, which makes it slow to file new bug reports or update active bug reports. These old bugs were about 1/3rd of the files in there. These projects want lists of their old bugs to still be accessible, and have the lists on their project pages, which will still list the old bugs. Commands used: for f in $(git grep -l '\[\[!tag projects/dandi\]\]'); do if grep -q 'done\]\]' "$f"; then git mv "$f" ../projects/dandi/bugs-done; g=$(echo "$f" | sed 's/.mdwn//'); if [ -d "$g" ]; then git mv "$g" ../projects/dandi/bugs-done; fi; fi; done for f in $(git grep -l '\[\[!tag projects/repronim\]\]'); do if grep -q 'done\]\]' "$f"; then git mv "$f" ../projects/repronim/bugs-done; g=$(echo "$f" | sed 's/.mdwn//'); if [ -d "$g" ]; then git mv "$g" ../projects/repronim/bugs-done; fi; fi; done for f in $(git grep -l '\[\[!tag projects/datalad\]\]'); do if grep -q 'done\]\]' "$f"; then git mv "$f" ../projects/datalad/bugs-done; g=$(echo "$f" | sed 's/.mdwn//'); if [ -d "$g" ]; then git mv "$g" ../projects/datalad/bugs-done; fi; fi; done That assumes that bugs are not tagged by multiple projects at the same time. Of the ones I moved, I've checked and none are. Could do the same with todo/ but there are only 370 files in there, and less than 84 of them could be moved this way, which does not seem likely to produce a sizeable speedup. Sponsored-by: Dartmouth College's Datalad project
This commit is contained in:
Joey Hess
parent
946fc20165
commit
bcc69f07e8
1011 changed files with 4 additions and 4 deletions
|
|
@ -0,0 +1,43 @@
|
|||
[[!format sh """
|
||||
|
||||
neurodebian@smaug ..6.20181011+git109-gff9ba1f4d-1~ndall+1 % grep -B10 FAIL git-annex_6.20181011+git109-gff9ba1f4d-1~ndall+1_amd64.build
|
||||
a03a722..0dd2b80 git-annex -> r2/git-annex
|
||||
To ../../.t/tmprepo331
|
||||
* [new branch] git-annex -> synced/git-annex
|
||||
To ../../.t/tmprepo330
|
||||
a03a722..22937c6 git-annex -> synced/git-annex
|
||||
OK (2.15s)
|
||||
adjusted branch merge regression: On branch master
|
||||
nothing to commit, working tree clean
|
||||
On branch master
|
||||
nothing to commit, working tree clean
|
||||
FAIL (0.50s)
|
||||
Test.hs:1463:
|
||||
adjust failed
|
||||
adjusted branch subtree regression: On branch master
|
||||
nothing to commit, working tree clean
|
||||
FAIL (0.31s)
|
||||
--
|
||||
059871b..4f97b0c annex/direct/master -> r2/annex/direct/master
|
||||
conflictor.variant-a507
|
||||
conflictor.variant-75dc
|
||||
conflictor.variant-a507
|
||||
conflictor.variant-75dc
|
||||
OK (1.37s)
|
||||
conflict resolution (adjusted branch): On branch master
|
||||
nothing to commit, working tree clean
|
||||
On branch master
|
||||
nothing to commit, working tree clean
|
||||
FAIL (0.70s)
|
||||
|
||||
"""]]
|
||||
|
||||
|
||||
Full log http://www.onerussian.com/tmp/git-annex_6.20181011+git109-gff9ba1f4d-1~ndall+1_amd64.build
|
||||
|
||||
(not sure if relevant - this is I think the first build without my custom no LOCPATH patch)
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="comment 1"
|
||||
date="2018-10-22T19:22:30Z"
|
||||
content="""
|
||||
See also (same?) failure when trying to build the conda-forge package:
|
||||
https://circleci.com/gh/conda-forge/git-annex-feedstock/116?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="comment 2"
|
||||
date="2018-10-24T16:01:53Z"
|
||||
content="""
|
||||
Maybe, post a new git-annex version on hackage, with the test failure fixed? I could then update the conda-forge recipe.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
### Please describe the problem.
|
||||
|
||||
Trying to build fresh snapshot for neurodebian but 3 tests report failing
|
||||
Full buildlogs are at [http://neuro.debian.net/_files/_buildlogs/git-annex/7.20181211+git29-gab4a1bed9](http://neuro.debian.net/_files/_buildlogs/git-annex/7.20181211+git29-gab4a1bed9)
|
||||
|
||||
|
||||
[[!format sh """
|
||||
neurodebian@smaug ../7.20181211+git29-gab4a1bed9-1~ndall+1 % grep -B3 '^FAIL' git-annex_7.20181211+git29-gab4a1bed9-1\~ndall+1_amd64.build
|
||||
bup remote: OK (0.11s)
|
||||
crypto: gpg: can't connect to the agent: IPC connect call failed
|
||||
gpg: error getting the KEK: No agent running
|
||||
FAIL
|
||||
--
|
||||
bup remote: OK (0.19s)
|
||||
crypto: gpg: can't connect to the agent: IPC connect call failed
|
||||
gpg: error getting the KEK: No agent running
|
||||
FAIL
|
||||
--
|
||||
nothing to commit, working tree clean
|
||||
gpg: can't connect to the agent: IPC connect call failed
|
||||
gpg: error getting the KEK: No agent running
|
||||
FAIL
|
||||
|
||||
"""]]
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[done]] I think.. --[[Joey]]
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2019-01-01T16:10:48Z"
|
||||
content="""
|
||||
I'll bet [[!commit 14971414dc263fcb8124b4cf6b14b9b7a19189af]]
|
||||
somehow led to this
|
||||
since the test suite now runs gpg with its home
|
||||
directory set to be inside the test repo and not in the system's temporary
|
||||
directory.
|
||||
|
||||
But I tested that commit on filesystems down to FAT, so it seems strange it
|
||||
would fail on whatever filesystem you are testing on.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="the build details"
|
||||
date="2019-01-02T01:14:39Z"
|
||||
content="""
|
||||
cowbuilder is used, underlying file system (dedicated to debian pkgs building) is ext4.
|
||||
but is it a filesystem effect somehow or just a \"partially\" isolated environment aspect (such as those pbuilder/cowbuilder chroots)? The message `gpg: can't connect to the agent: IPC connect call failed` suggests that may be tests pass for you locally as connecting to your gpg agent?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2019-01-18T17:51:09Z"
|
||||
content="""
|
||||
Having what seems a related problem on my i386 autobuilder:
|
||||
|
||||
gpg: keyblock resource '/home/builder/gitbuilder/build/.t/tmprepo89/.git/annex/othertmp/gpgtest/2/pubring.kbx': No such file or directory
|
||||
gpg: failed to create temporary file '/home/builder/gitbuilder/build/.t/tmprepo89/.git/annex/othertmp/gpgtest/2/.#lk0x57fed950.orca.13095': No such file or directory
|
||||
gpg: can't connect to the agent: No such file or directory
|
||||
gpg: problem with the agent: No agent running
|
||||
gpg: can't create `/home/builder/gitbuilder/build/.t/tmprepo89/.git/annex/othertmp/gpgtest/2/random_seed': No such file or directory
|
||||
user error (gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--batch","--passphrase-fd","28","--symmetric","--force-mdc","--no-textmode"] exited 2)
|
||||
copy: 1 failed
|
||||
FAIL (2.31s)
|
||||
Test.hs:1645:
|
||||
copy --to encrypted remote failed
|
||||
|
||||
Probably gpg error output has failed, but this is also failing to connect
|
||||
to the agent. And it seems a file/directory gpg expects to find is not present
|
||||
when it's running in the test harness.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2019-01-21T18:12:13Z"
|
||||
content="""
|
||||
I've fixed the problem I described above. But I don't think it was the
|
||||
same problem; the neurodebian build log is not complaining about missing
|
||||
files/directories.
|
||||
|
||||
gpg in the test suite is using a separate home directory, so it starts a
|
||||
separate gpg-agent process for that. If the test suite is run on a
|
||||
filesystem where gpg-agent can't work, due say to unix sockets not being
|
||||
supported, that might be why it fails.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 5"""
|
||||
date="2019-01-21T19:18:51Z"
|
||||
content="""
|
||||
Anyway, see if [[!commit f5f059e288b5e618e5c7c86d1db4bfd23d8b39ad]] somehow
|
||||
fixed it..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 6"
|
||||
date="2019-01-21T19:22:52Z"
|
||||
content="""
|
||||
tried to build 7.20181211+git284-g6f0d98f2a - it seems that the failing one is different one now?
|
||||
|
||||
prop_parse_show_Config: FAIL
|
||||
*** Failed! Falsifiable (after 3 tests and 1 shrink):
|
||||
fromList [(\"\791992\",\"\")]
|
||||
Use --quickcheck-replay=133129 to reproduce.
|
||||
prop_upFrom_basics: OK (0.03s)
|
||||
...
|
||||
1 out of 291 tests failed (254.87s)
|
||||
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 7"""
|
||||
date="2019-01-21T20:30:52Z"
|
||||
content="""
|
||||
If that unrelated test case is the only failure, then this problem is
|
||||
fixed.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
Since git-annex version is not always sufficient to pin point the source of regression (e.g. see [the get -J issue](http://git-annex.branchable.com/bugs/multiple_ssh_prompts__44___and_thread_blocked_indefinitely_in_an___63____63____63___transaction/)), it would be really nice if the version of ghc (and thus all its libraries if I get it right) was included in the output of `git annex version`
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> Closing as I was blind -- there is `dependency versions` reported [[done]] --[[yarikoptic]]
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
As of 436f10771 (make CommandStart return a StartMessage, 2019-06-06),
|
||||
`find --json` no longer outputs json.
|
||||
|
||||
For example, with
|
||||
|
||||
[[!format sh """
|
||||
cd $(mktemp -dt gx-find-json-XXXXXXX)
|
||||
git init && git annex init
|
||||
echo one >one && git annex add one && git commit -mone
|
||||
git annex find --json -- one
|
||||
"""]]
|
||||
|
||||
the last command outputs
|
||||
|
||||
```
|
||||
one
|
||||
```
|
||||
|
||||
rather than
|
||||
|
||||
```
|
||||
{"bytesize":"4","mtime":"unknown","keyname":"2c8b08da5ce60398e1f19af0e5dccc744df274b826abe585eaba68c525434806","backend":"SHA256E","key":"SHA256E-s4--2c8b08da5ce60398e1f19af0e5dccc744df274b826abe585eaba68c525434806","humansize":"4 B","hashdirmixed":"0J/J1/","file":"one","hashdirlower":"171/8dc/"}
|
||||
```
|
||||
|
||||
436f10771 appears to touch a lot of things, so perhaps other commands
|
||||
show similar issues and `find` is just the first I noticed.
|
||||
|
||||
### On what operating system?
|
||||
|
||||
GNU/Linux, building git-annex with Guix
|
||||
|
||||
[[!meta author=kyle]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 1"
|
||||
date="2019-07-08T16:37:50Z"
|
||||
content="""
|
||||
Thanks for the quick fix.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,80 @@
|
|||
Originally reported [here](https://github.com/CONP-PCNO/conp-dataset/pull/14)
|
||||
|
||||
[[!format sh """
|
||||
$> wget https://datahub-khvul4ng.udes.genap.ca/ALL.chr10.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
|
||||
--2019-04-10 09:26:54-- https://datahub-khvul4ng.udes.genap.ca/ALL.chr10.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
|
||||
Resolving datahub-khvul4ng.udes.genap.ca (datahub-khvul4ng.udes.genap.ca)... 204.19.23.238
|
||||
Connecting to datahub-khvul4ng.udes.genap.ca (datahub-khvul4ng.udes.genap.ca)|204.19.23.238|:443... connected.
|
||||
HTTP request sent, awaiting response... 200 OK
|
||||
Length: 773788987 (738M) [application/octet-stream]
|
||||
Saving to: ‘ALL.chr10.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz’
|
||||
|
||||
hr10.phase3_shapeit2_mvncall_integr 1%[ ] 9.44M 2.02MB/s eta 6m 16s ^C
|
||||
# so wget works - I interrupted
|
||||
|
||||
$> git annex addurl -d https://datahub-khvul4ng.udes.genap.ca/ALL.chr10.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
|
||||
[2019-04-10 09:27:04.412248] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
|
||||
[2019-04-10 09:27:04.416582] process done ExitSuccess
|
||||
[2019-04-10 09:27:04.416692] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2019-04-10 09:27:04.420688] process done ExitSuccess
|
||||
[2019-04-10 09:27:04.420852] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..3db4e3c3bf941c1cff0936d4441366bd10c9b2f1","--pretty=%H","-n1"]
|
||||
[2019-04-10 09:27:04.424774] process done ExitSuccess
|
||||
[2019-04-10 09:27:04.425311] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
|
||||
[2019-04-10 09:27:04.42596] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
|
||||
addurl https://datahub-khvul4ng.udes.genap.ca/ALL.chr10.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz [2019-04-10 09:27:04.449013] Request {
|
||||
host = "datahub-khvul4ng.udes.genap.ca"
|
||||
port = 443
|
||||
secure = True
|
||||
requestHeaders = [("Accept-Encoding",""),("User-Agent","git-annex/7.20190322+git23-g9662cb332-1~ndallstack+1")]
|
||||
path = "/ALL.chr10.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz"
|
||||
queryString = ""
|
||||
method = "HEAD"
|
||||
proxy = Nothing
|
||||
rawBody = False
|
||||
redirectCount = 10
|
||||
responseTimeout = ResponseTimeoutDefault
|
||||
requestVersion = HTTP/1.1
|
||||
}
|
||||
|
||||
|
||||
[2019-04-10 09:27:04.711525] Request {
|
||||
host = "datahub-khvul4ng.udes.genap.ca"
|
||||
port = 443
|
||||
secure = True
|
||||
requestHeaders = [("Accept-Encoding","identity"),("User-Agent","git-annex/7.20190322+git23-g9662cb332-1~ndallstack+1")]
|
||||
path = "/ALL.chr10.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz"
|
||||
queryString = ""
|
||||
method = "GET"
|
||||
proxy = Nothing
|
||||
rawBody = False
|
||||
redirectCount = 10
|
||||
responseTimeout = ResponseTimeoutDefault
|
||||
requestVersion = HTTP/1.1
|
||||
}
|
||||
|
||||
download failed: InvalidHeader "preload"
|
||||
failed
|
||||
[2019-04-10 09:27:04.956626] process done ExitSuccess
|
||||
[2019-04-10 09:27:04.957293] process done ExitSuccess
|
||||
git-annex: addurl: 1 failed
|
||||
|
||||
$> git annex version
|
||||
git-annex version: 7.20190322+git23-g9662cb332-1~ndallstack+1
|
||||
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.17.1 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.7.0 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 5 7
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6
|
||||
local repository version: 5
|
||||
|
||||
"""]]
|
||||
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> closing as there are no code changes to git-annex that can fix this,
|
||||
> it has to be fixed in either http-client or the web server. [[done]]
|
||||
> --[[Joey]]
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2019-04-10T13:58:03Z"
|
||||
content="""
|
||||
curl -v shows the problem response
|
||||
|
||||
< Server: openresty/1.7.10.1
|
||||
< Date: Wed, 10 Apr 2019 13:57:10 GMT
|
||||
< Content-Type: application/octet-stream
|
||||
< Content-Length: 773788987
|
||||
< Connection: keep-alive
|
||||
< Last-Modified: Mon, 11 Mar 2019 16:35:02 GMT
|
||||
< ETag: "5c868e36-2e1f153b"
|
||||
< Accept-Ranges: bytes
|
||||
< Strict-Transport-Security: max-age=15768000; includeSubdomains;
|
||||
< preload
|
||||
< X-Frame-Options: DENY
|
||||
< X-Content-Type-Options: nosniff
|
||||
|
||||
It looks like the server is sending "preload" on its own line rather
|
||||
than being part of the Strict-Transport-Security header.
|
||||
|
||||
Just in case, I found another url that uses HSTS preload, and git-annex
|
||||
can download that:
|
||||
|
||||
joey@darkstar:~>curl -v -o foo https://www.torproject.org/ 2>&1 | grep preload
|
||||
< Strict-Transport-Security: max-age=15768000; preload
|
||||
joey@darkstar:~/tmp/c>git annex addurl https://www.torproject.org/
|
||||
addurl https://www.torproject.org/
|
||||
(to www.torproject.org_) ok
|
||||
|
||||
The web server is probably violating standard HTTP to some extent with that
|
||||
response. Of course, it's possible to parse the response less strictly and
|
||||
not fail on the malformed header. Still, fixing the web server is probably the
|
||||
fastest way to fix the immediate problem (as well as make HSTS preloading
|
||||
actually be used).
|
||||
|
||||
Issue filed on http-client, <https://github.com/snoyberg/http-client/issues/398>
|
||||
with a prospective patch.
|
||||
|
||||
I don't see any changes to git-annex that can help with the problem, so
|
||||
I'll close this bug report.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
### Please describe the problem.
|
||||
|
||||
[Originally blamed datalad](https://github.com/datalad/datalad/issues/3685#issuecomment-533075845) but apparently `git annex add --json` does not include error message in the json output. In our case it was due to permission issues.
|
||||
|
||||
|
||||
[[!format sh """
|
||||
$ git annex add more --json
|
||||
more: setFileMode: permission denied (Operation not permitted)
|
||||
{"command":"add","success":false,"file":"more"}
|
||||
git-annex: add: 1 failed
|
||||
"""]]
|
||||
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
[[!meta project=repronim]]
|
||||
|
||||
> If you use --json-error-messages it does work, at least in this
|
||||
> permissions error case. So nothing more needs to be [[done]]. --[[Joey]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="about --json-error-messages"
|
||||
date="2019-09-24T19:19:20Z"
|
||||
content="""
|
||||
From \"any program parsing stderr for json would somehow have to deal with this non-json that could be mixed up with it, which seems very hard\" at
|
||||
[another bug page](https://git-annex.branchable.com/bugs/git-annex-copy_--json-error-messages_does_not_write_error_messages_to_stderr_) , it sounds like parsing the output of `--json-error-messages` might not be reliable?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2019-09-24T21:01:48Z"
|
||||
content="""
|
||||
--json-error-messages does not go to stderr, so does not have that problem.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
### Please describe the problem.
|
||||
|
||||
Trying to add non youtube URL causes an error demanding to set `annex.security.allowed-http-addresses`, which is IMHO not warranted:
|
||||
|
||||
[[!format sh """
|
||||
$> mkdir /tmp/testrepo; cd /tmp/testrepo; git init; git annex init; git annex addurl --file=index.html https://www.cs.toronto.edu/\~kriz/index.html
|
||||
Initialized empty Git repository in /tmp/testrepo/.git/
|
||||
init ok
|
||||
(recording state in git...)
|
||||
addurl https://www.cs.toronto.edu/~kriz/index.html
|
||||
|
||||
This url is supported by youtube-dl, but youtube-dl could potentially access any address, and the configuration of annex.security.allowed-http-addresses does not allow that. Not using youtube-dl.
|
||||
failed
|
||||
git-annex: addurl: 1 failed
|
||||
|
||||
$> git annex version
|
||||
git-annex version: 7.20190219+git191-g2d6a364d4-1~ndall+1
|
||||
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 5 7
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6
|
||||
local repository version: 5
|
||||
"""]]
|
||||
|
||||
|
||||
The same happens if I remove `youtube-dl` from the system entirely!
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> due to using --file which bypasses the usual check that youtube-dl
|
||||
> can extract media. [[fixed|done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,100 @@
|
|||
### Please describe the problem.
|
||||
|
||||
As of a14f6ce75 (fix repo description setting bugs, 2019-05-23), `git annex init` without a description sets the description to a blank string. AFAICT from the description of that change and from the corresponding [bug report][0], that isn't an intentional change.
|
||||
|
||||
[0]: https://git-annex.branchable.com/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description/
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
```
|
||||
cd $(mktemp -dt gx-XXXXXXX)
|
||||
git init
|
||||
git annex init
|
||||
git annex info --json | jq
|
||||
```
|
||||
|
||||
On the parent of a14f6ce75, this shows
|
||||
|
||||
```
|
||||
Initialized empty Git repository in /tmp/gx-BLmU7TJ/.git/
|
||||
init ok
|
||||
(recording state in git...)
|
||||
{
|
||||
"local annex size": "0 bytes",
|
||||
"size of annexed files in working tree": "0 bytes",
|
||||
"command": "info",
|
||||
"untrusted repositories": [],
|
||||
"success": true,
|
||||
"semitrusted repositories": [
|
||||
{
|
||||
"here": false,
|
||||
"uuid": "00000000-0000-0000-0000-000000000001",
|
||||
"description": "web"
|
||||
},
|
||||
{
|
||||
"here": false,
|
||||
"uuid": "00000000-0000-0000-0000-000000000002",
|
||||
"description": "bittorrent"
|
||||
},
|
||||
{
|
||||
"here": true,
|
||||
"uuid": "d1ef2ae8-9604-4c33-8c3c-181f81c4348f",
|
||||
"description": "kyle@hylob:/tmp/gx-BLmU7TJ"
|
||||
}
|
||||
],
|
||||
"bloom filter size": "32 mebibytes (0% full)",
|
||||
"repository mode": "indirect",
|
||||
"backend usage": {},
|
||||
"transfers in progress": [],
|
||||
"local annex keys": 0,
|
||||
"available local disk space": "301.33 gigabytes (+1 megabyte reserved)",
|
||||
"annexed files in working tree": 0,
|
||||
"file": null,
|
||||
"trusted repositories": []
|
||||
}
|
||||
```
|
||||
|
||||
On a14f6ce75, this shows
|
||||
|
||||
```
|
||||
Initialized empty Git repository in /tmp/gx-pK345zF/.git/
|
||||
init ok
|
||||
{
|
||||
"local annex size": "0 bytes",
|
||||
"size of annexed files in working tree": "0 bytes",
|
||||
"command": "info",
|
||||
"untrusted repositories": [],
|
||||
"success": true,
|
||||
"semitrusted repositories": [
|
||||
{
|
||||
"here": false,
|
||||
"uuid": "00000000-0000-0000-0000-000000000001",
|
||||
"description": "web"
|
||||
},
|
||||
{
|
||||
"here": false,
|
||||
"uuid": "00000000-0000-0000-0000-000000000002",
|
||||
"description": "bittorrent"
|
||||
},
|
||||
{
|
||||
"here": true,
|
||||
"uuid": "00b8da69-41a5-4de8-9a6a-9ed991289f81",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"bloom filter size": "32 mebibytes (0% full)",
|
||||
"repository mode": "indirect",
|
||||
"backend usage": {},
|
||||
"transfers in progress": [],
|
||||
"local annex keys": 0,
|
||||
"available local disk space": "301.22 gigabytes (+1 megabyte reserved)",
|
||||
"annexed files in working tree": 0,
|
||||
"file": null,
|
||||
"trusted repositories": []
|
||||
}
|
||||
```
|
||||
|
||||
Note that the description for here is a blank string.
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
[[!meta author=kyle]]
|
||||
[[!tag projects/repronim]]
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="uuid.log is also not created"
|
||||
date="2019-06-05T15:18:30Z"
|
||||
content="""
|
||||
As of that same commit (a14f6ce75), running `git annex init` in a fresh repository no longer creates uuid.log:
|
||||
|
||||
```
|
||||
% cd $(mktemp -dt gx-XXXXXXX)
|
||||
% git init
|
||||
Initialized empty Git repository in /tmp/gx-kpnGSjg/.git/
|
||||
% git annex init
|
||||
init ok
|
||||
% git ls-tree -r git-annex ;# no output
|
||||
```
|
||||
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="Issue with description cache?"
|
||||
date="2019-06-17T21:11:14Z"
|
||||
content="""
|
||||
I took another look at this thinking that maybe I could spot something obvious in the condition added in a14f6ce75
|
||||
|
||||
```
|
||||
{- Avoid overwriting existing description with a default
|
||||
- description. -}
|
||||
whenM (pure (isJust mdescription) <||> not . M.member u <$> uuidDescMap) $
|
||||
describeUUID u =<< genDescription mdescription
|
||||
```
|
||||
|
||||
but that looks fine. Grepping around for uuidDescMap, I spotted a couple of places where the cache needed to be invalidated. Blindly applying that same approach I did
|
||||
|
||||
```
|
||||
diff --git a/Annex/Init.hs b/Annex/Init.hs
|
||||
index cb7f8905e..6303103df 100644
|
||||
--- a/Annex/Init.hs
|
||||
+++ b/Annex/Init.hs
|
||||
@@ -76,6 +76,7 @@ genDescription Nothing = do
|
||||
|
||||
initialize :: Maybe String -> Maybe RepoVersion -> Annex ()
|
||||
initialize mdescription mversion = checkCanInitialize $ do
|
||||
+ uuidDescMapLoad
|
||||
{- Has to come before any commits are made as the shared
|
||||
- clone heuristic expects no local objects. -}
|
||||
sharedclone <- checkSharedClone
|
||||
```
|
||||
|
||||
That fixes the problem on my end (i.e., `git annex init` sets the description to the default user@host:dir description) while still addressing the original issue fixed by a14f6ce75.
|
||||
|
||||
So it seems like it has something to do with the uuidDescMap cache, but I haven't been able to figure out what in particular.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2019-06-21T00:12:51Z"
|
||||
content="""
|
||||
Thanks Kyle, that does point at the problem. In uuidDescMapLoad
|
||||
it implicitly adds the UUID of the current repo, with an empty
|
||||
description if none is set. So, the added check of uuidDescMap
|
||||
always finds the UUID in it already. Your uuidDescMapLoad hack
|
||||
avoids this by loading the map before the current repo has a UUID.
|
||||
|
||||
Fixed more cleanly..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="thanks"
|
||||
date="2019-06-21T02:05:24Z"
|
||||
content="""
|
||||
Thank you for the explanation and fix!
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="bug fix release"
|
||||
date="2019-06-22T16:14:58Z"
|
||||
content="""
|
||||
@joeyh Thanks for fixing this. Would it be possible to make a new release on Hackage? I'd like to make a new [[install/conda]] release that incorporates the improvements in version `7.20190615`, but don't want to make a release that fails DataLad tests the default one. Thanks!
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 6"""
|
||||
date="2019-06-25T17:14:03Z"
|
||||
content="""
|
||||
Release soon.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
### Please describe the problem.
|
||||
|
||||
Initially reported in [heudiconv](https://github.com/nipy/heudiconv/pull/259). I am not yet sure what is going on, but git-annex (tried "bleeding edge" from a few days back too) seems to believe being unable to get .git/config file, while I do not see any request being logged on the server side, and `curl` gets it just ok.
|
||||
|
||||
To reproduce you can use ```docker run -it --rm --entrypoint bash nipy/heudiconv:latest -c "cd /tmp/ && rm -rf /tmp/MEEPI3; git clone http://datasets-tests.datalad.org/dicoms/velasco/MEEPI/.git MEEPI3 && cd MEEPI3 && git annex info --debug; echo -e '\n\nCURL:'; curl http://datasets-tests.datalad.org/dicoms/velasco/MEEPI/.git/config"```
|
||||
|
||||
|
||||
<details>
|
||||
<summary>A sample docker run</summary>
|
||||
|
||||
[[!format sh """
|
||||
docker run -it --rm --entrypoint bash nipy/heudiconv:latest -c "cd /tmp/ && rm -rf /tmp/MEEPI3; git clone http://datasets-tests.datalad.org/dicoms/velasco/MEEPI/.git MEEPI3 && cd MEEPI3 && git annex info --debug; echo -e '\n\nCURL:'; curl http://datasets-tests.datalad.org/dicoms/velasco/MEEPI/.git/config"
|
||||
...
|
||||
[2018-11-12 17:58:02.246908731] Request {
|
||||
host = "datasets-tests.datalad.org"
|
||||
port = 80
|
||||
secure = False
|
||||
requestHeaders = [("Range","bytes=0-"),("Accept-Encoding","identity"),("User-Agent","git-annex/6.20181011+git124-g94aa0e2f6-1~ndall+1")]
|
||||
path = "/dicoms/velasco/MEEPI/.git/config"
|
||||
queryString = ""
|
||||
method = "GET"
|
||||
proxy = Nothing
|
||||
rawBody = False
|
||||
redirectCount = 10
|
||||
responseTimeout = ResponseTimeoutDefault
|
||||
requestVersion = HTTP/1.1
|
||||
}
|
||||
|
||||
|
||||
Remote origin not usable by git-annex; setting annex-ignore
|
||||
[2018-11-12 17:58:02.247631556] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","config","remote.origin.annex-ignore","true"]
|
||||
[2018-11-12 17:58:02.249993683] process done ExitSuccess
|
||||
[2018-11-12 17:58:02.250056119] read: git ["config","--null","--list"]
|
||||
[2018-11-12 17:58:02.252212701] process done ExitSuccess
|
||||
repository mode: indirect
|
||||
trusted repositories: 0
|
||||
semitrusted repositories: 6
|
||||
00000000-0000-0000-0000-000000000001 -- web
|
||||
00000000-0000-0000-0000-000000000002 -- bittorrent
|
||||
41bc6812-269a-47af-a7c9-ba2d30e55642 -- yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/dicoms/velasco/MEEPI
|
||||
757a81e7-f905-4796-9941-a2772ec190b6 -- yoh@falkor:/srv/datasets.datalad.org/www/dicoms/velasco/MEEPI
|
||||
c04eb54b-4b4e-5755-8436-866b043170fa -- datalad-archives
|
||||
ffa6417c-5511-4b43-a67a-036dd57ab974 -- root@d9efe0626d93:/tmp/MEEPI3 [here]
|
||||
untrusted repositories: 0
|
||||
transfers in progress: none
|
||||
available local disk space: 12.08 gigabytes (+1 megabyte reserved)
|
||||
local annex keys: 0
|
||||
local annex size: 0 bytes
|
||||
annexed files in working tree: [2018-11-12 17:58:02.253291443] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","--others","-z","--","."]
|
||||
[2018-11-12 17:58:02.255650007] process done ExitSuccess
|
||||
3
|
||||
size of annexed files in working tree: 5.87 megabytes
|
||||
bloom filter size: 32 mebibytes (0% full)
|
||||
backend usage:
|
||||
MD5E: 3
|
||||
[2018-11-12 17:58:02.256351183] process done ExitSuccess
|
||||
[2018-11-12 17:58:02.256558475] process done ExitSuccess
|
||||
[2018-11-12 17:58:02.256945107] process done ExitSuccess
|
||||
|
||||
|
||||
CURL:
|
||||
[core]
|
||||
repositoryformatversion = 0
|
||||
filemode = true
|
||||
bare = false
|
||||
logallrefupdates = true
|
||||
sharedrepository = 2
|
||||
[receive]
|
||||
denyNonFastforwards = true
|
||||
denyCurrentBranch = updateInstead
|
||||
[annex]
|
||||
uuid = 757a81e7-f905-4796-9941-a2772ec190b6
|
||||
version = 5
|
||||
|
||||
|
||||
$> docker run -it --rm --entrypoint bash nipy/heudiconv:latest -c "git annex version | head -n1"
|
||||
git-annex version: 6.20181011+git124-g94aa0e2f6-1~ndall+1
|
||||
"""]]
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[done]]; git-annex displays a good error message and the dependency is
|
||||
> there. --[[Joey]]
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2018-11-12T18:42:53Z"
|
||||
content="""
|
||||
It does seem to work just fine with current git-annex outside
|
||||
docker.
|
||||
|
||||
Given the insecurity-to-weight ratio of docker and the number of hoops
|
||||
I would have to jump through to run that docker command,
|
||||
it may be several days until I have time, bandwidth, and power to try
|
||||
it myself.. If you want to show the debug output, that generally doesn't
|
||||
hurt.
|
||||
|
||||
(Please include the git-annex version information in all bug reports.)
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,351 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 2"
|
||||
date="2018-11-12T19:27:48Z"
|
||||
content="""
|
||||
Just want to check first -- you didn't miss my fancy foldable \"A sample docker run\" in the original submission? (has annex version at the bottom), did you?
|
||||
|
||||
and yes -- works fine outside of docker on the same box (well - my laptop).
|
||||
|
||||
FWIW, you are welcome to try it on smaug, you are a part of `docker` group there. For some reason docker is slowish there.
|
||||
|
||||
Interestingly, the version of that image being different on smaug, output might be more informative (?):
|
||||
|
||||
[[!format sh \"\"\"
|
||||
[2018-11-12 19:09:33.816486231] process done ExitSuccess
|
||||
download failed: ConnectionFailure Network.BSD.getProtocolByName: does not exist (no such protocol name: tcp)
|
||||
|
||||
Remote origin not usable by git-annex; setting annex-ignore
|
||||
\"\"\"]]
|
||||
|
||||
And here is (folded) full output of that run:
|
||||
|
||||
<details>
|
||||
<summary>Docker run with older git annex 6.20180626+gitg12cd64369-1~ndall+1</summary>
|
||||
|
||||
[[!format sh \"\"\"
|
||||
$> docker run -it --rm --entrypoint bash nipy/heudiconv:annex-6.20180626 -c \"cd /tmp/ && rm -rf /tmp/MEEPI3; git clone http://datasets-tests.datalad.org/dicoms/velasco/MEEPI/.git MEEPI3 && cd MEEPI3 && git annex info --debug; echo -e '\n\nCURL:'; curl http://datasets-tests.datalad.org/dicoms/velasco/MEEPI/.git/config; git annex version\"
|
||||
Cloning into 'MEEPI3'...
|
||||
remote: Counting objects: 101, done.
|
||||
remote: Compressing objects: 100% (82/82), done.
|
||||
remote: Total 101 (delta 35), reused 0 (delta 0)
|
||||
Receiving objects: 100% (101/101), 11.70 KiB | 0 bytes/s, done.
|
||||
Resolving deltas: 100% (35/35), done.
|
||||
[2018-11-12 19:09:33.497644702] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
|
||||
[2018-11-12 19:09:33.502984282] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.503282134] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
|
||||
[2018-11-12 19:09:33.508604943] process done ExitFailure 1
|
||||
[2018-11-12 19:09:33.508770986] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--verify\",\"-q\",\"origin/git-annex\"]
|
||||
[2018-11-12 19:09:33.512907558] process done ExitFailure 1
|
||||
[2018-11-12 19:09:33.513545552] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"write-tree\"]
|
||||
[2018-11-12 19:09:33.678146279] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.678410423] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"commit-tree\",\"4b825dc642cb6eb9a060e54bf8d69288fbee4904\",\"--no-gpg-sign\"]
|
||||
|
||||
*** Please tell me who you are.
|
||||
|
||||
Run
|
||||
|
||||
git config --global user.email \"you@example.com\"
|
||||
git config --global user.name \"Your Name\"
|
||||
|
||||
to set your account's default identity.
|
||||
Omit --global to set the identity only in this repository.
|
||||
|
||||
fatal: unable to auto-detect email address (got 'root@5d3e0680b6fc.(none)')
|
||||
[2018-11-12 19:09:33.684364456] process done ExitFailure 128
|
||||
[2018-11-12 19:09:33.684636234] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"user.name\",\"root\"]
|
||||
[2018-11-12 19:09:33.68955458] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.689684113] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2018-11-12 19:09:33.693793841] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.693980718] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"user.email\",\"root\"]
|
||||
[2018-11-12 19:09:33.697545036] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.697671055] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2018-11-12 19:09:33.701348909] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.701520785] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
|
||||
[2018-11-12 19:09:33.704126701] process done ExitFailure 1
|
||||
[2018-11-12 19:09:33.704281927] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--verify\",\"-q\",\"origin/git-annex\"]
|
||||
[2018-11-12 19:09:33.708300009] process done ExitFailure 1
|
||||
[2018-11-12 19:09:33.708510969] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"write-tree\"]
|
||||
[2018-11-12 19:09:33.712223058] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.712455349] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"commit-tree\",\"4b825dc642cb6eb9a060e54bf8d69288fbee4904\",\"--no-gpg-sign\"]
|
||||
[2018-11-12 19:09:33.717700724] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.717878364] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-ref\",\"refs/heads/git-annex\",\"b6d1fa4a9fd6e1e2b92cc4472374ff788f4ffe82\"]
|
||||
[2018-11-12 19:09:33.72191105] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.72266107] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"annex.uuid\",\"f7e33e3d-b3aa-473a-9fa3-213ef312c646\"]
|
||||
[2018-11-12 19:09:33.726190775] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.726346741] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2018-11-12 19:09:33.729706319] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.73202387] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
|
||||
[2018-11-12 19:09:33.73679675] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.73698363] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
|
||||
[2018-11-12 19:09:33.74067435] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.741001325] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..b6d1fa4a9fd6e1e2b92cc4472374ff788f4ffe82\",\"--pretty=%H\",\"-n1\"]
|
||||
[2018-11-12 19:09:33.745857469] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.74605887] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..04619d91febf86de056a8d091a5f06d464a01c04\",\"--pretty=%H\",\"-n1\"]
|
||||
[2018-11-12 19:09:33.751124219] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.751630501] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
|
||||
[2018-11-12 19:09:33.752216628] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
|
||||
(merging origin/git-annex into git-annex...)
|
||||
[2018-11-12 19:09:33.755344426] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"]
|
||||
[2018-11-12 19:09:33.756221301] feed: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-index\",\"-z\",\"--index-info\"]
|
||||
[2018-11-12 19:09:33.756880319] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"diff-index\",\"--raw\",\"-z\",\"-r\",\"--no-renames\",\"-l0\",\"--cached\",\"04619d91febf86de056a8d091a5f06d464a01c04\",\"--\"]
|
||||
[2018-11-12 19:09:33.7622414] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.763122925] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.763684956] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"04619d91febf86de056a8d091a5f06d464a01c04..refs/heads/git-annex\",\"--pretty=%H\",\"-n1\"]
|
||||
[2018-11-12 19:09:33.768141316] process done ExitSuccess
|
||||
(recording state in git...)
|
||||
[2018-11-12 19:09:33.768438348] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"write-tree\"]
|
||||
[2018-11-12 19:09:33.77206771] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.772262959] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"commit-tree\",\"f21c20ffe799304525ca1aeace21935c47926873\",\"--no-gpg-sign\",\"-p\",\"refs/heads/git-annex\",\"-p\",\"04619d91febf86de056a8d091a5f06d464a01c04\"]
|
||||
[2018-11-12 19:09:33.776274369] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.77643436] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-ref\",\"refs/heads/git-annex\",\"d076e4f8669ea8f8ddce8685e85af824a9697100\"]
|
||||
[2018-11-12 19:09:33.781233336] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.783050992] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"annex.version\",\"5\"]
|
||||
[2018-11-12 19:09:33.787781391] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.78793579] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2018-11-12 19:09:33.790756389] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.790954849] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"]
|
||||
[2018-11-12 19:09:33.795171566] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.795368409] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"]
|
||||
[2018-11-12 19:09:33.800060961] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.800259221] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"]
|
||||
[2018-11-12 19:09:33.803306615] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.803473861] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/master\"]
|
||||
[2018-11-12 19:09:33.807579032] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.807783985] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"checkout\",\"-q\",\"-B\",\"master\"]
|
||||
[2018-11-12 19:09:33.814124796] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.815125257] read: uname [\"-n\"]
|
||||
[2018-11-12 19:09:33.816486231] process done ExitSuccess
|
||||
download failed: ConnectionFailure Network.BSD.getProtocolByName: does not exist (no such protocol name: tcp)
|
||||
|
||||
Remote origin not usable by git-annex; setting annex-ignore
|
||||
[2018-11-12 19:09:33.94194622] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"remote.origin.annex-ignore\",\"true\"]
|
||||
[2018-11-12 19:09:33.947114461] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.9474738] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2018-11-12 19:09:33.951509986] process done ExitSuccess
|
||||
repository mode: indirect
|
||||
trusted repositories: 0
|
||||
semitrusted repositories: 6
|
||||
00000000-0000-0000-0000-000000000001 -- web
|
||||
00000000-0000-0000-0000-000000000002 -- bittorrent
|
||||
41bc6812-269a-47af-a7c9-ba2d30e55642 -- yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/dicoms/velasco/MEEPI
|
||||
757a81e7-f905-4796-9941-a2772ec190b6 -- yoh@falkor:/srv/datasets.datalad.org/www/dicoms/velasco/MEEPI
|
||||
c04eb54b-4b4e-5755-8436-866b043170fa -- datalad-archives
|
||||
f7e33e3d-b3aa-473a-9fa3-213ef312c646 -- root@5d3e0680b6fc:/tmp/MEEPI3 [here]
|
||||
untrusted repositories: 0
|
||||
transfers in progress: none
|
||||
available local disk space: 34.84 terabytes (+1 megabyte reserved)
|
||||
local annex keys: 0
|
||||
local annex size: 0 bytes
|
||||
annexed files in working tree: [2018-11-12 19:09:33.953212804] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"--cached\",\"--others\",\"-z\",\"--\",\".\"]
|
||||
[2018-11-12 19:09:33.957668639] process done ExitSuccess
|
||||
3
|
||||
size of annexed files in working tree: 5.87 megabytes
|
||||
bloom filter size: 32 mebibytes (0% full)
|
||||
backend usage:
|
||||
MD5E: 3
|
||||
[2018-11-12 19:09:33.958614844] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.95905526] process done ExitSuccess
|
||||
[2018-11-12 19:09:33.959474605] process done ExitSuccess
|
||||
|
||||
|
||||
CURL:
|
||||
[core]
|
||||
repositoryformatversion = 0
|
||||
filemode = true
|
||||
bare = false
|
||||
logallrefupdates = true
|
||||
sharedrepository = 2
|
||||
[receive]
|
||||
denyNonFastforwards = true
|
||||
denyCurrentBranch = updateInstead
|
||||
[annex]
|
||||
uuid = 757a81e7-f905-4796-9941-a2772ec190b6
|
||||
version = 5
|
||||
git-annex version: 6.20180626+gitg12cd64369-1~ndall+1
|
||||
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.16 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.7.1 persistent-sqlite-2.6.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
local repository version: 5
|
||||
|
||||
\"\"\"]]
|
||||
</details>
|
||||
|
||||
Having pulled the `nipy/heudiconv:latest`, original reported behavior reproduces
|
||||
|
||||
<details>
|
||||
<summary>Docker run with newer git annex </summary>
|
||||
|
||||
[[!format sh \"\"\"
|
||||
$> docker run -it --rm --entrypoint bash nipy/heudiconv:latest -c \"cd /tmp/ && rm -rf /tmp/MEEPI3; git clone http://datasets-tests.datalad.org/dicoms/velasco/MEEPI/.git MEEPI3 && cd MEEPI3 && git annex info --debug; echo -e '\n\nCURL:'; curl http://datasets-tests.datalad.org/dicoms/velasco/MEEPI/.git/config; git annex version\"
|
||||
Cloning into 'MEEPI3'...
|
||||
remote: Counting objects: 101, done.
|
||||
remote: Compressing objects: 100% (82/82), done.
|
||||
remote: Total 101 (delta 35), reused 0 (delta 0)
|
||||
Receiving objects: 100% (101/101), 11.70 KiB | 0 bytes/s, done.
|
||||
Resolving deltas: 100% (35/35), done.
|
||||
[2018-11-12 19:16:30.168203115] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
|
||||
[2018-11-12 19:16:30.173678013] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.174017869] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
|
||||
[2018-11-12 19:16:30.17921506] process done ExitFailure 1
|
||||
[2018-11-12 19:16:30.179386236] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--verify\",\"-q\",\"origin/git-annex\"]
|
||||
[2018-11-12 19:16:30.183673031] process done ExitFailure 1
|
||||
[2018-11-12 19:16:30.184420434] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"write-tree\"]
|
||||
[2018-11-12 19:16:30.189923909] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.190097446] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"commit-tree\",\"4b825dc642cb6eb9a060e54bf8d69288fbee4904\",\"--no-gpg-sign\"]
|
||||
|
||||
*** Please tell me who you are.
|
||||
|
||||
Run
|
||||
|
||||
git config --global user.email \"you@example.com\"
|
||||
git config --global user.name \"Your Name\"
|
||||
|
||||
to set your account's default identity.
|
||||
Omit --global to set the identity only in this repository.
|
||||
|
||||
fatal: unable to auto-detect email address (got 'root@b88726a4382b.(none)')
|
||||
[2018-11-12 19:16:30.194707379] process done ExitFailure 128
|
||||
[2018-11-12 19:16:30.194989101] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"user.name\",\"root\"]
|
||||
[2018-11-12 19:16:30.199736477] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.199882502] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2018-11-12 19:16:30.204311006] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.204467196] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"user.email\",\"root\"]
|
||||
[2018-11-12 19:16:30.209146004] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.2092902] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2018-11-12 19:16:30.21315675] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.213311976] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
|
||||
[2018-11-12 19:16:30.217993425] process done ExitFailure 1
|
||||
[2018-11-12 19:16:30.218146321] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--verify\",\"-q\",\"origin/git-annex\"]
|
||||
[2018-11-12 19:16:30.22226559] process done ExitFailure 1
|
||||
[2018-11-12 19:16:30.596337472] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"write-tree\"]
|
||||
[2018-11-12 19:16:30.601348306] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.601517345] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"commit-tree\",\"4b825dc642cb6eb9a060e54bf8d69288fbee4904\",\"--no-gpg-sign\"]
|
||||
[2018-11-12 19:16:30.607398795] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.607513593] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-ref\",\"refs/heads/git-annex\",\"44bd175787872acfd21593ef7f75224da84f728e\"]
|
||||
[2018-11-12 19:16:30.612739367] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.613560258] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"annex.uuid\",\"d886638a-7427-46c1-911d-2febef7fc831\"]
|
||||
[2018-11-12 19:16:30.617982319] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.618139741] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2018-11-12 19:16:30.622404815] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.624792574] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
|
||||
[2018-11-12 19:16:30.629506669] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.629670469] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
|
||||
[2018-11-12 19:16:30.633873559] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.634159315] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..44bd175787872acfd21593ef7f75224da84f728e\",\"--pretty=%H\",\"-n1\"]
|
||||
[2018-11-12 19:16:30.638868994] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.63901631] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..04619d91febf86de056a8d091a5f06d464a01c04\",\"--pretty=%H\",\"-n1\"]
|
||||
[2018-11-12 19:16:30.644133907] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.644558474] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
|
||||
[2018-11-12 19:16:30.645069766] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
|
||||
(merging origin/git-annex into git-annex...)
|
||||
[2018-11-12 19:16:30.648787618] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"]
|
||||
[2018-11-12 19:16:30.649353618] feed: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-index\",\"-z\",\"--index-info\"]
|
||||
[2018-11-12 19:16:30.650115411] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"diff-index\",\"--raw\",\"-z\",\"-r\",\"--no-renames\",\"-l0\",\"--cached\",\"04619d91febf86de056a8d091a5f06d464a01c04\",\"--\"]
|
||||
[2018-11-12 19:16:30.654663147] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.655618594] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.656176449] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"04619d91febf86de056a8d091a5f06d464a01c04..refs/heads/git-annex\",\"--pretty=%H\",\"-n1\"]
|
||||
[2018-11-12 19:16:30.661120546] process done ExitSuccess
|
||||
(recording state in git...)
|
||||
[2018-11-12 19:16:30.661335622] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"write-tree\"]
|
||||
[2018-11-12 19:16:30.666151568] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.666446029] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"commit-tree\",\"f21c20ffe799304525ca1aeace21935c47926873\",\"--no-gpg-sign\",\"-p\",\"refs/heads/git-annex\",\"-p\",\"04619d91febf86de056a8d091a5f06d464a01c04\"]
|
||||
[2018-11-12 19:16:30.67115132] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.671318437] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-ref\",\"refs/heads/git-annex\",\"e9b0bb4be7614f8dab19660fc8b3c00fec9c1180\"]
|
||||
[2018-11-12 19:16:30.676528891] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.678481925] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"annex.version\",\"5\"]
|
||||
[2018-11-12 19:16:30.682563316] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.682715955] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2018-11-12 19:16:30.686466067] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.68663411] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"]
|
||||
[2018-11-12 19:16:30.690971999] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.691134218] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"]
|
||||
[2018-11-12 19:16:30.695344845] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.695518841] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"]
|
||||
[2018-11-12 19:16:30.699919641] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.700082394] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/master\"]
|
||||
[2018-11-12 19:16:30.704659184] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.704830677] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"checkout\",\"-q\",\"-B\",\"master\"]
|
||||
[2018-11-12 19:16:30.709278877] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.710297742] read: uname [\"-n\"]
|
||||
[2018-11-12 19:16:30.730187288] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.964899187] Request {
|
||||
host = \"datasets-tests.datalad.org\"
|
||||
port = 80
|
||||
secure = False
|
||||
requestHeaders = [(\"Range\",\"bytes=0-\"),(\"Accept-Encoding\",\"identity\"),(\"User-Agent\",\"git-annex/6.20181011+git124-g94aa0e2f6-1~ndall+1\")]
|
||||
path = \"/dicoms/velasco/MEEPI/.git/config\"
|
||||
queryString = \"\"
|
||||
method = \"GET\"
|
||||
proxy = Nothing
|
||||
rawBody = False
|
||||
redirectCount = 10
|
||||
responseTimeout = ResponseTimeoutDefault
|
||||
requestVersion = HTTP/1.1
|
||||
}
|
||||
|
||||
|
||||
Remote origin not usable by git-annex; setting annex-ignore
|
||||
[2018-11-12 19:16:30.965795741] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"remote.origin.annex-ignore\",\"true\"]
|
||||
[2018-11-12 19:16:30.9707034] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.970879863] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2018-11-12 19:16:30.975621212] process done ExitSuccess
|
||||
repository mode: indirect
|
||||
trusted repositories: 0
|
||||
semitrusted repositories: 6
|
||||
00000000-0000-0000-0000-000000000001 -- web
|
||||
00000000-0000-0000-0000-000000000002 -- bittorrent
|
||||
41bc6812-269a-47af-a7c9-ba2d30e55642 -- yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/dicoms/velasco/MEEPI
|
||||
757a81e7-f905-4796-9941-a2772ec190b6 -- yoh@falkor:/srv/datasets.datalad.org/www/dicoms/velasco/MEEPI
|
||||
c04eb54b-4b4e-5755-8436-866b043170fa -- datalad-archives
|
||||
d886638a-7427-46c1-911d-2febef7fc831 -- root@b88726a4382b:/tmp/MEEPI3 [here]
|
||||
untrusted repositories: 0
|
||||
transfers in progress: none
|
||||
available local disk space: 34.83 terabytes (+1 megabyte reserved)
|
||||
local annex keys: 0
|
||||
local annex size: 0 bytes
|
||||
annexed files in working tree: [2018-11-12 19:16:30.978614061] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"--cached\",\"--others\",\"-z\",\"--\",\".\"]
|
||||
[2018-11-12 19:16:30.982161106] process done ExitSuccess
|
||||
3
|
||||
size of annexed files in working tree: 5.87 megabytes
|
||||
bloom filter size: 32 mebibytes (0% full)
|
||||
backend usage:
|
||||
MD5E: 3
|
||||
[2018-11-12 19:16:30.983219045] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.984207325] process done ExitSuccess
|
||||
[2018-11-12 19:16:30.984715431] process done ExitSuccess
|
||||
|
||||
|
||||
CURL:
|
||||
[core]
|
||||
repositoryformatversion = 0
|
||||
filemode = true
|
||||
bare = false
|
||||
logallrefupdates = true
|
||||
sharedrepository = 2
|
||||
[receive]
|
||||
denyNonFastforwards = true
|
||||
denyCurrentBranch = updateInstead
|
||||
[annex]
|
||||
uuid = 757a81e7-f905-4796-9941-a2772ec190b6
|
||||
version = 5
|
||||
git-annex version: 6.20181011+git124-g94aa0e2f6-1~ndall+1
|
||||
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.19 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.2 feed-1.0.0.0 ghc-8.2.2 http-client-0.5.12 persistent-sqlite-2.8.1.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
local repository version: 5
|
||||
\"\"\"]]
|
||||
</details>
|
||||
|
||||
Reflecting on the discovery of that \"no such protocol name\", googled [github hit](https://github.com/fpco/haskell-scratch/issues/2#issuecomment-285923941), added `apt-get install -y libnss3 libnss-lwres libnss-mdns` to the new version of the docker image - that didn't help.
|
||||
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 3"
|
||||
date="2018-11-12T19:34:00Z"
|
||||
content="""
|
||||
even though not that -- probably due to `--no-install-recommends` while installing git-annex-standalone... may be at least core needed libs needed for annex operation could be added to Depends for the git-annex debian pkg?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="yeap -- it is netbase which is needed"
|
||||
date="2018-11-13T04:20:28Z"
|
||||
content="""
|
||||
spotted that netbase is among recommends thus is not installed while `apt-get install --no-install-recommends git-annex-standalone` . Installed and issue is gone.
|
||||
|
||||
- I feel like we had that issue/discussion before, but don't you think it would be valuable to make `netbase` a Depends not just Recommends?
|
||||
|
||||
- Also it would be nice if there was a more logical error, info or at least debug message whenever annex fails to perform network operations, instead of just marking remote as \"non-annex\".
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 5"""
|
||||
date="2018-11-13T17:30:33Z"
|
||||
content="""
|
||||
netbase was already added to depends in version 6.20180926.
|
||||
|
||||
Yes, I missed the folded data, the UI provides almost no clue that it's
|
||||
an interactive element.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="my bad"
|
||||
date="2018-11-15T12:34:58Z"
|
||||
content="""
|
||||
eh, and when you added it, I incorrectly adjusted the standalone patch in 042e20d895e022f283d7fb210e61bc19a77b0c39 by not including netbase into Depends.
|
||||
I have emailed you the patch, and I think this issue could be closed with it
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
### Please describe the problem.
|
||||
|
||||
It is probably not a regression (since happens with now elderly git-annex in debian sid), it could be something either changed on debian sid or on datalad side (originally ran into it while trying to build 0.11.7 package for debian sid) to make test directory name even more "tricky" than before.
|
||||
|
||||
Summary: `git annex init` fails even though `git init` (and other commands, not shown here) works just fine:
|
||||
|
||||
|
||||
[[!format sh """
|
||||
root@smaug:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1# pwd
|
||||
/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1
|
||||
|
||||
# git init was ran before already
|
||||
root@smaug:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1# git init
|
||||
Reinitialized existing Git repository in /tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1/.git/
|
||||
|
||||
root@smaug:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1# git annex init
|
||||
init fatal: Unable to create '/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&c `| 1/.git/annex/index.lock': No such file or directory
|
||||
fatal: Unable to create '/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&c `| 1/.git/annex/index.lock': No such file or directory
|
||||
|
||||
git-annex: failed to read sha from git write-tree
|
||||
CallStack (from HasCallStack):
|
||||
error, called at ./Git/Sha.hs:18:15 in main:Git.Sha
|
||||
failed
|
||||
git-annex: init: 1 failed
|
||||
|
||||
root@smaug:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1# git annex version
|
||||
git-annex version: 7.20190129
|
||||
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 5 7
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6
|
||||
|
||||
"""]]
|
||||
|
||||
so above version is the one in Debian proper (apparently quite dated!). I have tried with 7.20190819+git2-g908476a9b-1~ndall+1 and 7.20190819+git60-gcdb679818-1~ndall+1
|
||||
|
||||
You can see that `git-annex` got unicode portion of the directory name stripped (reports ``` "';a&b&c `| 1``` instead of ``` "';a&b&cΔЙקم๗あ `| 1```).
|
||||
|
||||
My wild guess is that full path (instead of a relative path) is used somewhere, encoded/decoded with losses, and then kaboom comes.
|
||||
|
||||
<details>
|
||||
<summary>locale is just C</summary>
|
||||
|
||||
[[!format sh """
|
||||
root@smaug:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/ "';a&b&cΔЙקم๗あ `| 1# locale
|
||||
LANG=C
|
||||
LANGUAGE=
|
||||
LC_CTYPE="C"
|
||||
LC_NUMERIC="C"
|
||||
LC_TIME="C"
|
||||
LC_COLLATE="C"
|
||||
LC_MONETARY="C"
|
||||
LC_MESSAGES="C"
|
||||
LC_PAPER="C"
|
||||
LC_NAME="C"
|
||||
LC_ADDRESS="C"
|
||||
LC_TELEPHONE="C"
|
||||
LC_MEASUREMENT="C"
|
||||
LC_IDENTIFICATION="C"
|
||||
LC_ALL=C
|
||||
"""]]
|
||||
</details>
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> fixed in the haskell process library. [[done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="archaeological expedition summary"
|
||||
date="2019-09-09T12:53:16Z"
|
||||
content="""
|
||||
just notes on the mystery on why didn't trigger before:
|
||||
- example shown above was triggered by a `test_create_with_obscure_name` which was added only recently in 0.11.7~13^2~2
|
||||
- (here I was going to provide analysis for other tests, but ran out of time... ;))
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2019-09-09T13:04:29Z"
|
||||
content="""
|
||||
To reproduce this, set LANG=C. In a unicode locale, it does not have the
|
||||
problem.
|
||||
|
||||
"קم๗あ" is a sufficient amount of unicode to cause the failure (so is "¡").
|
||||
The ascii punctuation is not involved, eg it works in a directory named
|
||||
"';a&b&c `| 1
|
||||
|
||||
joey@darkstar:/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg/קم๗あ>LANG=C git-annex init
|
||||
init fatal: Unable to create '/tmp/datalad_temp_test_create_with_obscure_name18qdmqwg//.git/annex/index.lock': No such file or directory
|
||||
|
||||
Notice that all the unicode has been stripped out of the directory name
|
||||
somehow.
|
||||
|
||||
This is not happening to git-annex generally; .git/annex/ get created and
|
||||
populated with several files; it's the call to git update-index to create the
|
||||
annex index file that is failing.
|
||||
|
||||
Dumping the env that git-annex sets when running that command, it includes eg
|
||||
`("GIT_INDEX_FILE","/tmp/\56514\56481/.git/annex/index")`; the "¡"
|
||||
has been decomposed into what I think are two unicode surrigates; that's
|
||||
produced when getting the working directory.
|
||||
|
||||
joey@darkstar:/tmp/¡>LANG=C ghci
|
||||
GHCi, version 8.6.5: http://www.haskell.org/ghc/ :? for help
|
||||
Loaded GHCi configuration from /home/joey/.etc/.ghci
|
||||
ghci> import System.Posix.Directory
|
||||
ghci> getWorkingDirectory
|
||||
"/tmp/\56514\56481"
|
||||
|
||||
This shell command does not exhibit the problem:
|
||||
|
||||
printf '100644 8a1218a1024a212bb3db30becd860315f9f3ac52 1\tfrotz' | GIT_INDEX_FILE=`pwd`/.git/annex/index LANG=C git update-index --index-info
|
||||
|
||||
So, the problem is probably in the process library's passing
|
||||
of unicode environment variables to exec. I've verified the problem in ghci,
|
||||
running createProcess with this:
|
||||
|
||||
CreateProcess {cmdspec = RawCommand "git" ["update-index","--index-info"], cwd = Nothing, env = Just [("GIT_INDEX_FILE","/tmp/\56514\56481/.git/annex/index")], std_in = Inherit, std_out = Inherit, std_err = Inherit, close_fds = False, create_group = False, delegate_ctlc = False, detach_console = False, create_new_console = False, new_session = False, child_group = Nothing, child_user = Nothing, use_process_jobs = False}
|
||||
|
||||
This bug needs to be forwarded to process.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2019-09-09T18:41:05Z"
|
||||
content="""
|
||||
<https://github.com/haskell/process/issues/152> forwarded with a
|
||||
prospective patch.
|
||||
|
||||
----
|
||||
|
||||
Normally git-annex tries to use relative paths, and if it used a relative
|
||||
path to .git/annex/index, it would avoid this problem. Except perhaps
|
||||
if `GIT_DIR` pointed to a different directory with unicode in its path.
|
||||
A relative `GIT_INDEX_FILE` would not suffice to close this bug, but would
|
||||
fix your test suite. But, git has some very strange behavior with a relative
|
||||
`GIT_INDEX_FILE`, it's not necessarily treated as relative to pwd.
|
||||
So I'd rather not open that can of worms.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""fixed in process"""
|
||||
date="2019-09-11T17:06:53Z"
|
||||
content="""
|
||||
I've gotten the process library fixed, the version will probably be
|
||||
process-1.6.5.2.
|
||||
|
||||
Doesn't seem to make sense to make git-annex require the fixed version,
|
||||
just use it when available. So closing this issue.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 5"
|
||||
date="2019-09-11T18:03:11Z"
|
||||
content="""
|
||||
GREAT! thank you! I should not forget to try backporting it after it gets released for buster so our -standalone package uses it.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 6"""
|
||||
date="2019-09-11T19:50:43Z"
|
||||
content="""
|
||||
Backporting it would be trivial, it's a single word change.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
[[!meta title="double rsync run and test failure"]]
|
||||
|
||||
might be a flaky test/operation, happened while building in i386 buster environment:
|
||||
|
||||
[[!format sh """
|
||||
neurodebian@smaug ..7.20190730+git103-g070fbd693-1~ndall+1 % grep -B20 -A1 FAIL git-annex_7.20190730+git103-g070fbd693-1~ndall+1_i386.build
|
||||
20 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
(checksum...) (recording state in git...)
|
||||
ok
|
||||
(recording state in git...)
|
||||
OK (1.00s)
|
||||
move (ssh remote): move foo (from origin...)
|
||||
(checksum...) ok
|
||||
(recording state in git...)
|
||||
move foo (to origin...)
|
||||
ok
|
||||
(recording state in git...)
|
||||
OK (0.99s)
|
||||
copy: copy foo (from origin...)
|
||||
SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77
|
||||
20 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
|
||||
SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77
|
||||
20 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
failed
|
||||
copy: 1 failed
|
||||
FAIL (0.34s)
|
||||
Test.hs:530:
|
||||
"""]]
|
||||
|
||||
I have tried to rebuild (`debian/rules binary`) second time while in the same environment -- didn't succeed... but then I proceeded to build amd64 fine, and redid i386 build just fine
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[fixed|done]] though w/o a true root cause analysis --[[Joey]]
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2019-08-13T17:28:15Z"
|
||||
content="""
|
||||
This is the same problem I tried to deal with in
|
||||
[[!commit f27c5db5c566bdc0baae256b67df04a50027679f]].
|
||||
Apparently not fully
|
||||
|
||||
Or at least the double run of rsync with no indication
|
||||
why it thought the first one failed appears to be the same.
|
||||
And IIRC I saw that sometimes without the permissions error and it still
|
||||
failed.
|
||||
|
||||
I was having to run git-annex test in a loop for an hour or so to reproduce
|
||||
it.
|
||||
|
||||
So, for some reason git-annex thinks the rsync failed, although it got to
|
||||
100%. If that happens once, it retries automatically. The second time it
|
||||
thinks rsync has failed, it sees the file didn't get any larger,
|
||||
and so gives up. That makes sense, but I don't know how rsync would get
|
||||
to 100% and then apparently exit nonzero, without displaying any error
|
||||
message.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 2"
|
||||
date="2019-08-13T18:15:36Z"
|
||||
content="""
|
||||
and git-annex cannot rely on the check that size of the receiving file has reached the target size?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2019-08-13T19:14:41Z"
|
||||
content="""
|
||||
No, it cannot in general rely on that. Consider a file transfer program
|
||||
that allocates the whole file full of 0 to start.
|
||||
|
||||
This needs to be reproduced in strace so we can see what it happening with
|
||||
rsync.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2019-08-15T18:00:42Z"
|
||||
content="""
|
||||
And even if we assume rsync never pre-allocates a file before receiving it, it
|
||||
probably does do some things at the end, like setting the final permissions and
|
||||
timestamp.
|
||||
|
||||
The permissions error looked like this:
|
||||
|
||||
get foo (from origin...)
|
||||
SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77
|
||||
20 100% 0.00kB/s 0:00:00 ^M 20 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
(from origin...)
|
||||
SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77
|
||||
20 100% 0.00kB/s 0:00:00 ^M 20 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
rsync: open "/home/joey/src/git-annex/.t/tmprepo1103/.git/annex/tmp/SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77" failed: Permission denied (13)
|
||||
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
|
||||
|
||||
That looked as if the first rsync had gotten as far as removing the write bit
|
||||
from the file. Interestingly, the second rsync seems to have received the whole
|
||||
(small) file content before ever trying to open the file for write.
|
||||
|
||||
The only recent change I can think of involving rsync was the CoW probing
|
||||
change, but I don't see how that could possibly lead to this behavior.
|
||||
|
||||
And I'm not sure this is a new problem. The test suite has been intermittently
|
||||
failing for several months, around this same point. The failure did not
|
||||
include any useful error message, so I could not debug it, and I have IIRC
|
||||
done a few things to try to get the test suite to display an error message.
|
||||
Perhaps I succeeded.
|
||||
|
||||
The intermittent test suite failure looks like this:
|
||||
|
||||
copy: [adjusted/master(unlocked) 05b89a6] empty
|
||||
adjust ok
|
||||
copy foo (from origin...)
|
||||
SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77
|
||||
20 100% 0.00kB/s 0:00:00 20 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
|
||||
SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77
|
||||
20 100% 0.00kB/s 0:00:00 20 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
failed
|
||||
copy: 1 failed
|
||||
FAIL (1.12s)
|
||||
|
||||
I am not sure if it only happens when testing adjusted unlocked branches/
|
||||
v7 unlocked files.
|
||||
|
||||
I've run git-annex get; git-annex drop in a tight loop for
|
||||
thousands of iterations on an adjusted unlocked branch,
|
||||
and can't reproduce the failure that way.
|
||||
|
||||
I've made git-annex display rsync's exit status when it's not 0 or 1,
|
||||
it has a lot of other exit statuses, so perhaps that will help track
|
||||
down how it's failing.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 5"""
|
||||
date="2019-08-15T21:18:20Z"
|
||||
content="""
|
||||
I tested a git-annex made to say when the rsync process succeeded,
|
||||
and rsync is actually not failing.
|
||||
|
||||
So something in git-annex is silently making the transfer fail.
|
||||
Could be, for example, that it thinks the file was modified while
|
||||
it was being transferred.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 6"""
|
||||
date="2019-08-15T21:55:43Z"
|
||||
content="""
|
||||
It is indeed detecting that the file it was sending
|
||||
appears to have been modified after the download started.
|
||||
|
||||
Probably the file has not really been modified. Instead the inode cache
|
||||
for it is somehow wrong.
|
||||
|
||||
An instrumented run had cached:
|
||||
|
||||
InodeCache (InodeCachePrim 21765306 20 (MTimeHighRes 1565905801.196804558s)
|
||||
InodeCache (InodeCachePrim 22158907 20 (MTimeHighRes 1565905801.196804558s))
|
||||
|
||||
And after the download the file it was sending had:
|
||||
|
||||
InodeCache (InodeCachePrim 21765305 20 (MTimeHighRes 1565905802.380791792s))
|
||||
|
||||
Note that the test suite moves the same file from origin,
|
||||
and then moves it back, and a while after that get fails.
|
||||
So at least one of the cached inodes is for the old copy of the file.
|
||||
The other one is probably the work tree copy.
|
||||
|
||||
Still have not had luck reproducing outside the test suite with a tight loop
|
||||
of move --from and --to and get.
|
||||
|
||||
Hypothesis: The test suite uses Annex.main for many of its runs of git-annex,
|
||||
and so sqlite does not do whatever it normally does atexit. If flushing
|
||||
a change to the db file gets deferred for whatever reason, a later call to
|
||||
Annex.main might see old information.
|
||||
|
||||
If so, converting the test suite to run git-annex instead of Annex.main
|
||||
would avoid the problem.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 7"""
|
||||
date="2019-08-16T15:03:11Z"
|
||||
content="""
|
||||
Confirmed; avoiding calling main has gotten the test suite to be able to
|
||||
run at least 500 iterations over 12 hours w/o failing. It had been failing
|
||||
in under 1 hour.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
Origin: [ReproNim/containers](https://github.com/ReproNim/containers/pull/17) repo where we were brave enough to start using v7 with unlocked files to avoid having symlinks for an image. TL;DR: here is command to replicate:
|
||||
|
||||
[[!format sh """
|
||||
$> git clone --depth=50 http://github.com/repronim/containers && cd containers && git fetch origin +refs/pull/19/merge: && git checkout -qf 'HEAD^{}' && git branch && git fetch origin git-annex && git remote add --fetch datalad.datasets.org http://datasets.datalad.org/repronim/containers/.git && git annex upgrade && git annex get scripts/tests/ && ls -l scripts/tests/
|
||||
Cloning into 'containers'...
|
||||
warning: redirecting to https://github.com/repronim/containers/
|
||||
remote: Enumerating objects: 404, done.
|
||||
remote: Counting objects: 100% (404/404), done.
|
||||
remote: Compressing objects: 100% (208/208), done.
|
||||
remote: Total 404 (delta 198), reused 367 (delta 171), pack-reused 0
|
||||
Receiving objects: 100% (404/404), 59.13 KiB | 1.34 MiB/s, done.
|
||||
Resolving deltas: 100% (198/198), done.
|
||||
LICENSE README.md binds/ images/ scripts/
|
||||
warning: redirecting to https://github.com/repronim/containers/
|
||||
remote: Enumerating objects: 21, done.
|
||||
remote: Counting objects: 100% (21/21), done.
|
||||
remote: Compressing objects: 100% (10/10), done.
|
||||
remote: Total 12 (delta 9), reused 4 (delta 2), pack-reused 0
|
||||
Unpacking objects: 100% (12/12), done.
|
||||
From http://github.com/repronim/containers
|
||||
* branch refs/pull/19/merge -> FETCH_HEAD
|
||||
* (HEAD detached at 44d55ad)
|
||||
master
|
||||
warning: redirecting to https://github.com/repronim/containers/
|
||||
remote: Enumerating objects: 798, done.
|
||||
remote: Counting objects: 100% (798/798), done.
|
||||
remote: Compressing objects: 100% (167/167), done.
|
||||
remote: Total 795 (delta 423), reused 788 (delta 416), pack-reused 0
|
||||
Receiving objects: 100% (795/795), 55.50 KiB | 1.50 MiB/s, done.
|
||||
Resolving deltas: 100% (423/423), done.
|
||||
From http://github.com/repronim/containers
|
||||
* branch git-annex -> FETCH_HEAD
|
||||
Updating datalad.datasets.org
|
||||
From http://datasets.datalad.org/repronim/containers/
|
||||
* [new branch] git-annex -> datalad.datasets.org/git-annex
|
||||
* [new branch] master -> datalad.datasets.org/master
|
||||
* [new branch] synced/master -> datalad.datasets.org/synced/master
|
||||
upgrade (merging datalad.datasets.org/git-annex into git-annex...)
|
||||
(recording state in git...)
|
||||
(v5 to v6...) (v6 to v7...) ok
|
||||
(recording state in git...)
|
||||
get scripts/tests/arg-test.simg
|
||||
Remote origin not usable by git-annex; setting annex-ignore
|
||||
(from datalad.datasets.org...)
|
||||
(checksum...) ok
|
||||
(recording state in git...)
|
||||
total 12
|
||||
-rw------- 1 yoh yoh 259 Jul 11 16:10 Singularity.arg-test
|
||||
-rw------- 1 yoh yoh 68 Jul 11 16:10 arg-test.simg
|
||||
-rwx------ 1 yoh yoh 981 Jul 11 16:10 test_singularity_cmd.bats*
|
||||
"""]]
|
||||
|
||||
as you can see the arg-test.sim is only 68 bytes, not 2MB or a symlink.
|
||||
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 1"
|
||||
date="2019-07-11T20:15:45Z"
|
||||
content="""
|
||||
further debugging showed that the bug appears if `annex upgrade` is done in detached HEAD mode.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="a reduced script"
|
||||
date="2019-07-11T20:19:23Z"
|
||||
content="""
|
||||
```
|
||||
#!/bin/sh
|
||||
|
||||
cd $(mktemp -dt gx-XXXXXXX)
|
||||
|
||||
git init a
|
||||
cd a
|
||||
git annex init --version=7
|
||||
echo foo >foo
|
||||
git add foo
|
||||
git commit -mfoo
|
||||
|
||||
cd ..
|
||||
git clone a b
|
||||
cd b
|
||||
|
||||
git checkout HEAD^{}
|
||||
git annex init
|
||||
git annex upgrade
|
||||
|
||||
git annex get foo
|
||||
cat foo
|
||||
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
Initialized empty Git repository in /tmp/gx-RqdQyBF/a/.git/
|
||||
init ok
|
||||
(recording state in git...)
|
||||
(recording state in git...)
|
||||
[master (root-commit) 2b5f739] foo
|
||||
1 file changed, 1 insertion(+)
|
||||
create mode 100644 foo
|
||||
Cloning into 'b'...
|
||||
done.
|
||||
Note: checking out 'HEAD^{}'.
|
||||
|
||||
You are in 'detached HEAD' state. You can look around, make experimental
|
||||
changes and commit them, and you can discard any commits you make in this
|
||||
state without impacting any branches by performing another checkout.
|
||||
|
||||
If you want to create a new branch to retain commits you create, you may
|
||||
do so (now or later) by using -b with the checkout command again. Example:
|
||||
|
||||
git checkout -b <new-branch-name>
|
||||
|
||||
HEAD is now at 2b5f739 foo
|
||||
init (merging origin/git-annex into git-annex...)
|
||||
ok
|
||||
(recording state in git...)
|
||||
upgrade (v5 to v6...) (v6 to v7...) ok
|
||||
get foo (from origin...) (checksum...) ok
|
||||
(recording state in git...)
|
||||
/annex/objects/SHA256E-s4--b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
|
||||
|
||||
```
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2019-07-16T16:13:49Z"
|
||||
content="""
|
||||
I notice that the keys database is not populated in the clone.
|
||||
|
||||
Also, the upgrade does not display "scanning for unlocked files".
|
||||
And in Annex.WorkTree, we can see why:
|
||||
|
||||
scanUnlockedFiles = whenM (isJust <$> inRepo Git.Branch.current) $ do
|
||||
showSideAction "scanning for unlocked files"
|
||||
|
||||
There is no current git branch in this case.
|
||||
|
||||
That check was added in [[!commit 9b995954731e05727d77c7bff487af10da9cb4b9]]
|
||||
"only do scan when there's a branch, not in freshly created new repo"
|
||||
|
||||
Since it does a ls-tree of HEAD, what it really ought to check for is that
|
||||
HEAD is set, which it's not in a fresh new repo. Done.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
### Please describe the problem.
|
||||
|
||||
TL;DR - it seems that obsolete `.git/annex/index.lck .git/annex/journal.lck .git/annex/precommit.lck` could cause a new process to hang on `git commit` or `git annex add` without announcing any problem. git annex version installed is 7.20181211-g1cdb7a2. But that seems to be not a complete reason - and after removing them still gets stuck
|
||||
|
||||
|
||||
I think we initiated `git annex add` invocation over remote ssh session, and then connection eventually dropped before `git annex add` finished. The repository is large - 47k files.
|
||||
So we ended up with some files annex/staged, some just annexed (symlinks to keys), and some not yet tracked by git and in their original form.
|
||||
|
||||
I thought to commit first at least what we had already staged, but `git commit` pretty much hanged there for an hour. It is an NFS mount on an HPC, so IO is not stellar but an hour with little to no CPU consumed sounds like it just got stuck.
|
||||
I have checked ps and saw that it is in the `pre-commit` hook with two zombies. Interrupted and ran manually with `--debug`:
|
||||
|
||||
[[!format sh """
|
||||
fmriprep001]$ git annex --debug pre-commit .
|
||||
[2019-03-18 18:08:36.049933] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
|
||||
[2019-03-18 18:08:36.060284] process done ExitSuccess
|
||||
[2019-03-18 18:08:36.060364] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
|
||||
[2019-03-18 18:08:36.081517] process done ExitSuccess
|
||||
[2019-03-18 18:08:36.081776] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff","--cached","--name-only","-z","--diff-filter=ACMRT","--","."]
|
||||
[2019-03-18 18:08:44.780003] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff","--name-only","--diff-filter=T","-z","--cached","--","."]
|
||||
[2019-03-18 18:08:44.817133] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
|
||||
[2019-03-18 18:08:44.827942] process done ExitSuccess
|
||||
[2019-03-18 18:08:44.827996] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
|
||||
[2019-03-18 18:08:44.837793] process done ExitSuccess
|
||||
|
||||
# just hanging there, no notable CPU utilization
|
||||
# ps output -- I think I did mistake and didn't grep for git but for annex
|
||||
5765 0.0 0.0 127208 3800 pts/145 Ss Mar07 0:00 /bin/bash
|
||||
21340 0.0 0.0 113940 1400 pts/145 S+ 18:08 0:00 git annex --debug pre-commit .
|
||||
21341 2.0 0.0 1074154224 19164 pts/145 Sl+ 18:08 0:01 /ihome/crc/install/datalad/python3.7/bin/git-annex --debug pre-commit .
|
||||
21353 0.0 0.0 0 0 pts/145 Z+ 18:08 0:00 [git] <defunct>
|
||||
21397 0.0 0.0 0 0 pts/145 Z+ 18:08 0:00 [git] <defunct>
|
||||
|
||||
$ git annex version
|
||||
git-annex version: 7.20181211-g1cdb7a2
|
||||
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.2 feed-1.0.0.0 ghc-8.2.2 http-client-0.5.13.1 persistent-sqlite-2.8.1.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 5 7
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6
|
||||
|
||||
"""]]
|
||||
|
||||
Retrying to add everything again seems to lead to a stall as well:
|
||||
|
||||
[[!format sh """
|
||||
fmriprep001]$ git annex --debug add *
|
||||
[2019-03-18 18:19:54.527228] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
|
||||
[2019-03-18 18:19:54.537078] process done ExitSuccess
|
||||
[2019-03-18 18:19:54.537377] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
|
||||
[2019-03-18 18:19:54.547586] process done ExitSuccess
|
||||
[2019-03-18 18:19:54.547873] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--others","--exclude-standard","-z","--","fmriprep","freesurfer"]
|
||||
[2019-03-18 18:19:55.557577] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
|
||||
[2019-03-18 18:19:55.558288] read: git ["--version"]
|
||||
[2019-03-18 18:19:55.560309] process done ExitSuccess
|
||||
add freesurfer/sub-000457/mri/lh.ribbon.mgz
|
||||
"""]]
|
||||
|
||||
and that file seems to be freshly touched and initially was not staged for a commit according to `git status`:
|
||||
|
||||
[[!format sh """
|
||||
fmriprep001]$ ls -l freesurfer/sub-000457/mri/lh.ribbon.mgz
|
||||
... Mar 18 18:19 freesurfer/sub-000457/mri/lh.ribbon.mgz -> ../../../.git/annex/objects/61/87/MD5E-s103445--f9f251a9f80317fbcf46cf11fa7be520.mgz/MD5E-s103445--f9f251a9f80317fbcf46cf11fa7be520.mgz
|
||||
"""]]
|
||||
|
||||
here is ps
|
||||
|
||||
[[!format sh """
|
||||
# ps auxw -H | grep annex -- showing only relevant portion, no other annex processes detected;
|
||||
# should have grepped for git! :-/
|
||||
26125 0.0 0.0 113940 1404 pts/145 S+ 18:19 0:00 git annex --debug add fmriprep freesurfer
|
||||
26126 0.4 0.0 1074154224 108492 pts/145 Sl+ 18:19 0:00 /ihome/crc/install/datalad/python3.7/bin/git-annex --debug add fmriprep freesurfer
|
||||
26138 0.0 0.0 115916 3432 pts/145 S+ 18:19 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs check-attr -z --stdin annex.backend annex.numcopies annex.largefiles --
|
||||
"""]]
|
||||
|
||||
looking at the list of files for 26126 process I see `.git/annex/journal.lck` which seems to date to ` Feb 27 ` (3 weeks ago!), and altogether there are 3 lock files from that date:
|
||||
`.git/annex/index.lck .git/annex/journal.lck .git/annex/precommit.lck`.
|
||||
FWIW, NFS mount options `type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.201.0.82,local_lock=none,addr=10.201.2.178)`
|
||||
|
||||
I have removed them, and then called `git commit` which seemed to proceed normally, got to report `(recording state in git...)` but then seemed to get stuck as well:
|
||||
|
||||
[[!format sh """
|
||||
31711 0.0 0.0 1460028 3876 pts/145 S+ 18:27 0:00 git commit -m Annexed portion of the results, probably was disconnected, so not all Also there were some stale lock files present .git/annex/index.lck .git/annex/journal.lck .git/annex/precommit.lck which caused precommit and add to hang, removed them
|
||||
31788 0.0 0.0 113184 1412 pts/145 S+ 18:27 0:00 /bin/sh .git/hooks/pre-commit
|
||||
31789 0.0 0.0 113940 1404 pts/145 S+ 18:27 0:00 git annex pre-commit .
|
||||
31791 1.1 0.0 1074154220 19240 pts/145 Dl+ 18:27 0:04 /ihome/crc/install/datalad/python3.7/bin/git-annex pre-commit .
|
||||
31802 0.0 0.0 0 0 pts/145 Z+ 18:27 0:00 [git] <defunct>
|
||||
31889 0.0 0.0 0 0 pts/145 Z+ 18:27 0:00 [git] <defunct>
|
||||
31901 1.6 0.0 113964 1608 pts/145 S+ 18:27 0:06 git --git-dir=.git --work-tree=. --literal-pathspecs hash-object -w --stdin-paths --no-filters
|
||||
1075 0.0 0.0 113944 1624 pts/145 S+ 18:29 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
|
||||
1076 0.0 0.0 113944 1408 pts/145 S+ 18:29 0:00 git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize)
|
||||
"""]]
|
||||
|
||||
There is also two `.git/annex/misctmp/jlog*` files with one from Feb 28 and one from today. Today's one is bigger, and used by 31901 but seems not to change for the last 15 minutes.
|
||||
|
||||
what could be a reason and any way to mitigate?
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[closing|done]] per comments --[[Joey]]
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2019-03-19T02:33:02Z"
|
||||
content="""
|
||||
Since this is NFS, does it have annex.pidlock set?
|
||||
|
||||
If so, an stale pid lock (.git/annex/pidlock) can cause a stall, for up to
|
||||
annex.pidlockimeout seconds.
|
||||
|
||||
The hang points seem consistent with it preparing to commit the journal to
|
||||
the git-annex branch and indeed something related to locking.
|
||||
An strace would tell for sure.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="reporting back"
|
||||
date="2019-03-19T17:01:10Z"
|
||||
content="""
|
||||
Removing .lck, setting `annex.pidlock=true` worked! got just one complaint from git:
|
||||
|
||||
[[!format sh \"\"\"
|
||||
$ git commit -m 'Committing staged changes after annex process was killed
|
||||
>
|
||||
> Also removed stale .git/annex/*.lck and now set annex.pidlock=True for the repo'
|
||||
(recording state in git...)
|
||||
git-annex: .git/annex/journal/e64_e4a_MD5E-s55076--a7352bf6fd37902fc1dfb6228f2e8a53.log: removeLink: does not exist (No such file or directory)
|
||||
|
||||
\"\"\"]]
|
||||
I found that file (via `find * -lname *MD5E-s55076--a7352bf6fd37902fc1dfb6228f2e8a53*` FTR), it was not known to git yet, so I am now running `annex add` on it (takes awhile -- I guess repo is indeed got large + NFS) -- will report back when done/if there is more.
|
||||
|
||||
|
||||
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 3"
|
||||
date="2019-03-19T17:41:12Z"
|
||||
content="""
|
||||
Managed to login to that node again, after hours (>3) of \"running\" `annex add` is still there with current process tree being
|
||||
|
||||
[[!format sh \"\"\"
|
||||
PID TTY STAT TIME COMMAND
|
||||
22168 pts/50 Ss 0:00 /bin/bash
|
||||
15560 pts/50 S+ 0:00 \_ git annex add freesurfer/sub-000458/label/lh.BA4p_exvivo.thresh.label
|
||||
15565 pts/50 Dl+ 0:18 \_ /ihome/crc/install/datalad/python3.7/bin/git-annex add freesurfer/sub-000458/label/lh.BA4p_exvivo.thresh.label
|
||||
15591 pts/50 Z+ 0:00 \_ [git] <defunct>
|
||||
15601 pts/50 S+ 0:00 \_ git --git-dir=.git --work-tree=. --literal-pathspecs check-attr -z --stdin annex.backend annex.numcopies annex.largefiles --
|
||||
15625 pts/50 S+ 0:00 \_ git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
|
||||
15626 pts/50 S+ 0:00 \_ git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize)
|
||||
15642 pts/50 Z+ 0:00 \_ [git] <defunct>
|
||||
15650 pts/50 Z+ 0:00 \_ [git] <defunct>
|
||||
15703 pts/50 S+ 0:02 \_ git --git-dir=.git --work-tree=. --literal-pathspecs hash-object -w --stdin-paths --no-filters
|
||||
|
||||
\"\"\"]]
|
||||
|
||||
and all the locks there, and I believe being generated by this `annex add` run:
|
||||
|
||||
[[!format sh \"\"\"
|
||||
$ /bin/ls .git/annex/
|
||||
index index.lck journal journal.lck misctmp objects pidlock precommit.lck sentinal sentinal.cache
|
||||
\"\"\"]]
|
||||
|
||||
the process 15565 is slowly increasing its seconds (now 19)... is that a good sign (no lockup, just very slow IO) or not (it is locked up, and all the time corresponds just to the periodic querying of the lock)? if to strace -- which particular process? or should I interrupt and run the entire `annex add` under strace?
|
||||
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2019-03-19T17:59:05Z"
|
||||
content="""
|
||||
Certianly sounds like the process is not deadlocked.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 5"""
|
||||
date="2019-03-19T19:55:24Z"
|
||||
content="""
|
||||
Although, the haskell RTS uses a select loop with a timeout and so might use
|
||||
a small amount of CPU while otherwise blocked waiting on a lock..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="more info on hanging"
|
||||
date="2019-03-19T21:04:54Z"
|
||||
content="""
|
||||
[[!format sh \"\"\"
|
||||
$ strace -f -p 15565
|
||||
strace: Process 15565 attached with 5 threads
|
||||
[pid 15570] futex(0x7f0c2c000924, FUTEX_WAIT_PRIVATE, 15, NULL <unfinished ...>
|
||||
[pid 15568] restart_syscall(<... resuming interrupted poll ...> <unfinished ...>
|
||||
[pid 15566] read(3, <unfinished ...>
|
||||
[pid 15567] futex(0x4a454a4, FUTEX_WAIT_PRIVATE, 27143, NULL <unfinished ...>
|
||||
[pid 15566] <... read resumed> \"\1\0\0\0\0\0\0\0\", 8) = 8
|
||||
[pid 15566] read(3, \"\1\0\0\0\0\0\0\0\", 8) = 8
|
||||
[pid 15566] read(3, \"\1\0\0\0\0\0\0\0\", 8) = 8
|
||||
.... keeps going, had to bf, kill -9 strace process
|
||||
\"\"\"]]
|
||||
and that 15566 is a thread:
|
||||
[[!format sh \"\"\"
|
||||
$ ps -L -p 15565
|
||||
PID LWP TTY TIME CMD
|
||||
15565 15565 pts/50 00:00:01 git-annex
|
||||
15565 15566 pts/50 00:00:31 ghc_ticker
|
||||
15565 15567 pts/50 00:00:00 ghc_worker
|
||||
15565 15568 pts/50 00:00:00 ghc_worker
|
||||
15565 15570 pts/50 00:00:00 ghc_worker
|
||||
|
||||
$ ls -l /proc/15566/fd | anon
|
||||
total 0
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 17:03 0 -> /dev/pts/50
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 17:03 1 -> /dev/pts/50
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 17:03 10 -> anon_inode:[eventfd]
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 17:03 11 -> /dev/shm/cd6281f9a8c5f62e1da78a60ca9ca1b4D_fmriprep001_.git_annex_pidlock.lck
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 17:03 12 -> /zfs1/GROUP/USER/R03/WM-R03/DIAMOND/fmriprep001/.git/annex/misctmp/jlog15565-2
|
||||
l-wx------ 1 USER GROUP 64 Mar 19 17:03 13 -> pipe:[233188705]
|
||||
lr-x------ 1 USER GROUP 64 Mar 19 17:03 14 -> pipe:[233188706]
|
||||
l-wx------ 1 USER GROUP 64 Mar 19 17:03 16 -> pipe:[233188711]
|
||||
lr-x------ 1 USER GROUP 64 Mar 19 17:03 17 -> pipe:[233188712]
|
||||
l-wx------ 1 USER GROUP 64 Mar 19 17:03 18 -> pipe:[233188714]
|
||||
lr-x------ 1 USER GROUP 64 Mar 19 17:03 19 -> pipe:[233188715]
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 16:59 2 -> /dev/pts/50
|
||||
l-wx------ 1 USER GROUP 64 Mar 19 17:03 20 -> pipe:[233188718]
|
||||
lr-x------ 1 USER GROUP 64 Mar 19 17:03 21 -> pipe:[233188719]
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 17:03 3 -> anon_inode:[timerfd]
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 17:03 4 -> anon_inode:[eventpoll]
|
||||
lr-x------ 1 USER GROUP 64 Mar 19 17:03 5 -> pipe:[233191520]
|
||||
l-wx------ 1 USER GROUP 64 Mar 19 17:03 6 -> pipe:[233191520]
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 17:03 7 -> anon_inode:[eventfd]
|
||||
lr-x------ 1 USER GROUP 64 Mar 19 17:03 8 -> pipe:[233191523]
|
||||
l-wx------ 1 USER GROUP 64 Mar 19 17:03 9 -> pipe:[233191523]
|
||||
|
||||
\"\"\"]]
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 7"""
|
||||
date="2019-03-20T15:52:45Z"
|
||||
content="""
|
||||
Hard to see for sure from this strace tail end, but the fd number sequence
|
||||
suggests it first took the pid lock:
|
||||
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 17:03 11 -> /dev/shm/cd6281f9a8c5f62e1da78a60ca9ca1b4D_fmriprep001_.git_annex_pidlock.lck
|
||||
|
||||
And since it then seems to have proceeded to start to write to this file,
|
||||
the pid lock is apparently not the hang point.
|
||||
|
||||
lrwx------ 1 USER GROUP 64 Mar 19 17:03 12 -> /zfs1/GROUP/USER/R03/WM-R03/DIAMOND/fmriprep001/.git/annex/misctmp/jlog15565-2
|
||||
|
||||
So stageJournal is running, after openjlog. That does not involve any
|
||||
further locks, and would not normally take a long time unless
|
||||
.git/annex/journal/ has an unusual number of files in it. It seems to have
|
||||
stalled for some reason. It should have started a git update-index process
|
||||
and fed data into it.
|
||||
|
||||
Do you know if this process had a child git update-index running
|
||||
at this point? It could be that child is what is stalled, somehow.
|
||||
|
||||
The number of lines in the jlog15565-2 file would also be a clue to how far
|
||||
it got before it got stuck.
|
||||
|
||||
But I think a full strace may be needed to find the sticking point.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 8"""
|
||||
date="2020-03-30T15:59:33Z"
|
||||
content="""
|
||||
The precommit.lck is no longer used, it was not needed since v8. But
|
||||
this bug does not really appear to be about that lock at all.
|
||||
|
||||
Pity there was no followup to my last questions. At this point I can either
|
||||
close the bug report, or mark it moreinfo or something, but it seems pretty
|
||||
unlikely we will get to the bottom of it.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="Let's just close"
|
||||
date="2020-03-30T16:32:24Z"
|
||||
content="""
|
||||
I guess let's just close it for now -- I have managed to miss your reply back then (may be didn't check \"email replies to me\"), so the path got cold.
|
||||
I hope that I will get back to exercising such workflows soon, so we might get back to it in a new manifestation if it happens again.
|
||||
Cheers
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
### Please describe the problem.
|
||||
|
||||
I got a new server for the repronim project (welcome `typhon` to the family of `smaug`, `drogon`, and `falkor`). Wanted to redo https://www.datalad.org/test_fs_analysis.html benchmarking. Added `git annex test` call to see how its time would vary across file systems etc.
|
||||
|
||||
My surprise was that my script never exited.
|
||||
|
||||
Current ps tree is (yeah, running as `root`, likely will reinstall the thing anyways fully after I am done benchmarking)
|
||||
|
||||
```
|
||||
root 185696 0.0 0.0 62536 54768 pts/1 S+ Nov15 2:56 python test_fs.py benchmark
|
||||
root 3655423 0.0 0.0 2484 516 pts/1 S+ Nov18 0:00 /bin/sh -c cd test9; git annex test
|
||||
root 3655424 0.0 0.0 9728 3340 pts/1 S+ Nov18 0:00 git annex test
|
||||
root 3655425 0.0 0.0 1076293924 28360 pts/1 Sl+ Nov18 0:00 /usr/bin/git-annex test
|
||||
root 3696268 0.2 0.0 1074196636 41932 pts/1 Sl+ Nov18 13:14 /usr/bin/git-annex --color=never test
|
||||
root 3710526 0.0 0.0 1074122856 44736 pts/1 Sl+ Nov18 0:00 /usr/bin/git-annex move --to origin foo
|
||||
root 3710593 0.0 0.0 0 0 pts/1 Z+ Nov18 0:00 [git] <defunct>
|
||||
root 3710594 0.0 0.0 0 0 pts/1 Z+ Nov18 0:00 [git] <defunct>
|
||||
root 3710595 0.0 0.0 0 0 pts/1 Z+ Nov18 0:00 [git] <defunct>
|
||||
root 3710596 0.0 0.0 0 0 pts/1 Z+ Nov18 0:00 [git] <defunct>
|
||||
root 3710597 0.0 0.0 2484 516 pts/1 S+ Nov18 0:00 sh -c git-annex test --fakessh -- 'localhost' 'git-annex-shell '"'"'p2pstdio'"'"' '"'"'/mnt/test/test9/.t/75/main0'"'"' '"'"'6129b8ec-34af-47d1-b7e6-3c4fcc0d33e3'"'"' --uuid c129397d-8209-40ea-8347-16a8c3fe69de'
|
||||
root 3710598 0.0 0.0 1074048960 20408 pts/1 Sl+ Nov18 0:00 git-annex test --fakessh -- localhost git-annex-shell 'p2pstdio' '/mnt/test/test9/.t/75/main0' '6129b8ec-34af-47d1-b7e6-3c4fcc0d33e3' --uuid c129397d-8209-40ea-8347-16a8c3fe69de
|
||||
root 3710604 0.0 0.0 2484 580 pts/1 S+ Nov18 0:00 /bin/sh -c git-annex-shell 'p2pstdio' '/mnt/test/test9/.t/75/main0' '6129b8ec-34af-47d1-b7e6-3c4fcc0d33e3' --uuid c129397d-8209-40ea-8347-16a8c3fe69de
|
||||
root 3710605 0.0 0.0 1074122860 18948 pts/1 Sl+ Nov18 2:39 git-annex-shell p2pstdio /mnt/test/test9/.t/75/main0 6129b8ec-34af-47d1-b7e6-3c4fcc0d33e3 --uuid c129397d-8209-40ea-8347-16a8c3fe69de
|
||||
|
||||
```
|
||||
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
```
|
||||
git-annex version: 10.20221003
|
||||
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
|
||||
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 8 9 10
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
|
||||
|
||||
```
|
||||
|
||||
any ideas? need access to troubleshoot?
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 10"""
|
||||
date="2022-12-09T16:16:10Z"
|
||||
content="""
|
||||
The retest seemed to hang again, but I'm not sure I was testing the right
|
||||
binary. Running it again.
|
||||
|
||||
This patch exposes when the lock is held and so illustrates the bug
|
||||
reliably and quickly:
|
||||
|
||||
diff --git a/Command/Smudge.hs b/Command/Smudge.hs
|
||||
index a5d8871998..cf6e2973f8 100644
|
||||
--- a/Command/Smudge.hs
|
||||
+++ b/Command/Smudge.hs
|
||||
@@ -127,7 +129,8 @@ clean'
|
||||
-> (Key -> Annex ())
|
||||
-- ^ emitpointer: Emit a pointer file for the key.
|
||||
-> Annex ()
|
||||
-clean' file mk passthrough discardreststdin emitpointer =
|
||||
+clean' file mk passthrough discardreststdin emitpointer = do
|
||||
+ calcRestageLog () (\_ _ -> ())
|
||||
ifM (fileOutsideRepo file)
|
||||
( passthrough
|
||||
, inSmudgeCleanFilter go
|
||||
|
||||
With that patch, the unfixed git-annex hangs immediately. With
|
||||
my fix, it does not hang, which seems to show I fixed it correctly..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 11"""
|
||||
date="2022-12-10T16:02:15Z"
|
||||
content="""
|
||||
Got a clean test run with the right binary for sure, and I've confirmed
|
||||
this bug is really fixed. Test ran 20k iterations successfully.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2022-11-28T18:11:41Z"
|
||||
content="""
|
||||
Seems like the test it is stuck in is "move (ssh remote)".
|
||||
|
||||
Is this replicable?
|
||||
|
||||
I wonder if `test_fs.py` is doing something that causes the problem.
|
||||
There is some complexity in there. Maybe it's happening on one
|
||||
particular FS.
|
||||
|
||||
If it happens when not run by `test_fs.py`, it would be useful to get a
|
||||
debug log, because I think that seeing the P2P protocol dump would help
|
||||
explain what has happened.
|
||||
|
||||
There was not a good way to get the test suite to output a full debug
|
||||
log, so I had to implement one. With an updated build of git-annex,
|
||||
you can use:
|
||||
|
||||
git-annex test -J1 --test-debug
|
||||
|
||||
Here's the expected output of that section of the test suite, when it
|
||||
is working properly:
|
||||
|
||||
move foo [2022-11-28 14:54:11.10011679] (Utility.Process) process [390399] chat: sh ["-c","git-annex test --fakessh -- 'localhost' 'git-annex-shell '\"'\"'p2pstdio'\"'\"' '\"'\"'/home/joey/src/git-annex/.t/3/main1'\"'\"' '\"'\"'--debug'\"'\"' '\"'\"'c76e0406-38ea-413e-9fb0-56cc32847734'\"'\"' --uuid 000c7285-ec19-4c15-9c67-4dd4b7c74775'"]
|
||||
[2022-11-28 14:54:11.129754519] (P2P.IO) [ThreadId 4] P2P > AUTH-SUCCESS 000c7285-ec19-4c15-9c67-4dd4b7c74775
|
||||
[2022-11-28 14:54:11.130259741] (P2P.IO) [ssh connection Just 390399] [ThreadId 4] P2P < AUTH-SUCCESS 000c7285-ec19-4c15-9c67-4dd4b7c74775
|
||||
[2022-11-28 14:54:11.13041388] (P2P.IO) [ssh connection Just 390399] [ThreadId 4] P2P > VERSION 1
|
||||
[2022-11-28 14:54:11.130680315] (P2P.IO) [ThreadId 4] P2P < VERSION 1
|
||||
[2022-11-28 14:54:11.13087982] (P2P.IO) [ThreadId 4] P2P > VERSION 1
|
||||
[2022-11-28 14:54:11.131127415] (P2P.IO) [ssh connection Just 390399] [ThreadId 4] P2P < VERSION 1
|
||||
[2022-11-28 14:54:11.131262307] (P2P.IO) [ssh connection Just 390399] [ThreadId 4] P2P > CHECKPRESENT SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77
|
||||
[2022-11-28 14:54:11.131679501] (P2P.IO) [ThreadId 4] P2P < CHECKPRESENT SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77
|
||||
[2022-11-28 14:54:11.132388822] (P2P.IO) [ThreadId 4] P2P > FAILURE
|
||||
[2022-11-28 14:54:11.13261351] (P2P.IO) [ssh connection Just 390399] [ThreadId 4] P2P < FAILURE
|
||||
(to origin...)
|
||||
[2022-11-28 14:54:11.135583194] (P2P.IO) [ssh connection Just 390399] [ThreadId 4] P2P > PUT foo SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77
|
||||
[2022-11-28 14:54:11.136134981] (P2P.IO) [ThreadId 4] P2P < PUT foo SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77
|
||||
[2022-11-28 14:54:11.136824468] (P2P.IO) [ThreadId 4] P2P > PUT-FROM 0
|
||||
[2022-11-28 14:54:11.137123219] (P2P.IO) [ssh connection Just 390399] [ThreadId 4] P2P < PUT-FROM 0
|
||||
...
|
||||
[2022-11-28 14:54:11.148194515] (P2P.IO) [ssh connection Just 390399] [ThreadId 4] P2P > DATA 20
|
||||
[2022-11-28 14:54:11.148414855] (P2P.IO) [ThreadId 4] P2P < DATA 20
|
||||
[2022-11-28 14:54:11.148739395] (P2P.IO) [ssh connection Just 390399] [ThreadId 4] P2P > VALID
|
||||
^M100% 20 B 2 KiB/s 0s[2022-11-28 14:54:11.152428942] (P2P.IO) [ThreadId 4] P2P < VALID
|
||||
...
|
||||
[2022-11-28 14:54:11.177599567] (P2P.IO) [ThreadId 4] P2P > SUCCESS
|
||||
[2022-11-28 14:54:11.177858296] (P2P.IO) [ssh connection Just 390399] [ThreadId 4] P2P < SUCCESS
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2022-11-29T21:45:15Z"
|
||||
content="""
|
||||
I examined this situation on the machine. I had to ctrl-z the process to
|
||||
get a shell.
|
||||
|
||||
Then I checked what the stdin of 3710605 was, and it was still
|
||||
open, and was a pipe from 3710526. So that's why the p2pstdio process
|
||||
was still running; that pipe should be closed when the parent git-annex
|
||||
is done, but the parent was apparently stuck on something.
|
||||
|
||||
Then I straced 3710605 but it was suspended (oops). So I ran `fg`. This
|
||||
somehow unstuck everything! The test suite finished up very fast.
|
||||
|
||||
Here is what it output for the test that had gotten stuck:
|
||||
|
||||
Tests
|
||||
Repo Tests v8 unlocked
|
||||
Init Tests
|
||||
init: OK (2.54s)
|
||||
add: OK (5.48s)
|
||||
move (ssh remote): FAIL (958561.16s)
|
||||
./Test/Framework.hs:398:
|
||||
bad content in location log for foo key SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77 uuid UUID "c129397d-8209-40ea-8347-16a8c3fe69de"
|
||||
expected: True
|
||||
but got: False
|
||||
|
||||
That seems like it must come from here in the test suite:
|
||||
|
||||
git_annex "move" ["--to", "origin", annexedfile] "move --to of file"
|
||||
inmainrepo $ annexed_present annexedfile
|
||||
|
||||
So it seems that the content was moved back to origin successfully
|
||||
(`annexed_present` checks that the object file is present before checking
|
||||
the location log), but that the location log didn't get updated. Need
|
||||
to check if that update would have been done by the `p2pstdio` process
|
||||
or the `move` process.
|
||||
|
||||
Why would a SIGCONT have unstuck it I wonder?
|
||||
|
||||
I have re-ran the command to see if the bug replicates..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 3"
|
||||
date="2022-11-29T22:25:42Z"
|
||||
content="""
|
||||
> I have re-ran the command to see if the bug replicates..
|
||||
|
||||
note: it was considerable amount of time (days?) for it to take there ;) I made a copy `test_fs_testonly.py` where I removed all other \"benchmarks\" prior running `annex test` -- might get there faster if works so feel free to interrupt and rerun that one. That code is old (circa 2014 ;)) , and I should do some face lift but haven't had a chance yet :-/
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2022-12-05T17:48:52Z"
|
||||
content="""
|
||||
The rerun did not get stuck. So it's not something directly the fault
|
||||
of the script running `git-annex test`, I think.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 5"
|
||||
date="2022-12-05T18:08:02Z"
|
||||
content="""
|
||||
what kind of a \"fault\" could it be? ;)
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 6"""
|
||||
date="2022-12-07T16:36:44Z"
|
||||
content="""
|
||||
Well, a specific filesystem or mount option that the script sets up, for
|
||||
example.
|
||||
|
||||
Or if it's something of that kind, it's also intermittent.
|
||||
|
||||
This will limit the test suite to only running the that test:
|
||||
|
||||
git-annex test --pattern '/move (ssh remote)/ && /Repo Tests v8 unlocked/'
|
||||
|
||||
I have that repeating locally in a loop, in case it's just something that
|
||||
happens every 1000 test runs or something like that.
|
||||
|
||||
(I've patched git-annex to speed up the above command 400%.)
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,114 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 7"""
|
||||
date="2022-12-07T18:19:19Z"
|
||||
content="""
|
||||
Very good news! Reproduced the hang on kite, after 500 iterations of test
|
||||
suite.
|
||||
|
||||
All 0 tests passed (0.01s)
|
||||
Tests
|
||||
Repo Tests v10 adjusted unlocked branch
|
||||
Init Tests
|
||||
init: OK (0.44s)
|
||||
add: OK (0.81s)
|
||||
move (ssh remote):
|
||||
|
||||
strace of the `git-annex move` process:
|
||||
|
||||
joey@kite:~/tmp>strace -p 1710605 -f
|
||||
strace: Process 1710605 attached with 6 threads
|
||||
[pid 1710671] futex(0x7fab6801c398, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
|
||||
[pid 1710614] futex(0x5a21968, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
|
||||
[pid 1710613] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
|
||||
[pid 1710611] futex(0x4868648, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
|
||||
[pid 1710610] futex(0x4868648, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
|
||||
[pid 1710605] futex(0x5a2164c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
|
||||
[pid 1710611] <... futex resumed>) = 0
|
||||
[pid 1710610] <... futex resumed>) = -1 EAGAIN (Resource temporarily unavailable)
|
||||
[pid 1710611] epoll_wait(3, <unfinished ...>
|
||||
[pid 1710610] futex(0x48685e0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
|
||||
[pid 1710611] <... epoll_wait resumed>[], 64, 0) = 0
|
||||
[pid 1710610] <... futex resumed>) = 0
|
||||
[pid 1710611] epoll_wait(3, <unfinished ...>
|
||||
[pid 1710610] read(10, <unfinished ...>
|
||||
[pid 1710611] <... epoll_wait resumed>[], 64, 0) = 0
|
||||
[pid 1710610] <... read resumed>"\344\16\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710611] epoll_wait(3, <unfinished ...>
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] write(9, "\377\0\0\0\0\0\0\0", 8 <unfinished ...>
|
||||
[pid 1710613] <... restart_syscall resumed>) = 1
|
||||
[pid 1710610] <... write resumed>) = 8
|
||||
[pid 1710613] read(9, <unfinished ...>
|
||||
[pid 1710610] read(10, <unfinished ...>
|
||||
[pid 1710613] <... read resumed>"\377\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710613] futex(0x5a21968, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
|
||||
[pid 1710614] <... futex resumed>) = 0
|
||||
[pid 1710613] <... futex resumed>) = 1
|
||||
[pid 1710613] poll([{fd=7, events=POLLIN}, {fd=9, events=POLLIN}], 2, -1 <unfinished ...>
|
||||
[pid 1710614] futex(0x5a21970, FUTEX_WAKE_PRIVATE, 1) = 0
|
||||
[pid 1710614] sched_getaffinity(0, 128, [0, 1]) = 8
|
||||
[pid 1710614] rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM CONT TSTP], [], 8) = 0
|
||||
[pid 1710610] <... read resumed>"\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] read(10, <unfinished ...>
|
||||
[pid 1710614] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
|
||||
[pid 1710614] futex(0x5a2196c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
|
||||
[pid 1710610] <... read resumed>"\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710610] futex(0x486864c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY
|
||||
|
||||
Strace of the child:
|
||||
|
||||
joey@kite:~/tmp>strace -p 1710650 -f
|
||||
strace: Process 1710650 attached with 6 threads
|
||||
[pid 1710658] futex(0x51ef9ec, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
|
||||
[pid 1710655] futex(0x51ef7ec, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
|
||||
[pid 1710653] read(10, <unfinished ...>
|
||||
[pid 1710650] fcntl(12, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0} <unfinished ...>
|
||||
[pid 1710673] futex(0x7f6ec001c3e8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
|
||||
[pid 1710657] futex(0x7f6ed0000bf8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
|
||||
[pid 1710653] <... read resumed>"\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
[pid 1710653] read(10, "\1\0\0\0\0\0\0\0", 8) = 8
|
||||
|
||||
Which looks like it's blocked on taking a lock of
|
||||
`/home/joey/tmp/.t/3/main0/.git/annex/restage.lck`
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 8"""
|
||||
date="2022-12-07T20:55:35Z"
|
||||
content="""
|
||||
Oh interesting, the same process has restage.lck open twice.
|
||||
|
||||
lrwx------ 1 joey joey 64 Dec 7 14:20 /proc/1710650/fd/12 -> /home/joey/tmp/.t/3/main0/.git/annex/restage.lck
|
||||
lrwx------ 1 joey joey 64 Dec 7 16:53 /proc/1710650/fd/16 -> /home/joey/tmp/.t/3/main0/.git/annex/restage.lck
|
||||
|
||||
There are other processes that have the same restage.lck open too.
|
||||
In the process tree below, pid 1710382 does, and so does pid 1710483.
|
||||
|
||||
1710368 pts/8 S+ 0:00 sh -c git-annex test --fakessh -- 'localhost' 'git-annex-shell '"'"'p2pstdio'"'"' '"'"'/home/joey/tmp/.t/3/main0'"'"' '"'"'6e958d13-c3cb-4c78-a360-e14308368520'"'"' --uuid 3a4d8ec8-3f33-4b80-bd67-119b1c65fc88'
|
||||
1710369 pts/8 Sl+ 0:00 \_ git-annex test --fakessh -- localhost git-annex-shell 'p2pstdio' '/home/joey/tmp/.t/3/main0' '6e958d13-c3cb-4c78-a360-e14308368520' --uuid 3a4d8ec8-3f33-4b80-bd67-119b1c65fc88
|
||||
1710380 pts/8 S+ 0:00 \_ /bin/sh -c git-annex-shell 'p2pstdio' '/home/joey/tmp/.t/3/main0' '6e958d13-c3cb-4c78-a360-e14308368520' --uuid 3a4d8ec8-3f33-4b80-bd67-119b1c65fc88
|
||||
1710382 pts/8 Sl+ 0:00 \_ git-annex-shell p2pstdio /home/joey/tmp/.t/3/main0 6e958d13-c3cb-4c78-a360-e14308368520 --uuid 3a4d8ec8-3f33-4b80-bd67-119b1c65fc88
|
||||
1710411 pts/8 S+ 0:00 \_ git --git-dir=/home/joey/tmp/.t/3/main0/.git --work-tree=/home/joey/tmp/.t/3/main0 --literal-pathspecs cat-file --batch
|
||||
1710476 pts/8 S+ 0:00 \_ git --git-dir=/home/joey/tmp/.t/3/main0/.git --work-tree=/home/joey/tmp/.t/3/main0 --literal-pathspecs -c core.safecrlf=false update-index -q --refresh -z --stdin
|
||||
1710479 pts/8 S+ 0:00 \_ /bin/sh -c git-annex filter-process git-annex filter-process
|
||||
1710483 pts/8 Sl+ 0:21 \_ git-annex filter-process
|
||||
1710504 pts/8 S+ 0:00 \_ git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize)
|
||||
1710506 pts/8 S+ 0:00 \_ git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
|
||||
1710533 pts/8 S+ 0:00 \_ git --git-dir=.git --work-tree=. --literal-pathspecs hash-object -w --stdin-paths --no-filters
|
||||
1710540 pts/8 S+ 0:00 \_ git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
|
||||
|
||||
64 Dec 7 17:01 /proc/1710382/fd/13 -> /home/joey/tmp/.t/3/main0/.git/annex/restage.lck
|
||||
64 Dec 7 16:53 /proc/1710483/fd/11 -> /home/joey/tmp/.t/3/main0/.git/annex/restage.lck
|
||||
64 Dec 7 16:53 /proc/1710483/fd/19 -> /home/joey/tmp/.t/3/main0/.git/annex/restage.lck
|
||||
|
||||
(Strange that pid 1710368 is not a child process of the main git-annex test
|
||||
run. I'm unsure why that happens. )
|
||||
|
||||
Stracing 1710483, it is also stuck on `fcntl F_SETLKW`.
|
||||
This looks like it may be the main deadlock, because the parent is waiting on
|
||||
`git update-index`, which is waiting on `git-annex filter-process`, which is
|
||||
stuck waiting to take the lock... which its parent is holding!
|
||||
|
||||
(There are also several other process trees, running `git-annex test --fakessh`,
|
||||
that have gotten deparented from the main test run, and are hanging around.
|
||||
Those also have restage.lck open, though in their cases it is a since-deleted
|
||||
file.)
|
||||
|
||||
I think this is a bug in restagePointerFiles, which uses
|
||||
streamRestageLog while refreshIndex is running.
|
||||
The restage.log was added 2 months ago, in
|
||||
[[!commit 6a3bd283b8af53f810982e002e435c0d7c040c59]].
|
||||
|
||||
I suspect that `git-annex filter-process` is running populatePointerFile via
|
||||
Command.Smudge.getMoveRaceRecovery, which needs to write to the restage.log.
|
||||
That would explain why the deadlock only happens sometimes, because it
|
||||
involves a race that triggers that code.
|
||||
|
||||
Reading the whole
|
||||
restage.log into memory first would probably avoid the bug, but is not ideal.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 9"""
|
||||
date="2022-12-08T18:07:47Z"
|
||||
content="""
|
||||
I've developed a fix. Gonna re-run the reproducer long enough to be sure
|
||||
the bug is squashed..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
### Please describe the problem.
|
||||
|
||||
While running tests on a local HPC with
|
||||
|
||||
```
|
||||
(datalad) [discovery7 tmp]$ cat /etc/redhat-release
|
||||
CentOS Linux release 7.9.2009 (Core)
|
||||
|
||||
(datalad) [discovery7 tmp]$ gpgconf --version
|
||||
gpgconf (GnuPG) 2.0.22
|
||||
Copyright (C) 2013 Free Software Foundation, Inc.
|
||||
|
||||
```
|
||||
|
||||
3 tests (also failed in [another more "prolific in fails" run](https://git-annex.branchable.com/bugs/__34__357_out_of_984_tests_failed__34___on_NFS_lustre_mount/) failed due to
|
||||
|
||||
```
|
||||
(datalad) [d31548v@discovery7 tmp]$ grep -B1 FAIL annex-8.20210803-g99bb214-1.log
|
||||
gpgconf: invalid option "--kill"
|
||||
FAIL (18.37s)
|
||||
--
|
||||
gpgconf: invalid option "--kill"
|
||||
FAIL (16.43s)
|
||||
--
|
||||
gpgconf: invalid option "--kill"
|
||||
FAIL (17.07s)
|
||||
```
|
||||
|
||||
Not sure if you would like to do anything on git-annex level (skip those tests on outdated etc)...
|
||||
Meanwhile - adding `gnupg >=2.1.1 ` dependency to conda build: https://github.com/conda-forge/git-annex-feedstock/pull/126
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2021-08-24T15:54:15Z"
|
||||
content="""
|
||||
It already ignores nonzero exit status of that command, since
|
||||
old gpg's don't support it.
|
||||
|
||||
So, it must not really be the `gpgconf --kill` that is causing the test
|
||||
failure.
|
||||
|
||||
Please post the actual test failure message.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="more info"
|
||||
date="2021-08-24T16:35:26Z"
|
||||
content="""
|
||||
full log is available [for another issue](https://git-annex.branchable.com/bugs/__34__357_out_of_984_tests_failed__34___on_NFS_lustre_mount/) [http://www.onerussian.com/tmp/git-annex-noretry+pidlock1-1.log](http://www.onerussian.com/tmp/git-annex-noretry+pidlock1-1.log) which shows the same error as well.
|
||||
|
||||
FWIW
|
||||
|
||||
```
|
||||
[d31548v@discovery7 ~]$ /usr/bin/gpgconf --kill
|
||||
gpgconf: invalid option \"--kill\"
|
||||
[d31548v@discovery7 ~]$ echo $?
|
||||
2
|
||||
```
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2021-10-12T16:42:02Z"
|
||||
content="""
|
||||
From the log:
|
||||
|
||||
crypto: gpgconf: invalid option "--kill"
|
||||
gpgconf: invalid option "--kill"
|
||||
FAIL (22.13s)
|
||||
./Test/Framework.hs:57:
|
||||
copy --to encrypted remote failed (transcript follows)
|
||||
copy foo (to foo...)
|
||||
gpg: can't connect to the agent: Invalid value passed to IPC
|
||||
gpg: problem with the agent: No agent running
|
||||
|
||||
That is not really be a problem with gpgconf --kill, but a problem
|
||||
talking to gpg-agent.
|
||||
|
||||
The same crypto test fails a couple more times in that log, like this:
|
||||
|
||||
crypto: gpgconf: invalid option "--kill"
|
||||
gpgconf: invalid option "--kill"
|
||||
FAIL (12.00s)
|
||||
./Test/Framework.hs:57:
|
||||
get of file failed (transcript follows)
|
||||
get foo (not available)
|
||||
No other repository is known to contain the file.
|
||||
failed
|
||||
get: 1 failed
|
||||
|
||||
That is also not a problem with gpgconf --kill, it's actually due to an
|
||||
earlier test failure, unrelated to this. That earlier failure was
|
||||
the one [the other issue](https://git-annex.branchable.com/bugs/__34__357_out_of_984_tests_failed__34___on_NFS_lustre_mount/)
|
||||
was about, which has since been fixed. So we can ignore these I think,
|
||||
leaving only the one above as an unexplained failure.
|
||||
|
||||
"gpg: can't connect to the agent: Invalid value passed to IPC" could
|
||||
be some kind of gpg bug. I found some other instances of gpg failing that way.
|
||||
One involved using --homedir (similar to the test suite's
|
||||
use of GNUPGHOME) but on windows.
|
||||
<https://lists.gnupg.org/pipermail/gnupg-users/2016-October/056817.html>
|
||||
And here's another one, in WSL when apt runs
|
||||
gpg. <https://github.com/microsoft/WSL/issues/5125>
|
||||
|
||||
Perhaps this is a problem with the location of the gpg agent socket in the
|
||||
filesystem that git-annex test is running in. That somehow messes up not
|
||||
creation of that socket, but later use of it. It seems that the earlier
|
||||
self-test of the test harness did not trigger the problem though, which is
|
||||
odd because it sets up a gpg private key and I'd think would use the agent
|
||||
too.
|
||||
|
||||
In [[!commit b426ff682570d8600dc8025bbcd20aa95819a7e4]] I considered
|
||||
putting the gpg directory inside the system temp dir, which would perhaps
|
||||
avoid the problem here. I've made that change.
|
||||
|
||||
Please test a fresh build on this system again, if you can..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2022-06-02T16:23:44Z"
|
||||
content="""
|
||||
Any chance you can try to reproduce this with a recent build of git-annex?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 5"""
|
||||
date="2022-06-03T18:00:16Z"
|
||||
content="""
|
||||
Ok, that is behaving as expected then, it ignores failure to support older
|
||||
gpgs. Going to close this.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="still reports about invalid option but doesn't fail"
|
||||
date="2022-06-03T01:36:38Z"
|
||||
content="""
|
||||
so I should try with outdated gpgconf on some older CentOS? ..already done - we do run annex tests daily on ndoli (part of discovery) now and apparently: tests passed, but it did have those reports
|
||||
|
||||
```
|
||||
(git)smaug:/mnt/datasets/datalad/ci/git-annex-ci-client-jobs/builds/2022/06[master]git
|
||||
$> tail result-ndoli-715/handle-result.yaml-154-7a381a8a-success/result-ndoli-715/git-annex-tmp.rc
|
||||
0
|
||||
|
||||
$> tail result-ndoli-715/handle-result.yaml-154-7a381a8a-success/result-ndoli-715/git-annex-tmp.log
|
||||
gpgconf: invalid option \"--kill\"
|
||||
gpgconf: invalid option \"--kill\"
|
||||
gpgconf: invalid option \"--kill\"
|
||||
gpgconf: invalid option \"--kill\"
|
||||
gpgconf: invalid option \"--kill\"
|
||||
+ echo 'Elapsed time: 3278 seconds'
|
||||
Elapsed time: 3278 seconds
|
||||
+ cd /dartfs/rc/lab/D/DBIC/DBIC/archive/tmp
|
||||
+ chmod -R +w 715
|
||||
+ rm -rf 715
|
||||
```
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
I was excited to give a shot to https://github.com/OpenNeuroDatasets/ds001544/ which has proper publicurl set etc... unfortunately there is something forbidding immediate parallel get (in below example `-J8` but it is the same with `-J2`). It works on the 2nd try, or if not using -J. Error message states "unknown export location":
|
||||
|
||||
[[!format sh """
|
||||
$> git annex version | head -n 1; git clone https://github.com/OpenNeuroDatasets/ds001544/ ; cd ds001544; git annex enableremote s3-PUBLIC; git annex get -J8 . 2>&1 | head -n 20; echo "2nd run"; git annex get -J8 .
|
||||
git-annex version: 6.20181011+git7-g373c2abc2-1~ndall+1
|
||||
Cloning into 'ds001544'...
|
||||
remote: Enumerating objects: 866, done.
|
||||
remote: Counting objects: 100% (866/866), done.
|
||||
remote: Compressing objects: 100% (483/483), done.
|
||||
remote: Total 866 (delta 144), reused 863 (delta 141), pack-reused 0
|
||||
Receiving objects: 100% (866/866), 75.07 KiB | 4.69 MiB/s, done.
|
||||
Resolving deltas: 100% (144/144), done.
|
||||
|
||||
(merging origin/git-annex into git-annex...)
|
||||
(recording state in git...)
|
||||
|
||||
Remote origin not usable by git-annex; setting annex-ignore
|
||||
enableremote s3-PUBLIC ok
|
||||
(recording state in git...)
|
||||
get code/convert_sub01_ses01.R (from s3-PUBLIC...)
|
||||
unknown export location
|
||||
|
||||
Unable to access these remotes: s3-PUBLIC
|
||||
|
||||
Try making some of these repositories available:
|
||||
1c66b8f9-34c7-42d1-8e9f-d7bc1982311a -- root@460f24a504cc:/datalad/ds001544
|
||||
837e28c7-9e4a-4792-b1b1-aa69d3430a42 -- [s3-PUBLIC]
|
||||
a7294efc-f620-445d-8e9d-803b3ec748ef -- s3-PRIVATE
|
||||
|
||||
(Note that these git remotes have annex-ignore set: origin)
|
||||
failed
|
||||
get sub-01/ses-01/fmap/sub-01_ses-01_acq-cf0PA_run-03_epi.nii.gz (from s3-PUBLIC...)
|
||||
unknown export location
|
||||
|
||||
Unable to access these remotes: s3-PUBLIC
|
||||
|
||||
Try making some of these repositories available:
|
||||
1c66b8f9-34c7-42d1-8e9f-d7bc1982311a -- root@460f24a504cc:/datalad/ds001544
|
||||
837e28c7-9e4a-4792-b1b1-aa69d3430a42 -- [s3-PUBLIC]
|
||||
2nd run
|
||||
get code/convert_sub01_ses02.R (from s3-PUBLIC...) (checksum...) ok
|
||||
get code/convert_sub01_ses01.R (from s3-PUBLIC...) (checksum...) ok
|
||||
get sub-01/ses-01/fmap/sub-01_ses-01_acq-cf1PA_run-02_epi.nii.gz (from s3-PUBLIC...) (checksum...) ok
|
||||
get sub-01/ses-01/func/sub-01_ses-01_task-Stroop_acq-cf0AP_run-03_physio.tsv.gz (from s3-PUBLIC...) (checksum...) ok
|
||||
...
|
||||
"""]]
|
||||
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2018-10-22T16:37:06Z"
|
||||
content="""
|
||||
Seems that the problem is that updateExportTreeFromLog gets started by the
|
||||
first thread, and once it's started it won't be run again. Meanwhile,
|
||||
other threads try to access the export database that populates.
|
||||
|
||||
So, just needs some inter-thread locking..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
### Please describe the problem.
|
||||
|
||||
Originally reported (so more details could be found there) in [datalad issue #2988].
|
||||
|
||||
While trying to `annex get` on local cluster, over ssh without a ssh-agent, I was prompted for password multiple times, although previous version didn't do it:
|
||||
|
||||
[[!format sh """
|
||||
[yhalchen@discovery7 QA]$ git annex version | head -n1 ; git annex get -J2 sub-qa/
|
||||
git-annex version: 6.20180926-gc906aaf
|
||||
get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2Xs4_dwi.nii.gz get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2_dwi.nii.gz (from origin...)
|
||||
(from origin...)
|
||||
yoh@falkor.datalad.org's password: yoh@falkor.datalad.org's password:
|
||||
"""]]
|
||||
|
||||
|
||||
and when I try an "imputated" from NeuroDebian .deb package recent build of git-annex then I am observing
|
||||
|
||||
[[!format sh """
|
||||
[yhalchen@discovery7 QA]$ export PATH=/ihome/yhalchen/7.20181105+git22-g4c7236c58/usr/lib/git-annex.linux:$PATH
|
||||
[yhalchen@discovery7 QA]$ git annex version | head -n1 ; git annex get -J2 sub-qa/
|
||||
git-annex version: 7.20181105+git22-g4c7236c58-1~ndall+1
|
||||
get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2Xs4_dwi.nii.gz get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2_dwi.nii.gz (transfer already in progress, or unable to take transfer lock)
|
||||
Unable to access these remotes: origin
|
||||
|
||||
Try making some of these repositories available:
|
||||
6384a551-a41d-4290-b186-9258befede97 -- bids@rolando:/inbox/BIDS/dbic/QA
|
||||
7d9ed214-3e5f-4cc8-ac88-f397145b2d4c -- yoh@falkor:/srv/datasets.datalad.org/www/dbic/QA [origin]
|
||||
ba8f2cea-f229-422c-82be-6580e5e07ed5 -- yoh@smaug:/mnt/datasets/datalad/crawl/dbic/QA
|
||||
failed
|
||||
get sub-qa/ses-20161128/func/sub-qa_ses-20161128_task-rest_acq-p2Xs4X35mm_bold.nii.gz (from origin...)
|
||||
(from origin...)
|
||||
|
||||
|
||||
thread blocked indefinitely in an MVar operation
|
||||
thread blocked indefinitely in an STM transaction
|
||||
|
||||
Unable to access these remotes: origin
|
||||
|
||||
Unable to access these remotes: origin
|
||||
|
||||
Try making some of these repositories available:
|
||||
6384a551-a41d-4290-b186-9258befede97 -- bids@rolando:/inbox/BIDS/dbic/QA
|
||||
7d9ed214-3e5f-4cc8-ac88-f397145b2d4c -- yoh@falkor:/srv/datasets.datalad.org/www/dbic/QA [origin]
|
||||
ba8f2cea-f229-422c-82be-6580e5e07ed5 -- yoh@smaug:/mnt/datasets/datalad/crawl/dbic/QA
|
||||
failed
|
||||
|
||||
Try making some of these repositories available:
|
||||
6384a551-a41d-4290-b186-9258befede97 -- bids@rolando:/inbox/BIDS/dbic/QA
|
||||
7d9ed214-3e5f-4cc8-ac88-f397145b2d4c -- yoh@falkor:/srv/datasets.datalad.org/www/dbic/QA [origin]
|
||||
ba8f2cea-f229-422c-82be-6580e5e07ed5 -- yoh@smaug:/mnt/datasets/datalad/crawl/dbic/QA
|
||||
failed
|
||||
git-annex: thread blocked indefinitely in an STM transaction
|
||||
"""]]
|
||||
so smells like may be recent fixes caused some other locking problems
|
||||
|
||||
This is on an NFS mounted partition, but remains the same when run under `/tmp`
|
||||
|
||||
Running a significantly more outdated version I found laying around (6.20170815+gitg22da64d0f-1~ndall+1) shows a single password prompt
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 10"""
|
||||
date="2018-11-13T17:50:48Z"
|
||||
content="""
|
||||
Use the script command to get a typescript.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 11"""
|
||||
date="2018-11-13T18:04:15Z"
|
||||
content="""
|
||||
Since you can reproduce it, why don't you bisect it?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 12"
|
||||
date="2018-11-13T20:02:18Z"
|
||||
content="""
|
||||
bisection would require building, building would require build environment (which is not there), etc... and need to get to grant writing/reporting/etc...
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 13"""
|
||||
date="2018-11-15T17:06:39Z"
|
||||
content="""
|
||||
I don't think this bug will be able to be tracked down without
|
||||
rebuilding git-annex one way or another.
|
||||
|
||||
I could use the technique in
|
||||
<https://www.fpcomplete.com/blog/2018/05/pinpointing-deadlocks-in-haskell>
|
||||
to instrument every possible place in git-annex that could deadlock.
|
||||
This would be a lot of work, probably hundreds of lines of code changes.
|
||||
|
||||
It seems much easier to either find a way to reproduce the problem
|
||||
outside of this one machine with the problem, or bisect it.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 14"
|
||||
date="2018-11-15T18:06:10Z"
|
||||
content="""
|
||||
about to start bisection for the locking issue. Relevant PR on datalad: https://github.com/datalad/datalad/pull/2995 to get a helper script and Singularity environment to build
|
||||
|
||||
As for double password prompt - should I bisect too? that one seems to be not that easy since if `annex get` is invoked within script, typescript records nothing:
|
||||
[[!format sh \"\"\"
|
||||
[yhalchen@discovery7 ~]$ cat bisect-git-annex-doublepasswd.sh
|
||||
#!/bin/bash
|
||||
set -eu
|
||||
|
||||
err=\"thread blocked indefinitely\"
|
||||
|
||||
cd ~/QA
|
||||
# script doesn't work in a script since probably no tty
|
||||
timeout 10 script -f -c 'git annex get -J2 sub-*' || :
|
||||
test 1 -eq `sed -e 's, ,\n,g' typescript | grep -c 'password:' `
|
||||
[yhalchen@discovery7 ~]$ bash -x bisect-git-annex-doublepasswd.sh
|
||||
+ set -eu
|
||||
+ err='thread blocked indefinitely'
|
||||
+ cd /ihome/yhalchen/QA
|
||||
+ timeout 10 script -f -c 'git annex get -J2 sub-*'
|
||||
Script started, file is typescript
|
||||
+ :
|
||||
++ sed -e 's, ,\n,g' typescript
|
||||
++ grep -c password:
|
||||
+ test 1 -eq 0
|
||||
\"\"\"]]
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2018-11-15T18:33:11Z"
|
||||
content="""
|
||||
The problems at least so far appear to be related; it should be enough
|
||||
to bisect the easier one to deal with.
|
||||
|
||||
BTW, I noticed a couple of weird things that may or may not be related.
|
||||
Your repository is on NFS but annex.pidlock is not set, which I guess means
|
||||
that NFS is providing working real locking, or perhaps subtly broken real
|
||||
locking. And, the test suite failed with permission denied when writing a
|
||||
file, which is very strange.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 14"""
|
||||
date="2018-11-15T18:28:58Z"
|
||||
content="""
|
||||
[[!commit 872af2b2f1049e4eecf274ac70caf99a367f3818]] allows ruling out
|
||||
a concurrent-output bug. If git-annex with that commit is run with --quiet
|
||||
and still exhibits the problem, it shouldn't involve concurrent-output at
|
||||
all.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 16"
|
||||
date="2018-11-15T20:43:27Z"
|
||||
content="""
|
||||
woohoo `git bisect run ~/bisect-git-annex ~/git-annex-dev.img ~/bisect-git-annex-lock.sh` (on `git bisect start HEAD 6.20180926` with HEAD being 05bfce7ca8e67612c66e2cdb3c7c0c26fd8c5243 finished with ground breaking discovery
|
||||
|
||||
[[!format sh \"\"\"
|
||||
2c0300c8b9cd97b7d2a86e48e19283be8c9ef278 is the first bad commit
|
||||
commit 2c0300c8b9cd97b7d2a86e48e19283be8c9ef278
|
||||
Author: Ilya_Shlyakhter <Ilya_Shlyakhter@web>
|
||||
Date: Wed Sep 26 17:16:13 2018 +0000
|
||||
|
||||
Added a comment: question about special remote protocol
|
||||
|
||||
:040000 040000 6e8d46c0b393e4b96ff53f28224154939fbe4883 e9db3cc46114d98fe77bf0392d8a90c314c9326e M doc
|
||||
bisect run success
|
||||
\"\"\"]]
|
||||
so it is all destructive Ilya's comments! ;)
|
||||
|
||||
That is a commit right after the 6.20180926 which I chose as \"good\" since the good was the 6.20180926-gc906aaf (installed via conda build pl5.22.2.1_0).
|
||||
So may be the problem is not in git annex per se but in its build-depends, which changed in buster (I've been building standalones with buster for a while).
|
||||
I will try to do manual comparison across a number of builds from datalad.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 17"
|
||||
date="2018-11-16T22:12:58Z"
|
||||
content="""
|
||||
ok, not sure yet if it is git-annex change or ghc change or both here is the point where things changed radically:
|
||||
|
||||
going from 6.20180519+gitgf6f199be3 build which used ghc 8.0.1-17+b1 and where we had a nice single ssh password prompt:
|
||||
|
||||
[[!format sh \"\"\"
|
||||
[yhalchen@discovery7 QA]$ PATH=$HOME/6.20180519+gitgf6f199be3-1~ndall+1/usr/lib/git-annex.linux:$PATH git annex get -J2 sub-*
|
||||
get sub-amit/ses-20180508/anat/sub-amit_ses-20180508_acq-MPRAGE_T1w.nii.gz get sub-amit/ses-20180508/dwi/sub-amit_ses-20180508_acq-DTIX30Xp2Xs4_dwi.nii.gz (from origin...)
|
||||
(from origin...)
|
||||
yoh@falkor.datalad.org's password:
|
||||
|
||||
\"\"\"]]
|
||||
|
||||
to the next build I have 6.20180521+gitg0e3bebd71 which used ghc 8.0.2-11 and with which locking madness starts to kick in:
|
||||
|
||||
[[!format sh \"\"\"
|
||||
[yhalchen@discovery7 QA]$ PATH=$HOME/6.20180521+gitg0e3bebd71-1~ndall+1/usr/lib/git-annex.linux:$PATH git annex get -J2 sub-*
|
||||
get sub-amit/ses-20180508/anat/sub-amit_ses-20180508_acq-MPRAGE_T1w.nii.gz get sub-amit/ses-20180508/dwi/sub-amit_ses-20180508_acq-DTIX30Xp2Xs4_dwi.nii.gz (transfer already in progress, or unable to take transfer lock)
|
||||
Unable to access these remotes: origin
|
||||
|
||||
Try making some of these repositories available:
|
||||
6384a551-a41d-4290-b186-9258befede97 -- bids@rolando:/inbox/BIDS/dbic/QA
|
||||
7d9ed214-3e5f-4cc8-ac88-f397145b2d4c -- yoh@falkor:/srv/datasets.datalad.org/www/dbic/QA [origin]
|
||||
ba8f2cea-f229-422c-82be-6580e5e07ed5 -- yoh@smaug:/mnt/datasets/datalad/crawl/dbic/QA
|
||||
failed
|
||||
get sub-amit/ses-20180508/func/sub-amit_ses-20180508_task-rest_acq-p2_bold.nii.gz (from origin...)
|
||||
(from origin...)
|
||||
|
||||
|
||||
thread blocked indefinitely in an STM transaction
|
||||
thread blocked indefinitely in an MVar operation
|
||||
|
||||
Unable to access these remotes: origin
|
||||
|
||||
Unable to access these remotes: origin
|
||||
|
||||
Try making some of these repositories available:
|
||||
6384a551-a41d-4290-b186-9258befede97 -- bids@rolando:/inbox/BIDS/dbic/QA
|
||||
7d9ed214-3e5f-4cc8-ac88-f397145b2d4c -- yoh@falkor:/srv/datasets.datalad.org/www/dbic/QA [origin]
|
||||
ba8f2cea-f229-422c-82be-6580e5e07ed5 -- yoh@smaug:/mnt/datasets/datalad/crawl/dbic/QA
|
||||
failed
|
||||
|
||||
Try making some of these repositories available:
|
||||
6384a551-a41d-4290-b186-9258befede97 -- bids@rolando:/inbox/BIDS/dbic/QA
|
||||
7d9ed214-3e5f-4cc8-ac88-f397145b2d4c -- yoh@falkor:/srv/datasets.datalad.org/www/dbic/QA [origin]
|
||||
ba8f2cea-f229-422c-82be-6580e5e07ed5 -- yoh@smaug:/mnt/datasets/datalad/crawl/dbic/QA
|
||||
failed
|
||||
git-annex: thread blocked indefinitely in an STM transaction
|
||||
|
||||
\"\"\"]]
|
||||
|
||||
I hope that would give enough clues, Joey?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 18"
|
||||
date="2018-11-19T04:08:37Z"
|
||||
content="""
|
||||
FWIW 1, it was a switch from ghc 8.0.1-17+b1 because it was when (I guess) I switched from Debian stable (stretch) to testing (buster) build environment since build-dependencies were no longer satisfiable!
|
||||
|
||||
FWIW 2, I have now tested the build in debian unstable with ghc 8.4.3+dfsg1-4 -- the same issue with locks
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="dependencies information"
|
||||
date="2018-11-19T15:54:14Z"
|
||||
content="""
|
||||
not sure how I was blind that `git annex version` outputs dependencies information, so here is too nearby versions where
|
||||
|
||||
this one is good (no lock issues, but double password prompt):
|
||||
[[!format sh \"\"\"
|
||||
*[yhalchen@discovery7 QA]$ git annex version
|
||||
git-annex version: 6.20180926-gc906aaf
|
||||
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.17.1 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.7.0 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
local repository version: 5
|
||||
\"\"\"]]
|
||||
|
||||
and here is a bad one (locks issue)
|
||||
[[!format sh \"\"\"
|
||||
*[yhalchen@discovery7 QA]$ PATH=$HOME/6.20180926+git93-gdc47dfe/usr/lib/git-annex.linux:$PATH git annex version
|
||||
git-annex version: 6.20180926+git93-gdc47dfe90-1~ndall+1
|
||||
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.19 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.2 feed-1.0.0.0 ghc-8.2.2 http-client-0.5.12 persistent-sqlite-2.8.1.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
local repository version: 5
|
||||
\"\"\"]]
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
[[!comment format=mdwn
|
||||
username="andrew"
|
||||
avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435"
|
||||
subject="comment 1"
|
||||
date="2018-11-11T21:35:26Z"
|
||||
content="""
|
||||
This looks like a know issue: [get -J cannot be used with password-based authentication](http://git-annex.branchable.com/bugs/get_-J_cannot_be_used_with_password-based_authentication/).
|
||||
|
||||
Try without the -J2:
|
||||
|
||||
git annex version | head -n1 ; git annex get sub-qa/
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 20"""
|
||||
date="2018-11-19T17:04:18Z"
|
||||
content="""
|
||||
I don't see anything in ghc 8.0.2's release notes that points to a breaking
|
||||
change to STM/MVar handling. There were a couple of changes in that version
|
||||
that involve concurrency and MVars though, that could somehow have led to
|
||||
the problem.
|
||||
|
||||
Parallel programs should be significantly more reliable on platforms with
|
||||
weak memory consistency guarantees
|
||||
<https://ghc.haskell.org/trac/ghc/ticket/12469>
|
||||
|
||||
Scheduling bug with forkOS + MVar
|
||||
<https://ghc.haskell.org/trac/ghc/ticket/12419>
|
||||
|
||||
One thing that seems worth trying re the second of those
|
||||
is to edit CmdLine/Action.hs and delete the setNumCapabilities line:
|
||||
|
||||
--- a/CmdLine/Action.hs
|
||||
+++ b/CmdLine/Action.hs
|
||||
@@ -178,8 +178,6 @@ allowConcurrentOutput a = do
|
||||
where
|
||||
goconcurrent n = do
|
||||
c <- liftIO getNumCapabilities
|
||||
- when (n > c) $
|
||||
- liftIO $ setNumCapabilities n
|
||||
withMessageState $ \s -> case outputType s of
|
||||
NormalOutput -> ifM (liftIO concurrentOutputSupported)
|
||||
( Regions.displayConsoleRegions $
|
||||
|
||||
A compiler bug is looking not entirely unlikely..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 21"""
|
||||
date="2018-11-19T18:46:32Z"
|
||||
content="""
|
||||
I've added a DebugLocks build flag to try to track down the source
|
||||
of the deadlock. It's not enabled by default, so you'll need to eg modify
|
||||
the Makefile to set it:
|
||||
|
||||
cabal configure -fDebugLocks
|
||||
|
||||
Calls to `debugLocks` are scattered around in a several of the places I
|
||||
suspect may be involved, around ssh prompting, transfer locks, and
|
||||
general lock files. If one of them is, it will display line number
|
||||
information when the deadlock happens.
|
||||
|
||||
It will probably take several iterations of adding more calls to
|
||||
`debugLocks` to narrow in on the code that is involved in the
|
||||
deadlock.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 22"
|
||||
date="2018-11-19T19:25:35Z"
|
||||
content="""
|
||||
thanks! ... FYI: I am \"afraid\" that either I am completely lost my grip on things or may be bug was fixed somewhere in the recent days (within `05bfce7ca..900bf3436`)! bisecting now which change possibly fixed it
|
||||
"""]]
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue