move old fixed datalad/dandi/repronim bugs to the project pages
This is to cut down on the number of files in bugs/, which makes it slow to file new bug reports or update active bug reports. These old bugs were about 1/3rd of the files in there. These projects want lists of their old bugs to still be accessible, and have the lists on their project pages, which will still list the old bugs. Commands used: for f in $(git grep -l '\[\[!tag projects/dandi\]\]'); do if grep -q 'done\]\]' "$f"; then git mv "$f" ../projects/dandi/bugs-done; g=$(echo "$f" | sed 's/.mdwn//'); if [ -d "$g" ]; then git mv "$g" ../projects/dandi/bugs-done; fi; fi; done for f in $(git grep -l '\[\[!tag projects/repronim\]\]'); do if grep -q 'done\]\]' "$f"; then git mv "$f" ../projects/repronim/bugs-done; g=$(echo "$f" | sed 's/.mdwn//'); if [ -d "$g" ]; then git mv "$g" ../projects/repronim/bugs-done; fi; fi; done for f in $(git grep -l '\[\[!tag projects/datalad\]\]'); do if grep -q 'done\]\]' "$f"; then git mv "$f" ../projects/datalad/bugs-done; g=$(echo "$f" | sed 's/.mdwn//'); if [ -d "$g" ]; then git mv "$g" ../projects/datalad/bugs-done; fi; fi; done That assumes that bugs are not tagged by multiple projects at the same time. Of the ones I moved, I've checked and none are. Could do the same with todo/ but there are only 370 files in there, and less than 84 of them could be moved this way, which does not seem likely to produce a sizeable speedup. Sponsored-by: Dartmouth College's Datalad project
This commit is contained in:
parent
946fc20165
commit
bcc69f07e8
1011 changed files with 4 additions and 4 deletions
|
@ -1,69 +0,0 @@
|
|||
### Please describe the problem.
|
||||
|
||||
Identified while troubleshooting another [issue](https://git-annex.branchable.com/bugs/enableremote_stuck_with_a_recentish_git-annex/#comment-2116c5e109aaf39ffd62f3bdeeb14602)
|
||||
|
||||
[[!format sh """
|
||||
$> 'git-annex' 'enableremote' --debug -cremote.target1.blah=1 'target1'
|
||||
enableremote target1 ok
|
||||
|
||||
$> 'git-annex' 'enableremote' -cremote.target1.blah=1 --debug 'target1'
|
||||
[2020-02-26 14:46:47.789794028] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","show-ref","git-annex"]
|
||||
[2020-02-26 14:46:47.797917978] process done ExitSuccess
|
||||
[2020-02-26 14:46:47.798350533] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2020-02-26 14:46:47.802576899] process done ExitSuccess
|
||||
[2020-02-26 14:46:47.802884873] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","log","refs/heads/git-annex..b1ab0b11fbbc94ffd3d52adb7a0e93c3d45d8b52","--pretty=%H","-n1"]
|
||||
[2020-02-26 14:46:47.813289406] process done ExitSuccess
|
||||
[2020-02-26 14:46:47.815873454] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","cat-file","--batch"]
|
||||
[2020-02-26 14:46:47.818598891] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
|
||||
[2020-02-26 14:46:47.824657055] read: git ["config","--null","--list"]
|
||||
[2020-02-26 14:46:47.835897478] process done ExitSuccess
|
||||
enableremote target1 [2020-02-26 14:46:47.83652184] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","config","remote.target1.annex-ignore","false"]
|
||||
[2020-02-26 14:46:47.842277017] process done ExitSuccess
|
||||
[2020-02-26 14:46:47.842703576] read: git ["config","--null","--list"]
|
||||
[2020-02-26 14:46:47.853478328] process done ExitSuccess
|
||||
ok
|
||||
[2020-02-26 14:46:47.855317715] process done ExitSuccess
|
||||
[2020-02-26 14:46:47.856835556] process done ExitSuccess
|
||||
|
||||
"""]]
|
||||
|
||||
I consider it a bug since options shouldn't be order dependent, and even if they were -- `--debug` is listed before `-c` in `git annex enableremote --help`:
|
||||
|
||||
[[!format sh """
|
||||
$> git annex enableremote --help
|
||||
git-annex enableremote - enables git-annex to use a remote
|
||||
|
||||
Usage: git-annex enableremote [NAME K=V ...]
|
||||
|
||||
Available options:
|
||||
--force allow actions that may lose annexed data
|
||||
-F,--fast avoid slow operations
|
||||
-q,--quiet avoid verbose output
|
||||
-v,--verbose allow verbose output (default)
|
||||
-d,--debug show debug messages
|
||||
--no-debug don't show debug messages
|
||||
-b,--backend NAME specify key-value backend to use
|
||||
-N,--numcopies NUMBER override default number of copies
|
||||
--trust REMOTE override trust setting
|
||||
--semitrust REMOTE override trust setting back to default
|
||||
--untrust REMOTE override trust setting to untrusted
|
||||
-c,--config NAME=VALUE override git configuration setting
|
||||
--user-agent NAME override default User-Agent
|
||||
--trust-glacier Trust Amazon Glacier inventory
|
||||
--notify-finish show desktop notification after transfer finishes
|
||||
--notify-start show desktop notification after transfer starts
|
||||
-h,--help Show this help text
|
||||
|
||||
For details, run: git-annex help enableremote
|
||||
|
||||
$> git annex version
|
||||
git-annex version: 7.20190819+git2-g908476a9b-1~ndall+1
|
||||
|
||||
"""]]
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
|
||||
> fixed in [8.20200226-3-gc089f395b](https://git.kitenet.net/index.cgi/git-annex.git/commit/?id=c089f395b0c7d6416a3d4f2bf3211404acfd5b0e) --[[yarikoptic]]
|
|
@ -1,24 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2020-02-27T04:20:58Z"
|
||||
content="""
|
||||
-c uses adjustGitRepo which calls changeGitRepo, which
|
||||
re-extracts the GitConfig. --debug uses changeGitConfig which
|
||||
sets annexDebug in the GitConfig, which does not survive the changeGitRepo.
|
||||
|
||||
There might be a broader problem here, as changeGitRepo is also
|
||||
called by setConfig in many parts of the code. I think it narrowly
|
||||
escapes being a problem, because by the time a command is started,
|
||||
it's already enabled debug output, and so the GitConfig being reloaded
|
||||
doesn't disable debugging.
|
||||
|
||||
Other calls to changeGitConfig could also be a problem, if followed by
|
||||
an adjustGitRepo which loses those changes. There are only a few others,
|
||||
look probably ok, but this would be an easy gotcha to hit later.
|
||||
|
||||
So changeGitConfig needs to make a config change that persists across
|
||||
changeGitRepo.
|
||||
|
||||
Done.
|
||||
"""]]
|
|
@ -1,34 +0,0 @@
|
|||
### Please describe the problem.
|
||||
|
||||
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
|
||||
I am plowing through on making git-annex available within conda-forge "natively" for Windows. For now I just took the recently built installer, the one now available from [datasets.datalad.org](http://datasets.datalad.org/datalad/packages/windows/) and built on datalad-extensions github setup. I just extracted git-annex component from the installer and placed them within conda hierarchy (installed `posix` package with all the needed basic tools. Overall -- looks great, but:
|
||||
|
||||
[[!format sh """
|
||||
prop_view_roundtrips: FAIL (0.30s)
|
||||
*** Failed! Falsified (after 524 tests and 1 shrink):
|
||||
"a"
|
||||
MetaData (fromList [(MetaField "8",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet True) "\nD\EM",MetaValue (CurrentlySet True) "GO`!)",MetaValue (CurrentlySet False) "k\FS\CAN"]),(MetaField "dU",fromList [MetaValue (CurrentlySet True) "",MetaValue (CurrentlySet False) "\NUL44Vfm[\t",MetaValue (CurrentlySet True) "\nLMEgYc",MetaValue (CurrentlySet True) "\SO[",MetaValue (CurrentlySet True) "\FS\DC4\DLE\"3",MetaValue (CurrentlySet True) ";\f0&Wc\GS{^",MetaValue (CurrentlySet True) "D",MetaValue (CurrentlySet True) "c:"]),(MetaField "sV",fromList [MetaValue (CurrentlySet True) "",MetaValue (CurrentlySet False) "\STX8#w",MetaValue (CurrentlySet False) "\ny",MetaValue (CurrentlySet False) "\DC4qOq",MetaValue (CurrentlySet True) "\FSbqjq",MetaValue (CurrentlySet True) "T_bx%[lN",MetaValue (CurrentlySet True) "W0`",MetaValue (CurrentlySet True) "~ ueY"]),(MetaField "V",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet False) "\t\DC1~`\SOHv\DC1",MetaValue (CurrentlySet True) "\DLE3",MetaValue (CurrentlySet True) "/MZh$",MetaValue (CurrentlySet False) "0",MetaValue (CurrentlySet False) "MEulc",MetaValue (CurrentlySet True) "P5D",MetaValue (CurrentlySet True) "i|S,",MetaValue (CurrentlySet True) "x|C"])])
|
||||
True
|
||||
Use --quickcheck-replay=742853 to reproduce.
|
||||
"""]]
|
||||
|
||||
unfortunately I cannot tell from that output what could be the problem. Please let me know if hard to figure it out and I should provide access to such environment (ATM needs effort, so I do not want to spend time on that unless "no other way")
|
||||
|
||||
And it seems it might be a flaky test -- I started another run, it is still running but I this test did not fail
|
||||
|
||||
```
|
||||
$ grep prop_view_roundtrips git-annex-test-miniconda*.log
|
||||
git-annex-test-miniconda-2.log: prop_view_roundtrips: OK (2.51s)
|
||||
git-annex-test-miniconda.log: prop_view_roundtrips: FAIL (0.30s)
|
||||
```
|
||||
|
||||
Cheers,
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,17 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2020-10-26T16:03:40Z"
|
||||
content="""
|
||||
I can reproduce it in a windows VM running
|
||||
`git-annex test --quickcheck-replay=742853`
|
||||
|
||||
These quickcheck tests test random input so not flaky exactly.
|
||||
|
||||
Does not happen with that seed on linux, so it probably involves something
|
||||
encoding specific. An area where the windows port is known to have
|
||||
extensive problems.
|
||||
|
||||
([[!commit 1b8026b2cbc8df0274082c5f08a8b4f8ca47c5c9]] was similar,
|
||||
although that was MetaField and this appears to be MetaValue.)
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 2"
|
||||
date="2020-10-26T17:57:12Z"
|
||||
content="""
|
||||
you say \"windows port\", I say \"windows as a whole\", e.g. today revelation (or just a come back if I ran into it before but forgot) to me [was inability to have a file/directory named `con`...](https://github.com/datalad/datalad/issues/5097) - no bloody sense on how such design decision has happened and how it dragged all the way into the flagman of the 2020 product.
|
||||
"""]]
|
|
@ -1,42 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2020-10-26T18:41:27Z"
|
||||
content="""
|
||||
Hmm, this uses viewedFiles, which generates filenames
|
||||
based on the MetaValue. Note use of pathProduct, which uses
|
||||
System.FilePath.combine.
|
||||
|
||||
So, generating random ascii (including escape sequences)
|
||||
bytestrings, and passing them through decodeBS to generate FilePaths,
|
||||
and then operating on those filepaths. What could possibly go wrong.
|
||||
|
||||
And aha! I made pathProduct use System.FilePath.Windows.combine
|
||||
and was able to reproduce the test suite failure on Linux.
|
||||
|
||||
And aha again:
|
||||
|
||||
MetaValue (CurrentlySet True) "c:"
|
||||
|
||||
Which of course breaks it on windows because it wanted to generate
|
||||
something like "bar/c:/baz/a" but instead it gets "c:/bar/baz/a"
|
||||
|
||||
git-annex does replace '/' and '\' when generating these filenames.
|
||||
Not as a security measure (when the view branch is checked out, git's
|
||||
security checks apply same as any branch so it piggybacks on those),
|
||||
but to let the user build a view and successfully check it out
|
||||
when their metadata happens to include such stuff.
|
||||
|
||||
However, windows does have enough special filenames and gotchas
|
||||
that it simply does not seem to make sense for git-annex to try to work
|
||||
around them all in the view code. If a MetaValue happens to end with a
|
||||
period, or is "nul", and so the generated filename is illegal on Windows,
|
||||
it'll blow up at checkout time, and I am ok with that.
|
||||
|
||||
So I think it would make sense to also escape ':', but that's about as far
|
||||
as this should go. *Especially* because the filenames it generates need to
|
||||
roundtrip back to metadata cleanly, which is what this test case is
|
||||
testing. While I can finesse individual characters, it would be quite hard
|
||||
to make a filename w/o a trailing dot roundtrip back to one with it, for
|
||||
example.
|
||||
"""]]
|
|
@ -1,19 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 4"
|
||||
date="2021-01-22T16:44:58Z"
|
||||
content="""
|
||||
did it come back, I see
|
||||
|
||||
```
|
||||
2021-01-22T04:32:25.5012547Z prop_view_roundtrips: FAIL (0.09s)
|
||||
2021-01-22T04:32:25.5015902Z *** Failed! Falsified (after 218 tests):
|
||||
2021-01-22T04:32:25.5016251Z AssociatedFile (Just \"rdmBBP\")
|
||||
2021-01-22T04:32:25.5018130Z MetaData (fromList [(MetaField \"CkL\",fromList [MetaValue (CurrentlySet False) \"\",MetaValue (CurrentlySet True) \"\SOH5:R9\EM\DC4\",MetaValue (CurrentlySet True) \"\STX\US\fL2\ACK|\\\r[$\",MetaValue (CurrentlySet False) \"\ETBRi\",MetaValue (CurrentlySet False) \"/\FS}\",MetaValue (CurrentlySet True) \"W\",MetaValue (CurrentlySet False) \"X=sQh\NAK^\",MetaValue (CurrentlySet False) \"l\SUB\a\"]),(MetaField \"jM\",fromList [MetaValue (CurrentlySet False) \"\",MetaValue (CurrentlySet False) \"\FSSivk\",MetaValue (CurrentlySet True) \"J'<\SYN\STXGJP\"]),(MetaField \"V\",fromList [MetaValue (CurrentlySet False) \"\",MetaValue (CurrentlySet True) \"\n\NUL\",MetaValue (CurrentlySet True) \"\r\",MetaValue (CurrentlySet False) \"+X\",MetaValue (CurrentlySet True) \"@aN\t~c\SIy\",MetaValue (CurrentlySet False) \"K>xq\",MetaValue (CurrentlySet True) \"a:\"]),(MetaField \"W\",fromList [MetaValue (CurrentlySet True) \"0\DC4qL\",MetaValue (CurrentlySet False) \"K\",MetaValue (CurrentlySet False) \"LD\DC3<M\",MetaValue (CurrentlySet False) \"a\v\",MetaValue (CurrentlySet True) \"dO\",MetaValue (CurrentlySet True) \"w\EOT\"])])
|
||||
2021-01-22T04:32:25.5020545Z True
|
||||
2021-01-22T04:32:25.5020894Z Use --quickcheck-replay=455629 to reproduce.
|
||||
```
|
||||
|
||||
on https://github.com/datalad/git-annex/runs/1746587663?check_suite_focus=true with `8.20201129+git169-gaa07e68ed_x64`
|
||||
"""]]
|
|
@ -1,18 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2021-01-22T16:59:55Z"
|
||||
content="""
|
||||
Not sure if this is really the same bug, though certainly related. These
|
||||
quickcheck tests are fuzz tests, they can find numerous bugs, that's kind of
|
||||
the point of them. In any case, posting to a closed bug report risks your
|
||||
followup being lost and deprioritises it.
|
||||
|
||||
The problem this new failure shows is that toViewPath is failing to escape the
|
||||
final character in the path in some cases. Which is not a windows-specific
|
||||
bug at all really, it could also happen with a metadata value such as "foo/"
|
||||
being set on linux. Fixed that bug.
|
||||
|
||||
Which shows the point of these quickcheck fuzz tests: To be able to catch
|
||||
lots of different bugs with a single test case.
|
||||
"""]]
|
|
@ -1,42 +0,0 @@
|
|||
### Please describe the problem.
|
||||
|
||||
Since cron build of 20210828
|
||||
|
||||
```
|
||||
(git)smaug:/mnt/datasets/datalad/ci/git-annex/builds/2021/08[master]git
|
||||
$> git grep -l 'Unable to remove all write permissions'
|
||||
cron-20210828/build-macos.yaml-403-69466103-failed/2_test-annex (crippled-tmp).txt
|
||||
cron-20210828/build-macos.yaml-403-69466103-failed/test-annex (crippled-tmp)/8_Run tests.txt
|
||||
cron-20210829/build-macos.yaml-404-69466103-failed/2_test-annex (crippled-tmp).txt
|
||||
cron-20210829/build-macos.yaml-404-69466103-failed/test-annex (crippled-tmp)/8_Run tests.txt
|
||||
cron-20210830/build-macos.yaml-405-69466103-failed/2_test-annex (crippled-tmp).txt
|
||||
cron-20210830/build-macos.yaml-405-69466103-failed/test-annex (crippled-tmp)/8_Run tests.txt
|
||||
cron-20210831/build-macos.yaml-406-69466103-failed/2_test-annex (crippled-tmp).txt
|
||||
cron-20210831/build-macos.yaml-406-69466103-failed/test-annex (crippled-tmp)/8_Run tests.txt
|
||||
```
|
||||
|
||||
we got two test fails on a crippled FS on Mac (does not happen on linux afaik)
|
||||
|
||||
[example CI log](https://github.com/datalad/git-annex/runs/3468283573?check_suite_focus=true)
|
||||
|
||||
Both look like
|
||||
|
||||
```
|
||||
2021-08-31T02:15:42.0758760Z magic: OK (2.41s)
|
||||
2021-08-31T02:15:42.6972710Z import: FAIL (0.62s)
|
||||
2021-08-31T02:15:42.6973680Z ./Test/Framework.hs:57:
|
||||
2021-08-31T02:15:42.6974230Z import failed (transcript follows)
|
||||
2021-08-31T02:15:42.6974760Z import import1/f
|
||||
2021-08-31T02:15:42.6976570Z Unable to remove all write permissions from /Volumes/crippledfs/importtestvjfjz3/import1/f -- perhaps it has an xattr or ACL set.
|
||||
2021-08-31T02:15:42.6977430Z failed
|
||||
2021-08-31T02:15:42.6977830Z import: 1 failed
|
||||
2021-08-31T02:15:44.1985050Z reinject: OK (1.50s)
|
||||
```
|
||||
|
||||
[here is the script](https://github.com/datalad/git-annex/blob/master/.github/workflows/tools/setup_crippledfs#L24) to setup such a crippled (FAT32) FS on OSX.
|
||||
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] (provisionally, waiting on test run) --[[Joey]]
|
|
@ -1,34 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2021-09-01T13:48:57Z"
|
||||
content="""
|
||||
Seems that mounting that way on OSX results in a FS where files are always mode
|
||||
777 and the permissions cannot be changed.
|
||||
|
||||
When I tried using git-annex on such a FS, I saw:
|
||||
|
||||
datalads-imac:x joey$ git annex init
|
||||
init
|
||||
Detected a filesystem without fifo support.
|
||||
|
||||
Disabling ssh connection caching.
|
||||
|
||||
Filesystem allows writing to files whose write bit is not set.
|
||||
|
||||
Detected a crippled filesystem.
|
||||
|
||||
And it skips the new permissions check when on a crippled filesystem.
|
||||
|
||||
But in that that test run, it seems it is failing to detect a crippled
|
||||
filesystem. Both because of the failure and also the test suite does
|
||||
not even run the "v8 unlocked" tests when it detects a crippled filesystem.
|
||||
|
||||
Is the test suite running as root? Looks like probably yes. Running as
|
||||
root prevents detecting the issue that made it use a crippled FS above. And it
|
||||
seems that, when a FAT fs is mounted on OSX that way, symlinks actually work
|
||||
(!!!) so the other crippled FS tests also don't notice a problem.
|
||||
|
||||
So, the fix should be for init to also test if it can remove the write
|
||||
bits from a file, and it should try that test even when root.
|
||||
"""]]
|
|
@ -1,10 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 2"
|
||||
date="2021-09-01T16:22:22Z"
|
||||
content="""
|
||||
> Is the test suite running as root? Looks like probably yes.
|
||||
|
||||
FWIW, it is a `runner` user [ref](https://github.com/datalad/git-annex/pull/76/checks?check_run_id=3486443350#step:8:1) (did in a temp [PR](https://github.com/datalad/git-annex/pull/76)) who is not `root` but is part of the `admin` group thus with super privileges indeed (that is why I guess we can also use `hdiutil` directly to mount that crippled FS).
|
||||
"""]]
|
|
@ -1,7 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2021-09-02T16:22:56Z"
|
||||
content="""
|
||||
OSX test is still failing after that fix, reopened.
|
||||
"""]]
|
|
@ -1,27 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2021-09-02T16:26:14Z"
|
||||
content="""
|
||||
My prior analysis seems right, as far as it running as root would go, but it is
|
||||
not running as root. So I missed something.
|
||||
|
||||
The test failures are both of `git-annex import`.
|
||||
Otherwise locking down files does succeed. The difference with import
|
||||
must be that the file located in a directory outside the repository.
|
||||
|
||||
Aha... The test suite is being run with TEMPDIR set to the crippled FS,
|
||||
but `.t` is in another, non-crippled FS. A very smart idea to test that,
|
||||
although I think this import test is the only one that actually uses
|
||||
TEMPDIR. (Reading the workflow file, I think it was maybe expected that
|
||||
all the tests would run in TEMPDIR, but they don't; `git-annex test`
|
||||
writes to `./.t`, other than this one test.
|
||||
|
||||
When the import directory is on a crippled FS, and the repo
|
||||
is not, it will think the FS is not crippled. Then it fails
|
||||
to remove write perms from the file while it is in the import
|
||||
directory, and the perm check then fails.
|
||||
|
||||
So, I think it should skip the perm check when doing the initial lockdown
|
||||
of the file it's going to import.
|
||||
"""]]
|
|
@ -1,7 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 5"""
|
||||
date="2021-09-02T17:39:20Z"
|
||||
content="""
|
||||
Ok, fixed some more, hopefully all the way this time..
|
||||
"""]]
|
|
@ -1,66 +0,0 @@
|
|||
### Please describe the problem.
|
||||
|
||||
There was some recent work to ["centralize" such prompts](https://git-annex.branchable.com/devblog/day_457__improved_ssh_password_prompting/) but it seems some are still "leaking through" multiple times. May be it is because there are 2 available repos on that remote host, so annex generates one per each of those? (although it knows only about origin)
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
6.20170810+gitgff6f9e203-1~ndall+1
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
|
||||
$> git annex get -J5 .
|
||||
get R042/R042-2013-08-16/R042-2013-08-16-CSC01a.ncs get R042/R042-2013-08-16/R042-2013-08-16-CSC02a.ncs get R042/R042-2013-08-16/R042-2013-08-16-CSC03a.ncs get R042/R042-2013-08-16/R042-2013-08-16-CSC05a.ncs get R042/R042-2013-08-16/R042-2013-08-16-CSC04a.ncs (from datalad-archives...)
|
||||
(from datalad-archives...) (from datalad-archives...)
|
||||
|
||||
(from datalad-archives...)
|
||||
(from datalad-archives...)
|
||||
[ERROR] Failed to run ['git-annex', 'get', '--key', 'MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip'] under '/mnt/btrfs/datasets/datalad/crawl/workshops/mind-2017/MotivationalT'. Exit code=1. out=get MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip (transfer already in progress, or unable to take transfer lock)
|
||||
| Unable to access these remotes: origin
|
||||
|
|
||||
| Try making some of these repositories available:
|
||||
| aaa0bc14-51fc-45c8-81c2-76dff067755b -- mvdm@atlantis.hpcc.dartmouth.edu:~/data/mind-2017/MotivationalT_
|
||||
| f7f97046-ea49-4af1-9f5a-8475a5ea1e0a -- yhalchen@atlantis.hpcc.dartmouth.edu:~/mind-2017/MotivationalT [origin]
|
||||
| failed
|
||||
| err=git-annex: get: 1 failed
|
||||
|
|
||||
[ERROR] Failed to run ['git-annex', 'get', '--key', 'MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip'] under '/mnt/btrfs/datasets/datalad/crawl/workshops/mind-2017/MotivationalT'. Exit code=1. out=get MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip (transfer already in progress, or unable to take transfer lock)
|
||||
| Unable to access these remotes: origin
|
||||
|
|
||||
| Try making some of these repositories available:
|
||||
| aaa0bc14-51fc-45c8-81c2-76dff067755b -- mvdm@atlantis.hpcc.dartmouth.edu:~/data/mind-2017/MotivationalT_
|
||||
| f7f97046-ea49-4af1-9f5a-8475a5ea1e0a -- yhalchen@atlantis.hpcc.dartmouth.edu:~/mind-2017/MotivationalT [origin]
|
||||
| failed
|
||||
| err=git-annex: get: 1 failed
|
||||
|
|
||||
[ERROR] Failed to fetch any archive containing SHA256E-s17136940--0501aab6b4d1ce0565921728bc92ef74f81edf0d7bcd5a77946ca58f977f2537.ncs. Tried: ['MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip']
|
||||
[ERROR] Failed to fetch any archive containing SHA256E-s17136940--8b3b08310db20ca7e3e784a21f935a78f8669efdf1396168596411f1e355e43b.ncs. Tried: ['MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip']
|
||||
(from origin...) (from origin...) [ERROR] Failed to run ['git-annex', 'get', '--key', 'MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip'] under '/mnt/btrfs/datasets/datalad/crawl/workshops/mind-2017/MotivationalT'. Exit code=1. out=get MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip (transfer already in progress, or unable to take transfer lock)
|
||||
| Unable to access these remotes: origin
|
||||
|
|
||||
| Try making some of these repositories available:
|
||||
| aaa0bc14-51fc-45c8-81c2-76dff067755b -- mvdm@atlantis.hpcc.dartmouth.edu:~/data/mind-2017/MotivationalT_
|
||||
| f7f97046-ea49-4af1-9f5a-8475a5ea1e0a -- yhalchen@atlantis.hpcc.dartmouth.edu:~/mind-2017/MotivationalT [origin]
|
||||
| failed
|
||||
| err=git-annex: get: 1 failed
|
||||
|
|
||||
[ERROR] Failed to fetch any archive containing SHA256E-s17136940--08ce5a67c7fc09f02b994a3987812a75727eaf51f3e70fa7e1030dae934f9fbc.ncs. Tried: ['MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip']
|
||||
(from origin...) [ERROR] Failed to run ['git-annex', 'get', '--key', 'MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip'] under '/mnt/btrfs/datasets/datalad/crawl/workshops/mind-2017/MotivationalT'. Exit code=1. out=get MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip (transfer already in progress, or unable to take transfer lock)
|
||||
| Unable to access these remotes: origin
|
||||
|
|
||||
| Try making some of these repositories available:
|
||||
| aaa0bc14-51fc-45c8-81c2-76dff067755b -- mvdm@atlantis.hpcc.dartmouth.edu:~/data/mind-2017/MotivationalT_
|
||||
| f7f97046-ea49-4af1-9f5a-8475a5ea1e0a -- yhalchen@atlantis.hpcc.dartmouth.edu:~/mind-2017/MotivationalT [origin]
|
||||
| failed
|
||||
| err=git-annex: get: 1 failed
|
||||
|
|
||||
[ERROR] Failed to fetch any archive containing SHA256E-s17136940--bc145f07c79584181cad3763a763a2ea047282bd41153d20a63d85a44fb27a7f.ncs. Tried: ['MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip']
|
||||
(from origin...) yhalchen@discovery.dartmouth.edu's password: yhalchen@discovery.dartmouth.edu's password:
|
||||
|
||||
"""]]
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> warning added; [[done]] --[[Joey]]
|
|
@ -1,34 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 1"
|
||||
date="2017-08-12T04:08:04Z"
|
||||
content="""
|
||||
ha -- if I do specify `--from=origin` -- only 1 prompt
|
||||
|
||||
[[!format sh \"\"\"
|
||||
$> git annex get -J5 --from=origin .
|
||||
get R042/R042-2013-08-16/R042-2013-08-16-CSC01a.ncs (from origin...) get R042/R042-2013-08-16/R042-2013-08-16-CSC03a.ncs (from origin...) get R042/R042-2013-08-16/R042-2013-08-16-CSC02a.ncs (from origin...) get R042/R042-2013-08-16/R042-2013-08-16-CSC04a.ncs (from origin...) get R042/R042-2013-08-16/R042-2013-08-16-CSC05a.ncs (from origin...) yhalchen@discovery.dartmouth.edu's password:
|
||||
git-annex-shell: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
|
||||
|
||||
Unable to run git-annex-shell on remote .
|
||||
|
||||
git-annex-shell: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
|
||||
|
||||
Unable to run git-annex-shell on remote .
|
||||
|
||||
git-annex-shell: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
|
||||
|
||||
Unable to run git-annex-shell on remote .
|
||||
|
||||
git-annex-shell: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
|
||||
|
||||
Unable to run git-annex-shell on remote .
|
||||
|
||||
SHA256E-s17136940--08ce5a67c7fc09f02b994a3987812a75727eaf51f3e70fa7e1030dae934f9fbc.ncs
|
||||
0 0% 0.00kB/s 0:00:00 SHA256E-s17136940--bc145f07c79584181cad3763a763a2ea047282bd41153d20a63d85a44fb27a7f.ncs
|
||||
0 0% 0.00kB/s 0:00:00 SHA256E-s17136940--c3a8af948c77a2df422eae50807a6e7e6e5db7a3451a562bca529d3f1a1a234f.ncs
|
||||
0 0% 0.00kB/s 0:00:00 git-annex-shell: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
|
||||
|
||||
\"\"\"]]
|
||||
"""]]
|
|
@ -1,25 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2017-08-15T16:43:07Z"
|
||||
content="""
|
||||
The improvements around ssh password prompting require ssh connection
|
||||
caching to work. If a ssh connection fails because the wrong password is
|
||||
entered or because there's no usable tty or whatever, there's no cached
|
||||
ssh connection to reuse, so the next attempt to access that host will
|
||||
result in another password prompt.
|
||||
|
||||
Also, datalad does not seem to be running git-annex with -J. So it *can't*
|
||||
be trying to make two ssh connection at the same time. My recent work on
|
||||
ssh password prompting was mostly to fix cases where git-annex is run with
|
||||
-J.
|
||||
|
||||
It's also possible that some ssh configuration that I don't know of could
|
||||
make ssh password prompt even when git-annex is running it with
|
||||
`BatchMode=true` to avoid password prompts (in order to test if the ssh
|
||||
connection is already up). That would then result in two ssh password
|
||||
prompts, one after the other, which seems to match your transcript.
|
||||
|
||||
If you have only one remote, specifying `--from=origin` won't change
|
||||
anything. Entering the right password would change something there though..
|
||||
"""]]
|
|
@ -1,43 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="the issue persists"
|
||||
date="2019-11-01T18:12:27Z"
|
||||
content="""
|
||||
Ran into the same problem again, and it is not clear to me either connection caching is enabled or not (and why?):
|
||||
|
||||
[[!format sh \"\"\"
|
||||
[d31548v@discovery7 bids]$ git -c annex.sshcaching=true annex --debug get -J2 --from=origin sub-sid000005
|
||||
[2019-11-01 14:10:56.178577] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
|
||||
[2019-11-01 14:10:56.475956] process done ExitSuccess
|
||||
[2019-11-01 14:10:56.47622] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
|
||||
[2019-11-01 14:10:56.836271] process done ExitSuccess
|
||||
[2019-11-01 14:10:56.865928] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..8a694d5c54eb81b1e5c5446fa63bdcd13daa34b3\",\"--pretty=%H\",\"-n1\"]
|
||||
[2019-11-01 14:10:57.229787] process done ExitSuccess
|
||||
[2019-11-01 14:10:57.234655] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
|
||||
[2019-11-01 14:10:57.23592] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
|
||||
[2019-11-01 14:10:57.546203] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"]
|
||||
[2019-11-01 14:10:57.780246] process done ExitSuccess
|
||||
[2019-11-01 14:10:57.780454] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"]
|
||||
[2019-11-01 14:10:58.097345] process done ExitSuccess
|
||||
[2019-11-01 14:10:58.09754] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"sub-sid000005\"]
|
||||
[2019-11-01 14:10:58.298181] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
|
||||
[2019-11-01 14:10:58.29998] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
|
||||
[2019-11-01 14:10:58.305022] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
|
||||
[2019-11-01 14:10:58.306024] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
|
||||
[2019-11-01 14:10:58.62005] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
|
||||
[2019-11-01 14:10:58.621714] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
|
||||
[2019-11-01 14:10:58.632596] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
|
||||
[2019-11-01 14:10:58.6338] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
|
||||
get sub-sid000005/ses-actions1/fmap/sub-sid000005_ses-actions1_acq-25mm_magnitude2.nii.gz get sub-sid000005/ses-actions1/fmap/sub-sid000005_ses-actions1_acq-25mm_magnitude1.nii.gz (from origin...) (from origin...)
|
||||
[2019-11-01 14:10:59.489719] chat: ssh [\"yohtest@rolando.cns.dartmouth.edu\",\"-T\",\"git-annex-shell 'p2pstdio' '/inbox/BIDS/Haxby/Sam/1021_actions' '--debug' 'fd3f7af9-cf7d-4d7e-8efd-30e6bedf838d' --uuid d839134c-3afe-4456-920a-e280ce0fdf2a\"]
|
||||
|
||||
[2019-11-01 14:10:59.553029] chat: ssh [\"yohtest@rolando.cns.dartmouth.edu\",\"-T\",\"git-annex-shell 'p2pstdio' '/inbox/BIDS/Haxby/Sam/1021_actions' '--debug' 'fd3f7af9-cf7d-4d7e-8efd-30e6bedf838d' --uuid d839134c-3afe-4456-920a-e280ce0fdf2a\"]
|
||||
yohtest@rolando.cns.dartmouth.edu's password: yohtest@rolando.cns.dartmouth.edu's password:
|
||||
|
||||
|
||||
[d31548v@discovery7 bids]$ git annex version
|
||||
git-annex version: 7.20191024-g6dc2272
|
||||
\"\"\"]]
|
||||
Could you hint me on what/where to dig?
|
||||
"""]]
|
|
@ -1,21 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2020-01-23T15:51:46Z"
|
||||
content="""
|
||||
I notice that debug output has no BatchMode=true in any ssh call. But
|
||||
the version of git-annex you show always runs ssh with that when
|
||||
-J is used, unless sshcaching is disabled.
|
||||
|
||||
More evidence that sshcaching is disabled in your transcript is that when
|
||||
it does run ssh, it does not pass -S.
|
||||
|
||||
I think the repository must be on a crippled filesystem, on which
|
||||
git-annex can't do ssh connection caching, because the filesystem
|
||||
does not support unix sockets. (Or it potentially could be crippled in some
|
||||
other way.) So it ignores the annex.sshcaching setting.
|
||||
You could work around this by setting the (undocumented)
|
||||
GIT_ANNEX_TMP_DIR to some temporary directory on a non-crippled filesystem.
|
||||
|
||||
I'm going to add a warning message in this situation.
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 5"
|
||||
date="2020-01-23T17:51:58Z"
|
||||
content="""
|
||||
Thank you Joey! I can only confirm that the file system was likely a crippled/NFS one... So we would likely need to do some sensing on DataLad side and instruct git-annex. Will continue on our end at https://github.com/datalad/datalad/issues/4075
|
||||
"""]]
|
|
@ -1,43 +0,0 @@
|
|||
[[!format sh """
|
||||
|
||||
neurodebian@smaug ..6.20181011+git109-gff9ba1f4d-1~ndall+1 % grep -B10 FAIL git-annex_6.20181011+git109-gff9ba1f4d-1~ndall+1_amd64.build
|
||||
a03a722..0dd2b80 git-annex -> r2/git-annex
|
||||
To ../../.t/tmprepo331
|
||||
* [new branch] git-annex -> synced/git-annex
|
||||
To ../../.t/tmprepo330
|
||||
a03a722..22937c6 git-annex -> synced/git-annex
|
||||
OK (2.15s)
|
||||
adjusted branch merge regression: On branch master
|
||||
nothing to commit, working tree clean
|
||||
On branch master
|
||||
nothing to commit, working tree clean
|
||||
FAIL (0.50s)
|
||||
Test.hs:1463:
|
||||
adjust failed
|
||||
adjusted branch subtree regression: On branch master
|
||||
nothing to commit, working tree clean
|
||||
FAIL (0.31s)
|
||||
--
|
||||
059871b..4f97b0c annex/direct/master -> r2/annex/direct/master
|
||||
conflictor.variant-a507
|
||||
conflictor.variant-75dc
|
||||
conflictor.variant-a507
|
||||
conflictor.variant-75dc
|
||||
OK (1.37s)
|
||||
conflict resolution (adjusted branch): On branch master
|
||||
nothing to commit, working tree clean
|
||||
On branch master
|
||||
nothing to commit, working tree clean
|
||||
FAIL (0.70s)
|
||||
|
||||
"""]]
|
||||
|
||||
|
||||
Full log http://www.onerussian.com/tmp/git-annex_6.20181011+git109-gff9ba1f4d-1~ndall+1_amd64.build
|
||||
|
||||
(not sure if relevant - this is I think the first build without my custom no LOCPATH patch)
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,9 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="comment 1"
|
||||
date="2018-10-22T19:22:30Z"
|
||||
content="""
|
||||
See also (same?) failure when trying to build the conda-forge package:
|
||||
https://circleci.com/gh/conda-forge/git-annex-feedstock/116?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="comment 2"
|
||||
date="2018-10-24T16:01:53Z"
|
||||
content="""
|
||||
Maybe, post a new git-annex version on hackage, with the test failure fixed? I could then update the conda-forge recipe.
|
||||
"""]]
|
|
@ -1,29 +0,0 @@
|
|||
### Please describe the problem.
|
||||
|
||||
Trying to build fresh snapshot for neurodebian but 3 tests report failing
|
||||
Full buildlogs are at [http://neuro.debian.net/_files/_buildlogs/git-annex/7.20181211+git29-gab4a1bed9](http://neuro.debian.net/_files/_buildlogs/git-annex/7.20181211+git29-gab4a1bed9)
|
||||
|
||||
|
||||
[[!format sh """
|
||||
neurodebian@smaug ../7.20181211+git29-gab4a1bed9-1~ndall+1 % grep -B3 '^FAIL' git-annex_7.20181211+git29-gab4a1bed9-1\~ndall+1_amd64.build
|
||||
bup remote: OK (0.11s)
|
||||
crypto: gpg: can't connect to the agent: IPC connect call failed
|
||||
gpg: error getting the KEK: No agent running
|
||||
FAIL
|
||||
--
|
||||
bup remote: OK (0.19s)
|
||||
crypto: gpg: can't connect to the agent: IPC connect call failed
|
||||
gpg: error getting the KEK: No agent running
|
||||
FAIL
|
||||
--
|
||||
nothing to commit, working tree clean
|
||||
gpg: can't connect to the agent: IPC connect call failed
|
||||
gpg: error getting the KEK: No agent running
|
||||
FAIL
|
||||
|
||||
"""]]
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> [[done]] I think.. --[[Joey]]
|
|
@ -1,14 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2019-01-01T16:10:48Z"
|
||||
content="""
|
||||
I'll bet [[!commit 14971414dc263fcb8124b4cf6b14b9b7a19189af]]
|
||||
somehow led to this
|
||||
since the test suite now runs gpg with its home
|
||||
directory set to be inside the test repo and not in the system's temporary
|
||||
directory.
|
||||
|
||||
But I tested that commit on filesystems down to FAT, so it seems strange it
|
||||
would fail on whatever filesystem you are testing on.
|
||||
"""]]
|
|
@ -1,9 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="the build details"
|
||||
date="2019-01-02T01:14:39Z"
|
||||
content="""
|
||||
cowbuilder is used, underlying file system (dedicated to debian pkgs building) is ext4.
|
||||
but is it a filesystem effect somehow or just a \"partially\" isolated environment aspect (such as those pbuilder/cowbuilder chroots)? The message `gpg: can't connect to the agent: IPC connect call failed` suggests that may be tests pass for you locally as connecting to your gpg agent?
|
||||
"""]]
|
|
@ -1,22 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2019-01-18T17:51:09Z"
|
||||
content="""
|
||||
Having what seems a related problem on my i386 autobuilder:
|
||||
|
||||
gpg: keyblock resource '/home/builder/gitbuilder/build/.t/tmprepo89/.git/annex/othertmp/gpgtest/2/pubring.kbx': No such file or directory
|
||||
gpg: failed to create temporary file '/home/builder/gitbuilder/build/.t/tmprepo89/.git/annex/othertmp/gpgtest/2/.#lk0x57fed950.orca.13095': No such file or directory
|
||||
gpg: can't connect to the agent: No such file or directory
|
||||
gpg: problem with the agent: No agent running
|
||||
gpg: can't create `/home/builder/gitbuilder/build/.t/tmprepo89/.git/annex/othertmp/gpgtest/2/random_seed': No such file or directory
|
||||
user error (gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--batch","--passphrase-fd","28","--symmetric","--force-mdc","--no-textmode"] exited 2)
|
||||
copy: 1 failed
|
||||
FAIL (2.31s)
|
||||
Test.hs:1645:
|
||||
copy --to encrypted remote failed
|
||||
|
||||
Probably gpg error output has failed, but this is also failing to connect
|
||||
to the agent. And it seems a file/directory gpg expects to find is not present
|
||||
when it's running in the test harness.
|
||||
"""]]
|
|
@ -1,14 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2019-01-21T18:12:13Z"
|
||||
content="""
|
||||
I've fixed the problem I described above. But I don't think it was the
|
||||
same problem; the neurodebian build log is not complaining about missing
|
||||
files/directories.
|
||||
|
||||
gpg in the test suite is using a separate home directory, so it starts a
|
||||
separate gpg-agent process for that. If the test suite is run on a
|
||||
filesystem where gpg-agent can't work, due say to unix sockets not being
|
||||
supported, that might be why it fails.
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 5"""
|
||||
date="2019-01-21T19:18:51Z"
|
||||
content="""
|
||||
Anyway, see if [[!commit f5f059e288b5e618e5c7c86d1db4bfd23d8b39ad]] somehow
|
||||
fixed it..
|
||||
"""]]
|
|
@ -1,17 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 6"
|
||||
date="2019-01-21T19:22:52Z"
|
||||
content="""
|
||||
tried to build 7.20181211+git284-g6f0d98f2a - it seems that the failing one is different one now?
|
||||
|
||||
prop_parse_show_Config: FAIL
|
||||
*** Failed! Falsifiable (after 3 tests and 1 shrink):
|
||||
fromList [(\"\791992\",\"\")]
|
||||
Use --quickcheck-replay=133129 to reproduce.
|
||||
prop_upFrom_basics: OK (0.03s)
|
||||
...
|
||||
1 out of 291 tests failed (254.87s)
|
||||
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 7"""
|
||||
date="2019-01-21T20:30:52Z"
|
||||
content="""
|
||||
If that unrelated test case is the only failure, then this problem is
|
||||
fixed.
|
||||
"""]]
|
|
@ -1,79 +0,0 @@
|
|||
In datalad test builds with git-annex 7.20191114+git43-ge29663773, one
|
||||
of the new test failures is due to an unexpectedly dirty repository
|
||||
([related datalad issue][0]). The dirty status comes from a file that
|
||||
was tracked in Git switching over to an annex pointer file. Here's a
|
||||
script that distills enough of the test to trigger the failure on my
|
||||
end.
|
||||
|
||||
[[!format sh """
|
||||
#!/bin/sh
|
||||
|
||||
set -eu
|
||||
|
||||
assert_clean () {
|
||||
if test -n "$(git status --porcelain)"
|
||||
then
|
||||
printf "\n\nUnexpectedly dirty:\n" >&2
|
||||
git status >&2
|
||||
git diff >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
cd "$(mktemp -d --tmpdir gx-pointer-dirty-XXXXXXX)"
|
||||
git init && git annex init
|
||||
|
||||
printf content-git >file-git
|
||||
git -c annex.largefiles=nothing annex add -- file-git
|
||||
git commit -m'file-git added'
|
||||
assert_clean
|
||||
|
||||
printf content-annex >file-annex
|
||||
git -c annex.largefiles=anything annex add -- file-annex
|
||||
git commit -m'file-annex annexed'
|
||||
assert_clean
|
||||
"""]]
|
||||
|
||||
On Travis as well as my local machine, the failure is intermittent,
|
||||
but seems to happen much more often than not. In the failing case,
|
||||
the last assert_clean call shows:
|
||||
|
||||
```
|
||||
Unexpectedly dirty:
|
||||
On branch master
|
||||
Changes not staged for commit:
|
||||
modified: file-git
|
||||
|
||||
no changes added to commit
|
||||
diff --git a/file-git b/file-git
|
||||
index d1c416a..b41ca32 100644
|
||||
--- a/file-git
|
||||
+++ b/file-git
|
||||
@@ -1 +1 @@
|
||||
-content-git
|
||||
\ No newline at end of file
|
||||
+/annex/objects/SHA256E-s11--726732d25826965592478fcc7c145d5a10fa1aa70c49fe3a4f847174b6d8889c
|
||||
```
|
||||
|
||||
I see the failure with git-annex built from the latest master
|
||||
b962471c2 (2019-12-12). Bisecting against the git-annex repo (with a
|
||||
commit being marked "bad" if there was a failure within ten runs of the
|
||||
above script), points to ec08b66bd (shouldAnnex: check isInodeKnown,
|
||||
2019-10-23) as the first bad commit. Just looking at the topic of
|
||||
the commit, that result seems plausible to me.
|
||||
|
||||
### Other details
|
||||
|
||||
My git version 2.24.1 and locally I'm building git-annex through guix.
|
||||
On the failing Travis run, git-annex 7.20191114+git43-ge29663773 came
|
||||
from neurodebian, and the git version was 2.24.0.
|
||||
|
||||
Hopefully the script above is sufficient to trigger the issue on your end.
|
||||
Thanks for having a look.
|
||||
|
||||
[0]: https://github.com/datalad/datalad/issues/3890
|
||||
|
||||
[[!meta author=kyle]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,97 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2019-12-26T16:56:38Z"
|
||||
content="""
|
||||
The title makes it sound like a work tree file gets replaced with a
|
||||
dangling pointer file, which is not the case. A worktree file that was
|
||||
not annexed is is being added to the annex, if you choose to commit that
|
||||
state.
|
||||
|
||||
For whatever reason, git becomes confused about whether this file is
|
||||
modified. I seem to recall that git distrusts information it recorded in
|
||||
its own index if the mtime of the index file is too close to the
|
||||
mtime recorded inside it, or something like that. (Likely as a
|
||||
workaround for mtime granularity issues with various filesystems.) Whatever
|
||||
the reason, git-annex is not involved in it; it will happen sometimes even
|
||||
when git-annex has not initialized the repo and is not being used.
|
||||
|
||||
It's not normally a problem that git gets confused or distrusts its
|
||||
index or whatever, since all it does is stat the file, or
|
||||
feed it through the clean filter again, and if the file is not
|
||||
modified, nothing changes.
|
||||
|
||||
Why does the clean filter decide to add the file to annex in this case?
|
||||
|
||||
Well, because this is all happening inside this:
|
||||
|
||||
git -c annex.largefiles=anything annex add -- file-annex
|
||||
|
||||
And there you've told it to add all files to the annex with
|
||||
annex.largefiles=anything. So it does.
|
||||
|
||||
To complete the description of what happens:
|
||||
`git-annex add` runs `git add` on the `file-annex` symlink it's adding.
|
||||
`git add file-annex`, for whatever reason, decides to run the clean filter on
|
||||
file-git.
|
||||
The annex.largefiles=anything gets inherited through this chain of calls.
|
||||
|
||||
While the resulting "change" does not get staged by `git add`
|
||||
(it was never asked to operate on that file), the clean filter
|
||||
duly ingests the content into the annex, and remembers its inode.
|
||||
So when the clean filter later gets run by `git status`, it sees an inode
|
||||
it knows it saw before, and assumes it should remain annexed.
|
||||
(This is why the commit that checks for known inodes was fingered by the
|
||||
bisection.)
|
||||
|
||||
---
|
||||
|
||||
Note that, you can accomplish the same thing without setting
|
||||
annex.largefiles, assuming a current version of git-annex:
|
||||
|
||||
git add file-git
|
||||
git annex add file-annex
|
||||
|
||||
I think the only reason for setting annex.largefiles in either of the two
|
||||
places you did is if there's a default value that you want to
|
||||
temporarily override?
|
||||
|
||||
----
|
||||
|
||||
Also, just touching file-git before the annex.largefiles=anything
|
||||
operation causes the same problem, again git-annex add runs git add
|
||||
file-annex, which runs the clean filter on file-git, which this time
|
||||
is legitimately modified.
|
||||
|
||||
---
|
||||
|
||||
Possible ways to improve this short of improving git's behavior:
|
||||
|
||||
`git annex` could set annex.gitaddtoannex=false when it runs `git add`.
|
||||
Since git-annex never relies on `git add` adding files to the annex,
|
||||
that seems entirely safe to always do (perhaps even when running all git
|
||||
commands aside from git-annex commands of course). But, that would
|
||||
not help with a variant where rather than `git-annex add`,
|
||||
this is run:
|
||||
|
||||
git -c annex.largefiles=anything add file-annex
|
||||
|
||||
The clean filter could delay long enough that git stops distrusting
|
||||
its index based on timestamps. A 1 second sleep if the file's mtime
|
||||
is too close to the current time works; I prototyped a patch doing that.
|
||||
But, that does not deal with the case
|
||||
mentioned above where file-git gets touched or legitimately modified.
|
||||
|
||||
The clean filter could check if the file is already
|
||||
in the index but is not annexed, and avoid converting it to annexed.
|
||||
But that would prevent legitimate conversions from git to annexed
|
||||
as well, which rely on the same kind of use of annex.largefiles.
|
||||
|
||||
Temporary overrides of annex.largefiles could be ignored by the clean
|
||||
filter. Same problem as previous.
|
||||
|
||||
So, I think that fixing this will involve adding a new interface for
|
||||
converting between git and annexed files that does not involve
|
||||
-c annex.largefiles. That plus having the clean filter check for
|
||||
non-annexed files seems like the best approach.
|
||||
"""]]
|
|
@ -1,42 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2019-12-27T06:22:23Z"
|
||||
content="""
|
||||
On second thought, making the clean filter check for non-annexed files
|
||||
would prevent use cases like annex.largefiles=largerthan(100kb)
|
||||
from working as the user intended and letting a small file start out
|
||||
non-annexed and get annexed once it gets too large. Users certianly rely on
|
||||
that and this bug that only affects an edge case does not justify breaking
|
||||
that.
|
||||
|
||||
What would work to make the clean filter detect when a file's content
|
||||
has not changed, though its mtime (or inode) has changed. In that case,
|
||||
it's reasonable for the clean filter to ignore annex.largefiles and keep
|
||||
the content represented in git however it already was (non-annexed or
|
||||
annexed).
|
||||
|
||||
To detect that, in the case where the file in the index is not annexed:
|
||||
First check if the file size is the same as the
|
||||
size in the index. If it is, run git hash-object on the file, and see if
|
||||
the sha1 is the same as in the index. This avoids hashing any unusually
|
||||
large files, so the clean filter only gets a bit slower.
|
||||
|
||||
And when the file in the index is annexed, check if the file size is the
|
||||
same as the size of the annexed key. If it is, verify if the file content
|
||||
matches the key. (typically be hashing). Cases where keys lack size or
|
||||
don't use a checksum could lead to false positives or negatives though.
|
||||
Although, I've not managed to find a version of this bug that makes an
|
||||
annexed file get converted to git unintentionally, so maybe this part does
|
||||
not need to be done?
|
||||
|
||||
----
|
||||
|
||||
Or.. Since the root of the problem is temporarily overriding annex.largefiles,
|
||||
it could just be documented that it's not a good idea to use
|
||||
-c annex.largefiles=anything/nothing, because such broad overrides
|
||||
can affect other files than the ones you intended.
|
||||
(And since the documented methods of converting files from annexed to git and
|
||||
git to annexed use such overrides, that documentation would need to be
|
||||
changed.)
|
||||
"""]]
|
|
@ -1,16 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2019-12-27T17:11:42Z"
|
||||
content="""
|
||||
A variant of this where an annexed unlocked file is added first,
|
||||
then the file is touched, and then some other file is added
|
||||
with -c annex.largefiles=nothing does result in the clean filter sending
|
||||
the whole annexed file content back to git, rather than keeping it annexed.
|
||||
For whatever reason, git does not store that content in .git/objects or
|
||||
update the index for that file though, so it doesn't show up as a change.
|
||||
|
||||
So *apparently* that variant is only potentially an expensive cat of a
|
||||
large annexed file, and does not need to be dealt with. Unless git
|
||||
sometimes behaves otherwise.
|
||||
"""]]
|
|
@ -1,45 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2019-12-27T18:41:12Z"
|
||||
content="""
|
||||
It's almost possible to get the same unwanted conversion without any git
|
||||
races:
|
||||
|
||||
echo content-git > file-git
|
||||
sleep 2
|
||||
git add file-git
|
||||
git commit -m add
|
||||
|
||||
echo foo > file-git
|
||||
echo content-annex > file-annex
|
||||
git -c annex.largefiles=anything annex add file-annex
|
||||
|
||||
In this case, git currently does not run the modified file-git through the
|
||||
clean filter in the last line, so the annex.largefiles=anything doesn't
|
||||
affect it.
|
||||
|
||||
But, as far as I can see, there's nothing preventing a future version
|
||||
of git from deciding it does want to run file-git through the clean filter
|
||||
in this case.
|
||||
|
||||
I am not going to try to prevent against such a thing happening.
|
||||
As far as I can see, anything that the clean filter can possibly do to
|
||||
avoid such a situation will cripple existing uses cases of
|
||||
annex.largefiles, like largerthan() as mentioned above.
|
||||
The user has told git-annex to annex "anything", and if git
|
||||
decides to run the clean filter while that is in effect, caveat emptor.
|
||||
|
||||
Which is not to say I'm not going to fix the specific case this bug was
|
||||
filed about. I actually have a fix developed now. But just to say that
|
||||
setting annex.largefiles=anything/nothing temporarily is a blunt instrument,
|
||||
and you risk accidental conversion when using it, and so it would be a good
|
||||
idea to not do that.
|
||||
|
||||
One idea: Make `git-annex add --annex` and `git-annex add --git`
|
||||
add a specific file to annex or git, bypassing annex.largefiles and all
|
||||
other configuration and state. This could also be used to easily switch
|
||||
a file from one storage to the other. I'd hope the existence of that
|
||||
would prevent one-off setting of annex.largefiles=anything/nothing.
|
||||
[[todo/git_annex_add_option_to_control_to_where]]
|
||||
"""]]
|
|
@ -1,58 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 5"
|
||||
date="2019-12-28T21:06:46Z"
|
||||
content="""
|
||||
Thanks for the explanation and the fix.
|
||||
|
||||
> For whatever reason, git becomes confused about whether this file is
|
||||
> modified. I seem to recall that git distrusts information it recorded in
|
||||
> its own index if the mtime of the index file is too close to the
|
||||
> mtime recorded inside it, or something like that.
|
||||
|
||||
I see. I think the problem and associated workaround you're referring
|
||||
to is described in git's Documentation/technical/racy-git.txt.
|
||||
|
||||
> Note that, you can accomplish the same thing without setting
|
||||
> annex.largefiles, assuming a current version of git-annex:
|
||||
>
|
||||
> git add file-git
|
||||
> git annex add file-annex
|
||||
>
|
||||
> I think the only reason for setting annex.largefiles in either of the two
|
||||
> places you did is if there's a default value that you want to
|
||||
> temporarily override?
|
||||
|
||||
Right. DataLad's methods that are responsible for calling out to `git
|
||||
annex add` have a `git={None,False,True}` parameter. By default
|
||||
(`None`), DataLad just calls `git annex add ...` and let's any
|
||||
configuration in the repo control whether the file goes to git or is
|
||||
annexed. But with `git=True` or `git=False`, the `annex add` call
|
||||
includes a `-c annex.largefiles=` argument with a value of `nothing`
|
||||
or `anything`, respectively.
|
||||
|
||||
> But just to say that setting annex.largefiles=anything/nothing
|
||||
> temporarily is a blunt instrument, and you risk accidental
|
||||
> conversion when using it, and so it would be a good idea to not do
|
||||
> that.
|
||||
|
||||
Noted. As mentioned above, DataLad's default behavior is to honor the
|
||||
repo's `annex.largefiles` configuration. And the documentation for
|
||||
`datalad save`, DataLad's main user-facing entry point for `annex
|
||||
add`, recommends that the user configure .gitattributes rather than
|
||||
using the option that leads calling `annex add` with `-c
|
||||
annex.largefiles=nothing`.
|
||||
|
||||
> One idea: Make `git-annex add --annex` and `git-annex add --git`
|
||||
> add a specific file to annex or git, bypassing annex.largefiles and all
|
||||
> other configuration and state. This could also be used to easily switch
|
||||
> a file from one storage to the other. I'd hope the existence of that
|
||||
> would prevent one-off setting of annex.largefiles=anything/nothing.
|
||||
|
||||
As far as I can see, those flags would completely cover DataLad's
|
||||
one-off setting of `annex.largefiles=anything/nothing`. They map
|
||||
directly to DataLad's `git=False/True` option described above. So,
|
||||
from DataLad's perspective, they'd be very useful and welcome.
|
||||
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 6"""
|
||||
date="2020-01-01T17:41:13Z"
|
||||
content="""
|
||||
I've added git-annex add --force-large and --force-small, which would be
|
||||
good to use to avoid this kind of too-broad overriding problem in the future.
|
||||
"""]]
|
|
@ -1,66 +0,0 @@
|
|||
In the datalad test suite, a test involving one of our special remotes
|
||||
hangs after 7245a9ed5 (Improve shutdown process for external special
|
||||
remotes and external backends, 2020-11-02). The hang depends on the
|
||||
remote program creating an SSH socket but not cleaning it up on
|
||||
failure. That's being addressed on our end
|
||||
(<https://github.com/datalad/datalad/pull/5151>).
|
||||
|
||||
While that's clearly bad handling on the part of datalad's special
|
||||
remote, I'm opening this bug report in case the above stall indicates
|
||||
that 7245a9ed5 brings an increased susceptibility to hanging that's
|
||||
worth worrying about. Based on 7245a9ed5's commit message, it sounds
|
||||
like that resolved some hanging on Windows (and generally sounds like
|
||||
a good change), so it probably makes sense keep that at the expense of
|
||||
hanging that is restricted to buggy special remotes. But I don't have
|
||||
a good enough understanding of the issue to know if it's limited to
|
||||
misbehaving special remotes, or if perhaps there's a way to avoid both
|
||||
hangs.
|
||||
|
||||
In case it's helpful, here's a demo that tweaks the example special
|
||||
remote to trigger a hang:
|
||||
|
||||
[[!format sh """
|
||||
set -eu
|
||||
|
||||
ga_srcdir="$(realpath "${1?requires git-annex source directory as first argument}")"
|
||||
host="${2?requires ssh host as second argument}"
|
||||
cd "$(mktemp -d "${TMPDIR:-/tmp}"/gx-testremote-XXXXXXX)"
|
||||
top="$PWD"
|
||||
|
||||
mkdir bin
|
||||
git -C "$ga_srcdir" cat-file -p cfae74ae25a791a8dbf8b476fe17d5c6feacd8b0 \
|
||||
>bin/git-annex-remote-foo
|
||||
|
||||
# Open a socket in setupcreds().
|
||||
cmd="ssh -fN -o ControlMaster=auto -o ControlPersist=15m -o ControlPath=$top/socket $host"
|
||||
sed -i "s|setupcreds () {|&\\n\\t$cmd|" bin/git-annex-remote-foo
|
||||
|
||||
chmod +x bin/git-annex-remote-foo
|
||||
export PATH="$top/bin:$PATH"
|
||||
|
||||
git annex version
|
||||
git init a
|
||||
(
|
||||
cd a
|
||||
git annex init main
|
||||
# This will trigger an INITREMOTE-FAILURE failure because
|
||||
# MYPASSWORD and MYLOGIN are not set. This hangs after 7245a9ed5.
|
||||
git annex initremote d --debug \
|
||||
type=external externaltype=foo directory="$top/d" encryption=none
|
||||
)
|
||||
"""]]
|
||||
|
||||
```
|
||||
$ sh /tmp/gx-failed-dir-hang.sh ~/src/haskell/git-annex smaug
|
||||
git-annex version: 8.20201103
|
||||
build flags: Assistant Pairing Inotify MagicMime Feeds Testsuite S3 WebDAV
|
||||
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.4 feed-1.2.0.1 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.10.5.2 uuid-1.3.13
|
||||
[... 40 lines ...]
|
||||
[2020-11-12 11:40:01.886013862] /tmp/gx-testremote-3MudMuQ/bin/git-annex-remote-foo[1] --> SETCONFIG directory /tmp/gx-testremote-3MudMuQ/d
|
||||
[2020-11-12 11:40:02.451619481] /tmp/gx-testremote-3MudMuQ/bin/git-annex-remote-foo[1] --> INITREMOTE-FAILURE You need to set MYPASSWORD and MYLOGIN environment variables when running initremote.
|
||||
```
|
||||
|
||||
[[!meta author=kyle]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]]
|
|
@ -1,27 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 1"
|
||||
date="2020-11-16T18:22:05Z"
|
||||
content="""
|
||||
> In the datalad test suite, a test involving one of our special remotes
|
||||
> hangs after 7245a9ed5 [...] The hang depends on the remote program
|
||||
> creating an SSH socket but not cleaning it up on failure. That's being
|
||||
> addressed on our end (https://github.com/datalad/datalad/pull/5151).
|
||||
|
||||
That statement was based on me being able to trigger a stall locally
|
||||
in the same test on my Buster system. The changes from datalad's pr
|
||||
5151 resolved the hang on my end, as well as the hang from the
|
||||
stripped down demo I added here.
|
||||
|
||||
However, with the changes from pr 5151 merged, it turns out the the
|
||||
stall on the Ubuntu 18.04 CI is still there [1]. So, it's unclear
|
||||
what's going on. Perhaps 7245a9ed5 isn't even what triggered the hang
|
||||
on the CI, though the stall seems to have first appeared 14 days ago
|
||||
on a build using 7e24b2587 [2], so the timing is at least consistent
|
||||
with 7245a9ed5 being the culprit.
|
||||
|
||||
[1] https://github.com/datalad/git-annex/runs/1404305092?check_suite_focus=true
|
||||
[2] https://github.com/datalad/datalad-extensions/runs/1345445794?check_suite_focus=true
|
||||
|
||||
"""]]
|
|
@ -1,44 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2020-11-17T16:06:53Z"
|
||||
content="""
|
||||
The ssh process you run this way has the following file descriptor:
|
||||
|
||||
l-wx------ 1 joey joey 64 Nov 17 12:07 2 -> pipe:[6320144]
|
||||
|
||||
That pipe is connected to git-annex. git-annex is waiting to read any errors
|
||||
that the special remote might output to stderr, in order to relay them to the
|
||||
user. Yes, this was changed by the referenced commit.
|
||||
|
||||
Since git-annex has already waited on the process, the process is dead
|
||||
by the point it waits on the stderr relayer thread. The only purpose
|
||||
of waiting rather than closing the handle is to see anything the process
|
||||
might have output in its dying breath.
|
||||
|
||||
Also, this is very similar to the problem fixed
|
||||
in [[!commit aa492bc65904a19f22ffdfc20d7a5e7052e2f54d]] and in
|
||||
[[!commit cb74cefde782e542ad609b194792deabe55b1f5a]], also involving ssh.
|
||||
Those were solved using a rather ugly up to 2 second wait for any late
|
||||
stderr to arrive, though it only delays when the handle is kept open like this.
|
||||
|
||||
There are surely a ton of places where this could potentially happen.
|
||||
Not only stderr.. it's entirely possible that process that git-annex
|
||||
expects to read stdout from might spawn a daemon that keeps inherited
|
||||
stdout open, and exit, leaving git-annex waiting forever to read
|
||||
from the pipe.
|
||||
|
||||
I'm doubtful this is a bug everywhere that git-annex reads all the stdout from
|
||||
a pipe. Because it seems to me a great many programs would have the same
|
||||
problem if a program they were piping stdout from behaved in that way.
|
||||
I've never seen anything concern itself with this potential problem. This is
|
||||
why proper daemons close their handles, certianly their stdout handle. stderr
|
||||
is a slightly special case maybe.
|
||||
|
||||
processTranscript is one example of another place in git-annex that
|
||||
waits to consume all stderr from a process and would be hung by such a daemon.
|
||||
There are a dozen in all,
|
||||
`git grep 'std_err =' | egrep 'CreatePipe|UseHandle | grep -v nullh`
|
||||
I suppose they could all be audited and maybe something abstracted out to
|
||||
deal with them all.
|
||||
"""]]
|
|
@ -1,57 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2020-11-17T19:04:26Z"
|
||||
content="""
|
||||
One approach would be:
|
||||
|
||||
cancelOnExit :: ProcessHandle -> IO a -> IO (Maybe a)
|
||||
|
||||
The versions of this currently being used to work around the past
|
||||
occurances of the problem wait for the process to exit, and then wait 2
|
||||
seconds before canceling the action, to give it time to read anything that
|
||||
was buffered in the pipe.
|
||||
|
||||
That's annoyingly racey, what if the system somehow massively slows
|
||||
down for 2 seconds just after the process exits, and so buffered output is
|
||||
lost? Doing this in a lot of places, not just a few targeted places where
|
||||
maybe we didn't care too much about stderr anyway, does not feel safe.
|
||||
|
||||
There's GHC.IO.Handle.hWaitForInput that does not consume input from a
|
||||
Handle, but checks if there is any available to read. Digging into what
|
||||
that really means -- as well as checking its internal buffers, it calls a
|
||||
[fdReady C function](https://gitlab.haskell.org/ghc/ghc/-/blob/ba4dcc7cb77a37486368911fec854d112a1db93c/libraries/base/cbits/inputReady.c)
|
||||
which boils down to a `poll()`. So that seems like it could be used to
|
||||
avoid the race. I think it would be safest to use it with 0, which avoids a
|
||||
case where, it interrupted too long, it can return false when there's still
|
||||
input.
|
||||
|
||||
It should be possible to use that to implement this:
|
||||
|
||||
readUntilExit :: ProcessHandle -> (Handle -> IO a) -> Handle -> IO a
|
||||
|
||||
Wait for the process to exit, and then check hWaitForInput, and as long as
|
||||
there's input left, sleep breifly and loop. Once all buffered input as been
|
||||
consumed by the action, hClose the handle to get the action to finish
|
||||
reading from it.
|
||||
|
||||
One problem with this is that eg, this would throw an exception if it
|
||||
closed the handle out from under hGetLine before it read anything:
|
||||
|
||||
unlessM (hIsEOF h) $
|
||||
readUntilExit ph hGetLine h
|
||||
|
||||
So it's not entirely safe to drop this in just anywhere. Maybe it should
|
||||
catch the EOF exception?
|
||||
|
||||
readUntilExitOrEOF :: ProcessHandle -> (Handle -> IO a) -> Handle -> IO (Maybe a)
|
||||
|
||||
That should be fine for things like hGetLine, but would still be unsafe
|
||||
for hGetContents, which does *not* like having the handle it's reading
|
||||
closed out from under it ("hGetContents: illegal operation (delayed read on closed handle)").
|
||||
So maybe best to specialize it:
|
||||
|
||||
hGetLineUntilExitOrEOF :: ProcessHandle -> Handle -> IO (Maybe String)
|
||||
|
||||
I have a partial implementation on the ssh-hates-me branch.
|
||||
"""]]
|
|
@ -1,27 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2020-11-18T15:57:10Z"
|
||||
content="""
|
||||
hWaitForInput does not seem to behave as documented. It does not time out,
|
||||
so blocks forever in this situation. This is with a 0 timeout and with
|
||||
larger timeouts. Unsure why, it looked like it should work. Looking at the
|
||||
ghc bug tracker, it seems to have some open bugs.
|
||||
([[!commit 9af0000e0ff28c8eaae0ee1ce6838063d0a5ebef]] has my bench test
|
||||
demonstrating this problem.)
|
||||
|
||||
hGetLineUntilExitOrEOF could still be implemented, but rather than using
|
||||
hGetLine, it would need to use hGetBufNonBlocking to do one byte at a time
|
||||
reads, with a small sleep when there is nothing to read,
|
||||
until it gets an empty read at a point after the process has exited.
|
||||
|
||||
I can't see a way to implement it without the sleep, and that is
|
||||
problimatic, too long sleep will slow down reading, and too short a sleep
|
||||
will waste cpu. It might be possible to use hGetBufSome and cancel it
|
||||
once the process exits, and only then switch to hGetBufNonBlocking to
|
||||
read the rest of the buffer. But it's not clear what would happen hGetBufSome
|
||||
got as far as reading from the handle, but didn't quite manage to finish
|
||||
before being timed out. Seems likely any data it did read would then be lost.
|
||||
|
||||
Think I'll have to abandon the hGetLineUntilExitOrEOF idea.
|
||||
"""]]
|
|
@ -1,20 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2020-11-18T17:23:21Z"
|
||||
content="""
|
||||
Seems the problem with hWaitForInput happens when running it concurrently
|
||||
with hGetLine on the same handle. When I run it sequentially, it does seem
|
||||
to work. Ok.. I was able to implement hGetLineUntilExitOrEOF that seems to
|
||||
work.
|
||||
|
||||
I've converted it to use that, and the test case for this bug is fixed.
|
||||
Also, the sleep based workaround
|
||||
in [[!commit aa492bc65904a19f22ffdfc20d7a5e7052e2f54d]] is removed,
|
||||
and it uses hGetLineUntilExitOrEOF instead.
|
||||
|
||||
However, I still want to do the audit of other things that pipe stderr.
|
||||
Some of them don't do line based input, so will need a different function
|
||||
than hGetLineUntilExitOrEOF. Leaving this bug open until that audit is
|
||||
done.
|
||||
"""]]
|
|
@ -1,11 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 4"
|
||||
date="2020-11-18T15:30:06Z"
|
||||
content="""
|
||||
Thanks for the analysis. (I wish I had some sort of intelligent input
|
||||
to give rather than just being the reporter of several of these ssh
|
||||
issues :/)
|
||||
|
||||
"""]]
|
|
@ -1,10 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 7"
|
||||
date="2020-11-19T15:34:59Z"
|
||||
content="""
|
||||
Thanks. The hanging datalad CI builds are now resolved.
|
||||
|
||||
https://github.com/datalad/git-annex/actions/runs/371469598
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 8"""
|
||||
date="2020-11-19T20:37:57Z"
|
||||
content="""
|
||||
Finished auditing and fixing anywhere in git-annex this could possibly
|
||||
happen.
|
||||
"""]]
|
|
@ -1,15 +0,0 @@
|
|||
As of December 3, the GitHub Actions version of one of the programs (not sure which one) that git-annex bundles in its DMG installer for macOS uses `@loader_path` in the path for one of its dynamic libraries. (The value of the `@loader_path` variable is the directory containing the binary file in which the path appears.) `Build/OSXMkLibs.hs` does not resolve the `@loader_path` syntax, and so this leads to `./Build/Standalone` failing with:
|
||||
|
||||
```
|
||||
cp: ./@loader_path/libbrotlicommon.1.dylib: No such file or directory
|
||||
Standalone: library install failed
|
||||
CallStack (from HasCallStack):
|
||||
error, called at ./Build/OSXMkLibs.hs:72:41 in main:Build.OSXMkLibs
|
||||
```
|
||||
|
||||
I attempted to patch `Build/OSXMkLibs.hs` to handle this myself, but the file containing this syntax is another dynamic lib, and the code doesn't seem to record the original paths that dynamic libraries are copied from, and so the patch would be nontrivial.
|
||||
|
||||
[[!meta author=jwodder]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[done]] per commeents --[[Joey]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 1"
|
||||
date="2020-12-14T14:07:47Z"
|
||||
content="""
|
||||
For your convenience Joey here is a sample fresh (since still keeps failing) log: [http://www.onerussian.com/tmp/git-annex-osx-20201214-1.txt](http://www.onerussian.com/tmp/git-annex-osx-20201214-1.txt) . Link to workflow all results: [https://github.com/datalad/git-annex/actions?query=workflow%3A%22Build+git-annex+on+macOS%22](https://github.com/datalad/git-annex/actions?query=workflow%3A%22Build+git-annex+on+macOS%22)
|
||||
"""]]
|
|
@ -1,25 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2020-12-14T19:08:40Z"
|
||||
content="""
|
||||
install_name_tool: warning: changes being made to the file will invalidate the code signature in: tmp/build-dmg/git-annex.app/Contents/MacOS/bundle/R
|
||||
|
||||
This makes me doubtful that, even if the `@loader_path` problem gets resolved
|
||||
and it builds, it will actually work. It seems likely that if a library is
|
||||
signed, modifying it to have an invalid signature would make the OS refuse to
|
||||
use it.
|
||||
|
||||
The OSX build used for releases is still using an older OSX and does not have
|
||||
either problem. I do notice there is a similar workaround in Build/OSXMkLibs.h
|
||||
for `@executable_path`; it skips such libraries. So skipping `@loader_path`
|
||||
in the same place would probably build, but would it work?
|
||||
|
||||
Since git-annex can be installed from homebrew, I don't know if it's worth
|
||||
trying to keep the standalone app working in the face of such problems.
|
||||
Is there a use case for the .dmg file that this failing build is producing?
|
||||
|
||||
One approach would be to stop bundling system libraries, so the .dmg
|
||||
perhaps is tied to a specific version of OSX. I've added a way to build
|
||||
the dmg that way, make osxapp USE_SYSTEM_LIBS=1
|
||||
"""]]
|
|
@ -1,10 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 3"
|
||||
date="2020-12-14T20:59:06Z"
|
||||
content="""
|
||||
> Is there a use case for the .dmg file that this failing build is producing?
|
||||
|
||||
In our case we build .dmg to have it possibly (re)distributed later on as a build of git-annex to use. But primary use case is just to have it installed and run git-annex and then datalad tests to ensure correct operation.
|
||||
"""]]
|
|
@ -1,7 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2020-12-14T21:43:29Z"
|
||||
content="""
|
||||
In that case, seems fine to build with USE_SYSTEM_LIBS=1
|
||||
"""]]
|
|
@ -1,19 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 5"
|
||||
date="2020-12-15T16:18:39Z"
|
||||
content="""
|
||||
FTR: unfortunately not a complete workaround for us since the 'test' step is done in a clean environment (not build one), where we just install .dmg. That results in git-annex not being able to find those dynamic libraries it had in build environment
|
||||
|
||||
```
|
||||
Run git annex version
|
||||
dyld: Library not loaded: /usr/local/opt/libmagic/lib/libmagic.1.dylib
|
||||
Referenced from: /Applications/git-annex.app/Contents/MacOS/bundle/git-annex
|
||||
Reason: image not found
|
||||
error: git-annex died of signal 6
|
||||
Error: Process completed with exit code 134.
|
||||
```
|
||||
|
||||
ref: [https://github.com/datalad/git-annex/pull/31](https://github.com/datalad/git-annex/pull/31)
|
||||
"""]]
|
|
@ -1,13 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 6"""
|
||||
date="2020-12-29T16:49:52Z"
|
||||
content="""
|
||||
I later removed `USE_SYSTEM_LIBS` and made it skip
|
||||
`@loader_path` as it was already skipping `@executable_path`.
|
||||
|
||||
Looks like the OSX build is succeeding with building and using the
|
||||
app. I suppose this can probably be closed, although we don't know how
|
||||
portable that app might be to systems where the libraries this causes it to
|
||||
skip are at other versions than it needs.
|
||||
"""]]
|
|
@ -1,31 +0,0 @@
|
|||
As of commit b9c1aa925, trying to build git-annex on either Ubuntu or macOS fails with:
|
||||
|
||||
```
|
||||
Annex/TransferrerPool.hs:56:24: error:
|
||||
Not in scope: type constructor or class `MonadFail'
|
||||
|
|
||||
| :: (MonadIO m, MonadFail m, MonadMask m)
|
||||
| ^^^^^^^^^
|
||||
```
|
||||
|
||||
This can be fixed with the following patch:
|
||||
|
||||
```
|
||||
diff --git a/Annex/TransferrerPool.hs b/Annex/TransferrerPool.hs
|
||||
index 973f75629..0de145461 100644
|
||||
--- a/Annex/TransferrerPool.hs
|
||||
+++ b/Annex/TransferrerPool.hs
|
||||
@@ -28,6 +28,7 @@ import Utility.ThreadScheduler
|
||||
import Control.Concurrent
|
||||
import Control.Concurrent.Async
|
||||
import Control.Concurrent.STM hiding (check)
|
||||
+import Control.Monad.Fail (MonadFail)
|
||||
import Control.Monad.IO.Class (MonadIO)
|
||||
import Text.Read (readMaybe)
|
||||
import Data.Time.Clock.POSIX
|
||||
```
|
||||
|
||||
[[!meta author=jwodder]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,12 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2020-12-09T16:40:24Z"
|
||||
content="""
|
||||
Indeed, and thank you for the patch.
|
||||
|
||||
However, MonadFail often suggests a code smell, here it's the `Just ... <-`
|
||||
pattern match which could potentially fail. In fact, it's not possible for
|
||||
it to, so I have instead made that clear in a way that the type checker
|
||||
can understand.
|
||||
"""]]
|
|
@ -1,76 +0,0 @@
|
|||
Due to the type change of `tryMakeLockHandle` in commit ef3ab0769, compilation of git-annex is currently failing on Windows with the below error. I am unsure of the best way to fix it.
|
||||
|
||||
[[!format txt """
|
||||
Utility\LockPool\Windows.hs:25:19: error:
|
||||
* Couldn't match type `(LockHandle, t1)' with `LockHandle'
|
||||
Expected type: IO (Maybe LockHandle)
|
||||
Actual type: IO (Maybe (LockHandle, t1))
|
||||
* In the expression:
|
||||
tryMakeLockHandle
|
||||
P.lockPool file (\ p f -> P.tryTakeLock p f LockShared)
|
||||
(\ f _ -> fmap mk <$> F.lockShared f)
|
||||
In an equation for `lockShared':
|
||||
lockShared file
|
||||
= tryMakeLockHandle
|
||||
P.lockPool file (\ p f -> P.tryTakeLock p f LockShared)
|
||||
(\ f _ -> fmap mk <$> F.lockShared f)
|
||||
|
|
||||
25 | lockShared file = tryMakeLockHandle P.lockPool file
|
||||
|
||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
|
||||
|
||||
Utility\LockPool\Windows.hs:27:18: error:
|
||||
* Couldn't match type `FileLockOps' with `(FileLockOps, t1)'
|
||||
Expected type: IO (Maybe (FileLockOps, t1))
|
||||
Actual type: IO (Maybe FileLockOps)
|
||||
* In the expression: fmap mk <$> F.lockShared f
|
||||
In the fourth argument of `tryMakeLockHandle', namely
|
||||
`(\ f _ -> fmap mk <$> F.lockShared f)'
|
||||
In the expression:
|
||||
tryMakeLockHandle
|
||||
P.lockPool file (\ p f -> P.tryTakeLock p f LockShared)
|
||||
(\ f _ -> fmap mk <$> F.lockShared f)
|
||||
|
|
||||
27 | (\f _ -> fmap mk <$> F.lockShared f)
|
||||
|
||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Utility\LockPool\Windows.hs:36:22: error:
|
||||
* Couldn't match type `(LockHandle, t0)' with `LockHandle'
|
||||
Expected type: IO (Maybe LockHandle)
|
||||
Actual type: IO (Maybe (LockHandle, t0))
|
||||
* In the expression:
|
||||
tryMakeLockHandle
|
||||
P.lockPool file (\ p f -> P.tryTakeLock p f LockExclusive)
|
||||
(\ f _ -> fmap mk <$> F.lockExclusive f)
|
||||
In an equation for `lockExclusive':
|
||||
lockExclusive file
|
||||
= tryMakeLockHandle
|
||||
P.lockPool file (\ p f -> P.tryTakeLock p f LockExclusive)
|
||||
(\ f _ -> fmap mk <$> F.lockExclusive f)
|
||||
|
|
||||
36 | lockExclusive file = tryMakeLockHandle P.lockPool file
|
||||
|
||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
|
||||
|
||||
Utility\LockPool\Windows.hs:38:18: error:
|
||||
* Couldn't match type `FileLockOps' with `(FileLockOps, t0)'
|
||||
Expected type: IO (Maybe (FileLockOps, t0))
|
||||
Actual type: IO (Maybe FileLockOps)
|
||||
* In the expression: fmap mk <$> F.lockExclusive f
|
||||
In the fourth argument of `tryMakeLockHandle', namely
|
||||
`(\ f _ -> fmap mk <$> F.lockExclusive f)'
|
||||
In the expression:
|
||||
tryMakeLockHandle
|
||||
P.lockPool file (\ p f -> P.tryTakeLock p f LockExclusive)
|
||||
(\ f _ -> fmap mk <$> F.lockExclusive f)
|
||||
|
|
||||
38 | (\f _ -> fmap mk <$> F.lockExclusive f)
|
||||
|
||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
"""]]
|
||||
|
||||
[[!meta author=jwodder]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,6 +0,0 @@
|
|||
As of commit 0a9a9c44e, trying to build git-annex on Windows fails because due to a type error. [This patch](https://raw.githubusercontent.com/datalad/git-annex/master/patches/20210630-0a9a9c44e-fix-windows.patch) fixes it.
|
||||
|
||||
[[!meta author=jwodder]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[done]] thankyou! --[[Joey]]
|
|
@ -1,6 +0,0 @@
|
|||
As of commit 4b1b9d7a8, trying to build git-annex on Windows fails because of a missing function argument and a missing import, both in `Annex/Init.hs`. [This patch](https://raw.githubusercontent.com/datalad/git-annex/master/patches/20210622-4b1b9d7a8-fix-windows.patch) fixes both problems.
|
||||
|
||||
[[!meta author=jwodder]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[done]], thank you jwodder! --[[Joey]]
|
|
@ -1,6 +0,0 @@
|
|||
As of commit a706708d1, trying to build git-annex on Windows fails because the import of `oneSecond` from `Utility.ThreadScheduler` is not available. [This patch](https://raw.githubusercontent.com/datalad/git-annex/master/patches/20210607-a706708d1-fix-oneSecond.patch) fixes that.
|
||||
|
||||
[[!meta author=jwodder]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> Thank you for reporting and for the patch. [[applied|done]] --[[Joey]]
|
|
@ -1,6 +0,0 @@
|
|||
As of commit ddd7d1d11, trying to build git-annex on Windows fails due to a function being defined with too many arguments. [This patch](https://raw.githubusercontent.com/datalad/git-annex/master/patches/20210902-ddd7d1d11-fix-windows.patch) fixes it.
|
||||
|
||||
[[!meta author=jwodder]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,179 +0,0 @@
|
|||
### Please describe the problem.
|
||||
|
||||
When unlocking an annexed file and also adding a matching `annex.largefiles=nothing` entry to `.gitattributes`, leaves the file modified after committing both.
|
||||
This seems to happen only recently. Observed on macOS VM with recent annex in DataLad's CI builds. Same situation comes out clean on other machines (with not as recent git-annex). Moreover it appears to be a race condition, since with the reproducer below it happens almost always, but sometimes it does end up with a clean repository.
|
||||
|
||||
Note, that the issue is less about what exactly should happen to that file. git-annex-add reports it's adding it to the git repository and that's fine. If git-annex was to say: "Hey, that's stupid" - that too would probably be fine. What seems wrong either way is a modified file right after committing this file, plus the fact that we didn't observe this with earlier annex versions.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
[[!format sh """
|
||||
mkdir testrepo
|
||||
cd testrepo
|
||||
git init
|
||||
git annex init
|
||||
echo "**/.git* annex.largfiles=nothing" > .gitattributes
|
||||
git annex add .gitattributes
|
||||
echo "something" > somefile
|
||||
git annex add -c annex.dotfiles=true -- .gitattributes somefile
|
||||
git commit -m "to annex"
|
||||
ls -la
|
||||
git status
|
||||
#----------Setup done-----------------
|
||||
git annex unlock somefile
|
||||
echo "* annex.largefiles=nothing" >> .gitattributes
|
||||
git annex add -c annex.dotfiles=true -- .gitattributes somefile
|
||||
git commit -m "adding to git" .gitattributes somefile
|
||||
git status # <- this will (most of the time) show a modified `somefile`
|
||||
"""]]
|
||||
|
||||
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
|
||||
[[!format sh """
|
||||
> git annex version
|
||||
git-annex version: 10.20220222
|
||||
build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Feeds Testsuite S3 WebDAV
|
||||
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-8.10.7 http-client-0.7.11 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
|
||||
operating system: darwin x86_64
|
||||
supported repository versions: 8 9 10
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
|
||||
local repository version: 8
|
||||
> git --version
|
||||
git version 2.35.1
|
||||
"""]]
|
||||
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
On the said system the reproducer script above gives:
|
||||
|
||||
[[!format sh """
|
||||
++ mkdir testrepo
|
||||
++ cd testrepo
|
||||
++ git init
|
||||
Initialized empty Git repository in /Users/appveyor/projects/testrepo/.git/
|
||||
++ git annex init
|
||||
init ok
|
||||
(recording state in git...)
|
||||
++ echo '**/.git* annex.largfiles=nothing'
|
||||
++ git annex add .gitattributes
|
||||
add .gitattributes (non-large file; adding content to git repository) ok
|
||||
(recording state in git...)
|
||||
++ echo something
|
||||
++ git annex add -c annex.dotfiles=true -- .gitattributes somefile
|
||||
add somefile
|
||||
ok
|
||||
(recording state in git...)
|
||||
++ git commit -m 'to annex'
|
||||
[master (root-commit) fd4307e] to annex
|
||||
2 files changed, 2 insertions(+)
|
||||
create mode 100644 .gitattributes
|
||||
create mode 120000 somefile
|
||||
++ ls -la
|
||||
total 8
|
||||
drwxr-xr-x 5 appveyor staff 160 Mar 16 04:36 .
|
||||
drwxr-xr-x 5 appveyor staff 160 Mar 16 04:36 ..
|
||||
drwxr-xr-x 13 appveyor staff 416 Mar 16 04:36 .git
|
||||
-rw-r--r-- 1 appveyor staff 33 Mar 16 04:36 .gitattributes
|
||||
lrwxr-xr-x 1 appveyor staff 180 Mar 16 04:36 somefile -> .git/annex/objects/gG/2m/SHA256E-s10--4bc453b53cb3d914b45f4b250294236adba2c0e09ff6f03793949e7e39fd4cc1/SHA256E-s10--4bc453b53cb3d914b45f4b250294236adba2c0e09ff6f03793949e7e39fd4cc1
|
||||
++ git status
|
||||
On branch master
|
||||
nothing to commit, working tree clean
|
||||
++ git annex unlock somefile
|
||||
unlock somefile ok
|
||||
(recording state in git...)
|
||||
++ echo '* annex.largefiles=nothing'
|
||||
++ git annex add -c annex.dotfiles=true -- .gitattributes somefile
|
||||
add .gitattributes (non-large file; adding content to git repository) ok
|
||||
add somefile (non-large file; adding content to git repository) ok
|
||||
(recording state in git...)
|
||||
++ git commit -m 'adding to git' .gitattributes somefile
|
||||
[master 9a6e407] adding to git
|
||||
2 files changed, 2 insertions(+), 1 deletion(-)
|
||||
rewrite somefile (100%)
|
||||
mode change 120000 => 100644
|
||||
++ git status
|
||||
On branch master
|
||||
Changes to be committed:
|
||||
(use "git restore --staged <file>..." to unstage)
|
||||
modified: somefile
|
||||
"""]]
|
||||
|
||||
Whereas on a different machine it looks like this:
|
||||
|
||||
|
||||
[[!format sh """
|
||||
+ mkdir testrepo
|
||||
+ cd testrepo
|
||||
+ git init
|
||||
Initialized empty Git repository in /tmp/testrepo/.git/
|
||||
+ git annex init
|
||||
init ok
|
||||
(recording state in git...)
|
||||
+ echo **/.git* annex.largfiles=nothing
|
||||
+ git annex add .gitattributes
|
||||
add .gitattributes (non-large file; adding content to git repository) ok
|
||||
(recording state in git...)
|
||||
+ echo something
|
||||
+ git annex add -c annex.dotfiles=true -- .gitattributes somefile
|
||||
add somefile
|
||||
ok
|
||||
(recording state in git...)
|
||||
+ git commit -m to annex
|
||||
[master (root-commit) 408fed5] to annex
|
||||
2 files changed, 2 insertions(+)
|
||||
create mode 100644 .gitattributes
|
||||
create mode 120000 somefile
|
||||
+ ls -la
|
||||
total 36
|
||||
drwxr-xr-x 3 ben ben 4096 Mar 16 10:39 .
|
||||
drwxrwxrwt 29 root root 20480 Mar 16 10:39 ..
|
||||
drwxr-xr-x 9 ben ben 4096 Mar 16 10:39 .git
|
||||
-rw-r--r-- 1 ben ben 33 Mar 16 10:39 .gitattributes
|
||||
lrwxrwxrwx 1 ben ben 180 Mar 16 10:39 somefile -> .git/annex/objects/gG/2m/SHA256E-s10--4bc453b53cb3d914b45f4b250294236adba2c0e09ff6f03793949e7e39fd4cc1/SHA256E-s10--4bc453b53cb3d914b45f4b250294236adba2c0e09ff6f03793949e7e39fd4cc1
|
||||
+ git status
|
||||
On branch master
|
||||
nothing to commit, working tree clean
|
||||
+ git annex unlock somefile
|
||||
unlock somefile ok
|
||||
(recording state in git...)
|
||||
+ echo * annex.largefiles=nothing
|
||||
+ git annex add -c annex.dotfiles=true -- .gitattributes somefile
|
||||
add .gitattributes (non-large file; adding content to git repository) ok
|
||||
add somefile (non-large file; adding content to git repository) ok
|
||||
(recording state in git...)
|
||||
+ git commit -m adding to git .gitattributes somefile
|
||||
[master 865efeb] adding to git
|
||||
2 files changed, 2 insertions(+), 1 deletion(-)
|
||||
rewrite somefile (100%)
|
||||
mode change 120000 => 100644
|
||||
+ git status
|
||||
On branch master
|
||||
nothing to commit, working tree clean
|
||||
+ git annex version
|
||||
git-annex version: 8.20211123
|
||||
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
|
||||
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 8
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
|
||||
local repository version: 8
|
||||
+ git --version
|
||||
git version 2.34.1
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Lots! I love git-annex!
|
||||
|
||||
[[!meta author=bpoldrack]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,14 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="FWIW a detail"
|
||||
date="2022-03-16T14:38:45Z"
|
||||
content="""
|
||||
I believe our corresponding test in datalad started to fail on this date:
|
||||
```
|
||||
(git)smaug:/mnt/datasets/datalad/ci/git-annex/builds/2022/02[master]git
|
||||
$> git grep 'test_partial_unlocked.*FAIL' | head -n 1
|
||||
cron-20220219/build-macos.yaml-595-85dd9355-failed/1_test-datalad (master).txt:2022-02-19T02:37:34.9834050Z datalad.core.local.tests.test_save.test_partial_unlocked ... FAIL
|
||||
```
|
||||
which had git-annex-macos-dmg_10.20220127+git57-gb481ec273_x64 -- so that or some commit shortly before likely introduced that regression
|
||||
"""]]
|
|
@ -1,77 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2022-03-21T17:33:28Z"
|
||||
content="""
|
||||
Isolated to [[!commit c68f52c6a25ebd8a12f4572da53fbba827d682aa]]
|
||||
|
||||
Since that commit is only a speed improvement, it can be reverted, which
|
||||
does avoid the problem.
|
||||
|
||||
But I don't really see a bug in that commit either. It only causes
|
||||
git to smudge the file when it's unlocked, rather than at some later
|
||||
point. The result should be the same either way. Except, of course,
|
||||
that you're varying the annex.largefiles config, so when git smudges
|
||||
the file later, it sees a different result than when it smudges the file
|
||||
while unlock is running.
|
||||
|
||||
I think that the actually surprising thing here is that
|
||||
`git-annex add somefile` re-adds the file (adding it to git per the new
|
||||
annex.largefiles configuration). The file is in typechange state at that
|
||||
point but is not modified, so it does not need to be re-added at all.
|
||||
But add seeks `withUnmodifiedUnlockedPointers` which finds files in
|
||||
typechange state.
|
||||
|
||||
So git-annex add decides to add the file to git even though it was
|
||||
previously added to the annex. At that point, I'd expect `git add` would
|
||||
update the index and replace the smudged content with the unsmudged
|
||||
content. However, it actually does not!
|
||||
Looking at `git ls-files --stage` before and after git-annex add,
|
||||
the annex pointer content remains staged:
|
||||
|
||||
100644 bf1114c25db2833499838552f3fa69635fcc1292 0 somefile
|
||||
add .gitattributes (non-large file; adding content to git repository) ok
|
||||
add somefile (non-large file; adding content to git repository) ok
|
||||
(recording state in git...)
|
||||
100644 bf1114c25db2833499838552f3fa69635fcc1292 0 somefile
|
||||
|
||||
The only reason that git commit goes on to commit the non-annexed
|
||||
content is because you pass somefile to it, so it re-adds somefile
|
||||
at that point, which runs the smudge filter, which passes the content
|
||||
through.
|
||||
|
||||
And that's why git status then sees the file as modified --
|
||||
the index has the annex pointer still staged in it, which is different
|
||||
from what got committed.
|
||||
|
||||
So, probably this same thing could happen in another situation, even if
|
||||
the change to git-annex unlock were reverted. Eg, if something else
|
||||
caused git smudge to run just after unlock, before the annex.largefiles
|
||||
config got changed.
|
||||
|
||||
Why is `git add` not updating the index? Presumably because the inode
|
||||
and mtime have not changed, so it does not think it needs to. Even though
|
||||
it's being run with the smudge filter disabled, and so sees a different
|
||||
hash than the one in the index. I think a case could be made for this
|
||||
being a bug in `git add`. Although maybe the `--renormalize` option
|
||||
is intended to handle this kind of situation.
|
||||
|
||||
The documented way for a user to change a file from being stored in git-annex
|
||||
to git is to first `git rm --cached` and then `git annex add --force-small`.
|
||||
Which avoids that `git add` behavior.
|
||||
|
||||
I tracked git-annex add's behavior here back to
|
||||
[[!commit 2743224658cee33657ad30653ff97de2363366c6]].
|
||||
Which addressed an old bug, filed by Kyle, which can be seen at
|
||||
[[!commit 5567496518]]. So it was intentional that unlocking a file
|
||||
and then `git annex add` should lock it, for consistency with v5
|
||||
and for consistency with unlocking a file
|
||||
and then modifying it, and then `git annex add`. I am actually a bit
|
||||
surprised at this behavior, but it was intended to handle locking files,
|
||||
not adding files to git.
|
||||
|
||||
So, I think it would be fine for `git-annex add` to skip over a typechanged
|
||||
file when it would be added to git rather than to the annex.
|
||||
|
||||
I've implemented that, and it does fix this bug.
|
||||
"""]]
|
|
@ -1,80 +0,0 @@
|
|||
My understanding is that a preferred content value of "anything" (or
|
||||
equivalently the standard group "backup") should consider deleted
|
||||
files from a remote as wanted. I'm not a heavy user of preferred
|
||||
content myself, so this understanding is based primarily off of the
|
||||
[preferred_content/standard_groups][0] documentation saying
|
||||
|
||||
```
|
||||
### backup
|
||||
|
||||
All content is wanted. Even content of old/deleted files.
|
||||
|
||||
`anything`
|
||||
```
|
||||
|
||||
At least with the scenarios I've constructed, I haven't been able to
|
||||
find a case where old files are considered. As an example:
|
||||
|
||||
[[!format sh """
|
||||
cd "$(mktemp -d ${TMPDIR:-/tmp}/gx-XXXXXXX)"
|
||||
|
||||
git init a
|
||||
(
|
||||
cd a
|
||||
git annex init a
|
||||
>foo && git annex add foo && git commit -m"add foo"
|
||||
git rm foo && git commit -m"delete foo"
|
||||
)
|
||||
|
||||
git clone a b
|
||||
(
|
||||
cd b
|
||||
git annex init b
|
||||
git annex wanted . anything
|
||||
git annex sync
|
||||
|
||||
set -x
|
||||
|
||||
git show git-annex:uuid.log
|
||||
git show git-annex:preferred-content.log
|
||||
|
||||
git annex find --want-get
|
||||
git annex get --auto
|
||||
git annex unused
|
||||
)
|
||||
"""]]
|
||||
|
||||
```
|
||||
[...]
|
||||
+ git show git-annex:uuid.log
|
||||
17fae615-a90b-405a-85c2-9d0f28246bdf a timestamp=1586190820.385256186s
|
||||
1f0ad7d6-31ba-4d7e-a2e6-ab6acb3338c9 b timestamp=1586190820.750487084s
|
||||
+ git show git-annex:preferred-content.log
|
||||
1f0ad7d6-31ba-4d7e-a2e6-ab6acb3338c9 anything timestamp=1586190820.867633471s
|
||||
+ git annex find --want-get
|
||||
+ git annex get --auto
|
||||
+ git annex unused
|
||||
unused . (checking for unused data...) ok
|
||||
```
|
||||
|
||||
The output above is with 8.20200330, but I saw the same result when I
|
||||
tried with 7.20190819.
|
||||
|
||||
I haven't been able to think of another way to interpret "Even content
|
||||
of old/deleted files", so I think at least this is a documentation
|
||||
bug.
|
||||
|
||||
[Related DataLad issue](https://github.com/datalad/datalad/issues/4369#issuecomment-608603732)
|
||||
|
||||
[0]: https://git-annex.branchable.com/preferred_content/standard_groups/
|
||||
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Yes. Thank you! :>
|
||||
|
||||
|
||||
[[!meta author=kyle]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[done]] --[[Joey]]
|
|
@ -1,21 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2020-04-13T16:03:16Z"
|
||||
content="""
|
||||
All of the commands you ran looked only at files in the current tree,
|
||||
so of course did not operate on deleted files. Eg, "git annex get" and "git
|
||||
annex get ." are identical, and obviously the current directory does not
|
||||
contain any deleted files.
|
||||
|
||||
But there are commands that do operate on deleted files, including eg `git
|
||||
annex sync --content --all` and `git annex get --all`. Those commands will
|
||||
honor preferred content settings including "anything" when operating on
|
||||
deleted files. (The assistant also periodically does.)
|
||||
|
||||
I've added a note to that effect to the preferred content
|
||||
documentation. Also several command's man pages didn't make entirely
|
||||
explicit that they operated on files in the current tree
|
||||
(although the main git-annex man page does document that),
|
||||
so improved that documentation too.
|
||||
"""]]
|
|
@ -1,13 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 2"
|
||||
date="2020-04-13T18:58:50Z"
|
||||
content="""
|
||||
> All of the commands you ran looked only at files in the current
|
||||
> tree, so of course did not operate on deleted files.
|
||||
|
||||
Okay. I incorrectly thought `--auto` would bypass that.
|
||||
|
||||
Thank you for the doc updates.
|
||||
"""]]
|
|
@ -1,37 +0,0 @@
|
|||
I am using `git annex export` to export to a WEBDAV special remote configured with `exporttree=yes`. Works beautifully!
|
||||
|
||||
However, now I have a repository with a submodule, and I want to export them both, such that they look like a single worktree on the WEBDAV service. I can, in-principle, achieve this by using an appropriately adjusted WEBDAV URL. But git-annex exports an empty file with the name of the submodule in the location where I would need a directory to place the submodule export -- which the WEBDAV services rightfully acknowledges with a 409 (conflict) response, when I attempt running `initremote` on the submodule nevertheless.
|
||||
|
||||
I found no way to use an `wanted` expression to automatically exclude all submodules.
|
||||
|
||||
The options I am left with are:
|
||||
|
||||
1. Remove the empty submodule placeholders after an export.
|
||||
|
||||
2. Maintain a dedicated wanted expression with a continuously updated exclusion list covering all submodule mounts.
|
||||
|
||||
(2) Might allow me to not have to fiddle with WEBDAV requests myself, but it doesn't seem "right", given that the submodule layout and number can change with any state of the repository.
|
||||
|
||||
I wonder, though, what benefit the export of these placeholder files provides? Maybe they should not be exported at all. Or, alternatively, it might make sense to actually export them as directories right away. I might be biased, but that might be a more appropriate representation than an empty file.
|
||||
|
||||
|
||||
I have used a Nextcloud instance for me attempts, and a repository with a single submodule (both created with datalad in my case, but that should not be relevant here). Followed by
|
||||
|
||||
```
|
||||
git annex initremote wd type=webdav url=https://.../files/user/datalad/dummy encryption=none exporttree=yes
|
||||
git annex initremote wd type=webdav url=https://.../files/user/datalad/dummy/submod encryption=none exporttree=yes
|
||||
```
|
||||
|
||||
in both repositories respectively.
|
||||
|
||||
I have been using `git-annex version: 8.20201127`
|
||||
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
It is the rock that I stand on.
|
||||
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2021-03-12T15:46:04Z"
|
||||
content="""
|
||||
I think it's simply a bug that it exports submodules at all. Very easy to
|
||||
forget about the special case of those wacky submodule directories in git.
|
||||
"""]]
|
|
@ -1,14 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2021-03-12T16:16:44Z"
|
||||
content="""
|
||||
Easy enough to exclude them from export, that's done.
|
||||
|
||||
But the other side of the coin is, importing back from there will generate
|
||||
a remote-tracking branch without the submodule, and merging it will result
|
||||
in a tree that does not include the submodule, as if it were deleted.
|
||||
|
||||
So, importing needs to somehow add back submodules,
|
||||
unless the submodule location has been overwritten with a file on the remote.
|
||||
"""]]
|
|
@ -1,7 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2021-03-12T17:28:10Z"
|
||||
content="""
|
||||
All right, both export and import are fixed.
|
||||
"""]]
|
|
@ -1,10 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="mih"
|
||||
avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd"
|
||||
subject="Confirm fix"
|
||||
date="2021-03-13T16:23:34Z"
|
||||
content="""
|
||||
This is working now. `git-annex version: 8.20210311-gecee702b3`
|
||||
|
||||
Thx!
|
||||
"""]]
|
|
@ -1,59 +0,0 @@
|
|||
### Please describe the problem.
|
||||
|
||||
FTBFS for two days (this error matches only on linux, didn't check others why they are red in https://github.com/datalad/git-annex):
|
||||
|
||||
```
|
||||
2022-06-29T03:25:00.4893455Z E: build failed
|
||||
2022-06-29T03:25:00.4894450Z I: the tail of the log
|
||||
2022-06-29T03:25:00.4903612Z [310 of 677] Compiling Creds ( Creds.hs, dist/build/git-annex/git-annex-tmp/Creds.o )
|
||||
2022-06-29T03:25:00.4908788Z [311 of 677] Compiling Remote.Helper.AWS ( Remote/Helper/AWS.hs, dist/build/git-annex/git-annex-tmp/Remote/Helper/AWS.o )
|
||||
2022-06-29T03:25:00.4910166Z [312 of 677] Compiling P2P.Address ( P2P/Address.hs, dist/build/git-annex/git-annex-tmp/P2P/Address.o )
|
||||
2022-06-29T03:25:00.4911450Z [313 of 677] Compiling P2P.Auth ( P2P/Auth.hs, dist/build/git-annex/git-annex-tmp/P2P/Auth.o )
|
||||
2022-06-29T03:25:00.4912850Z [314 of 677] Compiling Annex.Notification ( Annex/Notification.hs, dist/build/git-annex/git-annex-tmp/Annex/Notification.o )
|
||||
2022-06-29T03:25:00.4913599Z
|
||||
2022-06-29T03:25:00.4913895Z Annex/Notification.hs:37:42: error:
|
||||
2022-06-29T03:25:00.4914739Z * Couldn't match type `Annex.AnnexState' with `Annex.AnnexRead'
|
||||
2022-06-29T03:25:00.4915686Z Expected type: Annex.AnnexState -> DesktopNotify
|
||||
2022-06-29T03:25:00.4916569Z Actual type: Annex.AnnexRead -> DesktopNotify
|
||||
2022-06-29T03:25:00.4917418Z * In the first argument of `Annex.getState', namely
|
||||
2022-06-29T03:25:00.4918159Z `Annex.desktopnotify'
|
||||
2022-06-29T03:25:00.4918842Z In a stmt of a 'do' block:
|
||||
2022-06-29T03:25:00.4919609Z wanted <- Annex.getState Annex.desktopnotify
|
||||
2022-06-29T03:25:00.4920362Z In the expression:
|
||||
2022-06-29T03:25:00.4921081Z do wanted <- Annex.getState Annex.desktopnotify
|
||||
2022-06-29T03:25:00.4921878Z if (notifyStart wanted || notifyFinish wanted) then
|
||||
2022-06-29T03:25:00.4922815Z do client <- liftIO DBus.Client.connectSession
|
||||
2022-06-29T03:25:00.4923576Z ....
|
||||
2022-06-29T03:25:00.4924147Z else
|
||||
2022-06-29T03:25:00.4924729Z a NotifyWitness
|
||||
2022-06-29T03:25:00.4925310Z |
|
||||
2022-06-29T03:25:00.4926060Z 37 | wanted <- Annex.getState Annex.desktopnotify
|
||||
2022-06-29T03:25:00.4926916Z | ^^^^^^^^^^^^^^^^^^^
|
||||
2022-06-29T03:25:00.4927396Z
|
||||
2022-06-29T03:25:00.4927702Z Annex/Notification.hs:60:34: error:
|
||||
2022-06-29T03:25:00.4928555Z * Couldn't match type `Annex.AnnexState' with `Annex.AnnexRead'
|
||||
2022-06-29T03:25:00.4929502Z Expected type: Annex.AnnexState -> DesktopNotify
|
||||
2022-06-29T03:25:00.4930367Z Actual type: Annex.AnnexRead -> DesktopNotify
|
||||
2022-06-29T03:25:00.4931212Z * In the first argument of `Annex.getState', namely
|
||||
2022-06-29T03:25:00.4931962Z `Annex.desktopnotify'
|
||||
2022-06-29T03:25:00.4935814Z In a stmt of a 'do' block:
|
||||
2022-06-29T03:25:00.4936937Z wanted <- Annex.getState Annex.desktopnotify
|
||||
2022-06-29T03:25:00.4937934Z In the expression:
|
||||
2022-06-29T03:25:00.4938808Z do wanted <- Annex.getState Annex.desktopnotify
|
||||
2022-06-29T03:25:00.4939703Z when (notifyFinish wanted)
|
||||
2022-06-29T03:25:00.4940467Z $ liftIO
|
||||
2022-06-29T03:25:00.4941336Z $ do client <- DBus.Client.connectSession
|
||||
2022-06-29T03:25:00.4942492Z void $ Notify.notify client (droppedNote ok (fromRawFilePath f))
|
||||
2022-06-29T03:25:00.4943446Z |
|
||||
2022-06-29T03:25:00.4944547Z 60 | wanted <- Annex.getState Annex.desktopnotify
|
||||
2022-06-29T03:25:00.4945576Z | ^^^^^^^^^^^^^^^^^^^
|
||||
2022-06-29T03:25:00.4946531Z make[3]: *** [Makefile:58: git-annex] Error 1
|
||||
|
||||
```
|
||||
|
||||
full log at https://github.com/datalad/git-annex/runs/7124197096?check_suite_focus=true or on smaug on `/mnt/datasets/datalad/ci/git-annex/builds/2022/06/cron-20220629/build-ubuntu.yaml-744-90215d4e-failed/1_build-package (1).txt`
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] earlier today --[[Joey]]
|
|
@ -1,119 +0,0 @@
|
|||
### Please describe the problem.
|
||||
I tried using GitLab as a `--type git-lfs` special remote. I followed the steps in https://git-annex.branchable.com/tips/storing_data_in_git-lfs/ for both GitHub and GitLab.
|
||||
While it worked as described for GitHub I get an error message when trying to push data to GitLab.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
[[!format sh """
|
||||
# basic repository setup
|
||||
mkdir annex
|
||||
cd annex
|
||||
git init
|
||||
git annex init
|
||||
|
||||
# create a random binary file
|
||||
head -c 100 /dev/urandom >random.bin
|
||||
git annex add .
|
||||
git commit -a -m added
|
||||
|
||||
# set GitHub as a remote (repo exists and is public: https://github.com/iimog/annex)
|
||||
git remote add github git@github.com:iimog/annex
|
||||
git push --all github
|
||||
git annex initremote lfs type=git-lfs encryption=none url=git@github.com:iimog/annex
|
||||
git annex copy * --to lfs
|
||||
#copy random.bin (to lfs...)
|
||||
#ok
|
||||
#(recording state in git...)
|
||||
|
||||
# set GitLab as a remote (repo exists and is public: https://gitlab.com/iimog/annex)
|
||||
git remote add gitlab git@gitlab.com:iimog/annex
|
||||
git push --all gitlab
|
||||
git annex initremote lfs-gitlab type=git-lfs encryption=none url=git@gitlab.com:iimog/annex
|
||||
#
|
||||
# Unable to parse git config from gitlab
|
||||
#
|
||||
# Remote gitlab does not have git-annex installed; setting annex-ignore
|
||||
#
|
||||
# This could be a problem with the git-annex installation on the remote. Please make sure that git-annex-shell is available in PATH when you ssh into the remote. Once you have fixed the git-annex installation, run: git annex enableremote gitlab
|
||||
#initremote lfs-gitlab ok
|
||||
#(recording state in git...)
|
||||
git annex copy * --to lfs-gitlab
|
||||
#copy random.bin (to lfs-gitlab...)
|
||||
#<long error message>
|
||||
#--> The error in short is a "Bad Request" (details see below)
|
||||
|
||||
# Try again after explicitly calling enableremote
|
||||
git annex enableremote gitlab
|
||||
#enableremote gitlab ok
|
||||
git annex copy * --to lfs-gitlab
|
||||
#copy random.bin (to lfs-gitlab...)
|
||||
#<long error message>
|
||||
#--> This still results in the same "Bad Request" error
|
||||
"""]]
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
git-annex version: 8.20210903-ga4d179c
|
||||
Ubuntu 20.04.3 LTS
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
I initially discovered this issue while using DataLad and reported the issue [here](https://github.com/datalad/datalad/issues/6126).
|
||||
As suspected by @yarikoptic the issue also exists when using git-annex directly, not through DataLad.
|
||||
|
||||
On a self-hosted GitLab instance the following error was found in the server log:
|
||||
[[!format sh """
|
||||
2021/10/28 21:07:13 [error] 7412#0: *1869253 client sent invalid chunked body while sending request to upstream, client: <...>, server: <...>, request: "PUT /iimog/my-cool-ds2.git/gitlab-lfs/objects/92f711c1799c5aa936c2be297171d08afc19ba97899b57a1f67f2e70486395aa/80547 HTTP/1.1", upstream: "http://unix/var/opt/gitlab/gitlab-workhorse/sockets/socket:/iimog/my-cool-ds2.git/gitlab-lfs/objects/92f711c1799c5aa936c2be297171d08afc19ba97899b57a1f67f2e70486395aa/80547", host: "<...>"
|
||||
"""]]
|
||||
|
||||
This is the full error message I got when calling `git annex copy * --to lfs-gitlab`:
|
||||
[[!format sh """
|
||||
# If you can, paste a complete transcript of the problem occurring here.
|
||||
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
|
||||
$ git annex copy * --to lfs-gitlab
|
||||
|
||||
copy random.bin (to lfs-gitlab...)
|
||||
|
||||
HttpExceptionRequest Request {
|
||||
host = "gitlab.com"
|
||||
port = 443
|
||||
secure = True
|
||||
requestHeaders = [("Authorization","<REDACTED>"),("Content-Type","application/octet-stream"),("Transfer-Encoding","chunked"),("User-Agent","git-annex/8.20210903-ga4d179c")]
|
||||
path = "/iimog/annex.git/gitlab-lfs/objects/04b38782b7f5990e090577047fb54b52e9771ba57c1c36e21f4702049adfcfe3/100"
|
||||
queryString = ""
|
||||
method = "PUT"
|
||||
proxy = Nothing
|
||||
rawBody = False
|
||||
redirectCount = 10
|
||||
responseTimeout = ResponseTimeoutDefault
|
||||
requestVersion = HTTP/1.1
|
||||
}
|
||||
(StatusCodeException (Response {responseStatus = Status {statusCode = 400, statusMessage = "Bad Request"}, responseVersion = HTTP/1.1, responseHeaders = [("Server","cloudflare"),("Date","Tue, 09 Nov 2021 15:03:11 GMT"),("Content-Type","text/html"),("Content-Length","155"),("Connection","close"),("CF-RAY","6ab7ed2afb9fdfcf-FRA")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}) "<html>\r\n<head><title>400 Bad Request</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n")
|
||||
|
||||
|
||||
HttpExceptionRequest Request {
|
||||
host = "gitlab.com"
|
||||
port = 443
|
||||
secure = True
|
||||
requestHeaders = [("Authorization","<REDACTED>"),("Content-Type","application/octet-stream"),("Transfer-Encoding","chunked"),("User-Agent","git-annex/8.20210903-ga4d179c")]
|
||||
path = "/iimog/annex.git/gitlab-lfs/objects/04b38782b7f5990e090577047fb54b52e9771ba57c1c36e21f4702049adfcfe3/100"
|
||||
queryString = ""
|
||||
method = "PUT"
|
||||
proxy = Nothing
|
||||
rawBody = False
|
||||
redirectCount = 10
|
||||
responseTimeout = ResponseTimeoutDefault
|
||||
requestVersion = HTTP/1.1
|
||||
}
|
||||
(StatusCodeException (Response {responseStatus = Status {statusCode = 400, statusMessage = "Bad Request"}, responseVersion = HTTP/1.1, responseHeaders = [("Server","cloudflare"),("Date","Tue, 09 Nov 2021 15:03:12 GMT"),("Content-Type","text/html"),("Content-Length","155"),("Connection","close"),("CF-RAY","6ab7ed2c9a4ad6b5-FRA")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}) "<html>\r\n<head><title>400 Bad Request</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n")
|
||||
failed
|
||||
copy: 1 failed
|
||||
|
||||
|
||||
# End of transcript or log.
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
Yes, I'm using DataLad for some of my projects and I'm really impressed how it makes use of git-annex to solve many of the tasks that I struggled with pure git before.
|
||||
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,56 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2021-11-09T20:09:01Z"
|
||||
content="""
|
||||
Let's see.. git-lfs endpoint discovery over ssh works.
|
||||
|
||||
The request to start a transfer works:
|
||||
|
||||
host = "gitlab.com"
|
||||
path = "/joeyh/test.git/info/lfs/objects/batch"
|
||||
...
|
||||
[2021-11-10 12:06:35.409182815] (Remote.GitLFS) Status {statusCode = 200, statusMessage = "OK"}
|
||||
|
||||
So it's the actual PUT that fails:
|
||||
|
||||
requestHeaders = [("Authorization","<REDACTED>"),("Content-Type","application/octet-stream"),("Transfer-Encoding","chunked"),("User-Agent","git-annex/8.20211029-ga5a7d8433")]
|
||||
path = "/joeyh/test.git/gitlab-lfs/objects/922d58c647a679e17ee7c30f7de0111b56b90e84129fa3663568b81822a2628a/30"
|
||||
|
||||
Seems that the Transfer-Encoding chunked header is the problem.
|
||||
That header is provided by the git-lfs endpoint as one to include in
|
||||
the PUT (along with the Authorization header), and git-annex dutifully does
|
||||
include it. But it seems that does not make the PUT use that
|
||||
transfer encoding. And then in the server error, we see
|
||||
"invalid chunked body".
|
||||
|
||||
I tried filtering out the Transfer-Encoding header, and that does
|
||||
fix the problem. But I dunno if that's the best fix. Should git-annex support
|
||||
Transfer-Encoding chunked?
|
||||
|
||||
git-lfs has itself supported Transfer-Encoding chunked since 2015,
|
||||
see <https://github.com/git-lfs/git-lfs/issues/385>. That says
|
||||
"client may send data via chunked Transfer-Encoding when the server
|
||||
explicitly advertises that it's supported". Which is an interesting
|
||||
wording -- "may", "supported" -- implying it's not required to use it.
|
||||
|
||||
The API docs <https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md>
|
||||
says the http headers are "Optional hash of String HTTP header key/value
|
||||
pairs to apply to the request". I think it means optional as in the
|
||||
server may optionally not send any, not necessarily
|
||||
that applying them to the request is optional, but that's not really clear.
|
||||
(Surely a header like Authorization is not optional to include.)
|
||||
|
||||
If the headers are not optional to include then the API would let the server
|
||||
specify any http headers at all, and the client has to send a PUT that includes
|
||||
those headers and that complies with them. So Transfer-Encoding deflate would
|
||||
need to use that compression method, etc.
|
||||
|
||||
But looking at the git-lfs implementation, it only actually handles
|
||||
Transfer-Encoding chunked and not other values. I think it may also
|
||||
not include other headers than Authorization in the PUT?
|
||||
|
||||
It seems possible there are other headers that might cause problems if they are
|
||||
blindly copied into the PUT. Content-Encoding is the only other obvious one,
|
||||
but who knows what may lurk in some odd corner of a HTTP spec.
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2021-11-10T17:21:01Z"
|
||||
content="""
|
||||
Fixed by not passing through those 2 problem headers. And also made it
|
||||
actually used chunked encoding when the server indicates it's supported.
|
||||
"""]]
|
|
@ -1,9 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="iimog@2c22a44141070c04b943932b697818a686859677"
|
||||
nickname="iimog"
|
||||
avatar="http://cdn.libravatar.org/avatar/e6b150bcd25f22fa74abde05e5d6d4eb"
|
||||
subject="It works"
|
||||
date="2021-11-12T08:14:32Z"
|
||||
content="""
|
||||
Thank you so much for this. I can confirm that using the latest version, I can push to GitLab LFS with git-annex and it also works when used through DataLad. This opens a whole range of options for me, so thanks a million!
|
||||
"""]]
|
|
@ -1,68 +0,0 @@
|
|||
I've been thinking a bit more about `annex.dotfiles` in the context of
|
||||
[this forum post][0]. It seems to me that annexed dotfiles can jump
|
||||
to git in a way that's surprising and worth raising as a possible bug.
|
||||
|
||||
Say that I have repo with `annex.dotfiles=true` in the .git/config,
|
||||
and I add some dotfiles to the annex. Then, someone clones that repo
|
||||
and goes into an adjusted state (either by running `git annex adjust
|
||||
--unlock` or by being on a crippled file system). In that clone,
|
||||
calling `annex get` on any of the annexed dotfiles will lead to those
|
||||
files being added to the index as regular files. (Demo included
|
||||
below.)
|
||||
|
||||
The above issue could be resolved by the user storing
|
||||
`annex.dotfiles=true` in `git-annex:config.log`, but perhaps it'd be
|
||||
feasible for git-annex to guard against already annexed dotfiles
|
||||
migrating to git?
|
||||
|
||||
Thanks in advance.
|
||||
|
||||
[[!format sh """
|
||||
git annex version | head -1
|
||||
|
||||
cd "$(mktemp -d --tmpdir gx-XXXXXXX)"
|
||||
git init a
|
||||
(
|
||||
cd a
|
||||
git annex init a
|
||||
git config annex.dotfiles true
|
||||
mkdir .reallybig
|
||||
echo "a" >.reallybig/foo
|
||||
git annex add .reallybig/foo
|
||||
git commit -m"add foo"
|
||||
)
|
||||
|
||||
git clone a b
|
||||
(
|
||||
cd b
|
||||
git annex init b
|
||||
git annex adjust --unlock
|
||||
git annex get .reallybig
|
||||
git status
|
||||
git diff --cached
|
||||
)
|
||||
"""]]
|
||||
|
||||
```
|
||||
git-annex version: 8.20200226
|
||||
[...]
|
||||
On branch adjusted/master(unlocked)
|
||||
Changes to be committed:
|
||||
modified: .reallybig/foo
|
||||
|
||||
diff --git a/.reallybig/foo b/.reallybig/foo
|
||||
index 3de500c..7898192 100644
|
||||
--- a/.reallybig/foo
|
||||
+++ b/.reallybig/foo
|
||||
@@ -1 +1 @@
|
||||
-/annex/objects/SHA256E-s2--87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7
|
||||
+a
|
||||
```
|
||||
|
||||
[0]: https://git-annex.branchable.com/forum/Get_annex.dotfiles__61__true_behavior_without_persistent_configuration__63__/
|
||||
|
||||
|
||||
[[!meta author=kyle]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,18 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2020-03-09T17:00:38Z"
|
||||
content="""
|
||||
This looks like a serious regression in 8.x. It is not an intentional part
|
||||
of the annex.dotfiles change. I have confirmed the last 7.x release did not
|
||||
have the behavior.
|
||||
|
||||
(Notice that annex.dotfiles is not used in the repo where the problem
|
||||
occurs. The regression was probably introduced by the annex.dotfiles
|
||||
changes, but the dotfile could just as well have been added by git-annex v5
|
||||
with git-annex add --include-dotfiles and the result committed to git would
|
||||
be the same.)
|
||||
|
||||
A simpler test case is to unlock the dotfile in the first repo, then
|
||||
all that needs to be run in its clone is `git annex get`.
|
||||
"""]]
|
|
@ -1,17 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2020-03-09T18:07:12Z"
|
||||
content="""
|
||||
Analysis: Command.Smudge.shouldAnnex was changed
|
||||
in [[!commit 3cd375723658398a4b15ff8ec19e562ec4622488]] to check, for dotfiles,
|
||||
if annex.dotfiles is set. If not, it always returns False, so the file
|
||||
always gets added to git when it's smudged.
|
||||
|
||||
After getting a file, git-annex tells git to update its index, which
|
||||
smudges the file. Of course, there are also plenty of other things that can
|
||||
cause a file to be smudged.
|
||||
|
||||
Bad. Fix is very easy, luckily. It just needs to, rather than returning
|
||||
False, checking if the file was annexed, same as is done for non-dotfiles.
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="thanks"
|
||||
date="2020-03-09T18:29:26Z"
|
||||
content="""
|
||||
Thank you for the analysis and the fix!
|
||||
"""]]
|
|
@ -1,26 +0,0 @@
|
|||
Neither successful nor failing exports via a WEBDAV special remote have appropriate `file` properties in the produced JSON records.
|
||||
|
||||
```
|
||||
% git annex export master --to 4shared --json --json-error-messages
|
||||
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
|
||||
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
|
||||
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
|
||||
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
|
||||
{"command":"export 4shared","success":false,"input":[],"error-messages":[" DAV failure: Status {statusCode = 500, statusMessage = \"Internal Server Error\"} \"<html><body><h1>Server Error</h1></body></html>\" HTTP request: \"MOVE\" \"/dummy/git-annex-webdav-tmp-MD5E-s143466--368857ff4d0e4b14c745d94db7f7d3ff.pdf\""," DAV failure: Status {statusCode = 500, statusMessage = \"Internal Server Error\"} \"<html><body><h1>Server Error</h1></body></html>\" HTTP request: \"MOVE\" \"/dummy/git-annex-webdav-tmp-MD5E-s143466--368857ff4d0e4b14c745d94db7f7d3ff.pdf\""],"file":null}
|
||||
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
|
||||
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
|
||||
git-annex: export: 1 failed
|
||||
```
|
||||
|
||||
```
|
||||
git-annex version: 8.20210223
|
||||
```
|
||||
|
||||
This behavior was observed using the 4shared service, but I believe git-annex knows which file a success or failure belongs to and could report it regardless of service oddities.
|
||||
|
||||
|
||||
Thx!
|
||||
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
|
@ -1,12 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2021-03-12T18:09:48Z"
|
||||
content="""
|
||||
I've fixed this.
|
||||
|
||||
However, there is still a case where the file field is null, which is when
|
||||
an already exported file is renamed. Since nothing is transferred in that
|
||||
case, and 2 files are involved which doesn't fit the one file in the json,
|
||||
I left it as-is.
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="mih"
|
||||
avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd"
|
||||
subject="Thx!"
|
||||
date="2021-03-13T16:05:51Z"
|
||||
content="""
|
||||
Confirmed working with git-annex version 8.20210311-gecee702b3
|
||||
"""]]
|
|
@ -1,42 +0,0 @@
|
|||
### Please describe the problem.
|
||||
|
||||
See transcript. Didn't bisect fully but 6.20160128-g599a3ba seems to work ok, 6.20160211-g479a094 already dies off
|
||||
|
||||
[[!format sh """
|
||||
|
||||
datalads-imac:~ yoh$ git annex version
|
||||
git-annex version: 6.20160221-g3d1895e
|
||||
...
|
||||
datalads-imac:~ yoh$ mkdir -p /tmp/123; cd /tmp/123; git init; git annex init;
|
||||
Initialized empty Git repository in /private/tmp/123/.git/
|
||||
init ok
|
||||
(recording state in git...)
|
||||
|
||||
datalads-imac:123 yoh$ rm -f /tmp/pipe; mkfifo /tmp/pipe; cat /tmp/pipe | git -c receive.autogc=false annex addurl -c annex.largefiles=exclude=*.txt --with-files --json --batch
|
||||
{"command":"addurl","file":"1.png","note":"downloading http://www.onerussian.com/tmp/banner.png ..."error: git-annex died of signal 4
|
||||
|
||||
# now with --debug
|
||||
datalads-imac:123 yoh$ rm -f /tmp/pipe; mkfifo /tmp/pipe; cat /tmp/pipe | git -c receive.autogc=false annex addurl -c annex.largefiles=exclude=*.txt --with-files --json --batch --debug
|
||||
[2016-02-23 08:32:49.160686] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.largefiles=exclude=*.txt","show-ref","git-annex"]
|
||||
[2016-02-23 08:32:49.167372] process done ExitSuccess
|
||||
[2016-02-23 08:32:49.167479] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.largefiles=exclude=*.txt","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2016-02-23 08:32:49.172795] process done ExitSuccess
|
||||
[2016-02-23 08:32:49.173039] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.largefiles=exclude=*.txt","log","refs/heads/git-annex..b2892e67682c5240dadb5d439da810edbdb882df","-n1","--pretty=%H"]
|
||||
[2016-02-23 08:32:49.178586] process done ExitSuccess
|
||||
[2016-02-23 08:32:49.179085] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.largefiles=exclude=*.txt","cat-file","--batch"]
|
||||
[2016-02-23 08:32:49.183985] read: quvi ["--version"]
|
||||
{"command":"addurl","file":"1.png","note":"downloading http://www.onerussian.com/tmp/banner.png ..."[2016-02-23 08:32:49.255949] call: curl ["-s","-f","-L","-C","-","-#","-o","/private/tmp/123/.git/annex/tmp/URL-s25319--http&c%%www.onerussian.com%tmp%banner.png","http://www.onerussian.com/tmp/banner.png","--user-agent","git-annex/6.20160221-g3d1895e"]
|
||||
[2016-02-23 08:32:49.302577] process done ExitSuccess
|
||||
[2016-02-23 08:32:49.303332] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.largefiles=exclude=*.txt","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
|
||||
[2016-02-23 08:32:49.305653] read: git ["--version"]
|
||||
[2016-02-23 08:32:49.310314] process done ExitSuccess
|
||||
error: git-annex died of signal 4
|
||||
|
||||
"""]]
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[done]]; this was indeed due to libmagic, and I've fixed it using brew
|
||||
> --build-bottle to make a portable one. Then had to disable the build
|
||||
> flag. --[[Joey]]
|
|
@ -1,38 +0,0 @@
|
|||
Compilation of the latest git-annex code (commit 04dca96) on Windows is currently failing due to some modules not being imported. The issues are fixed by the following patch:
|
||||
|
||||
```
|
||||
diff --git a/Annex/Content/Presence.hs b/Annex/Content/Presence.hs
|
||||
index 861288c0b..f5d4b593f 100644
|
||||
--- a/Annex/Content/Presence.hs
|
||||
+++ b/Annex/Content/Presence.hs
|
||||
@@ -25,6 +25,9 @@ module Annex.Content.Presence (
|
||||
) where
|
||||
|
||||
import Annex.Common
|
||||
+#ifdef mingw32_HOST_OS
|
||||
+import Annex.Perms
|
||||
+#endif
|
||||
import qualified Annex
|
||||
import Annex.LockPool
|
||||
import Annex.WorkerPool
|
||||
diff --git a/Git/Hook.hs b/Git/Hook.hs
|
||||
index 45662f236..788552305 100644
|
||||
--- a/Git/Hook.hs
|
||||
+++ b/Git/Hook.hs
|
||||
@@ -13,9 +13,7 @@ import Common
|
||||
import Git
|
||||
import Utility.Tmp
|
||||
import Utility.Shell
|
||||
-#ifndef mingw32_HOST_OS
|
||||
import Utility.FileMode
|
||||
-#endif
|
||||
|
||||
data Hook = Hook
|
||||
{ hookName :: FilePath
|
||||
|
||||
```
|
||||
|
||||
[[!meta author=jwodder]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> build is succeeding now [[fixed|done]] --[[Joey]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="jwodder"
|
||||
avatar="http://cdn.libravatar.org/avatar/b06e01332c949b895c681cc92934f36a"
|
||||
subject="comment 1"
|
||||
date="2020-11-23T15:02:28Z"
|
||||
content="""
|
||||
I see you patched `Git/Hook.hs`, but you did not patch `Annex/Content/Presence.hs`. The Windows builds are still failing.
|
||||
"""]]
|
|
@ -1,6 +0,0 @@
|
|||
Since git-annex version is not always sufficient to pin point the source of regression (e.g. see [the get -J issue](http://git-annex.branchable.com/bugs/multiple_ssh_prompts__44___and_thread_blocked_indefinitely_in_an___63____63____63___transaction/)), it would be really nice if the version of ghc (and thus all its libraries if I get it right) was included in the output of `git annex version`
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/repronim]]
|
||||
|
||||
> Closing as I was blind -- there is `dependency versions` reported [[done]] --[[yarikoptic]]
|
|
@ -1,58 +0,0 @@
|
|||
### Please describe the problem.
|
||||
|
||||
I have got data which I was trying to add to an existing annex. Unfortunately I have missed first that some files were not having write permissions so initial 'annex add' failed with similar to below messages, so I have pruned misctmp, adjusted permissions and have tried to add again -- but the same error keeps coming back
|
||||
|
||||
[[!format sh """
|
||||
% chmod +w -R S1/DDA-Freeviewing/surf2anat
|
||||
|
||||
% ls -l S1/DDA-Freeviewing/surf2anat/S1_run1_lh*
|
||||
-rw-r----- 2 yoh retino 241147956 Feb 28 21:46 S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz
|
||||
|
||||
% rm -rf .git/annex/misctmp/*
|
||||
zsh: sure you want to delete all the files in /mnt/datasets/mikemp/data-annex/.git/annex/misctmp [yn]? y
|
||||
|
||||
% git annex add S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz
|
||||
add S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz
|
||||
git-annex: .git/annex/misctmp/S1_run1_lh1804289383846930886: rename: permission denied (Permission denied)
|
||||
failed
|
||||
git-annex: add: 1 failed
|
||||
|
||||
% ls -l S1/DDA-Freeviewing/surf2anat/S1_run1_lh*
|
||||
-r--r----- 2 yoh retino 241147956 Feb 28 21:46 S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz
|
||||
|
||||
% git annex version
|
||||
git-annex version: 6.20160307+gitgb095561-1~ndall+1
|
||||
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
local repository version: 5
|
||||
supported repository versions: 5 6
|
||||
upgrade supported from repository versions: 0 1 2 4 5
|
||||
|
||||
|
||||
% rm -rf .git/annex/misctmp/*
|
||||
zsh: sure you want to delete all the files in /mnt/datasets/mikemp/data-annex/.git/annex/misctmp [yn]? y
|
||||
|
||||
% chmod +w -R S1/DDA-Freeviewing/surf2anat
|
||||
|
||||
% git annex add --debug S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz
|
||||
[2016-03-10 10:15:22.578786] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--others","--exclude-standard","-z","--","S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz"]
|
||||
[2016-03-10 10:15:22.583854] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
|
||||
[2016-03-10 10:15:22.584443] read: git ["--version"]
|
||||
[2016-03-10 10:15:22.588703] process done ExitSuccess
|
||||
add S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz [2016-03-10 10:15:23.186056] read: sha256sum [".git/annex/misctmp/S1_run1_lh1804289383846930886"]
|
||||
[2016-03-10 10:15:24.611133] process done ExitSuccess
|
||||
|
||||
git-annex: .git/annex/misctmp/S1_run1_lh1804289383846930886: rename: permission denied (Permission denied)
|
||||
failed
|
||||
[2016-03-10 10:15:24.648808] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff","--name-only","--diff-filter=T","-z","--","S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz"]
|
||||
git-annex: add: 1 failed
|
||||
% ls -l .git/annex/misctmp/S1_run1_lh1804289383846930886
|
||||
-r--r----- 2 yoh retino 241147956 Feb 28 21:46 .git/annex/misctmp/S1_run1_lh1804289383846930886
|
||||
|
||||
"""]]
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[unreproducible|done]] --[[Joey]]
|
|
@ -1,12 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-03-14T17:53:32Z"
|
||||
content="""
|
||||
A file's permissions cannot prevent it being renamed; so I diagnose some
|
||||
kind of directory permission problem.
|
||||
|
||||
(Also AFAICS git-annex add deals with files whose permissions don't allow
|
||||
being read by turning on the read bit. And it should never need the write
|
||||
bit. I can `chmod 000 foo; git annex add foo` and it succeeds..)
|
||||
"""]]
|
|
@ -1,167 +0,0 @@
|
|||
I'm looking into another SSH-based hang in the DataLad tests. I can't
|
||||
trigger the hang on my local system (Debian Buster), but in a Ubuntu
|
||||
Xenial VM I can. The hang bisects to 1f2e2d15e (async exception
|
||||
safety, 2020-06-03). It disappears on the parent of 1f2e2d15e or if I
|
||||
revert that commit on top of master (currently 3b6754e2a).
|
||||
|
||||
I was able to reduce the hang down to a `git annex get` from an rsync
|
||||
remote. Here is a script that triggers the hang via a Xenial Docker
|
||||
container. Sorry for the length; given the system interaction, it's
|
||||
the simplest reproducer I've managed to come up with.
|
||||
|
||||
[[!format sh """
|
||||
cd "$(mktemp -d ${TMPDIR:-/tmp}/ga-XXXXXXX)"
|
||||
|
||||
cat >demo.sh <<'EOF'
|
||||
git annex version
|
||||
|
||||
cd "$(mktemp -d /tmp/ga-XXXXXXX)"
|
||||
remdir="$PWD/store"
|
||||
mkdir "$remdir"
|
||||
git init repo
|
||||
(
|
||||
cd repo
|
||||
git annex init
|
||||
git annex initremote r type=rsync rsyncurl="localhost:$remdir" encryption=none
|
||||
echo 0 >f0
|
||||
git annex add f0
|
||||
git commit -m'f0'
|
||||
git annex copy --to=r f0
|
||||
git annex drop f0
|
||||
)
|
||||
|
||||
git clone repo clone
|
||||
(
|
||||
cd clone
|
||||
git annex init clone
|
||||
git annex enableremote r
|
||||
git annex get --debug f0
|
||||
)
|
||||
EOF
|
||||
|
||||
cat >Dockerfile <<'EOF'
|
||||
FROM ubuntu:xenial
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
RUN apt-get update
|
||||
RUN apt-get install -y --no-install-recommends \
|
||||
curl rsync openssh-client openssh-server ca-certificates
|
||||
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
|
||||
|
||||
RUN cd /root && curl -fSsL \
|
||||
https://downloads.kitenet.net/git-annex/autobuild/amd64/git-annex-standalone-amd64.tar.gz \
|
||||
| tar xz
|
||||
ENV PATH="/root/git-annex.linux:$PATH"
|
||||
|
||||
RUN git config --system user.name "u"
|
||||
RUN git config --system user.email "u@e"
|
||||
|
||||
RUN mkdir -p /root/.ssh
|
||||
RUN mkdir -p /var/run/sshd
|
||||
RUN ssh-keygen -f /root/.ssh/id_rsa -N ""
|
||||
RUN cat /root/.ssh/id_rsa.pub >>/root/.ssh/authorized_keys
|
||||
RUN echo "Host localhost\nStrictHostKeyChecking no\n" >>/root/.ssh/config
|
||||
|
||||
COPY demo.sh /root/demo.sh
|
||||
CMD /usr/sbin/sshd && sh /root/demo.sh
|
||||
EOF
|
||||
|
||||
docker build -t ga-rsync-hang:latest .
|
||||
docker run -it --rm ga-rsync-hang:latest
|
||||
"""]]
|
||||
|
||||
Output, where the last line stalled:
|
||||
|
||||
```
|
||||
[... 50 lines ...]
|
||||
git-annex version: 8.20200618-g3b6754e2a
|
||||
build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 8
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
|
||||
[... 32 lines ...]
|
||||
[2020-07-07 14:28:55.252423613] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
|
||||
[2020-07-07 14:28:55.256082009] process done ExitSuccess
|
||||
[2020-07-07 14:28:55.256316845] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
|
||||
[2020-07-07 14:28:55.26056207] process done ExitSuccess
|
||||
[2020-07-07 14:28:55.260917245] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","-z","--cached","--","f0"]
|
||||
get f0 [2020-07-07 14:28:55.26481346] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
|
||||
[2020-07-07 14:28:55.268786541] process done ExitSuccess
|
||||
[2020-07-07 14:28:55.268997933] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2020-07-07 14:28:55.272176401] process done ExitSuccess
|
||||
[2020-07-07 14:28:55.272550418] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..ea06186dfc3dab39316da534144d5d6ef3d6090c","--pretty=%H","-n1"]
|
||||
[2020-07-07 14:28:55.275989985] process done ExitSuccess
|
||||
[2020-07-07 14:28:55.27634403] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
|
||||
[2020-07-07 14:28:55.276782737] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
|
||||
[2020-07-07 14:28:55.279315206] read: git ["config","--null","--list"]
|
||||
[2020-07-07 14:28:55.281990769] process done ExitSuccess
|
||||
(from r...)
|
||||
[2020-07-07 14:28:55.288194172] read: rsync ["-e","'ssh' '-S' '.git/annex/ssh/localhost' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=yes' '-T'","--progress","--inplace","localhost:/tmp/ga-075MZKs/store/0c5/66e/'SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa/SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa'",".git/annex/tmp/SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa"]
|
||||
SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa
|
||||
|
||||
0 0% 0.00kB/s 0:00:00
|
||||
2 100% 1.95kB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
^C
|
||||
```
|
||||
|
||||
Replacing "FROM ubuntu:xenial" with "FROM ubuntu:bionic" (a later
|
||||
release) resolves the hang, so perhaps there is some interaction with
|
||||
an older rsync or openssh version. Here are the versions that are
|
||||
present in Xenial:
|
||||
|
||||
```
|
||||
OpenSSH_7.2p2 Ubuntu-4ubuntu2.10, OpenSSL 1.0.2g 1 Mar 2016
|
||||
|
||||
rsync version 3.1.1 protocol version 31
|
||||
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
|
||||
Web site: http://rsync.samba.org/
|
||||
Capabilities:
|
||||
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
|
||||
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
|
||||
append, ACLs, xattrs, iconv, symtimes, prealloc
|
||||
```
|
||||
|
||||
Thinking it's an older version of openssh or rsync, I tried with an older version of Debian. Using "FROM debian:stretch-slim" doesn't hang. Here are the versions there:
|
||||
|
||||
```
|
||||
OpenSSH_7.4p1 Debian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019
|
||||
|
||||
rsync version 3.1.2 protocol version 31
|
||||
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
|
||||
Web site: http://rsync.samba.org/
|
||||
Capabilities:
|
||||
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
|
||||
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
|
||||
append, ACLs, xattrs, iconv, symtimes, prealloc
|
||||
```
|
||||
|
||||
However, going back farther, "FROM debian:jessie-slim" does hang. The
|
||||
versions there:
|
||||
|
||||
```
|
||||
OpenSSH_6.7p1 Debian-5+deb8u8, OpenSSL 1.0.1t 3 May 2016
|
||||
|
||||
rsync version 3.1.1 protocol version 31
|
||||
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
|
||||
Web site: http://rsync.samba.org/
|
||||
Capabilities:
|
||||
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
|
||||
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
|
||||
append, ACLs, xattrs, iconv, symtimes, prealloc
|
||||
```
|
||||
|
||||
So perhaps there's some interaction with openssh before 7.4p1 or rsync
|
||||
before 3.1.2. Those are pretty old versions, and, in the case of
|
||||
Jessie, the release is now EOL. But I figured it was at least worth
|
||||
writing up given that the hang isn't triggered until a recent commit
|
||||
in git-annex.
|
||||
|
||||
[[!meta author=kyle]]
|
||||
[[!tag projects/datalad]]
|
||||
|
||||
> [[fixed|done]] made it cancel the threads 2 seconds after the process
|
||||
> exited if still running. So there will be a bit of a slow down when using
|
||||
> that broken ssh version, but it will work. --[[Joey]]
|
|
@ -1,34 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 10"
|
||||
date="2020-08-10T18:56:02Z"
|
||||
content="""
|
||||
Hmm. Well, I doubt it provides any useful information, but just in case, here's the output produced by the `git annex get --debug f0` call:
|
||||
|
||||
```
|
||||
[2020-08-10 14:49:51.227779797] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"]
|
||||
[2020-08-10 14:49:51.229945707] process done ExitSuccess
|
||||
[2020-08-10 14:49:51.230127447] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"]
|
||||
[2020-08-10 14:49:51.232132388] process done ExitSuccess
|
||||
[2020-08-10 14:49:51.232349474] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"-z\",\"--cached\",\"--\",\"f0\"]
|
||||
get f0 [2020-08-10 14:49:51.234243652] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
|
||||
[2020-08-10 14:49:51.236402808] process done ExitSuccess
|
||||
[2020-08-10 14:49:51.236609182] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
|
||||
[2020-08-10 14:49:51.238119863] process done ExitSuccess
|
||||
[2020-08-10 14:49:51.238409384] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..d8b33719c8ce515e7b5b99f8d0c51ab7d2de46e1\",\"--pretty=%H\",\"-n1\"]
|
||||
[2020-08-10 14:49:51.240088806] process done ExitSuccess
|
||||
[2020-08-10 14:49:51.240391883] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
|
||||
[2020-08-10 14:49:51.240716412] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
|
||||
[2020-08-10 14:49:51.247306917] read: git [\"config\",\"--null\",\"--list\"]
|
||||
[2020-08-10 14:49:51.24859076] process done ExitSuccess
|
||||
(from r...)
|
||||
[2020-08-10 14:49:51.250558418] read: rsync [\"-e\",\"'ssh' '-S' '.git/annex/ssh/localhost' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=yes' '-T'\",\"--progress\",\"--inplace\",\"localhost:/tmp/ga-urvfy8Q/store/0c5/66e/'SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa/SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa'\",\".git/annex/tmp/SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa\"]
|
||||
SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa
|
||||
|
||||
0 0% 0.00kB/s 0:00:00
|
||||
2 100% 1.95kB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
[2020-08-10 14:49:51.557365853] process done ExitSuccess
|
||||
git-annex: AsyncCancelled
|
||||
```
|
||||
"""]]
|
|
@ -1,9 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 11"
|
||||
date="2020-08-10T19:01:35Z"
|
||||
content="""
|
||||
And I should have checked the result the command rather than just
|
||||
noting that it didn't hang. f0 is not transferred.
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 12"""
|
||||
date="2020-08-10T19:36:30Z"
|
||||
content="""
|
||||
Wait, are you saying the command exited nonzero, or that the transfer
|
||||
didn't happen, or both?
|
||||
"""]]
|
|
@ -1,24 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 13"""
|
||||
date="2020-08-10T19:54:26Z"
|
||||
content="""
|
||||
Well, I ran the reproducer script, substituting a git-annex tarball with
|
||||
the fix (and adding an output of the exit code and dropping the --debug),
|
||||
and the end was:
|
||||
|
||||
get f0 (from r...)
|
||||
SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa2
|
||||
100% 1.95kB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
(checksum...) ok
|
||||
(recording state in git...)
|
||||
0
|
||||
|
||||
So, I guess this patch is ok, at least with the git-annex I've built.
|
||||
(Using haskell from debian unstable.) It seems possible that some older
|
||||
ghc, async, etc might have handled that AsyncCancelled differently and
|
||||
resulted in the behavior you're seeing.
|
||||
|
||||
Also tried with https://downloads.kitenet.net/git-annex/autobuild/i386-ancient/git-annex-standalone-i386-ancient.tar.gz
|
||||
and it's ok too.
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 13"
|
||||
date="2020-08-10T19:47:59Z"
|
||||
content="""
|
||||
The file isn't transferred, and the exit is non-zero (1).
|
||||
"""]]
|
|
@ -1,16 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 15"""
|
||||
date="2020-08-10T20:06:28Z"
|
||||
content="""
|
||||
Hmm, come to think of it, `wait errt` is racing
|
||||
`cancel errt`, so what if cancel kills the errt thread,
|
||||
and before `race` noticed cancel is done, the wait thread
|
||||
sees errt, died of an an AsyncCancelled exception and rethrows that?
|
||||
Result could be the exception leaking out, perhaps.
|
||||
|
||||
It might be that async now avoids that, and some other version didn't, or
|
||||
you could be getting unlucky with the race.
|
||||
|
||||
So, I've added something to catch that exception in case it leaks.
|
||||
"""]]
|
|
@ -1,78 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 16"
|
||||
date="2020-08-10T21:34:43Z"
|
||||
content="""
|
||||
I applied the change from c59a51a0651b and 4466c1001d88. Now I don't
|
||||
see \"git-annex: AsyncCancelled\", f0 is transferred, and the exit
|
||||
status is 0, so it looks to be working as intended. Thank you.
|
||||
|
||||
As described in comment 8, this is all in the Xenial VM on top of
|
||||
1df9e72a7829, because the hang goes away with the next commit
|
||||
(4c9ad1de46dd). For posterity, here is a diff of the changes on top
|
||||
1df9e72a7829, applied from aa492bc65904, c59a51a0651b, and
|
||||
4466c1001d88.
|
||||
|
||||
<details>
|
||||
<summary>diff</summary>
|
||||
|
||||
[[!format diff \"\"\"
|
||||
diff --git a/Utility/Metered.hs b/Utility/Metered.hs
|
||||
index 1c35a9a05..c4063c90b 100644
|
||||
--- a/Utility/Metered.hs
|
||||
+++ b/Utility/Metered.hs
|
||||
@@ -1,6 +1,6 @@
|
||||
{- Metered IO and actions
|
||||
-
|
||||
- - Copyright 2012-2018 Joey Hess <id@joeyh.name>
|
||||
+ - Copyright 2012-2020 Joey Hess <id@joeyh.name>
|
||||
-
|
||||
- License: BSD-2-clause
|
||||
-}
|
||||
@@ -46,6 +46,7 @@ import Common
|
||||
import Utility.Percentage
|
||||
import Utility.DataUnits
|
||||
import Utility.HumanTime
|
||||
+import Utility.ThreadScheduler
|
||||
|
||||
import qualified Data.ByteString.Lazy as L
|
||||
import qualified Data.ByteString as S
|
||||
@@ -314,10 +315,26 @@ outputFilter cmd params environ outfilter errfilter =
|
||||
catchMaybeIO $ withCreateProcess p go
|
||||
where
|
||||
go _ (Just outh) (Just errh) pid = do
|
||||
- void $ concurrently
|
||||
- (tryIO (outfilter outh) >> hClose outh)
|
||||
- (tryIO (errfilter errh) >> hClose errh)
|
||||
- waitForProcess pid
|
||||
+ outt <- async $ tryIO (outfilter outh) >> hClose outh
|
||||
+ errt <- async $ tryIO (errfilter errh) >> hClose errh
|
||||
+ ret <- waitForProcess pid
|
||||
+
|
||||
+ -- Normally, now that the process has exited, the threads
|
||||
+ -- will finish processing its output and terminate.
|
||||
+ -- But, just in case the process did something evil like
|
||||
+ -- forking to the background while inheriting stderr,
|
||||
+ -- it's possible that the threads will not finish, which
|
||||
+ -- would result in a deadlock. So, wait a few seconds
|
||||
+ -- maximum for them to finish and then cancel them.
|
||||
+ -- (One program that has behaved this way in the past is
|
||||
+ -- openssh.)
|
||||
+ void $ tryNonAsync $ race_
|
||||
+ (wait outt >> wait errt)
|
||||
+ (threadDelaySeconds (Seconds 2))
|
||||
+ cancel outt
|
||||
+ cancel errt
|
||||
+
|
||||
+ return ret
|
||||
go _ _ _ _ = error \"internal\"
|
||||
|
||||
p = (proc cmd (toCommand params))
|
||||
\"\"\"]]
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
|
||||
"""]]
|
|
@ -1,58 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2020-07-13T15:53:56Z"
|
||||
content="""
|
||||
Hmm, my guess is that the significant and unmentioned change in that commit
|
||||
is, it waits for one of the stdout or stderr to be fully processed before
|
||||
waiting on the process to terminate. So if it somehow fails to detect EOF
|
||||
(maybe because old rsync and ssh somehow end up with the handle not being
|
||||
closed when rsync exits eg if a background ssh process inherited them?) it
|
||||
would block forever.
|
||||
|
||||
If so, this patch would avoid the hang (and if this doesn't fix it, the
|
||||
problem must be somehow due to using with CreateProcess).
|
||||
|
||||
diff --git a/Utility/Metered.hs b/Utility/Metered.hs
|
||||
index 1c35a9a05..8e363681d 100644
|
||||
--- a/Utility/Metered.hs
|
||||
+++ b/Utility/Metered.hs
|
||||
@@ -314,9 +314,8 @@ outputFilter cmd params environ outfilter errfilter =
|
||||
catchMaybeIO $ withCreateProcess p go
|
||||
where
|
||||
go _ (Just outh) (Just errh) pid = do
|
||||
- void $ concurrently
|
||||
- (tryIO (outfilter outh) >> hClose outh)
|
||||
- (tryIO (errfilter errh) >> hClose errh)
|
||||
+ void $ async $ tryIO (outfilter outh) >> hClose outh
|
||||
+ void $ async $ tryIO (errfilter errh) >> hClose errh
|
||||
waitForProcess pid
|
||||
go _ _ _ _ = error "internal"
|
||||
|
||||
If that patch does avoid the hang, this one would be worth a try
|
||||
as it tries to accomplish the goal of the patch in a better way:
|
||||
|
||||
diff --git a/Utility/Metered.hs b/Utility/Metered.hs
|
||||
index 1c35a9a05..bd265ae09 100644
|
||||
--- a/Utility/Metered.hs
|
||||
+++ b/Utility/Metered.hs
|
||||
@@ -313,11 +313,10 @@ outputFilter
|
||||
outputFilter cmd params environ outfilter errfilter =
|
||||
catchMaybeIO $ withCreateProcess p go
|
||||
where
|
||||
- go _ (Just outh) (Just errh) pid = do
|
||||
- void $ concurrently
|
||||
- (tryIO (outfilter outh) >> hClose outh)
|
||||
- (tryIO (errfilter errh) >> hClose errh)
|
||||
- waitForProcess pid
|
||||
+ go _ (Just outh) (Just errh) pid =
|
||||
+ withAsync (tryIO (outfilter outh) >> hClose outh) $ const $
|
||||
+ withAsync (tryIO (errfilter errh) >> hClose errh) $ const $
|
||||
+ waitForProcess pid
|
||||
go _ _ _ _ = error "internal"
|
||||
|
||||
p = (proc cmd (toCommand params))
|
||||
|
||||
I can try these if necessary but don't have docker handy,
|
||||
so maybe you're in a better position to?
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="kyle"
|
||||
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
|
||||
subject="comment 2"
|
||||
date="2020-07-13T16:46:54Z"
|
||||
content="""
|
||||
Thanks for taking a look. I'll test out those changes and report back.
|
||||
"""]]
|
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue