mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

move old fixed datalad/dandi/repronim bugs to the project pages

Browse source 
This is to cut down on the number of files in bugs/, which makes it slow
to file new bug reports or update active bug reports. These old bugs
were about 1/3rd of the files in there. These projects want lists of
their old bugs to still be accessible, and have the lists on their
project pages, which will still list the old bugs.

Commands used:

for f in $(git grep -l '\[\[!tag projects/dandi\]\]'); do if grep -q 'done\]\]' "$f"; then git mv "$f" ../projects/dandi/bugs-done; g=$(echo "$f" | sed 's/.mdwn//'); if [ -d "$g" ]; then git mv "$g" ../projects/dandi/bugs-done; fi; fi; done
for f in $(git grep -l '\[\[!tag projects/repronim\]\]'); do if grep -q 'done\]\]' "$f"; then git mv "$f" ../projects/repronim/bugs-done; g=$(echo "$f" | sed 's/.mdwn//'); if [ -d "$g" ]; then git mv "$g" ../projects/repronim/bugs-done; fi; fi; done
for f in $(git grep -l '\[\[!tag projects/datalad\]\]'); do if grep -q 'done\]\]' "$f"; then git mv "$f" ../projects/datalad/bugs-done; g=$(echo "$f" | sed 's/.mdwn//'); if [ -d "$g" ]; then git mv "$g" ../projects/datalad/bugs-done; fi; fi; done

That assumes that bugs are not tagged by multiple projects at the same
time. Of the ones I moved, I've checked and none are.

Could do the same with todo/ but there are only 370 files in there, and
less than 84 of them could be moved this way, which does not seem likely
to produce a sizeable speedup.

Sponsored-by: Dartmouth College's Datalad project
This commit is contained in:
Joey Hess 2023-01-05 13:16:15 -04:00
parent 946fc20165
commit bcc69f07e8
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
1011 changed files with 4 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

69
doc/bugs/--debug_is_not_in_effect_if_precedes_-c.mdwn
View file

@ -1,69 +0,0 @@
### Please describe the problem.
Identified while troubleshooting another [issue](https://git-annex.branchable.com/bugs/enableremote_stuck_with_a_recentish_git-annex/#comment-2116c5e109aaf39ffd62f3bdeeb14602)
[[!format sh """
$> 'git-annex' 'enableremote' --debug -cremote.target1.blah=1 'target1'
enableremote target1 ok
$> 'git-annex' 'enableremote' -cremote.target1.blah=1 --debug 'target1'
[2020-02-26 14:46:47.789794028] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","show-ref","git-annex"]
[2020-02-26 14:46:47.797917978] process done ExitSuccess
[2020-02-26 14:46:47.798350533] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","show-ref","--hash","refs/heads/git-annex"]
[2020-02-26 14:46:47.802576899] process done ExitSuccess
[2020-02-26 14:46:47.802884873] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","log","refs/heads/git-annex..b1ab0b11fbbc94ffd3d52adb7a0e93c3d45d8b52","--pretty=%H","-n1"]
[2020-02-26 14:46:47.813289406] process done ExitSuccess
[2020-02-26 14:46:47.815873454] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","cat-file","--batch"]
[2020-02-26 14:46:47.818598891] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2020-02-26 14:46:47.824657055] read: git ["config","--null","--list"]
[2020-02-26 14:46:47.835897478] process done ExitSuccess
enableremote target1 [2020-02-26 14:46:47.83652184] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","remote.target1.blah=1","config","remote.target1.annex-ignore","false"]
[2020-02-26 14:46:47.842277017] process done ExitSuccess
[2020-02-26 14:46:47.842703576] read: git ["config","--null","--list"]
[2020-02-26 14:46:47.853478328] process done ExitSuccess
ok
[2020-02-26 14:46:47.855317715] process done ExitSuccess
[2020-02-26 14:46:47.856835556] process done ExitSuccess
"""]]
I consider it a bug since options shouldn't be order dependent, and even if they were -- `--debug` is listed before `-c` in `git annex enableremote --help`:
[[!format sh """
$> git annex enableremote --help
git-annex enableremote - enables git-annex to use a remote
Usage: git-annex enableremote [NAME K=V ...]
Available options:
--force allow actions that may lose annexed data
-F,--fast avoid slow operations
-q,--quiet avoid verbose output
-v,--verbose allow verbose output (default)
-d,--debug show debug messages
--no-debug don't show debug messages
-b,--backend NAME specify key-value backend to use
-N,--numcopies NUMBER override default number of copies
--trust REMOTE override trust setting
--semitrust REMOTE override trust setting back to default
--untrust REMOTE override trust setting to untrusted
-c,--config NAME=VALUE override git configuration setting
--user-agent NAME override default User-Agent
--trust-glacier Trust Amazon Glacier inventory
--notify-finish show desktop notification after transfer finishes
--notify-start show desktop notification after transfer starts
-h,--help Show this help text
For details, run: git-annex help enableremote
$> git annex version
git-annex version: 7.20190819+git2-g908476a9b-1~ndall+1
"""]]
[[!meta author=yoh]]
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]
> fixed in [8.20200226-3-gc089f395b](https://git.kitenet.net/index.cgi/git-annex.git/commit/?id=c089f395b0c7d6416a3d4f2bf3211404acfd5b0e) --[[yarikoptic]]

24
doc/bugs/--debug_is_not_in_effect_if_precedes_-c/comment_1_0f66414e629e31c2de868f8b7eeb7af5._comment
View file

@ -1,24 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2020-02-27T04:20:58Z"
content="""
-c uses adjustGitRepo which calls changeGitRepo, which
re-extracts the GitConfig. --debug uses changeGitConfig which
sets annexDebug in the GitConfig, which does not survive the changeGitRepo.
There might be a broader problem here, as changeGitRepo is also
called by setConfig in many parts of the code. I think it narrowly
escapes being a problem, because by the time a command is started,
it's already enabled debug output, and so the GitConfig being reloaded
doesn't disable debugging.
Other calls to changeGitConfig could also be a problem, if followed by
an adjustGitRepo which loses those changes. There are only a few others,
look probably ok, but this would be an easy gotcha to hit later.
So changeGitConfig needs to make a config change that persists across
changeGitRepo.
Done.
"""]]

34
doc/bugs/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips.mdwn
View file

@ -1,34 +0,0 @@
### Please describe the problem.
### What steps will reproduce the problem?
I am plowing through on making git-annex available within conda-forge "natively" for Windows. For now I just took the recently built installer, the one now available from [datasets.datalad.org](http://datasets.datalad.org/datalad/packages/windows/) and built on datalad-extensions github setup. I just extracted git-annex component from the installer and placed them within conda hierarchy (installed `posix` package with all the needed basic tools. Overall -- looks great, but:
[[!format sh """
prop_view_roundtrips: FAIL (0.30s)
*** Failed! Falsified (after 524 tests and 1 shrink):
"a"
MetaData (fromList [(MetaField "8",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet True) "\nD\EM",MetaValue (CurrentlySet True) "GO`!)",MetaValue (CurrentlySet False) "k\FS\CAN"]),(MetaField "dU",fromList [MetaValue (CurrentlySet True) "",MetaValue (CurrentlySet False) "\NUL44Vfm[\t",MetaValue (CurrentlySet True) "\nLMEgYc",MetaValue (CurrentlySet True) "\SO[",MetaValue (CurrentlySet True) "\FS\DC4\DLE\"3",MetaValue (CurrentlySet True) ";\f0&Wc\GS{^",MetaValue (CurrentlySet True) "D",MetaValue (CurrentlySet True) "c:"]),(MetaField "sV",fromList [MetaValue (CurrentlySet True) "",MetaValue (CurrentlySet False) "\STX8#w",MetaValue (CurrentlySet False) "\ny",MetaValue (CurrentlySet False) "\DC4qOq",MetaValue (CurrentlySet True) "\FSbqjq",MetaValue (CurrentlySet True) "T_bx%[lN",MetaValue (CurrentlySet True) "W0`",MetaValue (CurrentlySet True) "~ ueY"]),(MetaField "V",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet False) "\t\DC1~`\SOHv\DC1",MetaValue (CurrentlySet True) "\DLE3",MetaValue (CurrentlySet True) "/MZh$",MetaValue (CurrentlySet False) "0",MetaValue (CurrentlySet False) "MEulc",MetaValue (CurrentlySet True) "P5D",MetaValue (CurrentlySet True) "i|S,",MetaValue (CurrentlySet True) "x|C"])])
True
Use --quickcheck-replay=742853 to reproduce.
"""]]
unfortunately I cannot tell from that output what could be the problem. Please let me know if hard to figure it out and I should provide access to such environment (ATM needs effort, so I do not want to spend time on that unless "no other way")
And it seems it might be a flaky test -- I started another run, it is still running but I this test did not fail
```
$ grep prop_view_roundtrips git-annex-test-miniconda*.log
git-annex-test-miniconda-2.log: prop_view_roundtrips: OK (2.51s)
git-annex-test-miniconda.log: prop_view_roundtrips: FAIL (0.30s)
```
Cheers,
[[!meta author=yoh]]
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]

17
doc/bugs/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/comment_1_3a2be0a1cd74c7dba3b07e1300ce2a30._comment
View file

@ -1,17 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2020-10-26T16:03:40Z"
content="""
I can reproduce it in a windows VM running
`git-annex test --quickcheck-replay=742853`
These quickcheck tests test random input so not flaky exactly.
Does not happen with that seed on linux, so it probably involves something
encoding specific. An area where the windows port is known to have
extensive problems.
([[!commit 1b8026b2cbc8df0274082c5f08a8b4f8ca47c5c9]] was similar,
although that was MetaField and this appears to be MetaValue.)
"""]]

8
doc/bugs/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/comment_2_1945040bcdf209b9a0faecd4a77e0f35._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 2"
date="2020-10-26T17:57:12Z"
content="""
you say \"windows port\", I say \"windows as a whole\", e.g. today revelation (or just a come back if I ran into it before but forgot) to me [was inability to have a file/directory named `con`...](https://github.com/datalad/datalad/issues/5097) - no bloody sense on how such design decision has happened and how it dragged all the way into the flagman of the 2020 product.
"""]]

42
doc/bugs/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/comment_3_06dd9606225cf881f7681fc27fb4833d._comment
View file

@ -1,42 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2020-10-26T18:41:27Z"
content="""
Hmm, this uses viewedFiles, which generates filenames
based on the MetaValue. Note use of pathProduct, which uses
System.FilePath.combine.
So, generating random ascii (including escape sequences)
bytestrings, and passing them through decodeBS to generate FilePaths,
and then operating on those filepaths. What could possibly go wrong.
And aha! I made pathProduct use System.FilePath.Windows.combine
and was able to reproduce the test suite failure on Linux.
And aha again:
MetaValue (CurrentlySet True) "c:"
Which of course breaks it on windows because it wanted to generate
something like "bar/c:/baz/a" but instead it gets "c:/bar/baz/a"
git-annex does replace '/' and '\' when generating these filenames.
Not as a security measure (when the view branch is checked out, git's
security checks apply same as any branch so it piggybacks on those),
but to let the user build a view and successfully check it out
when their metadata happens to include such stuff.
However, windows does have enough special filenames and gotchas
that it simply does not seem to make sense for git-annex to try to work
around them all in the view code. If a MetaValue happens to end with a
period, or is "nul", and so the generated filename is illegal on Windows,
it'll blow up at checkout time, and I am ok with that.
So I think it would make sense to also escape ':', but that's about as far
as this should go. *Especially* because the filenames it generates need to
roundtrip back to metadata cleanly, which is what this test case is
testing. While I can finesse individual characters, it would be quite hard
to make a filename w/o a trailing dot roundtrip back to one with it, for
example.
"""]]

19
doc/bugs/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/comment_4_0c6b8a3b0a6fbd700b3c8934bfec199f._comment
View file

@ -1,19 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 4"
date="2021-01-22T16:44:58Z"
content="""
did it come back, I see
```
2021-01-22T04:32:25.5012547Z prop_view_roundtrips: FAIL (0.09s)
2021-01-22T04:32:25.5015902Z *** Failed! Falsified (after 218 tests):
2021-01-22T04:32:25.5016251Z AssociatedFile (Just \"rdmBBP\")
2021-01-22T04:32:25.5018130Z MetaData (fromList [(MetaField \"CkL\",fromList [MetaValue (CurrentlySet False) \"\",MetaValue (CurrentlySet True) \"\SOH5:R9\EM\DC4\",MetaValue (CurrentlySet True) \"\STX\US\fL2\ACK|\\\r[$\",MetaValue (CurrentlySet False) \"\ETBRi\",MetaValue (CurrentlySet False) \"/\FS}\",MetaValue (CurrentlySet True) \"W\",MetaValue (CurrentlySet False) \"X=sQh\NAK^\",MetaValue (CurrentlySet False) \"l\SUB\a\"]),(MetaField \"jM\",fromList [MetaValue (CurrentlySet False) \"\",MetaValue (CurrentlySet False) \"\FSSivk\",MetaValue (CurrentlySet True) \"J'<\SYN\STXGJP\"]),(MetaField \"V\",fromList [MetaValue (CurrentlySet False) \"\",MetaValue (CurrentlySet True) \"\n\NUL\",MetaValue (CurrentlySet True) \"\r\",MetaValue (CurrentlySet False) \"+X\",MetaValue (CurrentlySet True) \"@aN\t~c\SIy\",MetaValue (CurrentlySet False) \"K>xq\",MetaValue (CurrentlySet True) \"a:\"]),(MetaField \"W\",fromList [MetaValue (CurrentlySet True) \"0\DC4qL\",MetaValue (CurrentlySet False) \"K\",MetaValue (CurrentlySet False) \"LD\DC3<M\",MetaValue (CurrentlySet False) \"a\v\",MetaValue (CurrentlySet True) \"dO\",MetaValue (CurrentlySet True) \"w\EOT\"])])
2021-01-22T04:32:25.5020545Z True
2021-01-22T04:32:25.5020894Z Use --quickcheck-replay=455629 to reproduce.
```
on https://github.com/datalad/git-annex/runs/1746587663?check_suite_focus=true with `8.20201129+git169-gaa07e68ed_x64`
"""]]

18
doc/bugs/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/comment_4_c2f248745388b508337c5097baaa1979._comment
View file

@ -1,18 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2021-01-22T16:59:55Z"
content="""
Not sure if this is really the same bug, though certainly related. These
quickcheck tests are fuzz tests, they can find numerous bugs, that's kind of
the point of them. In any case, posting to a closed bug report risks your
followup being lost and deprioritises it.
The problem this new failure shows is that toViewPath is failing to escape the
final character in the path in some cases. Which is not a windows-specific
bug at all really, it could also happen with a metadata value such as "foo/"
being set on linux. Fixed that bug.
Which shows the point of these quickcheck fuzz tests: To be able to catch
lots of different bugs with a single test case.
"""]]

42
doc/bugs/2_mac_crippled_FS__58___Unable_to_remove_all_write.mdwn
View file

@ -1,42 +0,0 @@
### Please describe the problem.
Since cron build of 20210828
```
(git)smaug:/mnt/datasets/datalad/ci/git-annex/builds/2021/08[master]git
$> git grep -l 'Unable to remove all write permissions'
cron-20210828/build-macos.yaml-403-69466103-failed/2_test-annex (crippled-tmp).txt
cron-20210828/build-macos.yaml-403-69466103-failed/test-annex (crippled-tmp)/8_Run tests.txt
cron-20210829/build-macos.yaml-404-69466103-failed/2_test-annex (crippled-tmp).txt
cron-20210829/build-macos.yaml-404-69466103-failed/test-annex (crippled-tmp)/8_Run tests.txt
cron-20210830/build-macos.yaml-405-69466103-failed/2_test-annex (crippled-tmp).txt
cron-20210830/build-macos.yaml-405-69466103-failed/test-annex (crippled-tmp)/8_Run tests.txt
cron-20210831/build-macos.yaml-406-69466103-failed/2_test-annex (crippled-tmp).txt
cron-20210831/build-macos.yaml-406-69466103-failed/test-annex (crippled-tmp)/8_Run tests.txt
```
we got two test fails on a crippled FS on Mac (does not happen on linux afaik)
[example CI log](https://github.com/datalad/git-annex/runs/3468283573?check_suite_focus=true)
Both look like
```
2021-08-31T02:15:42.0758760Z magic: OK (2.41s)
2021-08-31T02:15:42.6972710Z import: FAIL (0.62s)
2021-08-31T02:15:42.6973680Z ./Test/Framework.hs:57:
2021-08-31T02:15:42.6974230Z import failed (transcript follows)
2021-08-31T02:15:42.6974760Z import import1/f
2021-08-31T02:15:42.6976570Z Unable to remove all write permissions from /Volumes/crippledfs/importtestvjfjz3/import1/f -- perhaps it has an xattr or ACL set.
2021-08-31T02:15:42.6977430Z failed
2021-08-31T02:15:42.6977830Z import: 1 failed
2021-08-31T02:15:44.1985050Z reinject: OK (1.50s)
```
[here is the script](https://github.com/datalad/git-annex/blob/master/.github/workflows/tools/setup_crippledfs#L24) to setup such a crippled (FAT32) FS on OSX.
[[!meta author=yoh]]
[[!tag projects/datalad]]
> [[fixed|done]] (provisionally, waiting on test run) --[[Joey]]

34
doc/bugs/2_mac_crippled_FS__58___Unable_to_remove_all_write/comment_1_22a1f4b025a6046e4d11f90590a6e4bd._comment
View file

@ -1,34 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2021-09-01T13:48:57Z"
content="""
Seems that mounting that way on OSX results in a FS where files are always mode
777 and the permissions cannot be changed.
When I tried using git-annex on such a FS, I saw:
datalads-imac:x joey$ git annex init
init
Detected a filesystem without fifo support.
Disabling ssh connection caching.
Filesystem allows writing to files whose write bit is not set.
Detected a crippled filesystem.
And it skips the new permissions check when on a crippled filesystem.
But in that that test run, it seems it is failing to detect a crippled
filesystem. Both because of the failure and also the test suite does
not even run the "v8 unlocked" tests when it detects a crippled filesystem.
Is the test suite running as root? Looks like probably yes. Running as
root prevents detecting the issue that made it use a crippled FS above. And it
seems that, when a FAT fs is mounted on OSX that way, symlinks actually work
(!!!) so the other crippled FS tests also don't notice a problem.
So, the fix should be for init to also test if it can remove the write
bits from a file, and it should try that test even when root.
"""]]

10
doc/bugs/2_mac_crippled_FS__58___Unable_to_remove_all_write/comment_2_71e985981c14780cc172a07beb0e9854._comment
View file

@ -1,10 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 2"
date="2021-09-01T16:22:22Z"
content="""
> Is the test suite running as root? Looks like probably yes.
FWIW, it is a `runner` user [ref](https://github.com/datalad/git-annex/pull/76/checks?check_run_id=3486443350#step:8:1) (did in a temp [PR](https://github.com/datalad/git-annex/pull/76)) who is not `root` but is part of the `admin` group thus with super privileges indeed (that is why I guess we can also use `hdiutil` directly to mount that crippled FS).
"""]]

7
doc/bugs/2_mac_crippled_FS__58___Unable_to_remove_all_write/comment_3_ac3fe75560c020015ecca615cf0e7abe._comment
View file

@ -1,7 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2021-09-02T16:22:56Z"
content="""
OSX test is still failing after that fix, reopened.
"""]]

27
doc/bugs/2_mac_crippled_FS__58___Unable_to_remove_all_write/comment_4_3bdf724f5e9203424b7cb5630fb49a46._comment
View file

@ -1,27 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2021-09-02T16:26:14Z"
content="""
My prior analysis seems right, as far as it running as root would go, but it is
not running as root. So I missed something.
The test failures are both of `git-annex import`.
Otherwise locking down files does succeed. The difference with import
must be that the file located in a directory outside the repository.
Aha... The test suite is being run with TEMPDIR set to the crippled FS,
but `.t` is in another, non-crippled FS. A very smart idea to test that,
although I think this import test is the only one that actually uses
TEMPDIR. (Reading the workflow file, I think it was maybe expected that
all the tests would run in TEMPDIR, but they don't; `git-annex test`
writes to `./.t`, other than this one test.
When the import directory is on a crippled FS, and the repo
is not, it will think the FS is not crippled. Then it fails
to remove write perms from the file while it is in the import
directory, and the perm check then fails.
So, I think it should skip the perm check when doing the initial lockdown
of the file it's going to import.
"""]]

7
doc/bugs/2_mac_crippled_FS__58___Unable_to_remove_all_write/comment_5_46acb87752c0f0574d8f3b91fdfb1697._comment
View file

@ -1,7 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2021-09-02T17:39:20Z"
content="""
Ok, fixed some more, hopefully all the way this time..
"""]]

66
doc/bugs/2_ssh_connection_prompts_for_password.mdwn
View file

@ -1,66 +0,0 @@
### Please describe the problem.
There was some recent work to ["centralize" such prompts](https://git-annex.branchable.com/devblog/day_457__improved_ssh_password_prompting/) but it seems some are still "leaking through" multiple times. May be it is because there are 2 available repos on that remote host, so annex generates one per each of those? (although it knows only about origin)
### What version of git-annex are you using? On what operating system?
6.20170810+gitgff6f9e203-1~ndall+1
### Please provide any additional information below.
[[!format sh """
$> git annex get -J5 .
get R042/R042-2013-08-16/R042-2013-08-16-CSC01a.ncs get R042/R042-2013-08-16/R042-2013-08-16-CSC02a.ncs get R042/R042-2013-08-16/R042-2013-08-16-CSC03a.ncs get R042/R042-2013-08-16/R042-2013-08-16-CSC05a.ncs get R042/R042-2013-08-16/R042-2013-08-16-CSC04a.ncs (from datalad-archives...)
(from datalad-archives...) (from datalad-archives...)
(from datalad-archives...)
(from datalad-archives...)
[ERROR] Failed to run ['git-annex', 'get', '--key', 'MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip'] under '/mnt/btrfs/datasets/datalad/crawl/workshops/mind-2017/MotivationalT'. Exit code=1. out=get MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip (transfer already in progress, or unable to take transfer lock)
| Unable to access these remotes: origin
|
| Try making some of these repositories available:
| aaa0bc14-51fc-45c8-81c2-76dff067755b -- mvdm@atlantis.hpcc.dartmouth.edu:~/data/mind-2017/MotivationalT_
| f7f97046-ea49-4af1-9f5a-8475a5ea1e0a -- yhalchen@atlantis.hpcc.dartmouth.edu:~/mind-2017/MotivationalT [origin]
| failed
| err=git-annex: get: 1 failed
|
[ERROR] Failed to run ['git-annex', 'get', '--key', 'MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip'] under '/mnt/btrfs/datasets/datalad/crawl/workshops/mind-2017/MotivationalT'. Exit code=1. out=get MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip (transfer already in progress, or unable to take transfer lock)
| Unable to access these remotes: origin
|
| Try making some of these repositories available:
| aaa0bc14-51fc-45c8-81c2-76dff067755b -- mvdm@atlantis.hpcc.dartmouth.edu:~/data/mind-2017/MotivationalT_
| f7f97046-ea49-4af1-9f5a-8475a5ea1e0a -- yhalchen@atlantis.hpcc.dartmouth.edu:~/mind-2017/MotivationalT [origin]
| failed
| err=git-annex: get: 1 failed
|
[ERROR] Failed to fetch any archive containing SHA256E-s17136940--0501aab6b4d1ce0565921728bc92ef74f81edf0d7bcd5a77946ca58f977f2537.ncs. Tried: ['MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip']
[ERROR] Failed to fetch any archive containing SHA256E-s17136940--8b3b08310db20ca7e3e784a21f935a78f8669efdf1396168596411f1e355e43b.ncs. Tried: ['MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip']
(from origin...) (from origin...) [ERROR] Failed to run ['git-annex', 'get', '--key', 'MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip'] under '/mnt/btrfs/datasets/datalad/crawl/workshops/mind-2017/MotivationalT'. Exit code=1. out=get MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip (transfer already in progress, or unable to take transfer lock)
| Unable to access these remotes: origin
|
| Try making some of these repositories available:
| aaa0bc14-51fc-45c8-81c2-76dff067755b -- mvdm@atlantis.hpcc.dartmouth.edu:~/data/mind-2017/MotivationalT_
| f7f97046-ea49-4af1-9f5a-8475a5ea1e0a -- yhalchen@atlantis.hpcc.dartmouth.edu:~/mind-2017/MotivationalT [origin]
| failed
| err=git-annex: get: 1 failed
|
[ERROR] Failed to fetch any archive containing SHA256E-s17136940--08ce5a67c7fc09f02b994a3987812a75727eaf51f3e70fa7e1030dae934f9fbc.ncs. Tried: ['MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip']
(from origin...) [ERROR] Failed to run ['git-annex', 'get', '--key', 'MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip'] under '/mnt/btrfs/datasets/datalad/crawl/workshops/mind-2017/MotivationalT'. Exit code=1. out=get MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip (transfer already in progress, or unable to take transfer lock)
| Unable to access these remotes: origin
|
| Try making some of these repositories available:
| aaa0bc14-51fc-45c8-81c2-76dff067755b -- mvdm@atlantis.hpcc.dartmouth.edu:~/data/mind-2017/MotivationalT_
| f7f97046-ea49-4af1-9f5a-8475a5ea1e0a -- yhalchen@atlantis.hpcc.dartmouth.edu:~/mind-2017/MotivationalT [origin]
| failed
| err=git-annex: get: 1 failed
|
[ERROR] Failed to fetch any archive containing SHA256E-s17136940--bc145f07c79584181cad3763a763a2ea047282bd41153d20a63d85a44fb27a7f.ncs. Tried: ['MD5E-s237624713--dbdc4079b005b8b7f1549e00647b36d6.zip']
(from origin...) yhalchen@discovery.dartmouth.edu's password: yhalchen@discovery.dartmouth.edu's password:
"""]]
[[!meta author=yoh]]
[[!tag projects/datalad]]
> warning added; [[done]] --[[Joey]]

34
doc/bugs/2_ssh_connection_prompts_for_password/comment_1_f78784304b46b77d334f74c827d0889b._comment
View file

@ -1,34 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 1"
date="2017-08-12T04:08:04Z"
content="""
ha -- if I do specify `--from=origin` -- only 1 prompt
[[!format sh \"\"\"
$> git annex get -J5 --from=origin .
get R042/R042-2013-08-16/R042-2013-08-16-CSC01a.ncs (from origin...) get R042/R042-2013-08-16/R042-2013-08-16-CSC03a.ncs (from origin...) get R042/R042-2013-08-16/R042-2013-08-16-CSC02a.ncs (from origin...) get R042/R042-2013-08-16/R042-2013-08-16-CSC04a.ncs (from origin...) get R042/R042-2013-08-16/R042-2013-08-16-CSC05a.ncs (from origin...) yhalchen@discovery.dartmouth.edu's password:
git-annex-shell: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
Unable to run git-annex-shell on remote .
git-annex-shell: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
Unable to run git-annex-shell on remote .
git-annex-shell: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
Unable to run git-annex-shell on remote .
git-annex-shell: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
Unable to run git-annex-shell on remote .
SHA256E-s17136940--08ce5a67c7fc09f02b994a3987812a75727eaf51f3e70fa7e1030dae934f9fbc.ncs
0 0% 0.00kB/s 0:00:00 SHA256E-s17136940--bc145f07c79584181cad3763a763a2ea047282bd41153d20a63d85a44fb27a7f.ncs
0 0% 0.00kB/s 0:00:00 SHA256E-s17136940--c3a8af948c77a2df422eae50807a6e7e6e5db7a3451a562bca529d3f1a1a234f.ncs
0 0% 0.00kB/s 0:00:00 git-annex-shell: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
\"\"\"]]
"""]]

25
doc/bugs/2_ssh_connection_prompts_for_password/comment_2_62a5cc7cd6b8b06850fb32b76121ad82._comment
View file

@ -1,25 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2017-08-15T16:43:07Z"
content="""
The improvements around ssh password prompting require ssh connection
caching to work. If a ssh connection fails because the wrong password is
entered or because there's no usable tty or whatever, there's no cached
ssh connection to reuse, so the next attempt to access that host will
result in another password prompt.
Also, datalad does not seem to be running git-annex with -J. So it *can't*
be trying to make two ssh connection at the same time. My recent work on
ssh password prompting was mostly to fix cases where git-annex is run with
-J.
It's also possible that some ssh configuration that I don't know of could
make ssh password prompt even when git-annex is running it with
`BatchMode=true` to avoid password prompts (in order to test if the ssh
connection is already up). That would then result in two ssh password
prompts, one after the other, which seems to match your transcript.
If you have only one remote, specifying `--from=origin` won't change
anything. Entering the right password would change something there though..
"""]]

43
doc/bugs/2_ssh_connection_prompts_for_password/comment_3_f621e615a745910b4e28f91b27132c0e._comment
View file

@ -1,43 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="the issue persists"
date="2019-11-01T18:12:27Z"
content="""
Ran into the same problem again, and it is not clear to me either connection caching is enabled or not (and why?):
[[!format sh \"\"\"
[d31548v@discovery7 bids]$ git -c annex.sshcaching=true annex --debug get -J2 --from=origin sub-sid000005
[2019-11-01 14:10:56.178577] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
[2019-11-01 14:10:56.475956] process done ExitSuccess
[2019-11-01 14:10:56.47622] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2019-11-01 14:10:56.836271] process done ExitSuccess
[2019-11-01 14:10:56.865928] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..8a694d5c54eb81b1e5c5446fa63bdcd13daa34b3\",\"--pretty=%H\",\"-n1\"]
[2019-11-01 14:10:57.229787] process done ExitSuccess
[2019-11-01 14:10:57.234655] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
[2019-11-01 14:10:57.23592] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
[2019-11-01 14:10:57.546203] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"]
[2019-11-01 14:10:57.780246] process done ExitSuccess
[2019-11-01 14:10:57.780454] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"]
[2019-11-01 14:10:58.097345] process done ExitSuccess
[2019-11-01 14:10:58.09754] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"sub-sid000005\"]
[2019-11-01 14:10:58.298181] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
[2019-11-01 14:10:58.29998] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
[2019-11-01 14:10:58.305022] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
[2019-11-01 14:10:58.306024] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
[2019-11-01 14:10:58.62005] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
[2019-11-01 14:10:58.621714] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
[2019-11-01 14:10:58.632596] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
[2019-11-01 14:10:58.6338] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
get sub-sid000005/ses-actions1/fmap/sub-sid000005_ses-actions1_acq-25mm_magnitude2.nii.gz get sub-sid000005/ses-actions1/fmap/sub-sid000005_ses-actions1_acq-25mm_magnitude1.nii.gz (from origin...) (from origin...)
[2019-11-01 14:10:59.489719] chat: ssh [\"yohtest@rolando.cns.dartmouth.edu\",\"-T\",\"git-annex-shell 'p2pstdio' '/inbox/BIDS/Haxby/Sam/1021_actions' '--debug' 'fd3f7af9-cf7d-4d7e-8efd-30e6bedf838d' --uuid d839134c-3afe-4456-920a-e280ce0fdf2a\"]
[2019-11-01 14:10:59.553029] chat: ssh [\"yohtest@rolando.cns.dartmouth.edu\",\"-T\",\"git-annex-shell 'p2pstdio' '/inbox/BIDS/Haxby/Sam/1021_actions' '--debug' 'fd3f7af9-cf7d-4d7e-8efd-30e6bedf838d' --uuid d839134c-3afe-4456-920a-e280ce0fdf2a\"]
yohtest@rolando.cns.dartmouth.edu's password: yohtest@rolando.cns.dartmouth.edu's password:
[d31548v@discovery7 bids]$ git annex version
git-annex version: 7.20191024-g6dc2272
\"\"\"]]
Could you hint me on what/where to dig?
"""]]

21
doc/bugs/2_ssh_connection_prompts_for_password/comment_4_43e5a90d6215b414f4e3357ea91af663._comment
View file

@ -1,21 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2020-01-23T15:51:46Z"
content="""
I notice that debug output has no BatchMode=true in any ssh call. But
the version of git-annex you show always runs ssh with that when
-J is used, unless sshcaching is disabled.
More evidence that sshcaching is disabled in your transcript is that when
it does run ssh, it does not pass -S.
I think the repository must be on a crippled filesystem, on which
git-annex can't do ssh connection caching, because the filesystem
does not support unix sockets. (Or it potentially could be crippled in some
other way.) So it ignores the annex.sshcaching setting.
You could work around this by setting the (undocumented)
GIT_ANNEX_TMP_DIR to some temporary directory on a non-crippled filesystem.
I'm going to add a warning message in this situation.
"""]]

8
doc/bugs/2_ssh_connection_prompts_for_password/comment_5_27b5669cecba4863ce296049804e82af._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 5"
date="2020-01-23T17:51:58Z"
content="""
Thank you Joey! I can only confirm that the file system was likely a crippled/NFS one... So we would likely need to do some sensing on DataLad side and instruct git-annex. Will continue on our end at https://github.com/datalad/datalad/issues/4075
"""]]

43
doc/bugs/3_tests_fail_while_trying_to_build_fresh_git_annex_6.20181011+git109-gff9ba1f4d.mdwn
View file

@ -1,43 +0,0 @@
[[!format sh """
neurodebian@smaug ..6.20181011+git109-gff9ba1f4d-1~ndall+1 % grep -B10 FAIL git-annex_6.20181011+git109-gff9ba1f4d-1~ndall+1_amd64.build
a03a722..0dd2b80 git-annex -> r2/git-annex
To ../../.t/tmprepo331
* [new branch] git-annex -> synced/git-annex
To ../../.t/tmprepo330
a03a722..22937c6 git-annex -> synced/git-annex
OK (2.15s)
adjusted branch merge regression: On branch master
nothing to commit, working tree clean
On branch master
nothing to commit, working tree clean
FAIL (0.50s)
Test.hs:1463:
adjust failed
adjusted branch subtree regression: On branch master
nothing to commit, working tree clean
FAIL (0.31s)
--
059871b..4f97b0c annex/direct/master -> r2/annex/direct/master
conflictor.variant-a507
conflictor.variant-75dc
conflictor.variant-a507
conflictor.variant-75dc
OK (1.37s)
conflict resolution (adjusted branch): On branch master
nothing to commit, working tree clean
On branch master
nothing to commit, working tree clean
FAIL (0.70s)
"""]]
Full log http://www.onerussian.com/tmp/git-annex_6.20181011+git109-gff9ba1f4d-1~ndall+1_amd64.build
(not sure if relevant - this is I think the first build without my custom no LOCPATH patch)
[[!meta author=yoh]]
[[!tag projects/repronim]]
> [[fixed|done]] --[[Joey]]

9
doc/bugs/3_tests_fail_while_trying_to_build_fresh_git_annex_6.20181011+git109-gff9ba1f4d/comment_1_244e138f7621c901dd7763159db84677._comment
View file

@ -1,9 +0,0 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="comment 1"
date="2018-10-22T19:22:30Z"
content="""
See also (same?) failure when trying to build the conda-forge package:
https://circleci.com/gh/conda-forge/git-annex-feedstock/116?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
"""]]

8
doc/bugs/3_tests_fail_while_trying_to_build_fresh_git_annex_6.20181011+git109-gff9ba1f4d/comment_2_cbdfb3dea5c125b52d53811b96b3e01e._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="comment 2"
date="2018-10-24T16:01:53Z"
content="""
Maybe, post a new git-annex version on hackage, with the test failure fixed? I could then update the conda-forge recipe.
"""]]

29
doc/bugs/7.20181211+git29-gab4a1bed9_fails_tests_during_neurodebian_build.mdwn
View file

@ -1,29 +0,0 @@
### Please describe the problem.
Trying to build fresh snapshot for neurodebian but 3 tests report failing
Full buildlogs are at [http://neuro.debian.net/_files/_buildlogs/git-annex/7.20181211+git29-gab4a1bed9](http://neuro.debian.net/_files/_buildlogs/git-annex/7.20181211+git29-gab4a1bed9)
[[!format sh """
neurodebian@smaug ../7.20181211+git29-gab4a1bed9-1~ndall+1 % grep -B3 '^FAIL' git-annex_7.20181211+git29-gab4a1bed9-1\~ndall+1_amd64.build
bup remote: OK (0.11s)
crypto: gpg: can't connect to the agent: IPC connect call failed
gpg: error getting the KEK: No agent running
FAIL
--
bup remote: OK (0.19s)
crypto: gpg: can't connect to the agent: IPC connect call failed
gpg: error getting the KEK: No agent running
FAIL
--
nothing to commit, working tree clean
gpg: can't connect to the agent: IPC connect call failed
gpg: error getting the KEK: No agent running
FAIL
"""]]
[[!meta author=yoh]]
[[!tag projects/repronim]]
> [[done]] I think.. --[[Joey]]

14
doc/bugs/7.20181211+git29-gab4a1bed9_fails_tests_during_neurodebian_build/comment_1_ddd60096d409a820358d5a57a3a449c9._comment
View file

@ -1,14 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2019-01-01T16:10:48Z"
content="""
I'll bet [[!commit 14971414dc263fcb8124b4cf6b14b9b7a19189af]]
somehow led to this
since the test suite now runs gpg with its home
directory set to be inside the test repo and not in the system's temporary
directory.
But I tested that commit on filesystems down to FAT, so it seems strange it
would fail on whatever filesystem you are testing on.
"""]]

9
doc/bugs/7.20181211+git29-gab4a1bed9_fails_tests_during_neurodebian_build/comment_2_b496a4967b6bcc0a4e1cc55e06ba90c8._comment
View file

@ -1,9 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="the build details"
date="2019-01-02T01:14:39Z"
content="""
cowbuilder is used, underlying file system (dedicated to debian pkgs building) is ext4.
but is it a filesystem effect somehow or just a \"partially\" isolated environment aspect (such as those pbuilder/cowbuilder chroots)? The message `gpg: can't connect to the agent: IPC connect call failed` suggests that may be tests pass for you locally as connecting to your gpg agent?
"""]]

22
doc/bugs/7.20181211+git29-gab4a1bed9_fails_tests_during_neurodebian_build/comment_3_bdadf679a2f82c2a1cae747d437add85._comment
View file

@ -1,22 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2019-01-18T17:51:09Z"
content="""
Having what seems a related problem on my i386 autobuilder:
gpg: keyblock resource '/home/builder/gitbuilder/build/.t/tmprepo89/.git/annex/othertmp/gpgtest/2/pubring.kbx': No such file or directory
gpg: failed to create temporary file '/home/builder/gitbuilder/build/.t/tmprepo89/.git/annex/othertmp/gpgtest/2/.#lk0x57fed950.orca.13095': No such file or directory
gpg: can't connect to the agent: No such file or directory
gpg: problem with the agent: No agent running
gpg: can't create `/home/builder/gitbuilder/build/.t/tmprepo89/.git/annex/othertmp/gpgtest/2/random_seed': No such file or directory
user error (gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--batch","--passphrase-fd","28","--symmetric","--force-mdc","--no-textmode"] exited 2)
copy: 1 failed
FAIL (2.31s)
Test.hs:1645:
copy --to encrypted remote failed
Probably gpg error output has failed, but this is also failing to connect
to the agent. And it seems a file/directory gpg expects to find is not present
when it's running in the test harness.
"""]]

14
doc/bugs/7.20181211+git29-gab4a1bed9_fails_tests_during_neurodebian_build/comment_4_90bc2857c83afae250834135a5884898._comment
View file

@ -1,14 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2019-01-21T18:12:13Z"
content="""
I've fixed the problem I described above. But I don't think it was the
same problem; the neurodebian build log is not complaining about missing
files/directories.
gpg in the test suite is using a separate home directory, so it starts a
separate gpg-agent process for that. If the test suite is run on a
filesystem where gpg-agent can't work, due say to unix sockets not being
supported, that might be why it fails.
"""]]

8
doc/bugs/7.20181211+git29-gab4a1bed9_fails_tests_during_neurodebian_build/comment_5_d29518fef1b3b0b8937e078f9303d41c._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2019-01-21T19:18:51Z"
content="""
Anyway, see if [[!commit f5f059e288b5e618e5c7c86d1db4bfd23d8b39ad]] somehow
fixed it..
"""]]

17
doc/bugs/7.20181211+git29-gab4a1bed9_fails_tests_during_neurodebian_build/comment_6_24eca56dc972ecff0b553e93b7c649c4._comment
View file

@ -1,17 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 6"
date="2019-01-21T19:22:52Z"
content="""
tried to build 7.20181211+git284-g6f0d98f2a - it seems that the failing one is different one now?
prop_parse_show_Config: FAIL
*** Failed! Falsifiable (after 3 tests and 1 shrink):
fromList [(\"\791992\",\"\")]
Use --quickcheck-replay=133129 to reproduce.
prop_upFrom_basics: OK (0.03s)
...
1 out of 291 tests failed (254.87s)
"""]]

8
doc/bugs/7.20181211+git29-gab4a1bed9_fails_tests_during_neurodebian_build/comment_7_3adce9695e4e3786f0c68b092d8f6400._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 7"""
date="2019-01-21T20:30:52Z"
content="""
If that unrelated test case is the only failure, then this problem is
fixed.
"""]]

79
doc/bugs/A_case_where_file_tracked_by_git_unexpectedly_becomes_annex_pointer_file.mdwn
View file

@ -1,79 +0,0 @@
In datalad test builds with git-annex 7.20191114+git43-ge29663773, one
of the new test failures is due to an unexpectedly dirty repository
([related datalad issue][0]). The dirty status comes from a file that
was tracked in Git switching over to an annex pointer file. Here's a
script that distills enough of the test to trigger the failure on my
end.
[[!format sh """
#!/bin/sh
set -eu
assert_clean () {
if test -n "$(git status --porcelain)"
then
printf "\n\nUnexpectedly dirty:\n" >&2
git status >&2
git diff >&2
exit 1
fi
}
cd "$(mktemp -d --tmpdir gx-pointer-dirty-XXXXXXX)"
git init && git annex init
printf content-git >file-git
git -c annex.largefiles=nothing annex add -- file-git
git commit -m'file-git added'
assert_clean
printf content-annex >file-annex
git -c annex.largefiles=anything annex add -- file-annex
git commit -m'file-annex annexed'
assert_clean
"""]]
On Travis as well as my local machine, the failure is intermittent,
but seems to happen much more often than not. In the failing case,
the last assert_clean call shows:
```
Unexpectedly dirty:
On branch master
Changes not staged for commit:
modified: file-git
no changes added to commit
diff --git a/file-git b/file-git
index d1c416a..b41ca32 100644
--- a/file-git
+++ b/file-git
@@ -1 +1 @@
-content-git
\ No newline at end of file
+/annex/objects/SHA256E-s11--726732d25826965592478fcc7c145d5a10fa1aa70c49fe3a4f847174b6d8889c
```
I see the failure with git-annex built from the latest master
b962471c2 (2019-12-12). Bisecting against the git-annex repo (with a
commit being marked "bad" if there was a failure within ten runs of the
above script), points to ec08b66bd (shouldAnnex: check isInodeKnown,
2019-10-23) as the first bad commit. Just looking at the topic of
the commit, that result seems plausible to me.
### Other details
My git version 2.24.1 and locally I'm building git-annex through guix.
On the failing Travis run, git-annex 7.20191114+git43-ge29663773 came
from neurodebian, and the git version was 2.24.0.
Hopefully the script above is sufficient to trigger the issue on your end.
Thanks for having a look.
[0]: https://github.com/datalad/datalad/issues/3890
[[!meta author=kyle]]
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]

97
doc/bugs/A_case_where_file_tracked_by_git_unexpectedly_becomes_annex_pointer_file/comment_1_ea82bf47752807531870651653160f1c._comment
View file

@ -1,97 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2019-12-26T16:56:38Z"
content="""
The title makes it sound like a work tree file gets replaced with a
dangling pointer file, which is not the case. A worktree file that was
not annexed is is being added to the annex, if you choose to commit that
state.
For whatever reason, git becomes confused about whether this file is
modified. I seem to recall that git distrusts information it recorded in
its own index if the mtime of the index file is too close to the
mtime recorded inside it, or something like that. (Likely as a
workaround for mtime granularity issues with various filesystems.) Whatever
the reason, git-annex is not involved in it; it will happen sometimes even
when git-annex has not initialized the repo and is not being used.
It's not normally a problem that git gets confused or distrusts its
index or whatever, since all it does is stat the file, or
feed it through the clean filter again, and if the file is not
modified, nothing changes.
Why does the clean filter decide to add the file to annex in this case?
Well, because this is all happening inside this:
git -c annex.largefiles=anything annex add -- file-annex
And there you've told it to add all files to the annex with
annex.largefiles=anything. So it does.
To complete the description of what happens:
`git-annex add` runs `git add` on the `file-annex` symlink it's adding.
`git add file-annex`, for whatever reason, decides to run the clean filter on
file-git.
The annex.largefiles=anything gets inherited through this chain of calls.
While the resulting "change" does not get staged by `git add`
(it was never asked to operate on that file), the clean filter
duly ingests the content into the annex, and remembers its inode.
So when the clean filter later gets run by `git status`, it sees an inode
it knows it saw before, and assumes it should remain annexed.
(This is why the commit that checks for known inodes was fingered by the
bisection.)
---
Note that, you can accomplish the same thing without setting
annex.largefiles, assuming a current version of git-annex:
git add file-git
git annex add file-annex
I think the only reason for setting annex.largefiles in either of the two
places you did is if there's a default value that you want to
temporarily override?
----
Also, just touching file-git before the annex.largefiles=anything
operation causes the same problem, again git-annex add runs git add
file-annex, which runs the clean filter on file-git, which this time
is legitimately modified.
---
Possible ways to improve this short of improving git's behavior:
`git annex` could set annex.gitaddtoannex=false when it runs `git add`.
Since git-annex never relies on `git add` adding files to the annex,
that seems entirely safe to always do (perhaps even when running all git
commands aside from git-annex commands of course). But, that would
not help with a variant where rather than `git-annex add`,
this is run:
git -c annex.largefiles=anything add file-annex
The clean filter could delay long enough that git stops distrusting
its index based on timestamps. A 1 second sleep if the file's mtime
is too close to the current time works; I prototyped a patch doing that.
But, that does not deal with the case
mentioned above where file-git gets touched or legitimately modified.
The clean filter could check if the file is already
in the index but is not annexed, and avoid converting it to annexed.
But that would prevent legitimate conversions from git to annexed
as well, which rely on the same kind of use of annex.largefiles.
Temporary overrides of annex.largefiles could be ignored by the clean
filter. Same problem as previous.
So, I think that fixing this will involve adding a new interface for
converting between git and annexed files that does not involve
-c annex.largefiles. That plus having the clean filter check for
non-annexed files seems like the best approach.
"""]]

42
doc/bugs/A_case_where_file_tracked_by_git_unexpectedly_becomes_annex_pointer_file/comment_2_27faf0663265bd3855e98f7f1f382bdd._comment
View file

@ -1,42 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2019-12-27T06:22:23Z"
content="""
On second thought, making the clean filter check for non-annexed files
would prevent use cases like annex.largefiles=largerthan(100kb)
from working as the user intended and letting a small file start out
non-annexed and get annexed once it gets too large. Users certianly rely on
that and this bug that only affects an edge case does not justify breaking
that.
What would work to make the clean filter detect when a file's content
has not changed, though its mtime (or inode) has changed. In that case,
it's reasonable for the clean filter to ignore annex.largefiles and keep
the content represented in git however it already was (non-annexed or
annexed).
To detect that, in the case where the file in the index is not annexed:
First check if the file size is the same as the
size in the index. If it is, run git hash-object on the file, and see if
the sha1 is the same as in the index. This avoids hashing any unusually
large files, so the clean filter only gets a bit slower.
And when the file in the index is annexed, check if the file size is the
same as the size of the annexed key. If it is, verify if the file content
matches the key. (typically be hashing). Cases where keys lack size or
don't use a checksum could lead to false positives or negatives though.
Although, I've not managed to find a version of this bug that makes an
annexed file get converted to git unintentionally, so maybe this part does
not need to be done?
----
Or.. Since the root of the problem is temporarily overriding annex.largefiles,
it could just be documented that it's not a good idea to use
-c annex.largefiles=anything/nothing, because such broad overrides
can affect other files than the ones you intended.
(And since the documented methods of converting files from annexed to git and
git to annexed use such overrides, that documentation would need to be
changed.)
"""]]

16
doc/bugs/A_case_where_file_tracked_by_git_unexpectedly_becomes_annex_pointer_file/comment_3_37fb6d6361068f57c2068270ceb72ac0._comment
View file

@ -1,16 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2019-12-27T17:11:42Z"
content="""
A variant of this where an annexed unlocked file is added first,
then the file is touched, and then some other file is added
with -c annex.largefiles=nothing does result in the clean filter sending
the whole annexed file content back to git, rather than keeping it annexed.
For whatever reason, git does not store that content in .git/objects or
update the index for that file though, so it doesn't show up as a change.
So *apparently* that variant is only potentially an expensive cat of a
large annexed file, and does not need to be dealt with. Unless git
sometimes behaves otherwise.
"""]]

45
doc/bugs/A_case_where_file_tracked_by_git_unexpectedly_becomes_annex_pointer_file/comment_4_4c3b233aa1c6b0b6c0b0d7519b5877a1._comment
View file

@ -1,45 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2019-12-27T18:41:12Z"
content="""
It's almost possible to get the same unwanted conversion without any git
races:
echo content-git > file-git
sleep 2
git add file-git
git commit -m add
echo foo > file-git
echo content-annex > file-annex
git -c annex.largefiles=anything annex add file-annex
In this case, git currently does not run the modified file-git through the
clean filter in the last line, so the annex.largefiles=anything doesn't
affect it.
But, as far as I can see, there's nothing preventing a future version
of git from deciding it does want to run file-git through the clean filter
in this case.
I am not going to try to prevent against such a thing happening.
As far as I can see, anything that the clean filter can possibly do to
avoid such a situation will cripple existing uses cases of
annex.largefiles, like largerthan() as mentioned above.
The user has told git-annex to annex "anything", and if git
decides to run the clean filter while that is in effect, caveat emptor.
Which is not to say I'm not going to fix the specific case this bug was
filed about. I actually have a fix developed now. But just to say that
setting annex.largefiles=anything/nothing temporarily is a blunt instrument,
and you risk accidental conversion when using it, and so it would be a good
idea to not do that.
One idea: Make `git-annex add --annex` and `git-annex add --git`
add a specific file to annex or git, bypassing annex.largefiles and all
other configuration and state. This could also be used to easily switch
a file from one storage to the other. I'd hope the existence of that
would prevent one-off setting of annex.largefiles=anything/nothing.
[[todo/git_annex_add_option_to_control_to_where]]
"""]]

58
doc/bugs/A_case_where_file_tracked_by_git_unexpectedly_becomes_annex_pointer_file/comment_5_d1d23d3884fcbef3c8715ebdf8791f01._comment
View file

@ -1,58 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="comment 5"
date="2019-12-28T21:06:46Z"
content="""
Thanks for the explanation and the fix.
> For whatever reason, git becomes confused about whether this file is
> modified. I seem to recall that git distrusts information it recorded in
> its own index if the mtime of the index file is too close to the
> mtime recorded inside it, or something like that.
I see. I think the problem and associated workaround you're referring
to is described in git's Documentation/technical/racy-git.txt.
> Note that, you can accomplish the same thing without setting
> annex.largefiles, assuming a current version of git-annex:
>
> git add file-git
> git annex add file-annex
>
> I think the only reason for setting annex.largefiles in either of the two
> places you did is if there's a default value that you want to
> temporarily override?
Right. DataLad's methods that are responsible for calling out to `git
annex add` have a `git={None,False,True}` parameter. By default
(`None`), DataLad just calls `git annex add ...` and let's any
configuration in the repo control whether the file goes to git or is
annexed. But with `git=True` or `git=False`, the `annex add` call
includes a `-c annex.largefiles=` argument with a value of `nothing`
or `anything`, respectively.
> But just to say that setting annex.largefiles=anything/nothing
> temporarily is a blunt instrument, and you risk accidental
> conversion when using it, and so it would be a good idea to not do
> that.
Noted. As mentioned above, DataLad's default behavior is to honor the
repo's `annex.largefiles` configuration. And the documentation for
`datalad save`, DataLad's main user-facing entry point for `annex
add`, recommends that the user configure .gitattributes rather than
using the option that leads calling `annex add` with `-c
annex.largefiles=nothing`.
> One idea: Make `git-annex add --annex` and `git-annex add --git`
> add a specific file to annex or git, bypassing annex.largefiles and all
> other configuration and state. This could also be used to easily switch
> a file from one storage to the other. I'd hope the existence of that
> would prevent one-off setting of annex.largefiles=anything/nothing.
As far as I can see, those flags would completely cover DataLad's
one-off setting of `annex.largefiles=anything/nothing`. They map
directly to DataLad's `git=False/True` option described above. So,
from DataLad's perspective, they'd be very useful and welcome.
"""]]

8
doc/bugs/A_case_where_file_tracked_by_git_unexpectedly_becomes_annex_pointer_file/comment_6_b713bf610c4f4d9c2463b61721bce406._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 6"""
date="2020-01-01T17:41:13Z"
content="""
I've added git-annex add --force-large and --force-small, which would be
good to use to avoid this kind of too-broad overriding problem in the future.
"""]]

66
doc/bugs/Buggy_external_special_remote_stalls_after_7245a9e.mdwn
View file

@ -1,66 +0,0 @@
In the datalad test suite, a test involving one of our special remotes
hangs after 7245a9ed5 (Improve shutdown process for external special
remotes and external backends, 2020-11-02). The hang depends on the
remote program creating an SSH socket but not cleaning it up on
failure. That's being addressed on our end
(<https://github.com/datalad/datalad/pull/5151>).
While that's clearly bad handling on the part of datalad's special
remote, I'm opening this bug report in case the above stall indicates
that 7245a9ed5 brings an increased susceptibility to hanging that's
worth worrying about. Based on 7245a9ed5's commit message, it sounds
like that resolved some hanging on Windows (and generally sounds like
a good change), so it probably makes sense keep that at the expense of
hanging that is restricted to buggy special remotes. But I don't have
a good enough understanding of the issue to know if it's limited to
misbehaving special remotes, or if perhaps there's a way to avoid both
hangs.
In case it's helpful, here's a demo that tweaks the example special
remote to trigger a hang:
[[!format sh """
set -eu
ga_srcdir="$(realpath "${1?requires git-annex source directory as first argument}")"
host="${2?requires ssh host as second argument}"
cd "$(mktemp -d "${TMPDIR:-/tmp}"/gx-testremote-XXXXXXX)"
top="$PWD"
mkdir bin
git -C "$ga_srcdir" cat-file -p cfae74ae25a791a8dbf8b476fe17d5c6feacd8b0 \
>bin/git-annex-remote-foo
# Open a socket in setupcreds().
cmd="ssh -fN -o ControlMaster=auto -o ControlPersist=15m -o ControlPath=$top/socket $host"
sed -i "s|setupcreds () {|&\\n\\t$cmd|" bin/git-annex-remote-foo
chmod +x bin/git-annex-remote-foo
export PATH="$top/bin:$PATH"
git annex version
git init a
(
cd a
git annex init main
# This will trigger an INITREMOTE-FAILURE failure because
# MYPASSWORD and MYLOGIN are not set. This hangs after 7245a9ed5.
git annex initremote d --debug \
type=external externaltype=foo directory="$top/d" encryption=none
)
"""]]
```
$ sh /tmp/gx-failed-dir-hang.sh ~/src/haskell/git-annex smaug
git-annex version: 8.20201103
build flags: Assistant Pairing Inotify MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.4 feed-1.2.0.1 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.10.5.2 uuid-1.3.13
[... 40 lines ...]
[2020-11-12 11:40:01.886013862] /tmp/gx-testremote-3MudMuQ/bin/git-annex-remote-foo[1] --> SETCONFIG directory /tmp/gx-testremote-3MudMuQ/d
[2020-11-12 11:40:02.451619481] /tmp/gx-testremote-3MudMuQ/bin/git-annex-remote-foo[1] --> INITREMOTE-FAILURE You need to set MYPASSWORD and MYLOGIN environment variables when running initremote.
```
[[!meta author=kyle]]
[[!tag projects/datalad]]
> [[fixed|done]]

27
doc/bugs/Buggy_external_special_remote_stalls_after_7245a9e/comment_1_c0a98d1391d4426182d3d43651ad67e5._comment
View file

@ -1,27 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="comment 1"
date="2020-11-16T18:22:05Z"
content="""
> In the datalad test suite, a test involving one of our special remotes
> hangs after 7245a9ed5 [...] The hang depends on the remote program
> creating an SSH socket but not cleaning it up on failure. That's being
> addressed on our end (https://github.com/datalad/datalad/pull/5151).
That statement was based on me being able to trigger a stall locally
in the same test on my Buster system. The changes from datalad's pr
5151 resolved the hang on my end, as well as the hang from the
stripped down demo I added here.
However, with the changes from pr 5151 merged, it turns out the the
stall on the Ubuntu 18.04 CI is still there [1]. So, it's unclear
what's going on. Perhaps 7245a9ed5 isn't even what triggered the hang
on the CI, though the stall seems to have first appeared 14 days ago
on a build using 7e24b2587 [2], so the timing is at least consistent
with 7245a9ed5 being the culprit.
[1] https://github.com/datalad/git-annex/runs/1404305092?check_suite_focus=true
[2] https://github.com/datalad/datalad-extensions/runs/1345445794?check_suite_focus=true
"""]]

44
doc/bugs/Buggy_external_special_remote_stalls_after_7245a9e/comment_2_952d0856b79cdf1637a84c74f396d9e8._comment
View file

@ -1,44 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2020-11-17T16:06:53Z"
content="""
The ssh process you run this way has the following file descriptor:
l-wx------ 1 joey joey 64 Nov 17 12:07 2 -> pipe:[6320144]
That pipe is connected to git-annex. git-annex is waiting to read any errors
that the special remote might output to stderr, in order to relay them to the
user. Yes, this was changed by the referenced commit.
Since git-annex has already waited on the process, the process is dead
by the point it waits on the stderr relayer thread. The only purpose
of waiting rather than closing the handle is to see anything the process
might have output in its dying breath.
Also, this is very similar to the problem fixed
in [[!commit aa492bc65904a19f22ffdfc20d7a5e7052e2f54d]] and in
[[!commit cb74cefde782e542ad609b194792deabe55b1f5a]], also involving ssh.
Those were solved using a rather ugly up to 2 second wait for any late
stderr to arrive, though it only delays when the handle is kept open like this.
There are surely a ton of places where this could potentially happen.
Not only stderr.. it's entirely possible that process that git-annex
expects to read stdout from might spawn a daemon that keeps inherited
stdout open, and exit, leaving git-annex waiting forever to read
from the pipe.
I'm doubtful this is a bug everywhere that git-annex reads all the stdout from
a pipe. Because it seems to me a great many programs would have the same
problem if a program they were piping stdout from behaved in that way.
I've never seen anything concern itself with this potential problem. This is
why proper daemons close their handles, certianly their stdout handle. stderr
is a slightly special case maybe.
processTranscript is one example of another place in git-annex that
waits to consume all stderr from a process and would be hung by such a daemon.
There are a dozen in all,
`git grep 'std_err =' | egrep 'CreatePipe|UseHandle | grep -v nullh`
I suppose they could all be audited and maybe something abstracted out to
deal with them all.
"""]]

57
doc/bugs/Buggy_external_special_remote_stalls_after_7245a9e/comment_3_ed02f45c96f7eedf2c15add673ca84f2._comment
View file

@ -1,57 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2020-11-17T19:04:26Z"
content="""
One approach would be:
cancelOnExit :: ProcessHandle -> IO a -> IO (Maybe a)
The versions of this currently being used to work around the past
occurances of the problem wait for the process to exit, and then wait 2
seconds before canceling the action, to give it time to read anything that
was buffered in the pipe.
That's annoyingly racey, what if the system somehow massively slows
down for 2 seconds just after the process exits, and so buffered output is
lost? Doing this in a lot of places, not just a few targeted places where
maybe we didn't care too much about stderr anyway, does not feel safe.
There's GHC.IO.Handle.hWaitForInput that does not consume input from a
Handle, but checks if there is any available to read. Digging into what
that really means -- as well as checking its internal buffers, it calls a
[fdReady C function](https://gitlab.haskell.org/ghc/ghc/-/blob/ba4dcc7cb77a37486368911fec854d112a1db93c/libraries/base/cbits/inputReady.c)
which boils down to a `poll()`. So that seems like it could be used to
avoid the race. I think it would be safest to use it with 0, which avoids a
case where, it interrupted too long, it can return false when there's still
input.
It should be possible to use that to implement this:
readUntilExit :: ProcessHandle -> (Handle -> IO a) -> Handle -> IO a
Wait for the process to exit, and then check hWaitForInput, and as long as
there's input left, sleep breifly and loop. Once all buffered input as been
consumed by the action, hClose the handle to get the action to finish
reading from it.
One problem with this is that eg, this would throw an exception if it
closed the handle out from under hGetLine before it read anything:
unlessM (hIsEOF h) $
readUntilExit ph hGetLine h
So it's not entirely safe to drop this in just anywhere. Maybe it should
catch the EOF exception?
readUntilExitOrEOF :: ProcessHandle -> (Handle -> IO a) -> Handle -> IO (Maybe a)
That should be fine for things like hGetLine, but would still be unsafe
for hGetContents, which does *not* like having the handle it's reading
closed out from under it ("hGetContents: illegal operation (delayed read on closed handle)").
So maybe best to specialize it:
hGetLineUntilExitOrEOF :: ProcessHandle -> Handle -> IO (Maybe String)
I have a partial implementation on the ssh-hates-me branch.
"""]]

27
doc/bugs/Buggy_external_special_remote_stalls_after_7245a9e/comment_4_639337906e441523a610c7080f215aa8._comment
View file

@ -1,27 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2020-11-18T15:57:10Z"
content="""
hWaitForInput does not seem to behave as documented. It does not time out,
so blocks forever in this situation. This is with a 0 timeout and with
larger timeouts. Unsure why, it looked like it should work. Looking at the
ghc bug tracker, it seems to have some open bugs.
([[!commit 9af0000e0ff28c8eaae0ee1ce6838063d0a5ebef]] has my bench test
demonstrating this problem.)
hGetLineUntilExitOrEOF could still be implemented, but rather than using
hGetLine, it would need to use hGetBufNonBlocking to do one byte at a time
reads, with a small sleep when there is nothing to read,
until it gets an empty read at a point after the process has exited.
I can't see a way to implement it without the sleep, and that is
problimatic, too long sleep will slow down reading, and too short a sleep
will waste cpu. It might be possible to use hGetBufSome and cancel it
once the process exits, and only then switch to hGetBufNonBlocking to
read the rest of the buffer. But it's not clear what would happen hGetBufSome
got as far as reading from the handle, but didn't quite manage to finish
before being timed out. Seems likely any data it did read would then be lost.
Think I'll have to abandon the hGetLineUntilExitOrEOF idea.
"""]]

20
doc/bugs/Buggy_external_special_remote_stalls_after_7245a9e/comment_4_746ffcf575f91bc3d156b279bcdf8ae2._comment
View file

@ -1,20 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2020-11-18T17:23:21Z"
content="""
Seems the problem with hWaitForInput happens when running it concurrently
with hGetLine on the same handle. When I run it sequentially, it does seem
to work. Ok.. I was able to implement hGetLineUntilExitOrEOF that seems to
work.
I've converted it to use that, and the test case for this bug is fixed.
Also, the sleep based workaround
in [[!commit aa492bc65904a19f22ffdfc20d7a5e7052e2f54d]] is removed,
and it uses hGetLineUntilExitOrEOF instead.
However, I still want to do the audit of other things that pipe stderr.
Some of them don't do line based input, so will need a different function
than hGetLineUntilExitOrEOF. Leaving this bug open until that audit is
done.
"""]]

11
doc/bugs/Buggy_external_special_remote_stalls_after_7245a9e/comment_4_e4f249bc78e870c91b3fcbf34596a9ed._comment
View file

@ -1,11 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="comment 4"
date="2020-11-18T15:30:06Z"
content="""
Thanks for the analysis. (I wish I had some sort of intelligent input
to give rather than just being the reporter of several of these ssh
issues :/)
"""]]

10
doc/bugs/Buggy_external_special_remote_stalls_after_7245a9e/comment_7_a9b594ecd43c0d2f35029c934ba7e143._comment
View file

@ -1,10 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="comment 7"
date="2020-11-19T15:34:59Z"
content="""
Thanks. The hanging datalad CI builds are now resolved.
https://github.com/datalad/git-annex/actions/runs/371469598
"""]]

8
doc/bugs/Buggy_external_special_remote_stalls_after_7245a9e/comment_8_9fdcef0d78d04ce05765693fe296bb3d._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 8"""
date="2020-11-19T20:37:57Z"
content="""
Finished auditing and fixing anywhere in git-annex this could possibly
happen.
"""]]

15
doc/bugs/Build__47__OSXMkLibs.hs_does_not_resolve___64__loader__95__path.mdwn
View file

@ -1,15 +0,0 @@
As of December 3, the GitHub Actions version of one of the programs (not sure which one) that git-annex bundles in its DMG installer for macOS uses `@loader_path` in the path for one of its dynamic libraries. (The value of the `@loader_path` variable is the directory containing the binary file in which the path appears.) `Build/OSXMkLibs.hs` does not resolve the `@loader_path` syntax, and so this leads to `./Build/Standalone` failing with:
```
cp: ./@loader_path/libbrotlicommon.1.dylib: No such file or directory
Standalone: library install failed
CallStack (from HasCallStack):
error, called at ./Build/OSXMkLibs.hs:72:41 in main:Build.OSXMkLibs
```
I attempted to patch `Build/OSXMkLibs.hs` to handle this myself, but the file containing this syntax is another dynamic lib, and the code doesn't seem to record the original paths that dynamic libraries are copied from, and so the patch would be nontrivial.
[[!meta author=jwodder]]
[[!tag projects/datalad]]
> [[done]] per commeents --[[Joey]]

8
doc/bugs/Build__47__OSXMkLibs.hs_does_not_resolve___64__loader__95__path/comment_1_138124e29ad5a27751fcd133e9dbf6df._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 1"
date="2020-12-14T14:07:47Z"
content="""
For your convenience Joey here is a sample fresh (since still keeps failing) log: [http://www.onerussian.com/tmp/git-annex-osx-20201214-1.txt](http://www.onerussian.com/tmp/git-annex-osx-20201214-1.txt) . Link to workflow all results: [https://github.com/datalad/git-annex/actions?query=workflow%3A%22Build+git-annex+on+macOS%22](https://github.com/datalad/git-annex/actions?query=workflow%3A%22Build+git-annex+on+macOS%22)
"""]]

25
doc/bugs/Build__47__OSXMkLibs.hs_does_not_resolve___64__loader__95__path/comment_2_4f68ed79045c93831cb8b41888057f29._comment
View file

@ -1,25 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2020-12-14T19:08:40Z"
content="""
install_name_tool: warning: changes being made to the file will invalidate the code signature in: tmp/build-dmg/git-annex.app/Contents/MacOS/bundle/R
This makes me doubtful that, even if the `@loader_path` problem gets resolved
and it builds, it will actually work. It seems likely that if a library is
signed, modifying it to have an invalid signature would make the OS refuse to
use it.
The OSX build used for releases is still using an older OSX and does not have
either problem. I do notice there is a similar workaround in Build/OSXMkLibs.h
for `@executable_path`; it skips such libraries. So skipping `@loader_path`
in the same place would probably build, but would it work?
Since git-annex can be installed from homebrew, I don't know if it's worth
trying to keep the standalone app working in the face of such problems.
Is there a use case for the .dmg file that this failing build is producing?
One approach would be to stop bundling system libraries, so the .dmg
perhaps is tied to a specific version of OSX. I've added a way to build
the dmg that way, make osxapp USE_SYSTEM_LIBS=1
"""]]

10
doc/bugs/Build__47__OSXMkLibs.hs_does_not_resolve___64__loader__95__path/comment_3_72d271ac0612e13b3ceaf736e9e50fd9._comment
View file

@ -1,10 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 3"
date="2020-12-14T20:59:06Z"
content="""
> Is there a use case for the .dmg file that this failing build is producing?
In our case we build .dmg to have it possibly (re)distributed later on as a build of git-annex to use. But primary use case is just to have it installed and run git-annex and then datalad tests to ensure correct operation.
"""]]

7
doc/bugs/Build__47__OSXMkLibs.hs_does_not_resolve___64__loader__95__path/comment_4_69ae4b847873d039d790cd2b1cfa97c0._comment
View file

@ -1,7 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2020-12-14T21:43:29Z"
content="""
In that case, seems fine to build with USE_SYSTEM_LIBS=1
"""]]

19
doc/bugs/Build__47__OSXMkLibs.hs_does_not_resolve___64__loader__95__path/comment_5_ca662cade7153349ce644cd93c379815._comment
View file

@ -1,19 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 5"
date="2020-12-15T16:18:39Z"
content="""
FTR: unfortunately not a complete workaround for us since the 'test' step is done in a clean environment (not build one), where we just install .dmg. That results in git-annex not being able to find those dynamic libraries it had in build environment
```
Run git annex version
dyld: Library not loaded: /usr/local/opt/libmagic/lib/libmagic.1.dylib
Referenced from: /Applications/git-annex.app/Contents/MacOS/bundle/git-annex
Reason: image not found
error: git-annex died of signal 6
Error: Process completed with exit code 134.
```
ref: [https://github.com/datalad/git-annex/pull/31](https://github.com/datalad/git-annex/pull/31)
"""]]

13
doc/bugs/Build__47__OSXMkLibs.hs_does_not_resolve___64__loader__95__path/comment_6_c75c9bcbcc8e68c60323dfde9c0248a7._comment
View file

@ -1,13 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 6"""
date="2020-12-29T16:49:52Z"
content="""
I later removed `USE_SYSTEM_LIBS` and made it skip
`@loader_path` as it was already skipping `@executable_path`.
Looks like the OSX build is succeeding with building and using the
app. I suppose this can probably be closed, although we don't know how
portable that app might be to systems where the libraries this causes it to
skip are at other versions than it needs.
"""]]

31
doc/bugs/Build_failing_because_MonadFail_is_not_in_scope.mdwn
View file

@ -1,31 +0,0 @@
As of commit b9c1aa925, trying to build git-annex on either Ubuntu or macOS fails with:
```
Annex/TransferrerPool.hs:56:24: error:
Not in scope: type constructor or class `MonadFail'
|
| :: (MonadIO m, MonadFail m, MonadMask m)
| ^^^^^^^^^
```
This can be fixed with the following patch:
```
diff --git a/Annex/TransferrerPool.hs b/Annex/TransferrerPool.hs
index 973f75629..0de145461 100644
--- a/Annex/TransferrerPool.hs
+++ b/Annex/TransferrerPool.hs
@@ -28,6 +28,7 @@ import Utility.ThreadScheduler
import Control.Concurrent
import Control.Concurrent.Async
import Control.Concurrent.STM hiding (check)
+import Control.Monad.Fail (MonadFail)
import Control.Monad.IO.Class (MonadIO)
import Text.Read (readMaybe)
import Data.Time.Clock.POSIX
```
[[!meta author=jwodder]]
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]

12
doc/bugs/Build_failing_because_MonadFail_is_not_in_scope/comment_1_3ebe2e53fd8e8ce673267ac2715d964c._comment
View file

@ -1,12 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2020-12-09T16:40:24Z"
content="""
Indeed, and thank you for the patch.
However, MonadFail often suggests a code smell, here it's the `Just ... <-`
pattern match which could potentially fail. In fact, it's not possible for
it to, so I have instead made that clear in a way that the type checker
can understand.
"""]]

76
doc/bugs/Build_failing_on_Windows_since_ef3ab0769.mdwn
View file

@ -1,76 +0,0 @@
Due to the type change of `tryMakeLockHandle` in commit ef3ab0769, compilation of git-annex is currently failing on Windows with the below error. I am unsure of the best way to fix it.
[[!format txt """
Utility\LockPool\Windows.hs:25:19: error:
* Couldn't match type `(LockHandle, t1)' with `LockHandle'
Expected type: IO (Maybe LockHandle)
Actual type: IO (Maybe (LockHandle, t1))
* In the expression:
tryMakeLockHandle
P.lockPool file (\ p f -> P.tryTakeLock p f LockShared)
(\ f _ -> fmap mk <$> F.lockShared f)
In an equation for `lockShared':
lockShared file
= tryMakeLockHandle
P.lockPool file (\ p f -> P.tryTakeLock p f LockShared)
(\ f _ -> fmap mk <$> F.lockShared f)
|
25 | lockShared file = tryMakeLockHandle P.lockPool file
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
Utility\LockPool\Windows.hs:27:18: error:
* Couldn't match type `FileLockOps' with `(FileLockOps, t1)'
Expected type: IO (Maybe (FileLockOps, t1))
Actual type: IO (Maybe FileLockOps)
* In the expression: fmap mk <$> F.lockShared f
In the fourth argument of `tryMakeLockHandle', namely
`(\ f _ -> fmap mk <$> F.lockShared f)'
In the expression:
tryMakeLockHandle
P.lockPool file (\ p f -> P.tryTakeLock p f LockShared)
(\ f _ -> fmap mk <$> F.lockShared f)
|
27 | (\f _ -> fmap mk <$> F.lockShared f)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
Utility\LockPool\Windows.hs:36:22: error:
* Couldn't match type `(LockHandle, t0)' with `LockHandle'
Expected type: IO (Maybe LockHandle)
Actual type: IO (Maybe (LockHandle, t0))
* In the expression:
tryMakeLockHandle
P.lockPool file (\ p f -> P.tryTakeLock p f LockExclusive)
(\ f _ -> fmap mk <$> F.lockExclusive f)
In an equation for `lockExclusive':
lockExclusive file
= tryMakeLockHandle
P.lockPool file (\ p f -> P.tryTakeLock p f LockExclusive)
(\ f _ -> fmap mk <$> F.lockExclusive f)
|
36 | lockExclusive file = tryMakeLockHandle P.lockPool file
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
Utility\LockPool\Windows.hs:38:18: error:
* Couldn't match type `FileLockOps' with `(FileLockOps, t0)'
Expected type: IO (Maybe (FileLockOps, t0))
Actual type: IO (Maybe FileLockOps)
* In the expression: fmap mk <$> F.lockExclusive f
In the fourth argument of `tryMakeLockHandle', namely
`(\ f _ -> fmap mk <$> F.lockExclusive f)'
In the expression:
tryMakeLockHandle
P.lockPool file (\ p f -> P.tryTakeLock p f LockExclusive)
(\ f _ -> fmap mk <$> F.lockExclusive f)
|
38 | (\f _ -> fmap mk <$> F.lockExclusive f)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
"""]]
[[!meta author=jwodder]]
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]

6
doc/bugs/Build_fails_on_Windows_as_of_commit_0a9a9c44e.mdwn
View file

@ -1,6 +0,0 @@
As of commit 0a9a9c44e, trying to build git-annex on Windows fails because due to a type error. [This patch](https://raw.githubusercontent.com/datalad/git-annex/master/patches/20210630-0a9a9c44e-fix-windows.patch) fixes it.
[[!meta author=jwodder]]
[[!tag projects/datalad]]
> [[done]] thankyou! --[[Joey]]

6
doc/bugs/Build_fails_on_Windows_as_of_commit_4b1b9d7a8.mdwn
View file

@ -1,6 +0,0 @@
As of commit 4b1b9d7a8, trying to build git-annex on Windows fails because of a missing function argument and a missing import, both in `Annex/Init.hs`. [This patch](https://raw.githubusercontent.com/datalad/git-annex/master/patches/20210622-4b1b9d7a8-fix-windows.patch) fixes both problems.
[[!meta author=jwodder]]
[[!tag projects/datalad]]
> [[done]], thank you jwodder! --[[Joey]]

6
doc/bugs/Build_fails_on_Windows_as_of_commit_a706708d1.mdwn
View file

@ -1,6 +0,0 @@
As of commit a706708d1, trying to build git-annex on Windows fails because the import of `oneSecond` from `Utility.ThreadScheduler` is not available. [This patch](https://raw.githubusercontent.com/datalad/git-annex/master/patches/20210607-a706708d1-fix-oneSecond.patch) fixes that.
[[!meta author=jwodder]]
[[!tag projects/datalad]]
> Thank you for reporting and for the patch. [[applied|done]] --[[Joey]]

6
doc/bugs/Build_fails_on_Windows_as_of_commit_ddd7d1d11.mdwn
View file

@ -1,6 +0,0 @@
As of commit ddd7d1d11, trying to build git-annex on Windows fails due to a function being defined with too many arguments. [This patch](https://raw.githubusercontent.com/datalad/git-annex/master/patches/20210902-ddd7d1d11-fix-windows.patch) fixes it.
[[!meta author=jwodder]]
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]

179
doc/bugs/Change_to_annex.largefiles_leaves_repo_modified.mdwn
View file

@ -1,179 +0,0 @@
### Please describe the problem.
When unlocking an annexed file and also adding a matching `annex.largefiles=nothing` entry to `.gitattributes`, leaves the file modified after committing both.
This seems to happen only recently. Observed on macOS VM with recent annex in DataLad's CI builds. Same situation comes out clean on other machines (with not as recent git-annex). Moreover it appears to be a race condition, since with the reproducer below it happens almost always, but sometimes it does end up with a clean repository.
Note, that the issue is less about what exactly should happen to that file. git-annex-add reports it's adding it to the git repository and that's fine. If git-annex was to say: "Hey, that's stupid" - that too would probably be fine. What seems wrong either way is a modified file right after committing this file, plus the fact that we didn't observe this with earlier annex versions.
### What steps will reproduce the problem?
[[!format sh """
mkdir testrepo
cd testrepo
git init
git annex init
echo "**/.git* annex.largfiles=nothing" > .gitattributes
git annex add .gitattributes
echo "something" > somefile
git annex add -c annex.dotfiles=true -- .gitattributes somefile
git commit -m "to annex"
ls -la
git status
#----------Setup done-----------------
git annex unlock somefile
echo "* annex.largefiles=nothing" >> .gitattributes
git annex add -c annex.dotfiles=true -- .gitattributes somefile
git commit -m "adding to git" .gitattributes somefile
git status # <- this will (most of the time) show a modified `somefile`
"""]]
### What version of git-annex are you using? On what operating system?
[[!format sh """
> git annex version
git-annex version: 10.20220222
build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-8.10.7 http-client-0.7.11 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: darwin x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 8
> git --version
git version 2.35.1
"""]]
### Please provide any additional information below.
On the said system the reproducer script above gives:
[[!format sh """
++ mkdir testrepo
++ cd testrepo
++ git init
Initialized empty Git repository in /Users/appveyor/projects/testrepo/.git/
++ git annex init
init ok
(recording state in git...)
++ echo '**/.git* annex.largfiles=nothing'
++ git annex add .gitattributes
add .gitattributes (non-large file; adding content to git repository) ok
(recording state in git...)
++ echo something
++ git annex add -c annex.dotfiles=true -- .gitattributes somefile
add somefile
ok
(recording state in git...)
++ git commit -m 'to annex'
[master (root-commit) fd4307e] to annex
2 files changed, 2 insertions(+)
create mode 100644 .gitattributes
create mode 120000 somefile
++ ls -la
total 8
drwxr-xr-x 5 appveyor staff 160 Mar 16 04:36 .
drwxr-xr-x 5 appveyor staff 160 Mar 16 04:36 ..
drwxr-xr-x 13 appveyor staff 416 Mar 16 04:36 .git
-rw-r--r-- 1 appveyor staff 33 Mar 16 04:36 .gitattributes
lrwxr-xr-x 1 appveyor staff 180 Mar 16 04:36 somefile -> .git/annex/objects/gG/2m/SHA256E-s10--4bc453b53cb3d914b45f4b250294236adba2c0e09ff6f03793949e7e39fd4cc1/SHA256E-s10--4bc453b53cb3d914b45f4b250294236adba2c0e09ff6f03793949e7e39fd4cc1
++ git status
On branch master
nothing to commit, working tree clean
++ git annex unlock somefile
unlock somefile ok
(recording state in git...)
++ echo '* annex.largefiles=nothing'
++ git annex add -c annex.dotfiles=true -- .gitattributes somefile
add .gitattributes (non-large file; adding content to git repository) ok
add somefile (non-large file; adding content to git repository) ok
(recording state in git...)
++ git commit -m 'adding to git' .gitattributes somefile
[master 9a6e407] adding to git
2 files changed, 2 insertions(+), 1 deletion(-)
rewrite somefile (100%)
mode change 120000 => 100644
++ git status
On branch master
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: somefile
"""]]
Whereas on a different machine it looks like this:
[[!format sh """
+ mkdir testrepo
+ cd testrepo
+ git init
Initialized empty Git repository in /tmp/testrepo/.git/
+ git annex init
init ok
(recording state in git...)
+ echo **/.git* annex.largfiles=nothing
+ git annex add .gitattributes
add .gitattributes (non-large file; adding content to git repository) ok
(recording state in git...)
+ echo something
+ git annex add -c annex.dotfiles=true -- .gitattributes somefile
add somefile
ok
(recording state in git...)
+ git commit -m to annex
[master (root-commit) 408fed5] to annex
2 files changed, 2 insertions(+)
create mode 100644 .gitattributes
create mode 120000 somefile
+ ls -la
total 36
drwxr-xr-x 3 ben ben 4096 Mar 16 10:39 .
drwxrwxrwt 29 root root 20480 Mar 16 10:39 ..
drwxr-xr-x 9 ben ben 4096 Mar 16 10:39 .git
-rw-r--r-- 1 ben ben 33 Mar 16 10:39 .gitattributes
lrwxrwxrwx 1 ben ben 180 Mar 16 10:39 somefile -> .git/annex/objects/gG/2m/SHA256E-s10--4bc453b53cb3d914b45f4b250294236adba2c0e09ff6f03793949e7e39fd4cc1/SHA256E-s10--4bc453b53cb3d914b45f4b250294236adba2c0e09ff6f03793949e7e39fd4cc1
+ git status
On branch master
nothing to commit, working tree clean
+ git annex unlock somefile
unlock somefile ok
(recording state in git...)
+ echo * annex.largefiles=nothing
+ git annex add -c annex.dotfiles=true -- .gitattributes somefile
add .gitattributes (non-large file; adding content to git repository) ok
add somefile (non-large file; adding content to git repository) ok
(recording state in git...)
+ git commit -m adding to git .gitattributes somefile
[master 865efeb] adding to git
2 files changed, 2 insertions(+), 1 deletion(-)
rewrite somefile (100%)
mode change 120000 => 100644
+ git status
On branch master
nothing to commit, working tree clean
+ git annex version
git-annex version: 8.20211123
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
+ git --version
git version 2.34.1
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Lots! I love git-annex!
[[!meta author=bpoldrack]]
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]

14
doc/bugs/Change_to_annex.largefiles_leaves_repo_modified/comment_1_7cd205f8f5560420d117d5a2bde48532._comment
View file

@ -1,14 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="FWIW a detail"
date="2022-03-16T14:38:45Z"
content="""
I believe our corresponding test in datalad started to fail on this date:
```
(git)smaug:/mnt/datasets/datalad/ci/git-annex/builds/2022/02[master]git
$> git grep 'test_partial_unlocked.*FAIL' | head -n 1
cron-20220219/build-macos.yaml-595-85dd9355-failed/1_test-datalad (master).txt:2022-02-19T02:37:34.9834050Z datalad.core.local.tests.test_save.test_partial_unlocked ... FAIL
```
which had git-annex-macos-dmg_10.20220127+git57-gb481ec273_x64 -- so that or some commit shortly before likely introduced that regression
"""]]

77
doc/bugs/Change_to_annex.largefiles_leaves_repo_modified/comment_2_725d46358366ea79e375fd47beac6acd._comment
View file

@ -1,77 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2022-03-21T17:33:28Z"
content="""
Isolated to [[!commit c68f52c6a25ebd8a12f4572da53fbba827d682aa]]
Since that commit is only a speed improvement, it can be reverted, which
does avoid the problem.
But I don't really see a bug in that commit either. It only causes
git to smudge the file when it's unlocked, rather than at some later
point. The result should be the same either way. Except, of course,
that you're varying the annex.largefiles config, so when git smudges
the file later, it sees a different result than when it smudges the file
while unlock is running.
I think that the actually surprising thing here is that
`git-annex add somefile` re-adds the file (adding it to git per the new
annex.largefiles configuration). The file is in typechange state at that
point but is not modified, so it does not need to be re-added at all.
But add seeks `withUnmodifiedUnlockedPointers` which finds files in
typechange state.
So git-annex add decides to add the file to git even though it was
previously added to the annex. At that point, I'd expect `git add` would
update the index and replace the smudged content with the unsmudged
content. However, it actually does not!
Looking at `git ls-files --stage` before and after git-annex add,
the annex pointer content remains staged:
100644 bf1114c25db2833499838552f3fa69635fcc1292 0 somefile
add .gitattributes (non-large file; adding content to git repository) ok
add somefile (non-large file; adding content to git repository) ok
(recording state in git...)
100644 bf1114c25db2833499838552f3fa69635fcc1292 0 somefile
The only reason that git commit goes on to commit the non-annexed
content is because you pass somefile to it, so it re-adds somefile
at that point, which runs the smudge filter, which passes the content
through.
And that's why git status then sees the file as modified --
the index has the annex pointer still staged in it, which is different
from what got committed.
So, probably this same thing could happen in another situation, even if
the change to git-annex unlock were reverted. Eg, if something else
caused git smudge to run just after unlock, before the annex.largefiles
config got changed.
Why is `git add` not updating the index? Presumably because the inode
and mtime have not changed, so it does not think it needs to. Even though
it's being run with the smudge filter disabled, and so sees a different
hash than the one in the index. I think a case could be made for this
being a bug in `git add`. Although maybe the `--renormalize` option
is intended to handle this kind of situation.
The documented way for a user to change a file from being stored in git-annex
to git is to first `git rm --cached` and then `git annex add --force-small`.
Which avoids that `git add` behavior.
I tracked git-annex add's behavior here back to
[[!commit 2743224658cee33657ad30653ff97de2363366c6]].
Which addressed an old bug, filed by Kyle, which can be seen at
[[!commit 5567496518]]. So it was intentional that unlocking a file
and then `git annex add` should lock it, for consistency with v5
and for consistency with unlocking a file
and then modifying it, and then `git annex add`. I am actually a bit
surprised at this behavior, but it was intended to handle locking files,
not adding files to git.
So, I think it would be fine for `git-annex add` to skip over a typechanged
file when it would be added to git rather than to the annex.
I've implemented that, and it does fix this bug.
"""]]

80
doc/bugs/Deleted_files_not_considered_with_wanted__61____34__anything__34__.mdwn
View file

@ -1,80 +0,0 @@
My understanding is that a preferred content value of "anything" (or
equivalently the standard group "backup") should consider deleted
files from a remote as wanted. I'm not a heavy user of preferred
content myself, so this understanding is based primarily off of the
[preferred_content/standard_groups][0] documentation saying
```
### backup
All content is wanted. Even content of old/deleted files.
`anything`
```
At least with the scenarios I've constructed, I haven't been able to
find a case where old files are considered. As an example:
[[!format sh """
cd "$(mktemp -d ${TMPDIR:-/tmp}/gx-XXXXXXX)"
git init a
(
cd a
git annex init a
>foo && git annex add foo && git commit -m"add foo"
git rm foo && git commit -m"delete foo"
)
git clone a b
(
cd b
git annex init b
git annex wanted . anything
git annex sync
set -x
git show git-annex:uuid.log
git show git-annex:preferred-content.log
git annex find --want-get
git annex get --auto
git annex unused
)
"""]]
```
[...]
+ git show git-annex:uuid.log
17fae615-a90b-405a-85c2-9d0f28246bdf a timestamp=1586190820.385256186s
1f0ad7d6-31ba-4d7e-a2e6-ab6acb3338c9 b timestamp=1586190820.750487084s
+ git show git-annex:preferred-content.log
1f0ad7d6-31ba-4d7e-a2e6-ab6acb3338c9 anything timestamp=1586190820.867633471s
+ git annex find --want-get
+ git annex get --auto
+ git annex unused
unused . (checking for unused data...) ok
```
The output above is with 8.20200330, but I saw the same result when I
tried with 7.20190819.
I haven't been able to think of another way to interpret "Even content
of old/deleted files", so I think at least this is a documentation
bug.
[Related DataLad issue](https://github.com/datalad/datalad/issues/4369#issuecomment-608603732)
[0]: https://git-annex.branchable.com/preferred_content/standard_groups/
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Yes. Thank you! :>
[[!meta author=kyle]]
[[!tag projects/datalad]]
> [[done]] --[[Joey]]

21
doc/bugs/Deleted_files_not_considered_with_wanted__61____34__anything__34__/comment_1_03eebb6527c364552b8f81c12616e926._comment
View file

@ -1,21 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2020-04-13T16:03:16Z"
content="""
All of the commands you ran looked only at files in the current tree,
so of course did not operate on deleted files. Eg, "git annex get" and "git
annex get ." are identical, and obviously the current directory does not
contain any deleted files.
But there are commands that do operate on deleted files, including eg `git
annex sync --content --all` and `git annex get --all`. Those commands will
honor preferred content settings including "anything" when operating on
deleted files. (The assistant also periodically does.)
I've added a note to that effect to the preferred content
documentation. Also several command's man pages didn't make entirely
explicit that they operated on files in the current tree
(although the main git-annex man page does document that),
so improved that documentation too.
"""]]

13
doc/bugs/Deleted_files_not_considered_with_wanted__61____34__anything__34__/comment_2_fded8e94841cb23a9d2715b00f63ae98._comment
View file

@ -1,13 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="comment 2"
date="2020-04-13T18:58:50Z"
content="""
> All of the commands you ran looked only at files in the current
> tree, so of course did not operate on deleted files.
Okay. I incorrectly thought `--auto` would bypass that.
Thank you for the doc updates.
"""]]

37
doc/bugs/Exclude_submodules_from_export_to_WEBDAV.mdwn
View file

@ -1,37 +0,0 @@
I am using `git annex export` to export to a WEBDAV special remote configured with `exporttree=yes`. Works beautifully!
However, now I have a repository with a submodule, and I want to export them both, such that they look like a single worktree on the WEBDAV service. I can, in-principle, achieve this by using an appropriately adjusted WEBDAV URL. But git-annex exports an empty file with the name of the submodule in the location where I would need a directory to place the submodule export -- which the WEBDAV services rightfully acknowledges with a 409 (conflict) response, when I attempt running `initremote` on the submodule nevertheless.
I found no way to use an `wanted` expression to automatically exclude all submodules.
The options I am left with are:
1. Remove the empty submodule placeholders after an export.
2. Maintain a dedicated wanted expression with a continuously updated exclusion list covering all submodule mounts.
(2) Might allow me to not have to fiddle with WEBDAV requests myself, but it doesn't seem "right", given that the submodule layout and number can change with any state of the repository.
I wonder, though, what benefit the export of these placeholder files provides? Maybe they should not be exported at all. Or, alternatively, it might make sense to actually export them as directories right away. I might be biased, but that might be a more appropriate representation than an empty file.
I have used a Nextcloud instance for me attempts, and a repository with a single submodule (both created with datalad in my case, but that should not be relevant here). Followed by
```
git annex initremote wd type=webdav url=https://.../files/user/datalad/dummy encryption=none exporttree=yes
git annex initremote wd type=webdav url=https://.../files/user/datalad/dummy/submod encryption=none exporttree=yes
```
in both repositories respectively.
I have been using `git-annex version: 8.20201127`
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
It is the rock that I stand on.
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]

8
doc/bugs/Exclude_submodules_from_export_to_WEBDAV/comment_1_43cde828704e54cad344c066ba1e2821._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2021-03-12T15:46:04Z"
content="""
I think it's simply a bug that it exports submodules at all. Very easy to
forget about the special case of those wacky submodule directories in git.
"""]]

14
doc/bugs/Exclude_submodules_from_export_to_WEBDAV/comment_2_f1d89a826b5a2847328f6eec36b9f452._comment
View file

@ -1,14 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2021-03-12T16:16:44Z"
content="""
Easy enough to exclude them from export, that's done.
But the other side of the coin is, importing back from there will generate
a remote-tracking branch without the submodule, and merging it will result
in a tree that does not include the submodule, as if it were deleted.
So, importing needs to somehow add back submodules,
unless the submodule location has been overwritten with a file on the remote.
"""]]

7
doc/bugs/Exclude_submodules_from_export_to_WEBDAV/comment_3_38d687454c618c3321ad6d9b3c4e7a38._comment
View file

@ -1,7 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2021-03-12T17:28:10Z"
content="""
All right, both export and import are fixed.
"""]]

10
doc/bugs/Exclude_submodules_from_export_to_WEBDAV/comment_4_8f31a065f5f97285adeb3f9614c1f3c9._comment
View file

@ -1,10 +0,0 @@
[[!comment format=mdwn
username="mih"
avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd"
subject="Confirm fix"
date="2021-03-13T16:23:34Z"
content="""
This is working now. `git-annex version: 8.20210311-gecee702b3`
Thx!
"""]]

59
doc/bugs/FTBFS__58___Couldn__39__t_match_type___96__Annex.AnnexState__39___..._.mdwn
View file

@ -1,59 +0,0 @@
### Please describe the problem.
FTBFS for two days (this error matches only on linux, didn't check others why they are red in https://github.com/datalad/git-annex):
```
2022-06-29T03:25:00.4893455Z E: build failed
2022-06-29T03:25:00.4894450Z I: the tail of the log
2022-06-29T03:25:00.4903612Z [310 of 677] Compiling Creds ( Creds.hs, dist/build/git-annex/git-annex-tmp/Creds.o )
2022-06-29T03:25:00.4908788Z [311 of 677] Compiling Remote.Helper.AWS ( Remote/Helper/AWS.hs, dist/build/git-annex/git-annex-tmp/Remote/Helper/AWS.o )
2022-06-29T03:25:00.4910166Z [312 of 677] Compiling P2P.Address ( P2P/Address.hs, dist/build/git-annex/git-annex-tmp/P2P/Address.o )
2022-06-29T03:25:00.4911450Z [313 of 677] Compiling P2P.Auth ( P2P/Auth.hs, dist/build/git-annex/git-annex-tmp/P2P/Auth.o )
2022-06-29T03:25:00.4912850Z [314 of 677] Compiling Annex.Notification ( Annex/Notification.hs, dist/build/git-annex/git-annex-tmp/Annex/Notification.o )
2022-06-29T03:25:00.4913599Z
2022-06-29T03:25:00.4913895Z Annex/Notification.hs:37:42: error:
2022-06-29T03:25:00.4914739Z * Couldn't match type `Annex.AnnexState' with `Annex.AnnexRead'
2022-06-29T03:25:00.4915686Z Expected type: Annex.AnnexState -> DesktopNotify
2022-06-29T03:25:00.4916569Z Actual type: Annex.AnnexRead -> DesktopNotify
2022-06-29T03:25:00.4917418Z * In the first argument of `Annex.getState', namely
2022-06-29T03:25:00.4918159Z `Annex.desktopnotify'
2022-06-29T03:25:00.4918842Z In a stmt of a 'do' block:
2022-06-29T03:25:00.4919609Z wanted <- Annex.getState Annex.desktopnotify
2022-06-29T03:25:00.4920362Z In the expression:
2022-06-29T03:25:00.4921081Z do wanted <- Annex.getState Annex.desktopnotify
2022-06-29T03:25:00.4921878Z if (notifyStart wanted || notifyFinish wanted) then
2022-06-29T03:25:00.4922815Z do client <- liftIO DBus.Client.connectSession
2022-06-29T03:25:00.4923576Z ....
2022-06-29T03:25:00.4924147Z else
2022-06-29T03:25:00.4924729Z a NotifyWitness
2022-06-29T03:25:00.4925310Z |
2022-06-29T03:25:00.4926060Z 37 | wanted <- Annex.getState Annex.desktopnotify
2022-06-29T03:25:00.4926916Z | ^^^^^^^^^^^^^^^^^^^
2022-06-29T03:25:00.4927396Z
2022-06-29T03:25:00.4927702Z Annex/Notification.hs:60:34: error:
2022-06-29T03:25:00.4928555Z * Couldn't match type `Annex.AnnexState' with `Annex.AnnexRead'
2022-06-29T03:25:00.4929502Z Expected type: Annex.AnnexState -> DesktopNotify
2022-06-29T03:25:00.4930367Z Actual type: Annex.AnnexRead -> DesktopNotify
2022-06-29T03:25:00.4931212Z * In the first argument of `Annex.getState', namely
2022-06-29T03:25:00.4931962Z `Annex.desktopnotify'
2022-06-29T03:25:00.4935814Z In a stmt of a 'do' block:
2022-06-29T03:25:00.4936937Z wanted <- Annex.getState Annex.desktopnotify
2022-06-29T03:25:00.4937934Z In the expression:
2022-06-29T03:25:00.4938808Z do wanted <- Annex.getState Annex.desktopnotify
2022-06-29T03:25:00.4939703Z when (notifyFinish wanted)
2022-06-29T03:25:00.4940467Z $ liftIO
2022-06-29T03:25:00.4941336Z $ do client <- DBus.Client.connectSession
2022-06-29T03:25:00.4942492Z void $ Notify.notify client (droppedNote ok (fromRawFilePath f))
2022-06-29T03:25:00.4943446Z |
2022-06-29T03:25:00.4944547Z 60 | wanted <- Annex.getState Annex.desktopnotify
2022-06-29T03:25:00.4945576Z | ^^^^^^^^^^^^^^^^^^^
2022-06-29T03:25:00.4946531Z make[3]: *** [Makefile:58: git-annex] Error 1
```
full log at https://github.com/datalad/git-annex/runs/7124197096?check_suite_focus=true or on smaug on `/mnt/datasets/datalad/ci/git-annex/builds/2022/06/cron-20220629/build-ubuntu.yaml-744-90215d4e-failed/1_build-package (1).txt`
[[!meta author=yoh]]
[[!tag projects/datalad]]
> [[fixed|done]] earlier today --[[Joey]]

119
doc/bugs/Git_LFS_as_a_special_remote_on_GitLab_not_working.mdwn
View file

@ -1,119 +0,0 @@
### Please describe the problem.
I tried using GitLab as a `--type git-lfs` special remote. I followed the steps in https://git-annex.branchable.com/tips/storing_data_in_git-lfs/ for both GitHub and GitLab.
While it worked as described for GitHub I get an error message when trying to push data to GitLab.
### What steps will reproduce the problem?
[[!format sh """
# basic repository setup
mkdir annex
cd annex
git init
git annex init
# create a random binary file
head -c 100 /dev/urandom >random.bin
git annex add .
git commit -a -m added
# set GitHub as a remote (repo exists and is public: https://github.com/iimog/annex)
git remote add github git@github.com:iimog/annex
git push --all github
git annex initremote lfs type=git-lfs encryption=none url=git@github.com:iimog/annex
git annex copy * --to lfs
#copy random.bin (to lfs...)
#ok
#(recording state in git...)
# set GitLab as a remote (repo exists and is public: https://gitlab.com/iimog/annex)
git remote add gitlab git@gitlab.com:iimog/annex
git push --all gitlab
git annex initremote lfs-gitlab type=git-lfs encryption=none url=git@gitlab.com:iimog/annex
#
# Unable to parse git config from gitlab
#
# Remote gitlab does not have git-annex installed; setting annex-ignore
#
# This could be a problem with the git-annex installation on the remote. Please make sure that git-annex-shell is available in PATH when you ssh into the remote. Once you have fixed the git-annex installation, run: git annex enableremote gitlab
#initremote lfs-gitlab ok
#(recording state in git...)
git annex copy * --to lfs-gitlab
#copy random.bin (to lfs-gitlab...)
#<long error message>
#--> The error in short is a "Bad Request" (details see below)
# Try again after explicitly calling enableremote
git annex enableremote gitlab
#enableremote gitlab ok
git annex copy * --to lfs-gitlab
#copy random.bin (to lfs-gitlab...)
#<long error message>
#--> This still results in the same "Bad Request" error
"""]]
### What version of git-annex are you using? On what operating system?
git-annex version: 8.20210903-ga4d179c
Ubuntu 20.04.3 LTS
### Please provide any additional information below.
I initially discovered this issue while using DataLad and reported the issue [here](https://github.com/datalad/datalad/issues/6126).
As suspected by @yarikoptic the issue also exists when using git-annex directly, not through DataLad.
On a self-hosted GitLab instance the following error was found in the server log:
[[!format sh """
2021/10/28 21:07:13 [error] 7412#0: *1869253 client sent invalid chunked body while sending request to upstream, client: <...>, server: <...>, request: "PUT /iimog/my-cool-ds2.git/gitlab-lfs/objects/92f711c1799c5aa936c2be297171d08afc19ba97899b57a1f67f2e70486395aa/80547 HTTP/1.1", upstream: "http://unix/var/opt/gitlab/gitlab-workhorse/sockets/socket:/iimog/my-cool-ds2.git/gitlab-lfs/objects/92f711c1799c5aa936c2be297171d08afc19ba97899b57a1f67f2e70486395aa/80547", host: "<...>"
"""]]
This is the full error message I got when calling `git annex copy * --to lfs-gitlab`:
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ git annex copy * --to lfs-gitlab
copy random.bin (to lfs-gitlab...)
HttpExceptionRequest Request {
host = "gitlab.com"
port = 443
secure = True
requestHeaders = [("Authorization","<REDACTED>"),("Content-Type","application/octet-stream"),("Transfer-Encoding","chunked"),("User-Agent","git-annex/8.20210903-ga4d179c")]
path = "/iimog/annex.git/gitlab-lfs/objects/04b38782b7f5990e090577047fb54b52e9771ba57c1c36e21f4702049adfcfe3/100"
queryString = ""
method = "PUT"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(StatusCodeException (Response {responseStatus = Status {statusCode = 400, statusMessage = "Bad Request"}, responseVersion = HTTP/1.1, responseHeaders = [("Server","cloudflare"),("Date","Tue, 09 Nov 2021 15:03:11 GMT"),("Content-Type","text/html"),("Content-Length","155"),("Connection","close"),("CF-RAY","6ab7ed2afb9fdfcf-FRA")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}) "<html>\r\n<head><title>400 Bad Request</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n")
HttpExceptionRequest Request {
host = "gitlab.com"
port = 443
secure = True
requestHeaders = [("Authorization","<REDACTED>"),("Content-Type","application/octet-stream"),("Transfer-Encoding","chunked"),("User-Agent","git-annex/8.20210903-ga4d179c")]
path = "/iimog/annex.git/gitlab-lfs/objects/04b38782b7f5990e090577047fb54b52e9771ba57c1c36e21f4702049adfcfe3/100"
queryString = ""
method = "PUT"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(StatusCodeException (Response {responseStatus = Status {statusCode = 400, statusMessage = "Bad Request"}, responseVersion = HTTP/1.1, responseHeaders = [("Server","cloudflare"),("Date","Tue, 09 Nov 2021 15:03:12 GMT"),("Content-Type","text/html"),("Content-Length","155"),("Connection","close"),("CF-RAY","6ab7ed2c9a4ad6b5-FRA")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}) "<html>\r\n<head><title>400 Bad Request</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n")
failed
copy: 1 failed
# End of transcript or log.
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Yes, I'm using DataLad for some of my projects and I'm really impressed how it makes use of git-annex to solve many of the tasks that I struggled with pure git before.
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]

56
doc/bugs/Git_LFS_as_a_special_remote_on_GitLab_not_working/comment_1_e821d8666ace3a760a2a335220844b29._comment
View file

@ -1,56 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2021-11-09T20:09:01Z"
content="""
Let's see.. git-lfs endpoint discovery over ssh works.
The request to start a transfer works:
host = "gitlab.com"
path = "/joeyh/test.git/info/lfs/objects/batch"
...
[2021-11-10 12:06:35.409182815] (Remote.GitLFS) Status {statusCode = 200, statusMessage = "OK"}
So it's the actual PUT that fails:
requestHeaders = [("Authorization","<REDACTED>"),("Content-Type","application/octet-stream"),("Transfer-Encoding","chunked"),("User-Agent","git-annex/8.20211029-ga5a7d8433")]
path = "/joeyh/test.git/gitlab-lfs/objects/922d58c647a679e17ee7c30f7de0111b56b90e84129fa3663568b81822a2628a/30"
Seems that the Transfer-Encoding chunked header is the problem.
That header is provided by the git-lfs endpoint as one to include in
the PUT (along with the Authorization header), and git-annex dutifully does
include it. But it seems that does not make the PUT use that
transfer encoding. And then in the server error, we see
"invalid chunked body".
I tried filtering out the Transfer-Encoding header, and that does
fix the problem. But I dunno if that's the best fix. Should git-annex support
Transfer-Encoding chunked?
git-lfs has itself supported Transfer-Encoding chunked since 2015,
see <https://github.com/git-lfs/git-lfs/issues/385>. That says
"client may send data via chunked Transfer-Encoding when the server
explicitly advertises that it's supported". Which is an interesting
wording -- "may", "supported" -- implying it's not required to use it.
The API docs <https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md>
says the http headers are "Optional hash of String HTTP header key/value
pairs to apply to the request". I think it means optional as in the
server may optionally not send any, not necessarily
that applying them to the request is optional, but that's not really clear.
(Surely a header like Authorization is not optional to include.)
If the headers are not optional to include then the API would let the server
specify any http headers at all, and the client has to send a PUT that includes
those headers and that complies with them. So Transfer-Encoding deflate would
need to use that compression method, etc.
But looking at the git-lfs implementation, it only actually handles
Transfer-Encoding chunked and not other values. I think it may also
not include other headers than Authorization in the PUT?
It seems possible there are other headers that might cause problems if they are
blindly copied into the PUT. Content-Encoding is the only other obvious one,
but who knows what may lurk in some odd corner of a HTTP spec.
"""]]

8
doc/bugs/Git_LFS_as_a_special_remote_on_GitLab_not_working/comment_2_3cdbe5b9b7150f9ad9fd5d7af01b247b._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2021-11-10T17:21:01Z"
content="""
Fixed by not passing through those 2 problem headers. And also made it
actually used chunked encoding when the server indicates it's supported.
"""]]

9
doc/bugs/Git_LFS_as_a_special_remote_on_GitLab_not_working/comment_3_1dcf8d6a37bec46a7e7c2d9a65becf28._comment
View file

@ -1,9 +0,0 @@
[[!comment format=mdwn
username="iimog@2c22a44141070c04b943932b697818a686859677"
nickname="iimog"
avatar="http://cdn.libravatar.org/avatar/e6b150bcd25f22fa74abde05e5d6d4eb"
subject="It works"
date="2021-11-12T08:14:32Z"
content="""
Thank you so much for this. I can confirm that using the latest version, I can push to GitLab LFS with git-annex and it also works when used through DataLad. This opens a whole range of options for me, so thanks a million!
"""]]

68
doc/bugs/Guard_against_previously_annexed_dotfiles_being_converted_to_git_files__63__.mdwn
View file

@ -1,68 +0,0 @@
I've been thinking a bit more about `annex.dotfiles` in the context of
[this forum post][0]. It seems to me that annexed dotfiles can jump
to git in a way that's surprising and worth raising as a possible bug.
Say that I have repo with `annex.dotfiles=true` in the .git/config,
and I add some dotfiles to the annex. Then, someone clones that repo
and goes into an adjusted state (either by running `git annex adjust
--unlock` or by being on a crippled file system). In that clone,
calling `annex get` on any of the annexed dotfiles will lead to those
files being added to the index as regular files. (Demo included
below.)
The above issue could be resolved by the user storing
`annex.dotfiles=true` in `git-annex:config.log`, but perhaps it'd be
feasible for git-annex to guard against already annexed dotfiles
migrating to git?
Thanks in advance.
[[!format sh """
git annex version | head -1
cd "$(mktemp -d --tmpdir gx-XXXXXXX)"
git init a
(
cd a
git annex init a
git config annex.dotfiles true
mkdir .reallybig
echo "a" >.reallybig/foo
git annex add .reallybig/foo
git commit -m"add foo"
)
git clone a b
(
cd b
git annex init b
git annex adjust --unlock
git annex get .reallybig
git status
git diff --cached
)
"""]]
```
git-annex version: 8.20200226
[...]
On branch adjusted/master(unlocked)
Changes to be committed:
modified: .reallybig/foo
diff --git a/.reallybig/foo b/.reallybig/foo
index 3de500c..7898192 100644
--- a/.reallybig/foo
+++ b/.reallybig/foo
@@ -1 +1 @@
-/annex/objects/SHA256E-s2--87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7
+a
```
[0]: https://git-annex.branchable.com/forum/Get_annex.dotfiles__61__true_behavior_without_persistent_configuration__63__/
[[!meta author=kyle]]
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]

18
doc/bugs/Guard_against_previously_annexed_dotfiles_being_converted_to_git_files__63__/comment_1_e9f5fdf1078c87718fb6ca2559323b1e._comment
View file

@ -1,18 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2020-03-09T17:00:38Z"
content="""
This looks like a serious regression in 8.x. It is not an intentional part
of the annex.dotfiles change. I have confirmed the last 7.x release did not
have the behavior.
(Notice that annex.dotfiles is not used in the repo where the problem
occurs. The regression was probably introduced by the annex.dotfiles
changes, but the dotfile could just as well have been added by git-annex v5
with git-annex add --include-dotfiles and the result committed to git would
be the same.)
A simpler test case is to unlock the dotfile in the first repo, then
all that needs to be run in its clone is `git annex get`.
"""]]

17
doc/bugs/Guard_against_previously_annexed_dotfiles_being_converted_to_git_files__63__/comment_2_5d0e1468f7ab2e8cd42452812d9ce49b._comment
View file

@ -1,17 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2020-03-09T18:07:12Z"
content="""
Analysis: Command.Smudge.shouldAnnex was changed
in [[!commit 3cd375723658398a4b15ff8ec19e562ec4622488]] to check, for dotfiles,
if annex.dotfiles is set. If not, it always returns False, so the file
always gets added to git when it's smudged.
After getting a file, git-annex tells git to update its index, which
smudges the file. Of course, there are also plenty of other things that can
cause a file to be smudged.
Bad. Fix is very easy, luckily. It just needs to, rather than returning
False, checking if the file was annexed, same as is done for non-dotfiles.
"""]]

8
doc/bugs/Guard_against_previously_annexed_dotfiles_being_converted_to_git_files__63__/comment_3_a9837edeacdf55c74d8fed2b8d7bf1f9._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="thanks"
date="2020-03-09T18:29:26Z"
content="""
Thank you for the analysis and the fix!
"""]]

26
doc/bugs/JSON_results_for___96__export__96___have_file__58__null_property.mdwn
View file

@ -1,26 +0,0 @@
Neither successful nor failing exports via a WEBDAV special remote have appropriate `file` properties in the produced JSON records.
```
% git annex export master --to 4shared --json --json-error-messages
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
{"command":"export 4shared","success":false,"input":[],"error-messages":[" DAV failure: Status {statusCode = 500, statusMessage = \"Internal Server Error\"} \"<html><body><h1>Server Error</h1></body></html>\" HTTP request: \"MOVE\" \"/dummy/git-annex-webdav-tmp-MD5E-s143466--368857ff4d0e4b14c745d94db7f7d3ff.pdf\""," DAV failure: Status {statusCode = 500, statusMessage = \"Internal Server Error\"} \"<html><body><h1>Server Error</h1></body></html>\" HTTP request: \"MOVE\" \"/dummy/git-annex-webdav-tmp-MD5E-s143466--368857ff4d0e4b14c745d94db7f7d3ff.pdf\""],"file":null}
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
{"command":"export 4shared","success":true,"input":[],"error-messages":[],"file":null}
git-annex: export: 1 failed
```
```
git-annex version: 8.20210223
```
This behavior was observed using the 4shared service, but I believe git-annex knows which file a success or failure belongs to and could report it regardless of service oddities.
Thx!
[[!tag projects/datalad]]
> [[fixed|done]] --[[Joey]]

12
doc/bugs/JSON_results_for___96__export__96___have_file__58__null_property/comment_1_bf743f047b8063b8ecbd8c87d12fa475._comment
View file

@ -1,12 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2021-03-12T18:09:48Z"
content="""
I've fixed this.
However, there is still a case where the file field is null, which is when
an already exported file is renamed. Since nothing is transferred in that
case, and 2 files are involved which doesn't fit the one file in the json,
I left it as-is.
"""]]

8
doc/bugs/JSON_results_for___96__export__96___have_file__58__null_property/comment_2_d6e28d84238cb79c1df927da491f50db._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="mih"
avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd"
subject="Thx!"
date="2021-03-13T16:05:51Z"
content="""
Confirmed working with git-annex version 8.20210311-gecee702b3
"""]]

42
doc/bugs/OSX__58___addurl_--batch_--json_spits_out_shortened_output_string__dies_off_with_4.mdwn
View file

@ -1,42 +0,0 @@
### Please describe the problem.
See transcript. Didn't bisect fully but 6.20160128-g599a3ba seems to work ok, 6.20160211-g479a094 already dies off
[[!format sh """
datalads-imac:~ yoh$ git annex version
git-annex version: 6.20160221-g3d1895e
...
datalads-imac:~ yoh$ mkdir -p /tmp/123; cd /tmp/123; git init; git annex init;
Initialized empty Git repository in /private/tmp/123/.git/
init ok
(recording state in git...)
datalads-imac:123 yoh$ rm -f /tmp/pipe; mkfifo /tmp/pipe; cat /tmp/pipe | git -c receive.autogc=false annex addurl -c annex.largefiles=exclude=*.txt --with-files --json --batch
{"command":"addurl","file":"1.png","note":"downloading http://www.onerussian.com/tmp/banner.png ..."error: git-annex died of signal 4
# now with --debug
datalads-imac:123 yoh$ rm -f /tmp/pipe; mkfifo /tmp/pipe; cat /tmp/pipe | git -c receive.autogc=false annex addurl -c annex.largefiles=exclude=*.txt --with-files --json --batch --debug
[2016-02-23 08:32:49.160686] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.largefiles=exclude=*.txt","show-ref","git-annex"]
[2016-02-23 08:32:49.167372] process done ExitSuccess
[2016-02-23 08:32:49.167479] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.largefiles=exclude=*.txt","show-ref","--hash","refs/heads/git-annex"]
[2016-02-23 08:32:49.172795] process done ExitSuccess
[2016-02-23 08:32:49.173039] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.largefiles=exclude=*.txt","log","refs/heads/git-annex..b2892e67682c5240dadb5d439da810edbdb882df","-n1","--pretty=%H"]
[2016-02-23 08:32:49.178586] process done ExitSuccess
[2016-02-23 08:32:49.179085] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.largefiles=exclude=*.txt","cat-file","--batch"]
[2016-02-23 08:32:49.183985] read: quvi ["--version"]
{"command":"addurl","file":"1.png","note":"downloading http://www.onerussian.com/tmp/banner.png ..."[2016-02-23 08:32:49.255949] call: curl ["-s","-f","-L","-C","-","-#","-o","/private/tmp/123/.git/annex/tmp/URL-s25319--http&c%%www.onerussian.com%tmp%banner.png","http://www.onerussian.com/tmp/banner.png","--user-agent","git-annex/6.20160221-g3d1895e"]
[2016-02-23 08:32:49.302577] process done ExitSuccess
[2016-02-23 08:32:49.303332] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.largefiles=exclude=*.txt","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
[2016-02-23 08:32:49.305653] read: git ["--version"]
[2016-02-23 08:32:49.310314] process done ExitSuccess
error: git-annex died of signal 4
"""]]
[[!meta author=yoh]]
[[!tag projects/datalad]]
> [[done]]; this was indeed due to libmagic, and I've fixed it using brew
> --build-bottle to make a portable one. Then had to disable the build
> flag. --[[Joey]]

38
doc/bugs/Patch_for_compilation_errors_on_Windows.mdwn
View file

@ -1,38 +0,0 @@
Compilation of the latest git-annex code (commit 04dca96) on Windows is currently failing due to some modules not being imported. The issues are fixed by the following patch:
```
diff --git a/Annex/Content/Presence.hs b/Annex/Content/Presence.hs
index 861288c0b..f5d4b593f 100644
--- a/Annex/Content/Presence.hs
+++ b/Annex/Content/Presence.hs
@@ -25,6 +25,9 @@ module Annex.Content.Presence (
) where
import Annex.Common
+#ifdef mingw32_HOST_OS
+import Annex.Perms
+#endif
import qualified Annex
import Annex.LockPool
import Annex.WorkerPool
diff --git a/Git/Hook.hs b/Git/Hook.hs
index 45662f236..788552305 100644
--- a/Git/Hook.hs
+++ b/Git/Hook.hs
@@ -13,9 +13,7 @@ import Common
import Git
import Utility.Tmp
import Utility.Shell
-#ifndef mingw32_HOST_OS
import Utility.FileMode
-#endif
data Hook = Hook
{ hookName :: FilePath
```
[[!meta author=jwodder]]
[[!tag projects/datalad]]
> build is succeeding now [[fixed|done]] --[[Joey]]

8
doc/bugs/Patch_for_compilation_errors_on_Windows/comment_1_eb8d464da14af1fead372b4b61b20f17._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="jwodder"
avatar="http://cdn.libravatar.org/avatar/b06e01332c949b895c681cc92934f36a"
subject="comment 1"
date="2020-11-23T15:02:28Z"
content="""
I see you patched `Git/Hook.hs`, but you did not patch `Annex/Content/Presence.hs`. The Windows builds are still failing.
"""]]

6
doc/bugs/Please_include_version_of_used_to_build_ghc_within___34__git_annex_version__34__.mdwn
View file

@ -1,6 +0,0 @@
Since git-annex version is not always sufficient to pin point the source of regression (e.g. see [the get -J issue](http://git-annex.branchable.com/bugs/multiple_ssh_prompts__44___and_thread_blocked_indefinitely_in_an___63____63____63___transaction/)), it would be really nice if the version of ghc (and thus all its libraries if I get it right) was included in the output of `git annex version`
[[!meta author=yoh]]
[[!tag projects/repronim]]
> Closing as I was blind -- there is `dependency versions` reported [[done]] --[[yarikoptic]]

58
doc/bugs/R__47__O_permissions_are_stalking_me_somehow.mdwn
View file

@ -1,58 +0,0 @@
### Please describe the problem.
I have got data which I was trying to add to an existing annex. Unfortunately I have missed first that some files were not having write permissions so initial 'annex add' failed with similar to below messages, so I have pruned misctmp, adjusted permissions and have tried to add again -- but the same error keeps coming back
[[!format sh """
% chmod +w -R S1/DDA-Freeviewing/surf2anat
% ls -l S1/DDA-Freeviewing/surf2anat/S1_run1_lh*
-rw-r----- 2 yoh retino 241147956 Feb 28 21:46 S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz
% rm -rf .git/annex/misctmp/*
zsh: sure you want to delete all the files in /mnt/datasets/mikemp/data-annex/.git/annex/misctmp [yn]? y
% git annex add S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz
add S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz
git-annex: .git/annex/misctmp/S1_run1_lh1804289383846930886: rename: permission denied (Permission denied)
failed
git-annex: add: 1 failed
% ls -l S1/DDA-Freeviewing/surf2anat/S1_run1_lh*
-r--r----- 2 yoh retino 241147956 Feb 28 21:46 S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz
% git annex version
git-annex version: 6.20160307+gitgb095561-1~ndall+1
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
local repository version: 5
supported repository versions: 5 6
upgrade supported from repository versions: 0 1 2 4 5
% rm -rf .git/annex/misctmp/*
zsh: sure you want to delete all the files in /mnt/datasets/mikemp/data-annex/.git/annex/misctmp [yn]? y
% chmod +w -R S1/DDA-Freeviewing/surf2anat
% git annex add --debug S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz
[2016-03-10 10:15:22.578786] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--others","--exclude-standard","-z","--","S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz"]
[2016-03-10 10:15:22.583854] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
[2016-03-10 10:15:22.584443] read: git ["--version"]
[2016-03-10 10:15:22.588703] process done ExitSuccess
add S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz [2016-03-10 10:15:23.186056] read: sha256sum [".git/annex/misctmp/S1_run1_lh1804289383846930886"]
[2016-03-10 10:15:24.611133] process done ExitSuccess
git-annex: .git/annex/misctmp/S1_run1_lh1804289383846930886: rename: permission denied (Permission denied)
failed
[2016-03-10 10:15:24.648808] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff","--name-only","--diff-filter=T","-z","--","S1/DDA-Freeviewing/surf2anat/S1_run1_lh_process_surf.nii.gz"]
git-annex: add: 1 failed
% ls -l .git/annex/misctmp/S1_run1_lh1804289383846930886
-r--r----- 2 yoh retino 241147956 Feb 28 21:46 .git/annex/misctmp/S1_run1_lh1804289383846930886
"""]]
[[!meta author=yoh]]
[[!tag projects/datalad]]
> [[unreproducible|done]] --[[Joey]]

12
doc/bugs/R__47__O_permissions_are_stalking_me_somehow/comment_1_9e4ba1e03a89c219c5e62280757311ba._comment
View file

@ -1,12 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2016-03-14T17:53:32Z"
content="""
A file's permissions cannot prevent it being renamed; so I diagnose some
kind of directory permission problem.
(Also AFAICS git-annex add deals with files whose permissions don't allow
being read by turning on the read bit. And it should never need the write
bit. I can `chmod 000 foo; git annex add foo` and it succeeds..)
"""]]

167
doc/bugs/Recent_hang_with_rsync_remote_with_older_systems___40__Xenial__44___Jessie__41__.mdwn
View file

@ -1,167 +0,0 @@
I'm looking into another SSH-based hang in the DataLad tests. I can't
trigger the hang on my local system (Debian Buster), but in a Ubuntu
Xenial VM I can. The hang bisects to 1f2e2d15e (async exception
safety, 2020-06-03). It disappears on the parent of 1f2e2d15e or if I
revert that commit on top of master (currently 3b6754e2a).
I was able to reduce the hang down to a `git annex get` from an rsync
remote. Here is a script that triggers the hang via a Xenial Docker
container. Sorry for the length; given the system interaction, it's
the simplest reproducer I've managed to come up with.
[[!format sh """
cd "$(mktemp -d ${TMPDIR:-/tmp}/ga-XXXXXXX)"
cat >demo.sh <<'EOF'
git annex version
cd "$(mktemp -d /tmp/ga-XXXXXXX)"
remdir="$PWD/store"
mkdir "$remdir"
git init repo
(
cd repo
git annex init
git annex initremote r type=rsync rsyncurl="localhost:$remdir" encryption=none
echo 0 >f0
git annex add f0
git commit -m'f0'
git annex copy --to=r f0
git annex drop f0
)
git clone repo clone
(
cd clone
git annex init clone
git annex enableremote r
git annex get --debug f0
)
EOF
cat >Dockerfile <<'EOF'
FROM ubuntu:xenial
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install -y --no-install-recommends \
curl rsync openssh-client openssh-server ca-certificates
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN cd /root && curl -fSsL \
https://downloads.kitenet.net/git-annex/autobuild/amd64/git-annex-standalone-amd64.tar.gz \
| tar xz
ENV PATH="/root/git-annex.linux:$PATH"
RUN git config --system user.name "u"
RUN git config --system user.email "u@e"
RUN mkdir -p /root/.ssh
RUN mkdir -p /var/run/sshd
RUN ssh-keygen -f /root/.ssh/id_rsa -N ""
RUN cat /root/.ssh/id_rsa.pub >>/root/.ssh/authorized_keys
RUN echo "Host localhost\nStrictHostKeyChecking no\n" >>/root/.ssh/config
COPY demo.sh /root/demo.sh
CMD /usr/sbin/sshd && sh /root/demo.sh
EOF
docker build -t ga-rsync-hang:latest .
docker run -it --rm ga-rsync-hang:latest
"""]]
Output, where the last line stalled:
```
[... 50 lines ...]
git-annex version: 8.20200618-g3b6754e2a
build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
[... 32 lines ...]
[2020-07-07 14:28:55.252423613] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
[2020-07-07 14:28:55.256082009] process done ExitSuccess
[2020-07-07 14:28:55.256316845] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
[2020-07-07 14:28:55.26056207] process done ExitSuccess
[2020-07-07 14:28:55.260917245] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","-z","--cached","--","f0"]
get f0 [2020-07-07 14:28:55.26481346] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2020-07-07 14:28:55.268786541] process done ExitSuccess
[2020-07-07 14:28:55.268997933] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2020-07-07 14:28:55.272176401] process done ExitSuccess
[2020-07-07 14:28:55.272550418] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..ea06186dfc3dab39316da534144d5d6ef3d6090c","--pretty=%H","-n1"]
[2020-07-07 14:28:55.275989985] process done ExitSuccess
[2020-07-07 14:28:55.27634403] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2020-07-07 14:28:55.276782737] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2020-07-07 14:28:55.279315206] read: git ["config","--null","--list"]
[2020-07-07 14:28:55.281990769] process done ExitSuccess
(from r...)
[2020-07-07 14:28:55.288194172] read: rsync ["-e","'ssh' '-S' '.git/annex/ssh/localhost' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=yes' '-T'","--progress","--inplace","localhost:/tmp/ga-075MZKs/store/0c5/66e/'SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa/SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa'",".git/annex/tmp/SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa"]
SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa
0 0% 0.00kB/s 0:00:00
2 100% 1.95kB/s 0:00:00 (xfr#1, to-chk=0/1)
^C
```
Replacing "FROM ubuntu:xenial" with "FROM ubuntu:bionic" (a later
release) resolves the hang, so perhaps there is some interaction with
an older rsync or openssh version. Here are the versions that are
present in Xenial:
```
OpenSSH_7.2p2 Ubuntu-4ubuntu2.10, OpenSSL 1.0.2g 1 Mar 2016
rsync version 3.1.1 protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, prealloc
```
Thinking it's an older version of openssh or rsync, I tried with an older version of Debian. Using "FROM debian:stretch-slim" doesn't hang. Here are the versions there:
```
OpenSSH_7.4p1 Debian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019
rsync version 3.1.2 protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, prealloc
```
However, going back farther, "FROM debian:jessie-slim" does hang. The
versions there:
```
OpenSSH_6.7p1 Debian-5+deb8u8, OpenSSL 1.0.1t 3 May 2016
rsync version 3.1.1 protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, prealloc
```
So perhaps there's some interaction with openssh before 7.4p1 or rsync
before 3.1.2. Those are pretty old versions, and, in the case of
Jessie, the release is now EOL. But I figured it was at least worth
writing up given that the hang isn't triggered until a recent commit
in git-annex.
[[!meta author=kyle]]
[[!tag projects/datalad]]
> [[fixed|done]] made it cancel the threads 2 seconds after the process
> exited if still running. So there will be a bit of a slow down when using
> that broken ssh version, but it will work. --[[Joey]]

34
doc/bugs/Recent_hang_with_rsync_remote_with_older_systems___40__Xenial__44___Jessie__41__/comment_10_d9dc3550012a618ba674119dd1163642._comment
View file

@ -1,34 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="comment 10"
date="2020-08-10T18:56:02Z"
content="""
Hmm. Well, I doubt it provides any useful information, but just in case, here's the output produced by the `git annex get --debug f0` call:
```
[2020-08-10 14:49:51.227779797] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"]
[2020-08-10 14:49:51.229945707] process done ExitSuccess
[2020-08-10 14:49:51.230127447] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"]
[2020-08-10 14:49:51.232132388] process done ExitSuccess
[2020-08-10 14:49:51.232349474] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"-z\",\"--cached\",\"--\",\"f0\"]
get f0 [2020-08-10 14:49:51.234243652] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
[2020-08-10 14:49:51.236402808] process done ExitSuccess
[2020-08-10 14:49:51.236609182] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2020-08-10 14:49:51.238119863] process done ExitSuccess
[2020-08-10 14:49:51.238409384] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..d8b33719c8ce515e7b5b99f8d0c51ab7d2de46e1\",\"--pretty=%H\",\"-n1\"]
[2020-08-10 14:49:51.240088806] process done ExitSuccess
[2020-08-10 14:49:51.240391883] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
[2020-08-10 14:49:51.240716412] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
[2020-08-10 14:49:51.247306917] read: git [\"config\",\"--null\",\"--list\"]
[2020-08-10 14:49:51.24859076] process done ExitSuccess
(from r...)
[2020-08-10 14:49:51.250558418] read: rsync [\"-e\",\"'ssh' '-S' '.git/annex/ssh/localhost' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=yes' '-T'\",\"--progress\",\"--inplace\",\"localhost:/tmp/ga-urvfy8Q/store/0c5/66e/'SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa/SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa'\",\".git/annex/tmp/SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa\"]
SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa
0 0% 0.00kB/s 0:00:00
2 100% 1.95kB/s 0:00:00 (xfr#1, to-chk=0/1)
[2020-08-10 14:49:51.557365853] process done ExitSuccess
git-annex: AsyncCancelled
```
"""]]

9
doc/bugs/Recent_hang_with_rsync_remote_with_older_systems___40__Xenial__44___Jessie__41__/comment_11_5dc1d7c54fc4768849d39affac51fcae._comment
View file

@ -1,9 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="comment 11"
date="2020-08-10T19:01:35Z"
content="""
And I should have checked the result the command rather than just
noting that it didn't hang. f0 is not transferred.
"""]]

8
doc/bugs/Recent_hang_with_rsync_remote_with_older_systems___40__Xenial__44___Jessie__41__/comment_12_7cb4c4e1bc100e8f3241ece74ce463e3._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 12"""
date="2020-08-10T19:36:30Z"
content="""
Wait, are you saying the command exited nonzero, or that the transfer
didn't happen, or both?
"""]]

24
doc/bugs/Recent_hang_with_rsync_remote_with_older_systems___40__Xenial__44___Jessie__41__/comment_13_a1bce716987b0d8d00a4a9aa07e9ac65._comment
View file

@ -1,24 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 13"""
date="2020-08-10T19:54:26Z"
content="""
Well, I ran the reproducer script, substituting a git-annex tarball with
the fix (and adding an output of the exit code and dropping the --debug),
and the end was:
get f0 (from r...)
SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa2
100% 1.95kB/s 0:00:00 (xfr#1, to-chk=0/1)
(checksum...) ok
(recording state in git...)
0
So, I guess this patch is ok, at least with the git-annex I've built.
(Using haskell from debian unstable.) It seems possible that some older
ghc, async, etc might have handled that AsyncCancelled differently and
resulted in the behavior you're seeing.
Also tried with https://downloads.kitenet.net/git-annex/autobuild/i386-ancient/git-annex-standalone-i386-ancient.tar.gz
and it's ok too.
"""]]

8
doc/bugs/Recent_hang_with_rsync_remote_with_older_systems___40__Xenial__44___Jessie__41__/comment_13_dd51471ec6a8f380663c62d6f9edb176._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="comment 13"
date="2020-08-10T19:47:59Z"
content="""
The file isn't transferred, and the exit is non-zero (1).
"""]]

16
doc/bugs/Recent_hang_with_rsync_remote_with_older_systems___40__Xenial__44___Jessie__41__/comment_15_93e2dd2b30cdeddd5e273eae78a422e0._comment
View file

@ -1,16 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 15"""
date="2020-08-10T20:06:28Z"
content="""
Hmm, come to think of it, `wait errt` is racing
`cancel errt`, so what if cancel kills the errt thread,
and before `race` noticed cancel is done, the wait thread
sees errt, died of an an AsyncCancelled exception and rethrows that?
Result could be the exception leaking out, perhaps.
It might be that async now avoids that, and some other version didn't, or
you could be getting unlucky with the race.
So, I've added something to catch that exception in case it leaks.
"""]]

78
doc/bugs/Recent_hang_with_rsync_remote_with_older_systems___40__Xenial__44___Jessie__41__/comment_16_b2d1550e20e8c4cc07dbd0ebba64c603._comment
View file

@ -1,78 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="comment 16"
date="2020-08-10T21:34:43Z"
content="""
I applied the change from c59a51a0651b and 4466c1001d88. Now I don't
see \"git-annex: AsyncCancelled\", f0 is transferred, and the exit
status is 0, so it looks to be working as intended. Thank you.
As described in comment 8, this is all in the Xenial VM on top of
1df9e72a7829, because the hang goes away with the next commit
(4c9ad1de46dd). For posterity, here is a diff of the changes on top
1df9e72a7829, applied from aa492bc65904, c59a51a0651b, and
4466c1001d88.
<details>
<summary>diff</summary>
[[!format diff \"\"\"
diff --git a/Utility/Metered.hs b/Utility/Metered.hs
index 1c35a9a05..c4063c90b 100644
--- a/Utility/Metered.hs
+++ b/Utility/Metered.hs
@@ -1,6 +1,6 @@
{- Metered IO and actions
-
- - Copyright 2012-2018 Joey Hess <id@joeyh.name>
+ - Copyright 2012-2020 Joey Hess <id@joeyh.name>
-
- License: BSD-2-clause
-}
@@ -46,6 +46,7 @@ import Common
import Utility.Percentage
import Utility.DataUnits
import Utility.HumanTime
+import Utility.ThreadScheduler
import qualified Data.ByteString.Lazy as L
import qualified Data.ByteString as S
@@ -314,10 +315,26 @@ outputFilter cmd params environ outfilter errfilter =
catchMaybeIO $ withCreateProcess p go
where
go _ (Just outh) (Just errh) pid = do
- void $ concurrently
- (tryIO (outfilter outh) >> hClose outh)
- (tryIO (errfilter errh) >> hClose errh)
- waitForProcess pid
+ outt <- async $ tryIO (outfilter outh) >> hClose outh
+ errt <- async $ tryIO (errfilter errh) >> hClose errh
+ ret <- waitForProcess pid
+
+ -- Normally, now that the process has exited, the threads
+ -- will finish processing its output and terminate.
+ -- But, just in case the process did something evil like
+ -- forking to the background while inheriting stderr,
+ -- it's possible that the threads will not finish, which
+ -- would result in a deadlock. So, wait a few seconds
+ -- maximum for them to finish and then cancel them.
+ -- (One program that has behaved this way in the past is
+ -- openssh.)
+ void $ tryNonAsync $ race_
+ (wait outt >> wait errt)
+ (threadDelaySeconds (Seconds 2))
+ cancel outt
+ cancel errt
+
+ return ret
go _ _ _ _ = error \"internal\"
p = (proc cmd (toCommand params))
\"\"\"]]
</details>
"""]]

58
doc/bugs/Recent_hang_with_rsync_remote_with_older_systems___40__Xenial__44___Jessie__41__/comment_1_5274e7f715ef060383046e8f5da08164._comment
View file

@ -1,58 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2020-07-13T15:53:56Z"
content="""
Hmm, my guess is that the significant and unmentioned change in that commit
is, it waits for one of the stdout or stderr to be fully processed before
waiting on the process to terminate. So if it somehow fails to detect EOF
(maybe because old rsync and ssh somehow end up with the handle not being
closed when rsync exits eg if a background ssh process inherited them?) it
would block forever.
If so, this patch would avoid the hang (and if this doesn't fix it, the
problem must be somehow due to using with CreateProcess).
diff --git a/Utility/Metered.hs b/Utility/Metered.hs
index 1c35a9a05..8e363681d 100644
--- a/Utility/Metered.hs
+++ b/Utility/Metered.hs
@@ -314,9 +314,8 @@ outputFilter cmd params environ outfilter errfilter =
catchMaybeIO $ withCreateProcess p go
where
go _ (Just outh) (Just errh) pid = do
- void $ concurrently
- (tryIO (outfilter outh) >> hClose outh)
- (tryIO (errfilter errh) >> hClose errh)
+ void $ async $ tryIO (outfilter outh) >> hClose outh
+ void $ async $ tryIO (errfilter errh) >> hClose errh
waitForProcess pid
go _ _ _ _ = error "internal"
If that patch does avoid the hang, this one would be worth a try
as it tries to accomplish the goal of the patch in a better way:
diff --git a/Utility/Metered.hs b/Utility/Metered.hs
index 1c35a9a05..bd265ae09 100644
--- a/Utility/Metered.hs
+++ b/Utility/Metered.hs
@@ -313,11 +313,10 @@ outputFilter
outputFilter cmd params environ outfilter errfilter =
catchMaybeIO $ withCreateProcess p go
where
- go _ (Just outh) (Just errh) pid = do
- void $ concurrently
- (tryIO (outfilter outh) >> hClose outh)
- (tryIO (errfilter errh) >> hClose errh)
- waitForProcess pid
+ go _ (Just outh) (Just errh) pid =
+ withAsync (tryIO (outfilter outh) >> hClose outh) $ const $
+ withAsync (tryIO (errfilter errh) >> hClose errh) $ const $
+ waitForProcess pid
go _ _ _ _ = error "internal"
p = (proc cmd (toCommand params))
I can try these if necessary but don't have docker handy,
so maybe you're in a better position to?
"""]]

8
doc/bugs/Recent_hang_with_rsync_remote_with_older_systems___40__Xenial__44___Jessie__41__/comment_2_8976d014def1910f5140b1e5cdd37fb0._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="kyle"
avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
subject="comment 2"
date="2020-07-13T16:46:54Z"
content="""
Thanks for taking a look. I'll test out those changes and report back.
"""]]

Some files were not shown because too many files have changed in this diff Show more