mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2020-03-02 11:55:50 -04:00
commit a1934434d5
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
4 changed files with 114 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

64
doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II.mdwn Normal file
View file

@ -0,0 +1,64 @@
### Please describe the problem.
When running git-annex tests on a specific machine I get the following error:
```
$ ./dist-newstyle/build/x86_64-linux/ghc-8.6.5/git-annex-8.20200226/build/git-annex/git-annex test -v -p Tests.QuickCheck.prop_hashes_stable
Tests
QuickCheck
prop_hashes_stable: Illegal instruction (core dumped)
```
(I get the same error when just running `./dist-newstyle/build/x86_64-linux/ghc-8.6.5/git-annex-8.20200226/build/git-annex/git-annex test`, but after successful passes of previous tests.)
### What steps will reproduce the problem?
Run the following on a machine with the CPU "AMD Phenom(tm) II X3 720 Processor":
```
git clone git://git-annex.branchable.com/ git-annex
cd git-annex
cabal install -j --only-dependencies
cabal configure
cabal build -j
./dist-newstyle/build/x86_64-linux/ghc-8.6.5/git-annex-8.20200226/build/git-annex/git-annex test -v -p Tests.QuickCheck.prop_hashes_stable
```
### What version of git-annex are you using? On what operating system?
I'm trying to build git-annex from the current "master". However, the same problem exists in 7.20190819, 7.20191230 and 7.20200204.
I'm building on NixOS, release 19.09, using GHC 8.6.5.
### Please provide any additional information below.
I believe the problem was initially encountered on NixOS' build farm, https://hydra.nixos.org/ , when trying to build the git-annex package version 7.20190819 for NixOS/nixpkgs. (As a result, the built package was not available when I tried to update my installation of NixOS, which led me to investigating this problem.)
So I tried to build the same package (and other versions, see above) on my own machine running NixOS (which has AMD Phenom II CPU). It consistently failed during testing on numerous attempts with different configurations. However, another machine (with AMD FX-8300 CPU) was able to build the same package, and never had the same test failures.
Then I built git-annex on both machines from source (not as a Nix package), using cabal, and tried to run tests, as shown in the beginning. I got the same results: Phenom II failed the test Tests.QuickCheck.prop_hashes_stable with "Illegal instruction", while FX-8300 passed it.
I do **not** think that my Phenom II machine has hardware problems: I ran MemTest recently, and this machine is able to build other packages (including GCC and GHC) just fine.
Moreover, the same problem occurs on hydra.hixos.org , and it looks like it's sporadic (repeating the build can fix it). I guess that this depends on which machine is assigned to build the package: some machines are able to build it, while others are not.
So, my hypothesis is that git-annex's tests (and maybe non-test code pieces as well) use some CPU instructions that are not supported on Phenom II (and other) CPUs, but supported on later CPUs such as FX-8300. Probably this is even a problem of GHC rather than of git-annex, but I'm not qualified to tell (I know nothing relevant about Haskell compilation or code generation).
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
[nix-shell:~/build/git-annex]$ ./dist-newstyle/build/x86_64-linux/ghc-8.6.5/git-annex-8.20200226/build/git-annex/git-annex test -v -p Tests.QuickCheck.prop_hashes_stable
Illegal instruction (core dumped)
[nix-shell:~/build/git-annex]$ dmesg | tail -n1
[151755.340563] traps: git-annex:w[20112] trap invalid opcode ip:2d6c567 sp:7fefa0a1fd98 error:0 in git-annex[407000+313b000]
# End of transcript or log.
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I'm using git-annex as a core of the simple in-house system that handles storage, transfer and tracking of multi-GB data sets, taking advantage of git for version tracking, git-annex for data storage and transfer, and a single SSH access point to both of them.
I must admit that it was a pain to adapt git-annex to my needs, and probably I did it in a manner that wasn't an intended/supported way of using git-annex. But it does the job.

32
doc/bugs/enableremote_stuck_with_a_recentish_git-annex/comment_6_362351708d543f1d219dfcf8d71b5a14._comment Normal file
View file

@ -0,0 +1,32 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 6"
date="2020-02-29T06:10:19Z"
content="""
FWIW -- two excerpts:
[good session](http://www.onerussian.com/tmp/travis-656403954.log), [bad session](http://www.onerussian.com/tmp/travis-655516047.log)
command to obtain
[[!format sh \"\"\"
> for j in 655516047 656403954; do wget -q -O- https://api.travis-ci.org/v3/job/$j/log.txt | sed -n -e'/Launching.*nableremote.*target2/,100000p' | sed -ne '1,/^I: done/p' | sed -e 's/2020-..-.. ..:..:..\........../TIMESTAMP/g' >| /tmp/travis-$j.log; done
\"\"\"]]
still too much details and too many differences unfortunately. But at least shows that the same invocation of ssh proceeds normally in the good case (first one is the only one in bad, in good ones, more come):
[[!format sh \"\"\"
$> grep 'read: ssh .\"datalad-test\",' travis-6*.log
travis-655516047.log:[Level 9] stderr| [TIMESTAMP] read: ssh [\"datalad-test\",\"-S\",\".git/annex/ssh/datalad-test\",\"-o\",\"ControlMaster=auto\",\"-o\",\"ControlPersist=yes\",\"-n\",\"-T\",\"git-annex-shell 'configlist' '/tmp/datalad_temp_test_publish_dependsymsjz245' '--debug'\"]
travis-656403954.log:[Level 9] stderr| [TIMESTAMP read: ssh [\"datalad-test\",\"-S\",\".git/annex/ssh/datalad-test\",\"-o\",\"ControlMaster=auto\",\"-o\",\"ControlPersist=yes\",\"-n\",\"-T\",\"git-annex-shell 'configlist' '/tmp/datalad_temp_test_publish_depends60annfb5' '--debug'\"]
travis-656403954.log:[Level 9] stderr| [TIMESTAMP] read: ssh [\"datalad-test\",\"-S\",\".git/annex/ssh/datalad-test\",\"-o\",\"ControlMaster=auto\",\"-o\",\"ControlPersist=yes\",\"-o\",\"ControlMaster=auto\",\"-S\",\"/home/travis/.cache/datalad/sockets/04825add\",\"-n\",\"-T\",\"git-annex-shell 'configlist' '/tmp/datalad_temp_test_publish_dependsgxlws0yt' '--debug'\"]
travis-656403954.log:[Level 11] stderr| [TIMESTAMP] read: ssh [\"datalad-test\",\"-S\",\".git/annex/ssh/datalad-test\",\"-o\",\"ControlMaster=auto\",\"-o\",\"ControlPersist=yes\",\"-o\",\"ControlMaster=auto\",\"-S\",\"/home/travis/.cache/datalad/sockets/04825add\",\"-n\",\"-T\",\"git-annex-shell 'commit' '/tmp/datalad_temp_test_publish_dependsgxlws0yt' '--debug' --uuid 7d69e4d6-f4d0-43c0-bf5a-0e6e8c363613\"]
travis-656403954.log:[Level 11] stderr| [TIMESTAMP] read: ssh [\"datalad-test\",\"
\"\"\"]]
and there is close to no chance for `\"git-annex-shell 'configlist' '/tmp/datalad_temp_test_publish_depends_q8i1lpv' '--debug'\"` to actually be the one which got stuck? (`ssh` is more likely of cause, but I am really not sure why...). Note, that if download full log (urls are above), in that \"bad\" session
```
$> grep configlist log.txt
[Level 9] stderr| [2020-02-26 20:21:35.087032521] read: ssh [\"localhost\",\"-S\",\".git/annex/ssh/localhost\",\"-o\",\"ControlMaster=auto\",\"-o\",\"ControlPersist=yes\",\"-o\",\"ControlMaster=auto\",\"-S\",\"/home/travis/.cache/datalad/sockets/2e18ea6b\",\"-n\",\"-T\",\"git-annex-shell 'configlist' '/tmp/datalad_temp_test_publish_depends7k9yct_j' '--debug'\"]
[Level 9] stderr| [2020-02-26 20:21:36.785462409] read: ssh [\"datalad-test\",\"-S\",\".git/annex/ssh/datalad-test\",\"-o\",\"ControlMaster=auto\",\"-o\",\"ControlPersist=yes\",\"-n\",\"-T\",\"git-annex-shell 'configlist' '/tmp/datalad_temp_test_publish_dependsymsjz245' '--debug'\"]
```
so -- there is a prior session (probably to another remote) with configlist and that one works out just fine... just that it goes to localhost, not datalad-test (set to 127.0.0.1 in /etc/hosts).
"""]]

8
doc/todo/making_it_easier_to_smudge_dotfiles/comment_6_62d03e7fccb9e8b824e145d5458e9796._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="spwhitton"
avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb"
subject="comment 6"
date="2020-02-29T16:34:16Z"
content="""
Thank you for working on this. Just upgraded my repo to v8 and removed my hack to edit .git/info/attributes.
"""]]

10
doc/todo/making_it_easier_to_smudge_dotfiles/comment_7_66e313842f388bad020d212885eeffd0._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="adding dotfiles"
date="2020-02-29T21:35:06Z"
content="""
Thanks -- this also resolves [[bugs/dotfiles_handled_differently]] (which wasn't actually a bug, just an unexpected special case).
[[Manpage|git-annex]] says \"Setting annex.dotfiles to true makes dotfiles be added to the annex the same as any other file\" -- \"same as any other file\" means that if `annex.largefiles` is *not* set, or if `annex.gitaddtoannex` is set to `false`, `git add .mydotfile` still adds to git, correct? So, basically, if `annex.dotfiles=false` (default), dotfiles are handled specially (always added to git regardless of other settings), while if `annex.dotfiles=true`, dotfiles are handled the same as non-dotfiles?
"""]]