Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2018-10-06 19:53:06 -04:00
commit e7ff1c6762
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
8 changed files with 212 additions and 0 deletions

View file

@ -0,0 +1,140 @@
[[!comment format=mdwn
username="Mowgli"
avatar="http://cdn.libravatar.org/avatar/17ab194dddf7b7da59ec039cbb3ac252"
subject="Gentoo-Build (Haskell nightmare)"
date="2018-10-05T19:36:01Z"
content="""
I try to build git-annex on Gentoo linux (the most current version to be able to interact with the same version on debian). Unfortunatelly the version in Gentoo itself is to old (6.20170818).
So I found the haskell overlay that includes newer libraries for haskell as well as git-annex version 6.20180626.
So for 3 days now I try to get the haskell stack compiled what failed many times with circular dependencies, conflicting versions or simply not working haskell versions. It was a real nightmare and only a hint from the mailinglist to simply allow any haskell library, even all broken ones, gave me the solution to fulfile all dependencies for git-annex.
But now, git-annex itself fails to compile. I have no idea how to solve that.
Here the relevant logs:
./setup configure --ghc --prefix=/usr --with-compiler=/usr/bin/ghc --with-hc-pkg=/usr/bin/ghc-pkg --prefix=/usr --libdir=/usr/lib64 --libsubdir=git-annex-6.20180626/ghc-8.4.3 --datadir=/usr/share/ --datasubdir=git-annex-6.20180626/ghc-8.4.3 --ghc-options=-j2 +RTS -A256M -qb0 -RTS --with-ar=x86_64-pc-linux-gnu-ar --ghc-option=-optc-march=native --ghc-option=-optc-O2 --ghc-option=-optc-pipe --ghc-option=-optl-Wl,-O1 --ghc-option=-optl-Wl,--as-needed --disable-executable-stripping --docdir=/usr/share/doc/git-annex-6.20180626 --verbose --enable-shared --enable-executable-dynamic --sysconfdir=/etc --disable-library-stripping --flag=-android --flag=-androidsplice --flags=-assistant --flags=-benchmark --flags=concurrentoutput --flags=-dbus --flags=magicmime --flags=network-uri --flags=pairing --flag=-production --flags=-s3 --flag=-testsuite --flags=torrentparser --flags=-webapp --flags=webdav
Assistant/Threads/MountWatcher.hs:36:2: error:
warning: #warning Building without dbus support; will use mtab polling [-Wcpp]
#warning Building without dbus support; will use mtab polling
^~~~~~~
|
36 | #warning Building without dbus support; will use mtab polling
| ^
Assistant/Threads/NetWatcher.hs:27:2: error:
warning: #warning Building without dbus support; will poll for network connection changes [-Wcpp]
#warning Building without dbus support; will poll for network connection changes
^~~~~~~
|
27 | #warning Building without dbus support; will poll for network connection changes
| ^
BuildFlags.hs:22:2: error:
warning: #warning Building without the assistant. [-Wcpp]
#warning Building without the assistant.
^~~~~~~
|
22 | #warning Building without the assistant.
| ^
BuildFlags.hs:27:2: error:
warning: #warning Building without the webapp. You probably need to install Yesod.. [-Wcpp]
#warning Building without the webapp. You probably need to install Yesod..
^~~~~~~
|
27 | #warning Building without the webapp. You probably need to install Yesod..
| ^
BuildFlags.hs:43:2: error:
warning: #warning Building without S3. [-Wcpp]
#warning Building without S3.
^~~~~~~
|
43 | #warning Building without S3.
| ^
Utility/DirWatcher.hs:42:2: error:
warning: #warning \"Building without inotify support\" [-Wcpp]
#warning \"Building without inotify support\"
^~~~~~~
|
42 | #warning \"Building without inotify support\"
| ^
Utility/IPAddress.hs:20:48: error:
Variable not in scope:
hostAddressToTuple :: HostAddress -> (Integer, b0, c0, d0)
|
20 | isLoopbackAddress (SockAddrInet _ ipv4) = case hostAddressToTuple ipv4 of
| ^^^^^^^^^^^^^^^^^^
Utility/IPAddress.hs:26:53: error:
Variable not in scope:
hostAddress6ToTuple
:: HostAddress6
-> (Word16, Word16, Word16, Word16, Word16, Word16, Word16, Word16)
|
26 | isLoopbackAddress (SockAddrInet6 _ _ ipv6 _) = case hostAddress6ToTuple ipv6 of
| ^^^^^^^^^^^^^^^^^^^
Utility/IPAddress.hs:39:47: error:
Variable not in scope:
hostAddressToTuple
:: HostAddress -> (Integer, Integer, Integer, Integer)
|
39 | isPrivateAddress (SockAddrInet _ ipv4) = case hostAddressToTuple ipv4 of
| ^^^^^^^^^^^^^^^^^^
Utility/IPAddress.hs:49:52: error:
Variable not in scope:
hostAddress6ToTuple
:: HostAddress6
-> (Word16, Word16, Word16, Word16, Word16, Word16, Word16, Word16)
|
49 | isPrivateAddress (SockAddrInet6 _ _ ipv6 _) = case hostAddress6ToTuple ipv6 of
| ^^^^^^^^^^^^^^^^^^^
Utility/IPAddress.hs:70:22: error:
Variable not in scope: htonl :: Integer -> t
|
70 | toipv4 a b = htonl $ fromIntegral a * (2^halfipv4bits) + fromIntegral b
| ^^^^^
Utility/Mounts.hs:13:1: error:
Could not find module `System.MountPoints'
Use -v to see a list of the files searched for.
|
13 | import qualified System.MountPoints
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Utility/Mounts.hs:14:1: error:
Could not find module `System.MountPoints'
Use -v to see a list of the files searched for.
|
14 | import System.MountPoints (Mntent(..))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Utility/SRV.hs:18:1: error:
Could not find module `Network.DNS.Lookup'
Use -v to see a list of the files searched for.
|
18 | import qualified Network.DNS.Lookup as DNS
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Utility/SRV.hs:19:1: error:
Could not find module `Network.DNS.Resolver'
Use -v to see a list of the files searched for.
|
19 | import Network.DNS.Resolver
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Full logs via e-mail.
As you can see, I try to compile without assistant and without webapp as I never use that two subsystems. From the ebuild file, that should exclude them via cabal flags and does not install the dns library (beside others).
Bt git-annex seems to ignore the cabal flags and try to install all the excluded stuff nevertheless (like s3, assistan or webapp).
Help! How to get out of the Haskell nightmare!?
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Mowgli"
avatar="http://cdn.libravatar.org/avatar/17ab194dddf7b7da59ec039cbb3ac252"
subject="Logs"
date="2018-10-05T19:38:06Z"
content="""
Sorry, your comment system ate up all the linebreaks and make all that beautiful error logs unreadable. :-(
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Teyras"
avatar="http://cdn.libravatar.org/avatar/d874f8fa5e6480461f368400b9089a2f"
subject="gcrypt support"
date="2018-10-06T12:14:44Z"
content="""
Is it possible to use gcrypt remotes when running git-annex in Termux?
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="comment 4"
date="2018-10-05T19:56:24Z"
content="""
If the s3 remote claims s3:// URLs, does the bucket name have to be a DNS domain? I thought, when a special remote claims a URL, it can interepret it however it wants?
"""]]

View file

@ -0,0 +1,18 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 5"
date="2018-10-05T20:20:42Z"
content="""
yes, for `s3://` urls there is only a bucket name, not a domain. Since bucket name is allowed to have `.`, some folks started to use their project domain name as a bucket name (e.g. `openneuro.org`, `images.cocodataset.org`). Then if you are to access them directly via url, full domain name would be e.g. http://images.cocodataset.org.s3.amazonaws.com, which would start causing troubles if you try to access it via https
[[!format sh \"\"\"
$> wget -S https://images.cocodataset.org.s3.amazonaws.com
--2018-10-05 16:19:48-- https://images.cocodataset.org.s3.amazonaws.com/
Resolving images.cocodataset.org.s3.amazonaws.com (images.cocodataset.org.s3.amazonaws.com)... 52.216.18.32
Connecting to images.cocodataset.org.s3.amazonaws.com (images.cocodataset.org.s3.amazonaws.com)|52.216.18.32|:443... connected.
The certificate's owner does not match hostname images.cocodataset.org.s3.amazonaws.com
\"\"\"]]
for which we started to provide workarounds in datalad.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="comment 6"
date="2018-10-05T21:31:33Z"
content="""
My current planned solution is to write an external special remote that claims s3:// URLs and downloads them. Then can use addurl --fast . My use case is that, if I run a batch job that reads inputs from s3 and writes outputs to s3, what I get at the end are pointers to s3, and I want to check these results into git-annex. Ideally there'd be a way for me to tell the batch system to use git-annex to send things to s3, but currently that's not possible.
Question: if an external special remote claims a URL that a built-in special remote could handle, does the external special remote take priority?
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 7"
date="2018-10-05T21:41:45Z"
content="""
FWIW datalad special remote already supports download from such s3:// URLs
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="comment 2"
date="2018-10-05T18:24:18Z"
content="""
Thanks for adding annex.jobs . Could you make it so that setting it to 0 means \"use all available processors\"? I use git-annex on AWS instances, and reserve instances with different processor counts at different times.
\"git-annex is rarely cpu-bound\" -- I though parallelization helps by parallelizing I/O operations such as file transfers?
"""]]