The OSX autobuilder is now using github CI, and can use a current
version of ghc, rather than the old one.
Sponsored-by: Dartmouth College's Datalad project
This is a cleaner build than on Jenkins because the whole environment setup
is handled by the CI config, at least up to the point of "get a random bag
of Windows bytes".
Also, the Jenkins autobuilder has been intermittently failing for a long
time, not due to any problem with git-annex but just a failure to clean up
directories.
Also, this build runs the test suite, and it is (mostly) passing. Test
suite always failed in the jenkins environment.
Also, this build includes libmagic.
Here is the build workflow used by github actions:
https://github.com/datalad/datalad-extensions/blob/master/.github/workflows/build-git-annex-windows.yaml
The libmagic build has its own workflow:
https://github.com/datalad/file-windows/blob/master/.github/workflows/build.yml
(Also cleaned up some windows build cruft I don't use anymore.)
There is no build-version file to link to. I've opened a todo requesting
one: https://github.com/datalad/datalad-extensions/issues/55
It was moved to avoid a race, but that's now avoided in another way.
I prefer having it here, because this way if it somehow fails and
deletes the locpath that is going to be used, at least it will get
re-created.
Use a different tmp directory so the cache cleanup won't delete the
locpath directory while it's being populated.
This does change the hash used for the locpath directory, but it already
changed in f0ec725234
This fixes a race between two runshells from two different
bundles. One could have run the cache cleanup code, seen the
LOCPATH the other one was in the process of populating, which didn't
have a base or a buildid file written yet, and so the cache cleanup code
would delete it out from under the other process.
Also, doing it fully atomically simplifies where the races between two
runshell processes from the same bundle. Now that needs to be
dealt with to only the mv that puts it in place.
Note that, if the same bundle has 2 runshells run first thing, they will
both generate locales, which is unncessary work, but that should be a
very unusual circumstance and after the LOCPATH is set up, it won't
happen again anyway.
This should fix doc/bugs/standalone_runshell_can_race_and_fail_to_remove___96____126____47__.cache__47__git-annex__47__locales__47____96___dirs
where 2 runshells were running and the second one tried to clean up
LOCPATH while the first one was still populating it.
By moving the cleanup until after LOCPATH is populated, we guarantee
it's populated, so don't need to worry about such a race with another
process populating our same LOCPATH.
This avoids the possibility that the bundle could be updated in place,
leading to LOCPATH existing but containing locales for the old version,
which needed to be tested for with code that was not race-free.
LOCPATH/buildid is still written and checked when cleaning up stale caches.
That is not actually necessary, except old versions of the standalone
bundle expect to see it, and this prevents them cleaning up the locale
cache of a new version. And still checking it prevents the new version
cleaning up the locale cache of the old version while the old version is
still in use.
Added explicit tests before creating LOCPATH and the base and buildid files.
The buildid file no longer needs to be updated every time, because it's
stable for the given LOCPATH directory.
And the base file actually did not need to be updated every time,
because the LOCPATH is derived from base, so if the bundle is moved
elsewhere, a different LOCPATH will be used.
Transitioning to this will mean that two git-annex builds that otherwise
have the same buildid -- the same git-annex md5sum -- will use different
LOCPATH values, but that's handled fine by the cache cleanup code, so at
most it will mean one extra generation of the locale files.
This seems to be the best way to deal with the race; if the first and
second runshell are running very close together, the first will generate
the locale directory, and a second test -d would still leave a race
window.
So these special remotes are always supported.
IIRC these build flags were added because the dep chains were a bit too
long, or perhaps because the libraries were not available in Debian stable,
or something like that. That was long ago, those reasons no longer apply,
and users get confused when builtin special remotes are not available, so
it seems best to remove the build flags now.
If this does cause a problem it can be reverted of course..
This commit was sponsored by Jochen Bartl on Patreon.
Seems 13.11 was not being used for a while, but 14.27 works.
For some reason, stack was getting confused when I set the magicmime
flag on the command line, but setting it in stack.yaml didn't confuse
it. Ugh.
Unfortunately, the autobuilder is using an old version of OSX, and
cannot use ghc newer than the one in lts-13.11. Newer ghc versions use a
libc symbol that is not available.
Since the main stack.yaml has been updated to a newer lts, and just
substituting in lts-13.11 no longer works due to other dependency
issues, it's simplest to use a separate one for the OSX build.
Hopefully this is temporary, and the autobuilder gets updated.
This includes making Build.Standalone run LinuxMkLibs or OSXMkLibs
rather than doing that separately. Which is groundwork for a later
optimisation.
Also it simplified the code some.
This reverts commit e930237f61.
Unbelievably, that worked, somehow the preprocessed file had gotten
cached from the wrong version. Windows has found a new way to amaze me in
its horribleness.
build is failing on commit d409f92318
but the error message quotes a version of Utility/Process/Transcript.hs
before a fix yesterday. All I can think is stack may have cached the old
version, perhaps after cpp?
md5sum is part of busybox, so is probably available unless it were compiled
out. If md5sum (or cut for that matter) is not available, it will
still use the whole path to $base, otherwise hash it.
Of course it's possible for md5sum to be available sometimes and not others
on the same system; in such an event the locales would be built twice for
the same bundle. The cleanup code will delete both sets once that
version of the bundle is upgraded.
A warning message is unsatisfying. But erroring out is too hard a failure,
especially since it may well work fine if the user has enabled passwordless
ssh.
I did think about falling back to one ssh connection at a time in this
case, but it would have needed a rework of every ssh call, which
seems far overboard for such a niche problem. There's no single place where
git-annex runs ssh, so no one place that it could block a concurrent call
on a semaphore. And, even if it did fall back to one ssh connection at a
time, it seems to me that doing so without warning the user about the
problem just invites bug reports like "git-annex is ignoring my -J2 and
only doing one download at a time". So a warning is needed, and I suppose
is good enough.
This makes it be the same as in the linux runshell.
It may be that bundle/ used to have all the necessary binaries in it,
but that seems unlikely. I think that the breakage fixed by my previous
commit masked this other breakage, and anything git execs from git-core
(as opposed to being buitin) has never worked in the OSX dmg.
This allows the rpm to be built anywhere the necessary build deps are
available (including on debian) and the resulting package will work on as
broad a range of rpm distributions as the libc/kernel supports.
The DistributionUpdate changes to use the new script have not yet been
tested.
rsync is only needed for rsync special remotes and git-annex-shell from
Debian oldstable. Since the library situation on windows for rsync required
a particular 32 bit build of git for it to work, and may also somehow need
git-annex to be 32 bit build, it's better to not include it.
This commit was sponsored by Jake Vosloo on Patreon.
stack upgrade --git is failing on windows, the ghc version it downloaded to
upgrade itself does not work:
11:06:25 Configuring base-compat-0.9.3...
11:06:25 Cabal-simple_Z6RU0evB_2.0.1.0_ghc-8.2.2.exe: The program 'ghc-pkg' is required
11:06:25 but the version of
11:06:25 C:\Users\jenkins\AppData\Local\Programs\stack\i386-windows\ghc-8.2.2\bin\ghc-pkg.EXE
11:06:25 could not be determined.
~/.profile works for bash, but not all other login shells.
This setting PATH is a minor convenience for users, particuarly since
typing on android is so much harder. The usual linux standalone bundle
just expects the user to know how to add it to PATH. I don't want this
code to grow special cases for every possible login shell. So displaying a
message to the presumably minority who don't use bash seems like the best
choice.
Longer term, I'd hope termux gets some way to set an environment variable
for all login shells. Systems using PAM can, via ~/.pam_environment. Or
alternatively, add a git-annex package to termux, even if just an installer
package. I'd rather spend time on either of those than on making this minor
thing support more login shells.
This commit was sponsored by mo on Patreon.
This is to work around https://github.com/datalad/datalad/issues/2769
which I don't know how to reproduce outside that environment, nor do I
understand the root cause of. For some time, Neurodebian has been working
around it by building its standalone debs with a patch that disables use
of the locales bundled with the standalone build, letting the system
locales be used.
Using the system locales is asking for trouble if there's
significant version skew between the system and bundled glibc, and
possibly also if the architeciture is different, or whatever. That's why
git-annex bundles and uses its own locales, because numerous users
reported real problems with using the system locales.
... However, in the specific case of the Neurodebian standalone debs,
the deb is built on a system very like the one it's targeted to be
installed on. Or well, so they assure me, although doc/install/Ubuntu.mdwn
also promotes those for use across all versions of Ubuntu, and the deb
is built avoiding xz so it will work with old versions of dpkg, so I wonder
how true it is. It does seem that, at least currently, there is no bad
version skew in the locales of the systems the deb is used on, since
it's already been using the system locales for some time.
Anyway, since the Neurodebian build already is setting
GIT_ANNEX_PACKAGE_INSTALL=1 in runshell, I made runshell use system
locales when that's set. This is a small scope creep for
GIT_ANNEX_PACKAGE_INSTALL, but it's not documented and AFAIK only used
for the Neurodebian build, so that seems ok. This will let them stop
carrying their patch for this forward.
This work is supported by the NIH-funded NICEMAN (ReproNim TR&D3) project.
The bundled uname -o says Linux in termux; have runshell on Android
delete it so the termux one is used instead.
This fixes the webapp so it will enter Android mode.
This commit was sponsored by mo on Patreon.
Running git-annex linux builds in termux seems to work well enough that the
only reason to keep the Android app would be to support Android 4-5, which
the old Android app supported, and which I don't know if the termux method
works on (although I see no reason why it would not).
According to [1], Android 4-5 remains on around 29% of devices, down from
51% one year ago.
[1] https://www.statista.com/statistics/271774/share-of-android-platforms-on-mobile-devices-with-android-os/
This is a rather large commit, but mostly very straightfoward removal of
android ifdefs and patches and associated cruft.
Also, removed support for building with very old ghc < 8.0.1, and with
yesod < 1.4.3, and without concurrent-output, which were only being used
by the cross build.
Some documentation specific to the Android app (screenshots etc) needs
to be updated still.
This commit was sponsored by Brett Eisenberg on Patreon.
* Added arm64 Linux standalone build. (No autobuilder yet.)
* Improved termux installation process.
Added git-annex-install.sh script to avoid user needing to type as much in
termux. The scope of this script is limited; runshell handles the rest.
Runshell runs termux-fix-shebang on the shell scripts. The problem is
the bundled bin/sh script, deleting that script also works, but then the
others probably use the system Android /bin/sh, which could be old or
broken or not posix or whatever. Using termux sh to run the scripts is
better.
This commit was sponsored by Eric Drechsel on Patreon.
On android arm64, I saw the cp fail with "Bad system call", because proot
has not run yet. runshell only recently started using cp, and it's bundled
with git-annex, so this fixes a reversion.
This commit was sponsored by Nick Piper on Patreon.
