Commit graph

114 commits

Author SHA1 Message Date
Joey Hess
78440ca37d
move assistant and webapp build-depends into main build-depends
For some reason, cabal 3.4.1.0 builds w/o the assistant and webapp,
even when the flag is explicitly turned on. Moving the build-depends from
inside the if flag section to the main build-depends somehow fixes this.

Since the webapp build deps are thus always available, there is no reason
not to build the webapp when building the assistant. So, got rid of the
webapp build flag. Kept the assistant build flag for now, since building
without it does at least still speed up the build.

Sponsored-by: Brock Spratlen on Patreon
2022-08-29 15:23:49 -04:00
Joey Hess
a460aa8b70
Removed the NetworkBSD build flag
Debian stable and the i386ancient build both have a new enough network
to not need this flag any longer.

Sponsored-by: Svenne Krap on Patreon
2022-03-22 11:52:52 -04:00
Joey Hess
982eb7ed0d
remove vendored http-client-restricted
Removed vendored copy of http-client-restricted, and removed the
HttpClientRestricted build flag that avoided that dependency.

http-client-restricted is in Debian stable, and the i386ancient build also
uses it, so I think this vendored copy is no longer needed.

Sponsored-by: Noam Kremen on Patreon
2022-03-22 11:50:06 -04:00
Joey Hess
c0185f0848
bump i386ancient deps to allow using http-client-restricted and git-lfs
This should let i386ancient limp along for a few years more, beyond the
removal of those vendored deps from git-annex.

Also networkbsd is set now, so probably the last thing to unset that
flag is gone, and the flag could be removed soon.

Sponsored-by: Jarkko Kniivilä on Patreon
2022-03-22 11:36:48 -04:00
Joey Hess
8bbd683f31
relax enough i386ancient deps to allow new tasty
The new ansi-terminal was needed for test concurrency, and the new
concurrent-output fixes several bugs. And it turns out this is all
that's needed to use the new tasty.

Sponsored-by: Kevin Mueller on Patreon
2022-03-22 10:59:22 -04:00
Joey Hess
a33f1a0815
re-relax tasty dependency version for i386ancient build
Dependency issues were looking difficult to support tasty-1.2 with that
build. Not using `after` only affects rerunning and limiting tests,
since tasty's concurrency is not used, so this build will just not
support that.

We are probably nearing end of life on this build; it also doesn't
support git-lfs or http-client-restricted. The 2.6.32 kernel it supports
is at this point 13 years old, and stopped being supported by linux LTS
developers 10 years ago. It was supported by RHEL 6.10 through November
2020. At this point, no new hardware should be shipping with this
kernel, but that probably does not stop certian embedded vendors from
shipping it. And there is certainly some hardware still using it. But
the returns from supporting it are diminishing, and the quality of the
build for it is also diminishing.

Sponsored-by: Nicholas Golder-Manning on Patreon
2022-03-22 10:31:10 -04:00
Joey Hess
6971ee8c41
bump ansi-terminal for tastyversion bump 2022-03-22 10:18:17 -04:00
Joey Hess
8f6b52b770
update tasty version to new minimum 2022-03-21 16:08:36 -04:00
Joey Hess
487f7f979e
bump filepath-bytestring dep 2020-11-11 11:49:19 -04:00
Joey Hess
0736383e98
Fix bug that prevented linux standalone bundle from working on a fresh install
Bug was introduced in version 8.20201007, lost a necessary mkdir.

This commit was sponsored by Noam Kremen on Patreon.
2020-10-23 12:19:40 -04:00
Joey Hess
db7fb5b8cf
avoid depending on cmp
If it was not available, the caches would get deleted unncessarily.
This should be just as fast as using it.
2020-10-06 11:27:05 -04:00
Joey Hess
d479a885de
move cache cleanup back to before cache creation
It was moved to avoid a race, but that's now avoided in another way.
I prefer having it here, because this way if it somehow fails and
deletes the locpath that is going to be used, at least it will get
re-created.
2020-10-06 08:46:35 -04:00
Joey Hess
224623e72e
fix another race
Use a different tmp directory so the cache cleanup won't delete the
locpath directory while it's being populated.

This does change the hash used for the locpath directory, but it already
changed in f0ec725234
2020-10-06 08:44:05 -04:00
Joey Hess
d74d978968
fully atomic LOCPATH populating
This fixes a race between two runshells from two different
bundles. One could have run the cache cleanup code, seen the
LOCPATH the other one was in the process of populating, which didn't
have a base or a buildid file written yet, and so the cache cleanup code
would delete it out from under the other process.

Also, doing it fully atomically simplifies where the races between two
runshell processes from the same bundle. Now that needs to be
dealt with to only the mv that puts it in place.

Note that, if the same bundle has 2 runshells run first thing, they will
both generate locales, which is unncessary work, but that should be a
very unusual circumstance and after the LOCPATH is set up, it won't
happen again anyway.
2020-10-05 14:17:46 -04:00
Joey Hess
12a248f823
move cache cleanup to avoid a race
This should fix doc/bugs/standalone_runshell_can_race_and_fail_to_remove___96____126____47__.cache__47__git-annex__47__locales__47____96___dirs
where 2 runshells were running and the second one tried to clean up
LOCPATH while the first one was still populating it.

By moving the cleanup until after LOCPATH is populated, we guarantee
it's populated, so don't need to worry about such a race with another
process populating our same LOCPATH.
2020-10-05 14:07:27 -04:00
Joey Hess
f0ec725234
include buildid in LOCPATH
This avoids the possibility that the bundle could be updated in place,
leading to LOCPATH existing but containing locales for the old version,
which needed to be tested for with code that was not race-free.

LOCPATH/buildid is still written and checked when cleaning up stale caches.
That is not actually necessary, except old versions of the standalone
bundle expect to see it, and this prevents them cleaning up the locale
cache of a new version. And still checking it prevents the new version
cleaning up the locale cache of the old version while the old version is
still in use.

Added explicit tests before creating LOCPATH and the base and buildid files.

The buildid file no longer needs to be updated every time, because it's
stable for the given LOCPATH directory.

And the base file actually did not need to be updated every time,
because the LOCPATH is derived from base, so if the bundle is moved
elsewhere, a different LOCPATH will be used.

Transitioning to this will mean that two git-annex builds that otherwise
have the same buildid -- the same git-annex md5sum -- will use different
LOCPATH values, but that's handled fine by the cache cleanup code, so at
most it will mean one extra generation of the locale files.
2020-10-05 14:04:49 -04:00
Joey Hess
5863d14fd2
runshell: fix some mess left in a race
This seems to be the best way to deal with the race; if the first and
second runshell are running very close together, the first will generate
the locale directory, and a second test -d would still leave a race
window.
2020-10-05 13:42:43 -04:00
Joey Hess
e0ca1236ee
runshell: Update files atomically when preparing to run git-annex
This does not make it entirely idempotent, but it's a start.
2020-10-05 13:38:34 -04:00
Joey Hess
b91d97c3d2
small runshell optimisation
avoid some test -d for locale variables that are not set
2020-10-05 13:37:00 -04:00
Joey Hess
cd9a60bc7d
runshell: Fix a edge case where rm errors were sent to stdout, which could confuse things parsing git-annex output. 2020-10-05 12:44:40 -04:00
Joey Hess
6ea511beb4
Removed the S3 and WebDAV build flags
So these special remotes are always supported.

IIRC these build flags were added because the dep chains were a bit too
long, or perhaps because the libraries were not available in Debian stable,
or something like that. That was long ago, those reasons no longer apply,
and users get confused when builtin special remotes are not available, so
it seems best to remove the build flags now.

If this does cause a problem it can be reverted of course..

This commit was sponsored by Jochen Bartl on Patreon.
2020-09-08 12:42:59 -04:00
Joey Hess
05dbc449f1
move stack file used by i386ancient autobuilder 2020-08-28 15:16:10 -04:00
Joey Hess
88e5ebcda7
runshell LD_HWCAP_MASK=0 optimisation 2020-08-03 14:34:15 -04:00
Joey Hess
e62817c00d
move standalone building code out of Makefile and into Build.Standalone
This includes making Build.Standalone run LinuxMkLibs or OSXMkLibs
rather than doing that separately. Which is groundwork for a later
optimisation.

Also it simplified the code some.
2020-08-03 14:15:03 -04:00
Joey Hess
00c5f04f20
Deal with unusual IFS settings in the shell scripts for linux standalone and OSX app.
Thanks, Yaroslav Halchenko
2020-07-24 14:46:50 -04:00
Joey Hess
79187a6eaf
Revert "Unset IFS in shell scripts in the linux standalone build and OSX app."
This reverts commit 24125e8dc4.

yoh has a better patch I see
2020-07-24 14:33:13 -04:00
Joey Hess
24125e8dc4
Unset IFS in shell scripts in the linux standalone build and OSX app. 2020-07-24 14:31:11 -04:00
Joey Hess
88f721549d
Linux standalone: Use md5sum to shorten paths in .cache/git-annex/locales
md5sum is part of busybox, so is probably available unless it were compiled
out. If md5sum (or cut for that matter) is not available, it will
still use the whole path to $base, otherwise hash it.

Of course it's possible for md5sum to be available sometimes and not others
on the same system; in such an event the locales would be built twice for
the same bundle. The cleanup code will delete both sets once that
version of the bundle is upgraded.
2020-03-04 13:04:56 -04:00
Joey Hess
5c3636037b
Display a warning when concurrency is enabled but ssh connection caching is not enabled or won't work due to a crippled filesystem
A warning message is unsatisfying. But erroring out is too hard a failure,
especially since it may well work fine if the user has enabled passwordless
ssh.

I did think about falling back to one ssh connection at a time in this
case, but it would have needed a rework of every ssh call, which
seems far overboard for such a niche problem. There's no single place where
git-annex runs ssh, so no one place that it could block a concurrent call
on a semaphore. And, even if it did fall back to one ssh connection at a
time, it seems to me that doing so without warning the user about the
problem just invites bug reports like "git-annex is ignoring my -J2 and
only doing one download at a time". So a warning is needed, and I suppose
is good enough.
2020-01-23 12:35:46 -04:00
Joey Hess
f2a54e3401
Android: Improve installation process when the user's login shell is not bash.
~/.profile works for bash, but not all other login shells.

This setting PATH is a minor convenience for users, particuarly since
typing on android is so much harder. The usual linux standalone bundle
just expects the user to know how to add it to PATH. I don't want this
code to grow special cases for every possible login shell. So displaying a
message to the presumably minority who don't use bash seems like the best
choice.

Longer term, I'd hope termux gets some way to set an environment variable
for all login shells. Systems using PAM can, via ~/.pam_environment. Or
alternatively, add a git-annex package to termux, even if just an installer
package. I'd rather spend time on either of those than on making this minor
thing support more login shells.

This commit was sponsored by mo on Patreon.
2019-05-23 13:06:31 -04:00
Joey Hess
42842ea0ea
runshell: Use system locales when built with GIT_ANNEX_PACKAGE_INSTALL set
This is to work around https://github.com/datalad/datalad/issues/2769
which I don't know how to reproduce outside that environment, nor do I
understand the root cause of. For some time, Neurodebian has been working
around it by building its standalone debs with a patch that disables use
of the locales bundled with the standalone build, letting the system
locales be used.

Using the system locales is asking for trouble if there's
significant version skew between the system and bundled glibc, and
possibly also if the architeciture is different, or whatever. That's why
git-annex bundles and uses its own locales, because numerous users
reported real problems with using the system locales.

... However, in the specific case of the Neurodebian standalone debs,
the deb is built on a system very like the one it's targeted to be
installed on. Or well, so they assure me, although doc/install/Ubuntu.mdwn
also promotes those for use across all versions of Ubuntu, and the deb
is built avoiding xz so it will work with old versions of dpkg, so I wonder
how true it is. It does seem that, at least currently, there is no bad
version skew in the locales of the systems the deb is used on, since
it's already been using the system locales for some time.

Anyway, since the Neurodebian build already is setting
GIT_ANNEX_PACKAGE_INSTALL=1 in runshell, I made runshell use system
locales when that's set. This is a small scope creep for
GIT_ANNEX_PACKAGE_INSTALL, but it's not documented and AFAIK only used
for the Neurodebian build, so that seems ok. This will let them stop
carrying their patch for this forward.

This work is supported by the NIH-funded NICEMAN (ReproNim TR&D3) project.
2018-10-13 15:04:10 -04:00
Joey Hess
d14983ee68
webapp: fix termux detection
The bundled uname -o says Linux in termux; have runshell on Android
delete it so the termux one is used instead.

This fixes the webapp so it will enter Android mode.

This commit was sponsored by mo on Patreon.
2018-10-13 12:08:27 -04:00
Joey Hess
0240775f32
adding arm64 build, and improved termux installation process
* Added arm64 Linux standalone build. (No autobuilder yet.)
* Improved termux installation process.

Added git-annex-install.sh script to avoid user needing to type as much in
termux. The scope of this script is limited; runshell handles the rest.

Runshell runs termux-fix-shebang on the shell scripts. The problem is
the bundled bin/sh script, deleting that script also works, but then the
others probably use the system Android /bin/sh, which could be old or
broken or not posix or whatever. Using termux sh to run the scripts is
better.

This commit was sponsored by Eric Drechsel on Patreon.
2018-10-11 13:32:00 -04:00
Joey Hess
a97ef366fa
Linux standalone: Avoid using bundled cp before envionment is fully set up.
On android arm64, I saw the cp fail with "Bad system call", because proot
has not run yet. runshell only recently started using cp, and it's bundled
with git-annex, so this fixes a reversion.

This commit was sponsored by Nick Piper on Patreon.
2018-10-10 16:02:25 -04:00
Joey Hess
142795993a
hide stderr if cmp is used on file that does not exist
seen on android in termux
2018-10-10 15:28:27 -04:00
Joey Hess
fa44bca8b3
linux standalone: When LOCPATH is already set, use it instead of the bundled locales.
It can be set to an empty string to use the system locales too. Of course
whether that will work depends on the amount of divergence.

This commit was supported by the NSF-funded DataLad project.
2018-08-20 12:20:54 -04:00
Joey Hess
a8a0f7fc58
refuse to run if we can't write to HOME 2018-07-10 14:20:23 -04:00
Joey Hess
eb08deefa5
Revert "fall back to /tmp for LOCPATH if HOME does not exist or not writable"
This reverts commit f991c8c526.

On second thought, this makes it unbearably slow and leaves far too much
cruft behind.
2018-07-10 14:15:47 -04:00
Joey Hess
f991c8c526
fall back to /tmp for LOCPATH if HOME does not exist or not writable
In this case, we can't clean up old locale caches, but presumably the
system has something to clean up /tmp eventually.
2018-07-10 13:26:35 -04:00
Joey Hess
e802323071
deal with the persistent locpath issue
linux standalone: Generate locale files in ~/.cache/git-annex/locales/ so
they're available even when the standalone tarball is installed in a
directory owned by root.

This avoids a full-on reference counting cleanup hell, by letting old
locale caches linger as long as the standalone bundle directory associated
with them is still around. Old ones get cleaned up.

In the case where the directory has a new bundle unpacked over top of it,
the old locale cache is invalidated and rebuilt. Of course, running
programs using that may get confused, but this was already the case, and
unpacking over top of a bundle is probably not a good idea anyhow.

To support that, added a buildid file, which only needs to be unique across
builds of git-annex with different libc versions. sha1sum of git-annex
seems good enough for that.

Removed debian/patches/standalone-no-LOCPATH as it's no longer
necessary.

This commit was supported by the NSF-funded DataLad project.
2018-07-10 12:13:19 -04:00
Joey Hess
71f450f677
use proot to support Android 8
runshell: Use proot when running on Android, to work around Android 8's
ill-advised seccomp filtering of system calls, including ones crucial for
reliable thread locking. (This will only work with termux's version of
proot.)

See https://github.com/termux/termux-packages/issues/420#issuecomment-386636938

This commit was sponsored by andrea rota.
2018-05-08 13:55:10 -04:00
Joey Hess
b2accf9da1
Assistant: Fix installation of menus, icons, etc when run from within runshell.
runshell followed by git annex webapp didn't install that stuff, because
GIT_ANNEX_APP_BASE is not set. Running git-annex.linux/git-annex-webapp did
install that stuff, since that script set the env var. I noticed this with
the termux port whose instructions currently go that way.

Seems the right thing to do is to move the env var setting to runshell.
2018-04-25 17:58:00 -04:00
Joey Hess
118ed8f92b
runshell: hacks for termux; add tip
Added some tweaks to make git-annex work in termux on Android. The regular
arm standalone tarball now works in termux.

I guess the test for "$base/bin/git" is not really necessary, since it
tests for git-annex. Since that gets deleted on android, removed that test.

These are pretty hackish hacks, especially adding it to PATH. The goal is
to make it work well enough out of the box on Android.

This commit was sponsored by Eric Drechsel on Patreon.
2018-04-25 13:48:37 -04:00
Joey Hess
dd7ab91f97
runshell: Unset LD_PRELOAD
Preloaded libraries from the host system may not get along with the bundled
linker.

This was observed by users in termux:

ERROR: ld.so: object '/data/data/com.termux/files/usr/lib/libtermux-exec.so' from LD_PRELOAD cannot be preloaded (wrong ELF class:
ELFCLASS64): ignored.
Bad system call

But it could also affect more usual systems; the preloaded library might rely
on symbols from the host libc that are not available or have the wrong versions
in the bundled libc. Unsetting LD_PRELOAD entirely seems safest.
2018-04-25 13:40:48 -04:00
Joey Hess
10390a6378
Linux standalone arm build no longer supports Linux kernels older than 2.6.28
As it was getting too expensive to patch out use of the "new" syscalls

We could revisit this if someone has hardware with an older kernel
that's still being maintained, but I've verified that the Synology
NAS that had used a too old kernel version has been updated to 2.6.32.
2017-11-21 13:29:14 -04:00
Joey Hess
abb14f529e
fix build of patched network 2017-11-21 13:20:56 -04:00
Joey Hess
b411d6c412
freshen patch 2017-11-21 13:11:16 -04:00
Joey Hess
cd97752626
update to lts-9,9
with aws updated to 0.17.1

The old version of this file no longer builds on the arm autobuilder,
running Debian unstable.
2017-11-21 13:03:20 -04:00
Edward Betts
0750913136
correct spelling mistakes 2017-02-12 17:30:23 -04:00
Joey Hess
95c8b37544
Linux standalone: Improve generation of locale definition files, supporting locales such as, en_GB.UTF-8. 2016-12-19 17:03:52 -04:00