This moves the responsibility to auto-repair filesystems to the
initramfs. (I think we don't do this at all right now).
We don't try to mount broken partitions at all, we tell the user
that the system is broken and fail_to_boot.
The flow is now:
1. mount the boot partition (read-only) for initramfs-extra
2. initramfs-extra is extracted (with the needed fsck executables)
3. fsck the root partition
4. mount root partition (read-write)
5. fsck the boot partition
6. mount the boot partition (read-write)
This helps with the systemd bringup, as we want to make use of
systemd-firstboot, which sets /etc and enabled services up. As
this service is responsible of setting up /etc, it is run before
/etc/fstab is read and before / is remounted read-write.
The referenced commit had surfaced a bug with the unlocking code, which
would attempt to change the output of find_root_partition while the
cached PMOS_ROOT is stale and active. Clear this variable to force
find_root_partition to find the new rootfs value.
Fixes: 7030e08e7b ("postmarketos-initramfs: check for root subpartition (MR 5048)")
[ci:skip-build]: already built successfully in CI
Drop the console-shell hook in favour of stuffing buffyboard into the
initramfs. This makes it possible to get a debug shell directly on the
device without needing to use a computer.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
[ci:skip-build]: already built successfully in CI
Rework the messages and avoid re-spawning the splash on every check.
Additionally, add a call to check_keys, this lets users drop to a debug
shell or export logs to triage this failure (particularly useful for
hard-to-reproduce issues).
Finally, add a (liberal) 30 second timeout and dump logs at the end. If
we're just waiting for a device to show up and it doesn't after 30
seconds then it's probably never going to...
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
In mount_subpartition we currently try to look for the boot partition,
to see if we're done (either we aren't using subpartitions or we found
the right one). However it is possible that a device might have
originally used the subpartition scheme, but was later converted to have
a real boot partition (for EFI for example).
In this scenario, the boot partition would be found immediately, but the
root subpartition would not be.
Address this by making mount_subpartition check for success by calling
find_root_partition instead of find_boot_partition. For most
installations this has no impact.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
pma!5000 added a new implicit dependency on libinput, because it
specified libinput files in the initramfs file hook. mkinitfs considers
it fatal if it cannot find files listed in a file hook.
Later this could be 'optimized' by adding a -libinput subpkg and an
`install_if=libinput-libs` to add this file hook so it's not installed
everywhere... but the size increase isn't too much for UIs.. someone can
do this later.
> libinput-libs-1.26.0-r0 installed size:
> 476 KiB
This fixes a missing dependency problem where a UI (e.g. "none") doesn't
normally depend on libinput, and mkinitfs would fail because these files
were missing.
mdev is slow and missing features, it also requires that we load all the
modules that we include at once making the boot process slower.
udev is required for unl0kr (and buffyboard), it can also load modules
on demand (and asynchronously). Making the boot process considerably
faster on devices, especially for generic images where we have lots of
drivers for different display panels in the initramfs.
Additionally, import full fat modprobe from the kmod package, this is
required to support compressed modules.
This brings us more in line with other distros and generally improves
compatibility.
If devices use broken kernel modules which don't correctly define
modalias', these drivers may not be loaded by udev. This should be fixed
by defining the missing modalias statements in the driver.
This also runs udev earlier in the init, so that display drivers are
loaded before the splash in case they are needed.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Co-authored-by: Clayton Craft <clayton@craftyguy.net>
[ci:ignore-count]
[ci:skip-build]: already built successfully in CI
This incorprates the debug-shell functionality directly into the
initramfs, so it's no longer necessary to build and boot a custom
initramfs in order to debug your device.
Additionally, the behaviour of the debug-shell is entirely reworked, It
now creates an ACM serial gadget which can be accessed via any normal
terminal emulator (picocom, minicom, etc; or PuTTY on windows). Rather
than just invoking sh, the debug-shell now creates a respawning getty on
both the new virtual console and the active console (this will either be
the UART console or tty0/1).
It is necessary to spawn these shells via getty since the logging rework
means we can no longer assume that stdin/out/err reference a TTY.
In addition to the above, it is now possible to trigger a log dump by
holding volume up during boot. This can be useful for helping users
debug their devices if the issue doesn't result in a failure that can be
detected in the initramfs.
With these changes, the console-shell and debug-shell hook packages are
reduced to only adding the additional tools/features. console-shell is
still required for fbkeyboard, and debug-shell for the setup_usb_storage
tool.
Co-Developed-by: Clayton Craft <craftyguy@postmarketos.org>
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
For some reason globbing doesn't work properly for empty directories,
adjust the check in run_hooks to ensure the directory both exists and
contains something, otherwise we get an error about
"/hooks-cleanup/*.sh" not existing.
/ # mkdir blah
/ # for x in blah/*; do echo $x; done
blah/*
/ # touch blah/beep
/ # for x in blah/*; do echo $x; done
blah/beep
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
I somehow hit a case where this wasn't installed and
/lib/mdev/persistent-storage wasn't available. Let's just explicitly
depend on mdevd...
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This step is the most likely to go wrong or have issues, and it has side
effects which can make it difficult to run multiple times on one boot.
Move it to after hooks so that e.g. when dropping to a debug shell, we
land before the first call.
This also makes booting to hooks a faster in many cases.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
fallocate seems to cause some strange behaviour where the logs.img file
is treated like it's empty. Given it's small and in a ramdisk let's just
create it with dd instead.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This fixes the annoying "sh: write error" and "Couldn't write to clear
UDC" messages that happen on every single boot.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Rework logging to always log the initramfs output to the kernel
ringbuffer and deprecate PMOS_NO_OUTPUT_REDIRECT in favour of following
the kernel loglevel.
I know it seems silly to use syslogd for this, but it's necessary to
buffer writes to /dev/kmsg per-line if we want to correctly set the log
level, and "tee" does not do this (it will write multiple lines at once,
resulting in "<14>" prints in the ringbuffer). The main advantage to
this is that we won't have kernel logs cut in half by initramfs logs
anymore, everything will be nicely line buffered!
The previous logging solution of multiple "tail" commands would actually
fail to log up to the last few lines before a crash due to how tail
works (it polls the file and buffers lines).
I attempted something like this before, but I stopped after running into
ratelimiting issues. These are now resolved by configuring the
printk_devkmsg sysctl.
Dropping PMOS_NO_OUTPUT_REDIRECT:
The general motivations behind PMOS_NO_OUTPUT_REDIRECT was to avoid
cluttering up the console with initramfs logs when they aren't wanted;
this is now handled instead by the kernels logging facility. We log to
the ringbuffer at LOGLEVEL_INFO, so if "quiet" is specified on the
cmdline (or the loglevel is otherwise set above info) then initramfs
logs will also not be shown.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We plan to make a lot of changes to the initramfs which will require
incresaing the size. There are some devices that have literally no free
space for this, so make a -minimal initramfs fork that can continue to
support those while we update the regular initramfs.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
I have noticed a bug. Because is some cases, uuid for the boot partition is set, it won't reach logic for the stowaway.
[ci:skip-build]: already built successfully in CI
This charset is the default and the recommended value, since boot fat
partitions should be case insensitive. linux-lts and linux-edge in
alpine use "utf-8", that might change, but add the options regardless,
so we can be safe
Fixes https://gitlab.com/postmarketOS/pmaports/-/issues/2782
Apparently f2fs has a soft dependency on this driver, so if it's =m in
the kernel then let's make sure it's always in the initramfs since f2fs
is a supported filesystem.
Fixes https://gitlab.com/postmarketOS/pmaports/-/issues/2711
[ci:skip-build]: already built successfully in CI
How to boot from USB drives:
1. Make sure the extcon has OTG support and included in initramfs
2. Make sure the PMIC supports the power to OTG devices, and included in initramfs. Otherwise, a hub with external power supply is required.
3. Make sure module `usb-storage` is enabled in initramfs
4. `pmbootstrap install --sdcard=/dev/<usb_drive>`
- postmarketOS/pmbootstrap@942ee20789 is required to ignore sdcard check. Thanks @ollieparanoid for the patch.
5. `pmbootstrap flasher flash_kernel`
6. Insert the USB drive to OTG and boot
[ci:skip-build]: already built successfully in CI
Reading the `ifname` of a gadget that has not been configured with a UDC
succeeds with the content `(unnamed net_device)` instead of failing.
So we need to explicitly check that a UDC was configured before we read
the interface name.
Ref: #2648
Sometimes, there can be more than 10M of stuff collected for logging,
which breaks things.. bump the size to 32M which should still be small
enough for low end / old devices.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
/dev/pmsg0 may be created by pstore driver. When it is available,
write logs into it so logs can be recovered through pstore after
a failed boot. This is useful for devices without console or hard
to get console.
Some devices take time to initialize storage, e.g. Qualcomm laptops
using ADSP need to load firmware firstly, and initramfs script does
not always manage to mount boot partition in time.
Unfortunately, there is no reliable way to reproduce the issue, but
this patch should fix cases like this.
Before this change the script assumed that the device has only one UDC.
However it is possible for a device to have multiple UDCs, in which case
the deviceinfo is the appropriate place to specify which one should be used.
This change adds an optional deviceinfo variable `deviceinfo_usb_network_udc`
to specify the UDC that should be used.
Also, before this change the script assumed the USB network interface name is
"usb0". But other gadget drivers like `g_cdc` can claim USB interfaces
before the initramfs script gets around to creating one for USB networking,
so the interface might be "usb1" or something else. The configfs gadget itself
can provide the actual interface name, so this change makes the script
use that.
Currently, when postmarketOS fails to boot up, retrieving any
information necessary to root cause the issue is non-trivial, requiring
building a custom initramfs with debug-shell enabled and then manually
copying out data.
Let's improve the situation by exporting logs automatically on boot
failure. This is safer than just exposing a root shell but still
provides a whole lot of useful info that should make duplicating and
triaging issues much easier for developers.
This commit implements the functionality and includes some generally
useful data. We then generate a README and expose files for each command/log
as well as an archive that can be easily attached to a GitLab issue.
To help with triaging, also record the version of the postmarketos-initramfs
package and hash the init.sh and init_functions.sh files.
For testing purposes, you can trigger the log recovery mode on-time by
creating an empty file named ".pmos_export_logs" in the /boot partition.
Signed-off-by: Caleb Connolly <caleb@connolly.tech>
Always write logs to the ramdisk, adjust setup_log() to spawn "tail" to
pipe the log to the console device when PMOS_NO_OUTPUT_REDIRECT is set.
This is necessary to ensure that the initramfs logs can be extracted on
a failed boot.
Signed-off-by: Caleb Connolly <caleb@connolly.tech>
As described in [0] let's switch away from RNDIS and use NCM instead.
Since we cannot force all kernels to switch at the same time, let's keep
a fallback to RNDIS in the setup function.
We can also remove usb_f_rndis from modprobe as the module gets loaded
automatically when needed, but instead we need to load libcomposite
manually so that $configfs/usb_gadget gets created.
[0] https://gitlab.com/postmarketOS/pmaports/-/issues/1797
[ci:skip-build] already built successfully in CI
This shows the OS version (from /etc/os-release), e.g. "edge", on the splash
screen. os-release is a tiny text file, adding it to the initramfs doesn't seem
too bad, and having this information on the splash screen can be helpful.
Note that this uses "VERSION" and not "PRETTY_NAME" from the os-release file,
since the splash already shows "postmarketOS" (and the pretty name includes
that too), it seemed redundant and took up valuable display space.
[ci:skip-build]: already built successfully in CI
Enable using the postmarketOS initramfs for boot-testing devices, in
this scenario we don't care about some components like the splash, mdev,
or subpartitions, instead we want to run full udev (to load all devices)
and then run hooks. The ci hook will deadloop after running tests.
Signed-off-by: Caleb Connolly <caleb@connolly.tech>
Explicitly pull in the blkid package (we already added it to
00-initramfs-base.files), and replace uses of busybox findfs. Full fat
blkid supports PARTLABEL (so the /dev/disk/by-partlabel symlinks will
work consistently now), and has some nice optimisations that make the
lookup times a lot faster than busybox findfs.
[ci:skip-build]: already built successfully in CI
The "pmos_[br]oot(_uuid)?" kernel cmdline args can be used to specify
how to mount the root and boot partitions. During a previous rework, the
behaviour of these was made inconsistent (becoming dependent on the
order they were specified on the cmdline).
Undo the previous "optimisation" by splitting them back out into two
FOR loops, where the _uuid variants take precedence.
Hides this error when FDE is *not* enabled, and ultimately skips the
block of logic for detecting the luks partition:
/init: line 437: cryptsetup: not found
This error has confused some users in the past, who were reporting
issues with the initramfs.
There's probably a minor speedup on non-FDE systems as well from
bailing early.
If a file exists in the initramfs at runtime, cpio won't overwrite it
by default when extracting the initramfs-extra archive. This adds the -u
option to cpio, to overwrite any existing files in the destination.
This is meant to fix issues where the -extra archive has an app that is
meant to replace a busybox app, busybox --install creates it in the
initramfs and gzip will skip it since it's newer than what is in the
archive.
With full-fat mdev supporting by-partlabel lookups, there is a small
delay during startup. As mdev and dynamic partitions aren't actually
needed for the framebuffer device, move show_splash earlier.
Furthermore, slightly rework setup_framebuffer to not dump the verbose
message about waiting for the framebuffer unless the framebuffer isn't
found.
Signed-off-by: Caleb Connolly <caleb@connolly.tech>
