We cannot use After=snapd.service as user services cannot synchronize
with system services. Using `snap system wait seed.loaded` should work,
except for the fact that it requires polkit authentication to perform
this operation.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
LXD is going to support launching riscv64 virtual machines,
and for riscv64 virtual machines to be usable the console
needs to be properly set. This and other fixes are currently
done in the hook 999-cpc-fixes.chroot, which was disabled for
riscv64 and which this commit enables.
Signed-off-by: Adriano Cordova <adriano.cordova@canonical.com>
We want the firmware updater and security center pointing to edge too.
The model only allow to select it, but we need to invoke them by
default in snap prepare-image
We need edge on the live session too so that subiquity knows about
latest and greatest on TPM FDE support. We will revert that once snapd
is released to the stable channel.
layer construction involves rsync, and that process ignores times to
avoid some of the layers being larger than they would otherwise where
the only difference is times. This saves a small amount of space,
around 14MiB, but results in files in the layers having non-intended
time values. Ensure mtime and atime in the source chroot match what is
found in the destination chroot.
To get 25.10 Desktop ISOs with TPMFDE bits, we need matching pc-kernel
and snapd otherwise we get errors like so when running
`snap prepare-image`:
WARNING: the kernel for the specified UC20+ model does not carry
assertion max formats information, assuming possibly incorrectly the
kernel revision can use the same formats as snapd
error: snapd 2.68+ is not compatible with a kernel containing snapd
prior to 2.68
Use the "dangerous" model, which allows overriding the channel, and pick
up the matching pc-kernel which is not yet on 25.10/stable, where the
non-dangerous model would expect to find it.
Also see https://bugs.launchpad.net/cloud-images/+bug/2106729.
Since Oracular[1]:
Ubuntu’s systemd-networkd no longer sets UseDomains=true for managed
network interfaces. In effect, this means that search domains
configured in DHCP leases will not be reflected in /etc/resolv.conf
by default. This change aligns Ubuntu’s default behavior with that
of upstream. System administrators may choose to override this
default on a global, or per-interface basis. See systemd.network 4
for details.
The default in systemd is UseDomains=false. From systemd.network(5)[2]:
DHCP=
Furthermore, note that by default the domain name specified
through DHCP is not used for name resolution. See option
UseDomains= below.
UseDomains=
It is recommended to enable this option only on trusted
networks, as setting this affects resolution of all hostnames,
in particular of single-label names. It is generally safer to
use the supplied domain only as routing domain, rather than as
search domain, in order to not have it affect local resolution
of single-label names.
It has been reported to us by few clouds that this breaks local name
resolution. For instance, in Google Cloud Compute, users can no longer
reach instances in the same zone[3] nor Google Cloud services[4] by
their names.
Arguably, the security concerns for having this option disabled are not
valid in cloud environments. As one of our partners said:
IIUC, the motivation to disable UseDomains by default is that a
laptop might be used on an untrusted network where the domains
provided by DHCP can be a security issue, directing users to places
they don't intend.
But it's not possible for a cloud instance to be connected to an
untrusted network (barring a breached account).
The way I'm looking at this is that DHCP option 119 exists for the
express purpose of allowing a network administrator to configure the
DNS search path for computers on that network. I understand there's
a security concern if that network isn't a datacenter. But in the
cloud there's no concern (in some clouds, it's not even possible for
DHCP response packets to come from anywhere but the cloud's own
DHCP).
We should restore this setting in cloud images.
[1] https://discourse.ubuntu.com/t/oracular-oriole-release-notes/44878
[2] https://manpages.ubuntu.com/manpages/plucky/en/man5/systemd.network.5.html
[3] https://cloud.google.com/compute/docs/internal-dns
[4] https://cloud.google.com/compute/docs/metadata/overview
This model intentionally uses pc-kernel from a branch, for components
testing purposes. We'll have to update this again before release when
the desired pc-kernel is on a stable channel.
* Again in ubuntu-server builds, configure LAYERFS_PATH in the kernel layer
and ensure the initrd is freshly regenerated in that layer. LAYERFS_PATH
was being set to the layer below the kernel layer, which meant that the
live session did not get access to all the modules in the case that the
kernel had not been installed in the base layer, which in turn means that
installs fail. (LP: #2100148)
* While we're at it, delete any initrd from any other layer than a kernel
layer, as they just waste space on the ISO.
patch create_manifest to produce an sbom when called by an ubuntu-cpc
project. Patch all the ubuntu-cpc hooks and series files to include the
newly generated manifests, filelists, and sboms. Generates a number of
new artifacts in the builds. the snap utilized, cpc-sbom, is an open
source repo and a provided via a hidden snap. there is no intention of
publisizing the snap or how we generate sboms, however partners require
the ability to audit if required.
defensively checks if the snap is already installed, in the case of
multiple hooks being called in a single build (thus sharing a build
host), and only if called in an ubuntu-cpc project.
(cherry picked from commit 7c7b7df89dc96169db1f255d6bba901ebb63a43c)
Plucky is currently on kernel 6.12 so preseeding fails with a apparmor
feature mismatch given that the live-build/apparmor/generic tree is
used. Adding a 6.12 tree (which is identical with the 6.11 tree)
solves this.
U-Boot with distroboot has:
efi_dtb_prefixes=/ /dtb/ /dtb/current/
So we should install the device-trees into dtb/ and not dtbs/ on the EFI
system partition.
Fixes: 365435ad2dbe ("riscv: copy device trees to the ESP")
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Commit f9c5020200ce ("riscv: directly copy device trees to /boot/dtbs")
incorrectly copied devicetrees into /boot/dtbs/$kvers instead of /boot/efi/dtbs,
inside the ESP and where U-boot expects them. This commit fixes this path.
Fixes: f9c5020200ce ("riscv: directly copy device trees to /boot/dtbs")
Signed-off-by: Adriano Cordova <adriano.cordova@canonical.com>
We are removing our different variants of wsl rootfs with the new
Microsoft format. We only keep one following the distribution policy:
- lts to lts
- intermediate release to next one
Co-authored-by: Carlos Nihelton <carlos.santanadeoliveira@canonical.com>
The previous Tegra kernel metapackage implementation (linux-nvidia-tegra-igx)
was initially planned to apply both for Jetson devices and IGX systems. It turned
out recently (LP: #2069179) that we now need to reserve the metapackage name
linux-nvidia-tegra-igx for IGX systems, and use the new linux-nvidia-tegra-jetson
metapackage for Jetson devices. For the sake of clarity, the image name, model,
sub-arch, variant should align with the kernel metapackage name.
LP:2083240
starting in noble, adduser no longer creates a homedir for system users.
The buildd user then does not have a home directory, causing snaps to be
unable to run, as well as possibly other issues from a missing assumed
homedir. Explicitly create /home/buildd
Version 1 of install-sources.yaml is a top-level list of the sources to
be offered.
Version 2 extends this by placing the list under a top-level key
`sources`, adding a `version` field, and adding a `kernel` field which
supplants the current kernel-meta-package file. `kernel.default` is
read to know which kernel to use - unless we need to fallback to the
bridge kernel.
This reverts commit c4e69348aed2e89bdef0187afe79da18d855eb8c as
the more debugging is needed for autopkgtest failures and is
therefore blocking apparmor fixes for cloud images.
In cloud-init version 24.3, single process mode where a shared python
systemd service cloud-init-main. In that release, cloud-init.service was
renamed cloud-init-network.service to better clarify cloud-init's
systemd unit names relative to the cloud-init boot stages.
This rename only applies to Oracular and newer releases.
See: https://discourse.ubuntu.com/t/announcement-cloud-init-perfomance-optimization-single-process/47505
functions drops in a complete override for cloud-init.service. That
override in /etc/systemd/system needs to be renamed and refreshed to
latest single process configuration.
LP: #2081325
kdump-tools uses ucf for config file management and naively
modifying the config file meant for the target system directly
will cause the file hash to not get updated in the ucf database.
This will then cause later modifications to fail because
"there's nothing to do". Although actually doing the modification
to the ucf database is messy. Let's just modify the file in the live
layer to get the behavior we want there.
We install the kdump-tools package to minimal layer via inclusion in the
desktop-minimal seed, but it is enabled by default. Include a new chroot
hook to set USE_KDUMP=0 to make sure it's disabled by default and let
the installer decide to enable it or not.
We install the kdump-tools package to minimal layer via inclusion in the
server-minimal seed, but it is enabled by default. Include a new chroot
hook to set USE_KDUMP=0 to make sure it's disabled by default and let
the installer decide to enable it or not.
By placing the kernel in minimal, we can achieve the following
improvements:
1. Space savings - there are redundant packages present in the ship-live
pool and in the live layer. Adding the kernel to minimal means that
the kernel is already in the live layer, and we don't then also need
it in the pool.
2. Time savings - informal vm testing suggests more than a minute
improvement to have the kernel preinstalled over installing it at
runtime.
As always, there is a cost tradeoff:
1. If a different kernel is desired, we need to be able to remove this
preinstalled kernel. Relevant curtin and subiquity changes are
already landed.
2. When installing that other kernel, it'll take longer than today due
to still needing to install a kernel at runtime + the time cost of
removing the preinstalled kernel.
Support some systems which don't handle partition numbers
higher than 15. (LP: #2072929)
Partition 16 was added for /boot to enable cloud FDE (commit a8b2a9b01)
Ubuntu Studio wants to add a minimal installation. The individual tasks
are metapackages that can be installed by the ubuntustudio-desktop task.
With that in mind, we would like to reintroduce
ubuntustudio-desktop-core as a minimal installation. This is made much
easier with the layered images compared to the package removal format
used by ubiquity. This also means ubuntustudio-desktop-core becomes the
base seed.
If I'm missing anything, please advise.
System override drop-ins cannot redact dependencies (Before or After) and
thus require a full unit override. Avoid writing the unit file delivered
by cloud-init deb package in /lib/systemd/system/cloud-init.service because
it will generate warnings fron debsums -c about modified files.
The correct place to provide a full unit override is in
/etc/systemd/system/cloud-init.service in order to drop
Before=sysinit.target from the packaged cloud-init.service file.
Note vigilance will be needed across cloud-init SRU boundaries to ensure
we sync any cloud-init.service unit changes that are introduced to
stable releases because livecd-rootfs is overriding the whole file.
LP: #2069391
The lowlatency kernel will eventually undergo deprecation. Rather than
wait for such a time to happen and be reactive, Ubuntu Studio would
rather be proactive about this now that the generic kernel can act as a
lowlatency kernel with certain command line parameters as outlined by
https://discourse.ubuntu.com/t/fine-tuning-the-ubuntu-24-04-kernel-for-low-latency-throughput-and-power-efficiency/44834.
As such, we have modified our `ubuntustudio-lowlatency-settings`
package, which installs `/etc/default/grub.d/ubuntustudio.cfg` with the
following line:
-GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT threadirqs"
+GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT preempt=full
nohz_full=all threadirqs"
Additionally, that same file used to set "GRUB_FLAVOUR_ORDER" which is
no longer needed.
unminimize is currently present at /usr/local/sbin/unminimize,
which is spit out by livecd-rootfs currently. We'd like to switch
that to use the packaged unminimize, which will be at
/usr/bin/unminimize instead.
There was a change made by me in https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/466388
as part of LP: #2066905 to remove references to LXD in the unminimize scripts
but I also removed the calls to `unminimize` in error.
This still needs to run but without any references to LXD which no longer
needs to be `unminimized` via snap installation.
The ubuntu-core-installer image is an installer that installs ubuntu
core. The environment the installer runs in is similar to the server
installer but it has a source catalog entry that points to the model
created in ubuntu-core-installer/hooks/05-prepare-image.binary, which
subiquity knows how to install.
With current kernel we need to specify the SBI driver
for the early console to work.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
fix: No longer install LXD snap in unminimize script (LP: #2066905)
The LXD snap is no longer seeded in any images since Noble+ so the LXD related unminimize logic in
./live-build/auto/build?h=ubuntu/noble and ./live-build/ubuntu-server/hooks/01-unminimize.chroot_early
is no longer required.
lxd-installer can remain installed.
MP: https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/466316
The LXD snap is no longer seeded in any images since Noble+ so the LXD related unminimize logic in
./live-build/auto/build?h=ubuntu/noble and ./live-build/ubuntu-server/hooks/01-unminimize.chroot_early
is no longer required.
lxd-installer can remain installed.
fix(HyperV desktop): Re-enable ability to build HyperV desktop images (LP: #2064280)
We have not built Hyperv desktop images since Jammy and with the re-introduction of HyperV for Noble we have encountered build issues caused by refactoring and removals of code assumed to be redundant but the HyperV desktop images were actually using these code paths.
In bbedffe6 we split the building of cloud images and non cloud to using an ddisk-image-uefi.binary and disk-image-uefi-non-cloud.binary respectively. In e38264ca there was a change which meant that any attempt to build hyperv images would result in incorrect disk size and incorrect disk label.
This has been fixed by ensuring that the ubuntu:desktop-preinstalled $PROJECT:$SUBPROJECT matches and sets the correct disk size and correct disk label.
A change in 76d79466 changed the logic of how the image size for amd64 images were being set. This overrode the sizes set for the desktop images incorrectly.
This MP ensures that hyperv desktop images can now be built and successfully launched with hyperv manager.
MP: https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/465288
For Ubuntu 24.04 and later cloud-init is included in desktop images. This is not applicable for Hyperv images so
we can disable cloud-init. This leaves the cloud-init package installed but disabled so users can still
use it if they want.
This is a documented way to disable cloud-init. See https://cloudinit.readthedocs.io/en/latest/howto/disable_cloud_init.html
A change in 8fb21808 also removed many of the dependencies that the hyperv images require.
This removal has been restored in this commit by adding them expliciltly in the hyperv hook.
We have not built Hyperv desktop images since Jammy and with the re-introduction of HyperV for Noble we have encountered build issues caused by refactoring and removals of code assumed to be redundant but the HyperV desktop images were actually using these code paths.
In bbedffe6 we split the building of cloud images and non cloud to using an ddisk-image-uefi.binary and disk-image-uefi-non-cloud.binary respectively. In e38264ca there was a change which meant that any attempt to build hyperv images would result in incorrect disk size and incorrect disk label.
This has been fixed by ensuring that the ubuntu:desktop-preinstalled $PROJECT:$SUBPROJECT matches and sets the correct disk size and correct disk label.
A change in 76d79466 changed the logic of how the image size for amd64 images were being set. This overrode the sizes set for the desktop images incorrectly.
This commit ensures that any desktop image being created uses the correct image size.
do_layered_desktop_image() is now the standard entry point for flavors using
ubuntu-desktop-bootstrap and handles minimal/standard/live layers in a
configurable and flavor-agnostic way to reduce code duplication.
Failing CPC tests show that the preseeded apparmor features don't
include policy:unconfined_restrictions for the 6.8 kernel. This
change adds the feature preseed with values based on a successfully
booted instance.
Fixes: LP: #2060558
It was found out that autopkgtests didn't pass the NOW env variable
which is generally provided by the launchpad-buildd build and thus
the autpkgtests fail.
ubuntu/include.* are the master location for these files.
Copy them over for projects with similar needs, while skipping ones that
are incorrect.
LP: #2055077
Ubuntu MATE is switching to a layered image in preparation to
use ubuntu-desktop-provision. Luckily, their seed structure is
already well-structured for layering, so this is easily done.
This has become moot now that the code block has been
moved out from live-build/functions to live-build/auto/build
so passing the argument is not needed anymore.
Presence of this field helps in determining if the image is an
unminimized image, which then can be leveraged in the unminimize
script to easily determine the image type.
fix: Set the required debconf settings to allow for non interactive grub updates in cloud images (LP: #2054103)
As part of addressing LP: #2054103 [1] an update to grub-pc added a feature to be able to ensure that grub-pc
installation can happen noninteractively on cloud images.
This change is equivalent to running
```
debconf-set-selections grub-pc grub-efi/cloud_style_installation boolean true
debconf-set-selections grub-pc grub-pc/cloud_style_installation boolean true
```
These were introduced optionally to determine the install device using
`grub-probe` dynamically instead of having to fill the `grub-pc/install-devices`
debconf entry.
[1] https://bugs.launchpad.net/cloud-images/+bug/2054103
MP: https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/461062
As part of addressing LP: #2054103 [1] an update to grub-pc added a feature to be able to ensure that grub-pc
installation can happen noninteractively on cloud images.
This change is equivalent to running
```
debconf-set-selections grub-pc grub-efi/cloud_style_installation boolean true
debconf-set-selections grub-pc grub-pc/cloud_style_installation boolean true
```
These were introduced optionally to determine the install device using
`grub-probe` dynamically instead of having to fill the `grub-pc/install-devices`
debconf entry.
[1] https://bugs.launchpad.net/cloud-images/+bug/2054103
There was a time historically where Launchpad buildd might have relied
on that behaviour, but this shouldn't be the case anymore as it sets
priority manually when building backports.
Meanwhile any other builds using buildd images (e.g. snapcraft)
shouldn't default to backports unless required. (lp: #2009871)
Refs:
- [1] https://git.launchpad.net/launchpad-buildd/commit?id=c2ebcb6752
Per the comments, BASE_SEED was initially used to identify the seed in the
flavor to use for identifying preseeded snaps, and later was also used to
identify which "minimal-remove" seed to apply to an image.
The first usage is now obsolete after a refactor; we now correctly detect
snaps from any of the included seeds without needing an explicit
declaration.
The second usage only applies to installer images that are NOT using layered
squashfs, since for these images 'minimal' is a separate squashfs layer
rather than a list of packages to remove after the fact.
Refactor this code to eliminate pointless definitions of BASE_SEED and
define it only for the subset of flavors today that:
- have a 'minimal-remove' seed
- are not using layered squashfs.
The cloud-images logic is now special case for ubuntu-wsl to not require
ending with project_prefix. Readd it first, which will allow us to
ensure backward compability on cloud-images.ubuntu.com
Also Use Signed-By: /etc/apt/keyrings/preinstalled-pool.gpg and
make sure we only update from that .sources file as we did before.
This code may all be dead, who can say.
FIXME: We should figure out how to do an armored export of that key
and then embed it in the signed-by field instead of using a keyring
file.
Template is based on the specification with some rewording for
Ubuntu Pro as agreed.
v2:
- Enabled backports by default (I did not see that!)
- Enabled restricted, multiverse security updates
- Replaced tweaked with adjusted
v3:
- Insert an explanatory sources.list
LP: #2048129
The publisher for cloud-images.ubuntu.com expects that artefacts
finishes with: file_url.endswith(project_prefix + suffix).
Now that we integrate app_id to the image name, we need thus to put it
before project_prefix and not between project_prefix and suffix.
The StarFive VisionFive 2 board can boot from SPI flash or SD-card.
Install U-Boot to the SD card.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Microsoft offically support systemd now and our Pro service relies on
it. This option is enabled by default via our launcher (Windows-side) on
first run for quite a while.
Remve this file creation from it, don’t ship it in a package as the
file may be altered by the user to add additional options and ship it as
part of the rootfs.
Co-authored-by: Jean-Baptiste Lallement <jean-baptiste@ubuntu.com>
On WSL, we have multiple applications with the same rootfs, but
different upgrade policy:
Ubuntu: should always track latest LTS and offer upgrade.
Ubuntu-<Version>: should never offer upgrade and will stick to Version
Ubuntu-Preview: current in development version.
Co-authored-by: Jean-Baptiste Lallement <jean-baptiste@ubuntu.com>
livecd-rootfs (24.04.25) noble; urgency=medium
.
* live-build/auto/config: for ubuntu-server, consider the actual kernel
flavor when dealing with netboot layers - even if we don't really care.
If the previous if statement checking if PASSES_TO_LAYERS is true,
then the last return code be non-zero and a return statement with no
argument will return the error code of the if statement thus exiting
the script. This is not our intent. So we need to return 0 here when
layer name as already been registered
The unminimize script will try to install the lxd snap using the shim script
`/usr/sbin/lxd` from the lxd-installer package.
Previously `unminimize` was using `snap`
to install `lxd` directly which was being diverted by diverting the `snap` command.
This is no longer the case so we can remove `/usr/sbin/lxd` from the lxd-installer package
if it exists and then redirect any calls to `/usr/sbin/lxd` to `/bin/true`
This is a cherry pick forward port from Jammy livecd-rootfs version 2.765.37.
(cherry picked from commit 8b83212372e0c1adb1dbdf7ead234f93c52a189e)
mount_disk_image function expects root partition to be at number 1. But
some images require the root partition to be at other some other number.
For example, EKS Anywhere images for bare metal are used with Tinkerbell
deployment with a default configuration that expects the root device to
be found at /dev/sda2. The knowledge of the root device path is needed
to modify certain files in the root filesystem (e.g. cloud-init configs)
for the machine to join Kubernetes cluster control plane.
The partition number can be changed in the hook by "sgidsk --transpose".
Allow the hook to use mount_disk_image with custom root partition number
by making it an optional third parameter that defaults to 1.
Noble moved to the 6.6 kernel now and the preseeding optimization
doesn't work anymore given that the apparmor features used during
preseeding do not match the apparmor features used on a running system
with kernel 6.6 .
By invoking LXD, lxd-installer will install LXD from the right
place, thereby make it simpler for us to not hardcode the
channel and manually snap install it.
This makes the hook ok to use cross-flavor.
We could also move glib-compile-schemas to a separate hook, to ensure we never
silently fail because glib-compile-schemas is broken/missing.
When the files we're creating in the live layer have static content, ship
them in live-build/ubuntu/includes.chroot.minimal.standard.live instead of
generating them from live-build/ubuntu/hooks/020-ubuntu-live.chroot_early.
Also fixes the fact that
live-build/ubuntu/hooks/020-ubuntu-live.chroot_early was incorrectly writing
to /root in the previous upload instead of /usr.
Without casper in the minimal.standard.live seed for flavors using the
new ubuntu-desktop-installer (or derivitives thereof), casper cannot
create a live user. Without this live user, Ubuntu Studio has been
experiencing the inability to login automatically from either the GUI or
manually from a TTY. This leaves the boot at a black screen with a mouse
cursor. This commit is an attempt to avoid the same situation. Previous
assessments of omitting casper from this line appear to have been
incorrect.
Remove this hook; it's only for pre-installed desktop images and
8fb2180842c452ff08dd41a5746c00bfd69521cf already removed the other bits
for pre-installed desktop images.
Somewhere along the line, we started trying to add packages to the live
environment of flavor "preinstalled" images. But:
- we don't build preinstalled images for any flavors
- the preinstalled images for projects like cloud images and wsl are
explicitly excepted from this code
- the only desktop project we do produce preinstalled images for, Ubuntu
on Raspberry Pi, uses ubuntu-image for building so this code is never
reached
fix: Enable snap preseeding with ppc64el images where /boot/vmlinux is used instead of /boot/vmlinuz. (LP: #2038957)
ppc64el still uses /boot/vmlinux so we need to determine the boot file name as non ppc64el use /boot/vmlinuz. This
is then used to determine the kernel major minor version installed so that the correct apparmor features can be used
during snap preseeding. This preseeding was failing for ppc64el for the mantic 6.5 kernel as the /boot/vmlinuz
being checked did not exist.
MP: https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/453306
ppc64el still uses /boot/vmlinux so we need to determine the boot file name as non ppc64el use /boot/vmlinuz. This
is then used to determine the kernel major minor version installed so that the correct apparmor features can be used
during snap preseeding. This preseeding was failing for ppc64el for the mantic 6.5 kernel as the /boot/vmlinuz
being checked did not exist.
On armhf and arm64 the QEMU virt machine provides the serial console as an
emulated AMBA PrimeCell UART which the kernel refers to as /dev/ttyAMA0.
Consider this when constructing GRUB_CMDLINE_LINUX_DEFAULT in file
/etc/default/grub.d/50-cloudimg-settings.cfg (LP: #2036730).
Reviewed-by: Gauthier Jolly <gauthier.jolly@canonical.com>
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
lp: #2037567. starting in kernel 6.5.0.1006, there's been an update to
apparmor features. Creates the 6.5 kernel directory, fully populates
with feats checked from a machine running 6.5.0.1006 installed from
proposed (as of 20230927).
Now that kernel names use expected -generic flavour, and kernels are
installed in the live layer, we can go back to stock behaviour of
auto/build noticing that binary hooks are called on a live layer and
executing the extraction & rename of the kernel flavours.
BTW we can even later expand that to support 2 generic abis, and
calling the bigger one the hwe generic such that can also remove
./live-build/ubuntu-server/hooks/04-kernel-bits.binary.
This fixes ubuntu arm64+x13s that is unable to find
ubuntu-x13s.kernel-laptop as due to this hook, which currently
produces ubuntu.kernel-generic in error which is not at all expected
by ubuntu-cdimage. Also this unbreaks producing oem & intel-iot
images, although we will build these in 24.04 only next.
This reverts ubuntu daily-live to use `--linux-flavours
laptop-generic-hwe-22.04` instead of `--linux-flavours none
--linux-packages=none --initramfs=none`, like it did in lunar and
pre-canary-image or images that install kernel in live layer.
Fixes: c00bbf3fb3 ("desktop: place kernel in the live layer")
Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com>
With the migration from linux-kvm to linux-virtual/generic for the mantic minimal
images we can now start building arm64 minimal cloud images.
When building initial test images we noticed that the flash-kernel package was being
installed. This is not required for EFI images.
This commit removes the flash-kernel package from the cloud images arm installs
The only images built using the livecd-rootfs ubuntu-cpc project for arm64 and armhf
which are not cloud images (which therefore do not require flash-kernel) are the
raspi images. raspi does require flash-kernel. But the raspi images use the
`ubuntu-server-raspi` task to install the flash-kernel. As such this non raspi
flash-kernel package install can be completely removed.
It also means that initramfs-tools and dracut-install are installed as these are dependencies
of flash-kernel package.
Add mapping to use laptop-23.10 kernel. Ensure that
enhanced-secureboot is only setup on amd64 arch.
LP: #2037099
Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com>
minimized cloud image policy, introduced in version 23.10.16, is to not install recommends for any package
installs during build. This is to keep the image as small as possible. This also extends to
the grub related packages.
This solves the problems detailed in LP: #2037075 and aligns other arches more with amd64 install of
grub/shim packages for both minimized and non minimized ubuntu-cpc cloud image builds.
Fix use of variable declared in conditional branch and used in parent
scope in snap_validate_seed. This would affect binary for images without
kernel and using "set -u". (LP: #2037338)
When trying to make changes to refactor livecd-rootfs, it is difficult to
know what side effects a change may have because of the use of globs on
arch/subarch and the lack of an authoritative list of supported arch
combinations.
This assembles a list of all possible values for $ARCH:$SUBARCH by looking at
all existing uses of $SUBARCH in live-build/auto/config and combining with
etc/default-arches from ubuntu-cdimage:
$ grep + etc/default-arches |grep -vE '(trusty|xenial|bionic|focal)[[:space:]]|appliance'
It also includes a special-case *appliance* glob, because there are many
Ubuntu Core appliance builds and there may be more in the future, and we
don't want to have to update livecd-rootfs with a hard-coded list.
Otherwise, this includes all currently used / supported combinations. The
amd64+kassel subarch is referenced in the code, but stopped being built a
while ago because "end of contract"; and there are some older no-longer-used
subarchs for particular raspi variants, which there is work to clean up
separately. So this should be a no-op wrt buildability of any existing
images on mantic - and if not, that's important for us to know!
In the past, we'd directly snap install lxd which defaults to
the latest/stable channel. However, whilst working on enhancing
unminimize, it was observed that we install this snap from
the stable/ubuntu-<version> channel instead.
This was also noted as a failure when running the CTF tests:
`lxd installed from latest/stable, not stable/ubuntu-23.10`
With the introduction of the 6.5 kernel for mantic on 13th September ago we are seeing image build failures
on the armhf builds. The build failure was `No kernel output for generic-lpae!`.
Introduced in the 6.4 kernel and therefore now also in 6.5 there is no generic-lpae flavor anymore. it's just generic now.
As such this commit updates the expected flavour for armhf to generic.
minimize-manual takes an argument of the path to the chroot but of the 3 commands that are run the argument
is only used twice and with the third hardcoding "chroot" as the path to the chroot.
Thankfully "chroot" has been the patch passed in for the current uses of minimize-manual but this
could cause issues later if that were to change.
This commit resolves the problem preventing future issues.
This is needed following the addition of the new boot partition. This
also gives us the opportunity to refactor the logic and use a case
statement instead of ifs
In order to support better support Full Disk Encryption on the clouds,
the boot assets have to sit on an un-encrypted partition. We've tried
mounting the ESP on /boot before but it didn't work as /boot has to
support linking for DPKG to work and the ESP has to be FAT.
In a minimized image, the linux headers are stripped, so when
unminimizing it, we should restore those stripped headers
by installing the linux-virtual package.
The unminimize script previously just restored the system documentation
and translations, man pages, and installed ubuntu-minimal and ubuntu-standard
packages to provide the familiar Ubuntu minimal system. But such an image
never became an equivalent of base image.
Upon investigation and looking at how the base image is constructed -
https://git.launchpad.net/livecd-rootfs/tree/live-build/auto/config#n1108 -
we use the following things:
- minimal task
- standard task
- cloud-image task (which involves ubuntu-server)
- ubuntu-minimal package
- server task if arch != amd64
OTOH, in the unminimize script, we use the following:
(https://git.launchpad.net/livecd-rootfs/tree/live-build/auto/build#n286)
- ubuntu-minimal package
- ubuntu-standard package
So upon running some tests, it was found that if we install ubuntu-server
(with --fix-policy flag), we get the resulting image equivalent to that
of a base image.
cf: https://warthogs.atlassian.net/browse/CPC-3033
The pi images contain redundant copies of cmdline.txt and config.txt in
the boot partition mount-point (which get hidden by the *actual* boot
partition). This commit removes those and simplifies the subarch check
(again, +raspi is the only subarch in use at this point on the Raspberry
Pi images)
/etc/hosts in an installed system comes from one of several places:
- the netbase package which ships a sane default
- scripts/casper-bottom/18hostname, which is mostly identical to the netbase
one but sets a hostname to the flavor name
- the installer which writes one to the target fs
- chroot includes in livecd-rootfs
A chroot include is only needed for images that don't include netbase (so
don't inherit from the minimal seed), don't boot casper, and don't run an
installer.
NONE of these conditions are true for the ubuntu-server live image. And in
addition, the /etc/hosts here differs from the others.
So, remove it.
systemd-resolved got moved out of the systemd package in kinetic and is
required for bootable buildd environement (as opposed to launchpad buildd)
(LP: #2007419)
Prior to dpkg/1.21.0, there was a bug where dpkg -V/--verify
couldn't list all the correct packages correctly but with
that being fix and in archive since Jammy, this works perfectly
but the syntax to report the missing files have changed. It
just prints 'missing' now. With that new format, we can now
fix the regex to simply list the packages.
With this patch, the unminimize script works flawlessly
on a minimized image.
ubuntu-cpc project binary hooks were not all producing .filelist files as they were not using
the create_manifest shared function.
This commit ensures the disk-image-uefi, disk-image-ppc64el and disk-image-uefi-non-cloud hooks create
a filelist during build.
The image filelists created during ubuntu-cpc project image builds were not sorted.
Soring the filelists makes it easier to compare the filelists without needing to sort first.
This package is needed by ubuntu-advantage-tools for cloud images
only. u-a-t ultimately will drop this entirely in their next release.
(cf: https://github.com/canonical/ubuntu-pro-client/issues/2692)
So instead of putting this in the seeds and then having to deal
with the seed changes to Mantic (after its release) and SRU of
ubuntu-meta, I'd rather have changes in livecd-rootfs done. Once
python3-systemd dependency is dropped from u-a-t, we'll drop it
from livecd-rootfs in Mantic+, too.
fuse3 was previously installed through recommends but with minimized images we no longer install recommends packages.
It is only required when preseeding snaps so does not need to be present in all minimized images so does not
need to be in the cloud-minimal seed.
As a result of not installing recommended packages the packages required to run `grub-install`
are no longer installed by default.
To ensure we can successfully run `grub-install` we install both `grub-pc` and `grub2-common`
packages.
As a result of not installing recommended packages we have dangling symlink `/boot/initrd.img.old`
As per the preceding `/boot/initrd.img` cleanup. Cleanup of `/boot/initrd.img.old`
only happens if it is a dangling symlink.
These `rm` commands also have `--verbose` flags now to make it easier when debugging logs
For minimized images we do not want to install any recommended packages.
We can do this by setting APT::Install-Recommends to false in apt config
or by passing --no-install-recommends to apt-get install.
While attempting to run autopkgtest locally, the test stops at the
following command:
ssh-keygen -t ed25519 -C ubuntu_vagrant_insecure_key -b 4096 -f
/tmp/tmp.VuAfnsBv1G/vagrant_insecure_key
This is found in live-build/ubuntu-cpc/hooks.d/base/vagrant.binary
It appears to be waiting for a passphrase, as running that outside of
adt gives a more helpful "Enter passphrase" prompt.
Explicitly set the passphrase to empty with the `-N` argument.
Armhf images install the `generic-lpae` kernel, while other ARCHes use the
standard `generic` kernel when building the "virtual" image flavour.
Code was looking for a kernel binary ending with -generic in armhf
builds, and failed. Add a special condition to handle armhf builds'
kernel ending with `generic-lpae`
References:
[1] https://bugs.launchpad.net/cloud-images/+bug/2029527
During Realtime kernel image build, there was an error during
validating snap seed which derivative images copied 5.19
apparmor feature and can't validate when Realtime kernel (5.15)
installed [0].
To prevent this, bind correct apparmor feature with kernel
version.
[0] https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2024639
(cherry picked from commit 6b54faa6be6286017eb2dc701534cf780ae462ce)
With the switch to the ubuntu-cloud-minimal seed, we
don't really need to purge anything now. On the contrary,
the purging of packages if not installed, fails with the
exit code of 100.
EDK II is available for the StarFive VisionFive 2 board. As it is larger
than U-Boot we need to increase the size of the loader 2 partition to
accommodate it.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Now that we have the cloud-minimal seed for minimized cloud
image builds, we should drop all the workarounds and hacks
we once needed when we were using the server seed. We can
directly use the new metapackage and get rid of the tasks and
other autoremoves, et al.
classic ubuntu-image builds will build the gadget from a repository
using make, and the reference pc-gadget snap uses chdist to pull the
latest GRUB/shim from the proper series x pocket; NB: chdist was used
instead of the more convenient pull-lp-debs as devscripts is in main
while ubuntu-dev-tools is in universe. It's inelegant for livecd-rootfs
to pick up dependencies needed to build gadgets, even if it's for the
official ones as other gadgets might want other dependencies, rather
this should be expressed as part of the build contract of the gadget, or
livecd-rootfs should only consume pre-built gadgets built in a standard
way (e.g. snap build, deb build etc.).
Revert this change for now as /boot then becomes a FAT partition which
breaks DPKG requirements[1]. This change is going to be re-evaluated and
maybe introduced in a different way.
This is not a clean revert because of 3282efb ("ubuntu-cpc: cleanup
disk-images-uefi.binary") which we want to keep.
[1] https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_What_are_the_filesystem_requirements_by_dpkg.3F
This reverts commit 6a66666e0a5ab1ad96cb0e388f278aafbd012ffe.
Package linux-allwinner has a kernel with the generic flavour as
dependency. Add this translation to our code checking the correct
installation.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
According to the kernel team the Linux Meta package linux-allwinner shall
continue to be supplied. It will depend on generic packages.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Remove kvm-image altogether.
Previously for minimal image replace_kernel function replaced virtual
images with kvm, and called force_boot_without_initramfs. Now simply
call force_boot_without_initramfs for minimal image without replacing
kernel flavour.
This also means minimal images can now be built for arm64 and armhf.
Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com>
Up to now we have used u-boot-menu for preinstalled images for the SiFive
HiFive Unmatched and Unleashed boards and GRUB for all other RISC-V images.
The choice was made because RISC-V GRUB was not available when the SiFive
boards where released.
Let the Unmatched and Unleashed board preinstalled images use GRUB.
Simplify the code.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Avoid lintian error bogus-mail-host-in-debian-changelog.
The name part of an email address containing a comma must
be enclosed in double quotation marks.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Canonical Public Cloud's project seems a bad place to build images for
hardware devices however this is how things were done a we now need to
maintain this.
The recent change to mount the ESP on /boot breaks those images, instead
of adding more hacky things in the hook, create a dedicated target for
those images and use a different hook to build UEFI images.
This is required by the new UEFI binary hook as we mount the ESP on
/boot and the ESP filesystem doesn't support symlinks.
We keep symlinks for s390x images which do not use UEFI anyway.
This is driven by online encryption scenarios. In order to efficiently
encrypt the root filesystem without modifying the partition layout, the
kernel should sit in an un-encrypted /boot partition. Instead of
creating a new partition that would change the default partition layout,
we mount the ESP on /boot. We also need to then bind mount /boot on
/boot/efi because that's where Grub expects the ESP to be located.
This now matches the cloud images (7c760864fdcb278ca37396f06f5e3f297428d63d)
fixing bootloader updates in the buildd images, but also fixing
compatibility with using devtmpfs for losetup.
kpartx on riscv64 appears to be racy. Rather than trying to debug these
fraught races somewhere between udev and libdevmapper, we can use losetup
which should be simpler and less error-prone.
live-build/auto/config:
- for Ubuntu Server live images and the arm64+tegra full arch, build a
tegra variant with linux-nvidia-tegra as the flavor and
linux-nvidia-tegra as the kernel meta-package
- default to nvidia-$SUBARCH as the kernel flavor for all images using
arm64+tegra as full arch
hooks/03-kernel-metapkg.chroot_early:
- use linux-nvidia-tegra as kernel meta-package for the nvidia-tegra
flavor
missing a && between icicle and visionfive, led to /boot/efi still being
in place, and grub-install running instead of exiting the func.
fixes LP:2015750
Cloud-init cannot write directly to
/etc/NetworkManager/system-connections because subiquity may
need to emit config to /etc/netplan/00-installer.yaml and call
netplan apply for autoinstall.network use-cases.
When cloud-init's config is written directly to
/etc/NetworkManager, neither netplan nor subiquity has knowledge of
this config and this results in namespace collisions in NetworkManager
due to `netplan-` named connections and `cloud-init` connection ids
fighting over which config own a given interface name.
Deleting this config overlay allows subiquity to manage all network
setup when it needs to with netplan directly.
Subiquity already has logic to rename any unwanted netplan
configuration when it intends to write cfg and run netplan apply[1].
This should allow subiquity full control of network config when needed.
[1] https://github.com/canonical/subiquity/blob/
92ac6544cdfedfd332d8cd94dbcfad0aab994575/subiquitycore/
controllers/network.py#L267
LP: #2015605
SUBARCH=visionfive2 is used to build images for the StarFive VisionFive 2
boards. For the device-tree we assume board revision 1.3B.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Autoinstall directives can be provided on the grub cmdline to
cloud-init via kernel parameters like the following:
autoinstall 'ds=nocloud-net;s=http://somedomain/'
In order to support DNS resolution for NoCloud datasource at
datasource discovery time, cloud-init.service needs to be
orderered after NetworkManager.service and
NetworkManager-wait-online.service
which will have brought up applicable NICs.
Since NetworkManager is After=dbus.service, the cloud-init.service
avoids systemd ordering cycles by also dropping
Before=sysinit.target when it adds, After=NetworkManager.service and
After=NetworkManager-wait-online.service
Add this file overlay for /lib/systemd/system/cloud-init.service
because systemd drop-in files can only add constraints and not
drop prexisting service constraints.
Also add an AUTOMATION_HEADER comment to any generated files to
add discoverability in the event of future bugs/concerns.
LP: #2008952
ipc was dropped as an apparmor feature. checked by grabbing the latest
lunar VM, installing the latest kernel, doing a reboot, and comparing
directories and files. compared all files and the only diff is the ipc
posix_mqueue
When building locally using the auto/build script unmounting fails.
Avoid mounting via bind. Mount mountpoint/dev as devtmpfs file system and
mountpoint/dev/pts as devpts file system.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Evolve the seed to only ship the specific part useful to WSL users. This
allows to trim down the image size.
Co-authored-by: Jean-Baptiste Lallement <jean-baptiste@ubuntu.com>
For the SiFive HiFive Unmatched board we create a pre-installed image using
u-boot-menu. Increase the watchdog threshold in this case too.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
With Radeon GPUs and kernel 5.19 a soft lockup was observed.
Increase the watchdog threshold.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Using numbered configuration fragments makes the order of application
easier to track
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
This allows to consolidate linux-kvm and linux-generic kernel
flavours. This brings the perfomance benefit of linux-kvm flavour to
all cloud and pre-installed images. It does trade data-safety.
LP: #2006511
Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com>
According to the EBBR specification the GPT partitions for firmware should
have attribute bit 0 (Required Partition) set.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Since version 2022.10 U-Boot SPL and U-Boot are installed onto the same partition.
Package nezha-boot0 is not needed anymore.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
RISC-V boards tend to boot slowly.
We should provide progress information when booting.
Use 'efi=debug earlycon' on the Linux command line via new file
/etc/default/grub.d/cmdline.cfg.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
We add a ubuntu user inside the image because we
want to have a operational nonroot user and also
be aligned with the other Ubuntu images.
Signed-off-by: Samir Akarioh <samir.akarioh@canonical.com>
This reverts commit 31d42bfd2ff86d175f389ee5bbed6f275597c185.
Disable the snap-preseed calls in the interest of
getting images built for the 22.10 beta. (LP: #1990884)"
The Nezha and the LicheeRV boards do not have enough memory for an initrd
with most modules. Therefore the number of included modules has to be
reduced.
Create file /etc/initramfs-tools/conf.d/modules_list.conf
to set MODULES=list.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Remove redirections of type
command &1>2
Executing the command in the background and creating and empty file '2'
was never intended.
As the messages are information only redirecting to stderr would not make
sense either.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
So that people without network access can download the package and
install it using a usb drive for example.
Signed-off-by: Alexandre Ghiti <alexandre.ghiti@canonical.com>
This reverts commit d5a7d6655f2fa653c8bc0f316613f37f58a9c2cc.
The Wifi driver package is in universe and can't be promoted in time for
the release, so revert this.
Signed-off-by: Alexandre Ghiti <alexandre.ghiti@canonical.com>
The LicheeRV Dock board comes with only 512MB of DRAM so the only difference
with a Nezha image is the fact that we have to remove
cryptsetup-initramfs package which makes the initrd too big for the
board to boot.
Signed-off-by: Alexandre Ghiti <alexandre.ghiti@canonical.com>
Current Kinetic GCE image builds are failing with the following error:
update-initramfs: Generating /boot/initrd.img-5.19.0-1004-gcp
zstd: error 25 : Write error : No space left on device (cannot write compressed block)
E: mkinitramfs failure zstd -q -1 -T0 25
Seems like after `linux-gcp` update from 5.15 to 5.19 `linux-modules` package
has gotten ~40MB larger and with that GCE image builds are over the edge wrt
available disk space in chroot.
Bumped up disk image size for amd64 to 3.5GB to match the sizes used by armhf
and generic images.
The cloud-init bug (see LP:1968873) got fixed now so using a sshd
config snippet should work now.
This partly reverts commit aa1be5eaaa1fdbb4c31104cc49e54b15f66c3343
but uses now 60-cloudimg-settings.conf instead of
10-cloudimg-settings.conf .
For now, all RISC-V hardware is SBC-like board which embed a Wifi
chipset so install wpasupplicant by default. We'll certainly split the
seeds between server and embedded hardware later.
Signed-off-by: Alexandre Ghiti <alexandre.ghiti@canonical.com>
* Fix some issues with the netboot tarballs:
- Include the signed shim (oops).
- Make the kernel path on disk and in the bootloader config match (more
oops).
- Make paths more architecture dependent as the code in grubnetXXX.efi to
probe a platform dependent path first doesn't work.
While merging the VisionFive support, we removed the installation of
u-boot-menu for the Unmatched by mistake: fix this by reinstating it.
Fixes: ce9f5caccadf ("riscv: Add support for StarFive VisionFive")
Signed-off-by: Alexandre Ghiti <alexandre.ghiti@canonical.com>
This change triggered a bug in cloud-init (see LP:1968873). cloud-init
does not recongnize sshd options set in /etc/ssh/sshd_config.d/ and
cloud-init modifies directly /etc/ssh/sshd_config which gets then
overwritten by settings from /etc/ssh/sshd_config.d/ .
This reverts commit b54d24ff3310f7ace00ab08e0dacfdc89e026f1c.
3.5G is not enough for riscv64 preinstalled as the creation of the initrd fails
with the following error:
Creating config file /etc/default/grub with new version
Processing triggers for initramfs-tools (0.140ubuntu13) ...
update-initramfs: Generating /boot/initrd.img-5.15.0-1011-generic
zstd: error 25 : Write error : No space left on device (cannot write compressed block)
E: mkinitramfs failure zstd -q -1 -T0 25
update-initramfs: failed for /boot/initrd.img-5.15.0-1011-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
installed initramfs-tools package post-installation script subprocess returned error exit status 1
Signed-off-by: Alexandre Ghiti <alexandre.ghiti@canonical.com>
The image created uses a UEFI bootflow, so we install grub for this board
only. We also need flash-kernel to install the dtb where grub can find
it.
This image is specifically architectured so that it can be installed on
a "factory" board, meaning using the u-boot firmware which was
originally implemented for Fedora, so we need the p3 partition that
embeds a uEnv.txt file to tell u-boot what/where to load next stage.
Signed-off-by: Alexandre Ghiti <alexandre.ghiti@canonical.com>
Define the image layout for the Nezha board.
The U-Boot SPL based boot0 may be installed starting in sector 16 or 256.
As sector 16 is incompatible with GPT partitioning use sector 256.
The primary U-Boot image is expected to start at sector 32800 and its
backup in sector 24576.
Cf. https://linux-sunxi.org/index.php?title=Allwinner_Nezha&oldid=24469
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Modifying directly /etc/ssh/sshd_config creates "problems" when
upgrading eg. from Focal to Jammy because the upgrade will ask the
user what to do with the modified config. To avoid that, put the
custom configuration into /etc/ssh/sshd_config.d/ so the upgrade of
openssh-server can just replace /etc/ssh/sshd_config without asking
the user.
This reverts part of a change causing regression with vmware import due to the
cdrom getting moved to SCSI while shifting controller IDs. (LP: #1970795)
Germinate doesn't take very long at all to run but downloading the
indices it operates on can take a while and nothing else in auto/config
does so not doing it every time you run "lb config" can be a real time
saver.
The code that invokes germinate already checked if the output was
already there but it was unconditionally deleted by the time control got
to that point.
LP: 1969664 tracks an issue related to the deprecation of rsa+ssh on
Jammy+ openssh server, coupled with upstream vagrant bugs, that cause
Jammy vagrant images fail to bootstrap due to ssh negotiation issues.
Moving to a different key algo from the upstream insecure key matches
Jammy's expectations, and works with older vagrant versions.
vagrant >= 2.2.16 hosts are unaffected by the issue, as an upstream
change was made. This change keep compatibility with newer vagrant
versions as well.
Readding this file per reviewer's request until CPC splits the
pipelines. Removing this file would make CPC image builds fail.
Co-authored-by: Didier Roche <didrocks@ubuntu.com>
Commit 245f7772bdb74 added code to abort the build if a snap wants to
install "core" (the 16.04 runtime). That's great but there are still
some CPC maintained image builds that use snaps based on "core". So
make it possible to continue the build if the "ALLOW_CORE_SNAP" env
variable is set.
Due to how `disk-image` file is structured, it builds BIOS and UEFI
images at the same time. However, certain images (e.g., GCE images)
require only UEFI image to be built, BIOS image is being simply
discarded. This results in longer build times.
Splitting out `disk-image-uefi` would allow images to use it instead of
`disk-image` and thus avoid building unused BIOS images.
`disk-image` now depends on `disk-image-uefi` for backward
compatibility.
The UNCONFIGURED FSTAB warning was being left in the result, the discard
option wasn't included, and the fsck flag was 0 (all in marked contrast
to the preinstalled server images).
Changes in either livecd-rootfs or ubuntu-image seem to periodically
break the transfer of the pre-allocated swapfile (copying it in such a
fashion that it winds up "with holes" and thus unable to be used as a
swapfile). Rather than fight this, just use a simple systemd service to
generate the swapfile if it doesn't exist (using fallocate to keep
things snappy).
This fixes GCE shielded VM instances integrity monitoring failures on
focal and later. Our images are built with an empty /boot/grub/grubenv
file, however after the first boot `initrdless_boot_fallback_triggered`
is set to 0. This change in `grubenv` results in integrity monitoring
`lateBootReportEvent` error.
It seems that the only thing that's checking for this `grubenv` variable
is `grub-common.service`, and it is looking specifically for a `1`
value:
if grub-editenv /boot/grub/grubenv list | grep -q
initrdless_boot_fallback_triggered=1; then echo "grub:
GRUB_FORCE_PARTUUID set, initrdless boot paniced, fallback triggered.";
fi
Unsetting this variable instead of setting it to 0 would prevent issues
with integrity monitoring.
LP: 1960537 illustrates an issue where the calls to e2fsck in the
umount_partition call are failing due to an open file handle. At this
time, we are unable to find a root cause, and it's causing many builds
to fail for CPC. Adding a sleep 30 as a workaround as the file handle
releases within that timeframe. This does not address root cause.
Currently the RISC-V preinstalled server images come with partitions that
are only 1 KiB aligned. Ext4 may use 4 KiB block size. The existing
misalignment leads to decreased performance.
Decrease the size of the loader2 partition by 34 512-byte blocks. This
results in 1 MiB alignment of the EFI and root partitions.
The remaining loader2 partition size of close to 4 MiB is still large
enough for U-Boot or a future EDK II.
Fixes: a808b28d47ec ("riscv64: build preinstalled riscv64 image with uboot SPL and CIDATA.")
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
livecd-rootfs creates non-private mounts. When building locally using
the auto/build script unmounting fails.
To unmount dev/pts it is insufficient to make the mount private. Its
parents must be private too. Change teardown_mountpoint() accordingly.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Current jammy builds fail with:
dpkg: error processing archive /var/cache/\
apt/archives/grub-common_2.04-1ubuntu48_armhf.deb (--unpack):
cannot copy extracted data for './usr/share/grub/unicode.pf2' \
to '/usr/share/grub/unicode.pf2.dpkg-new': \
failed to write (No space left on device)
It hangs during booting when upgrading hardware
version ESXi after deploying image in groovy.
(Current default version is 10)
It could be resolved by adding serial port in VM
when vm version is larger than 10.
Seriaol port1 has been configured as default so
we need to change setting serial0 as false.
As wsl is an image target of ubuntu-cpc, the base seed is hardcoded to
ubuntu-server instead of wsl one. For now, add it, as for the other
cpc images, in hooks.
A urllib.error.URLError.reason variable can either be a string or
another Exception[0]. In case it's another exception, the current code
fails because the exception is passed into send_error() which tries
call html.escape() on the Exception. That fails because the Exception
is not a string. Converting the Exception to a string fixes this.
This fixes:
AttributeError: 'TimeoutError' object has no attribute 'replace'
[0]
https://docs.python.org/3/library/urllib.error.html#urllib.error.URLError.reason
LP: 1944004 described an issue where a libc transition caused snapd
seccomp profiles to reference a path that no longer existed, leading to
permission denied errors. The committed fix for snapd then raised an
issue where running `snapd debug seeding` would present a
preseed-system-key and seed-restart-system-key due to a mismatch
between the running kernel capabilities and the profiles being loaded by
snapd. By mounting a cgroup2 type to /sys/fs/cgroup, the capabilities
match for snapd as mounted in the chroot. This is done similarly to
live-build/functions:138-140 where apparmour and seccomp actions are
mounted after updating the buildd.
Currently the uri that is passed into urllib.parse.urlparse() is not
prefixed with "http(s)://" which leads urlparse() to return a wrong
scheme/netloc/path. Currently it looks like:
ParseResult(scheme='', netloc='',
path='de.archive.ubuntu.com/ubuntu/dists/impish-backports/InRelease'
, params='', query='', fragment='')
That's wrong. The path should look like
'ubuntu/dists/impish-backports/InRelease'.
Prefixing the 'host' header with 'http://' in case it's not there does
fix the problem.
This fixes:
Traceback (most recent call last):
File "/usr/lib/python3.9/socketserver.py", line 683, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.9/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.9/socketserver.py", line 747, in __init__
self.handle()
File "/usr/lib/python3.9/http/server.py", line 427, in handle
self.handle_one_request()
File "/usr/lib/python3.9/http/server.py", line 415, in handle_one_request
method()
File "/home/tom/devel/livecd-rootfs/./magic-proxy", line 787, in do_GET
File "/home/tom/devel/livecd-rootfs/./magic-proxy", line 838, in __get_request
File "/home/tom/devel/livecd-rootfs/./magic-proxy", line 84, in get_uri
TypeError: can only concatenate str (not "NoneType") to str
Debian changelog.Debian.* files are already keept for minimized
builds. But those changelogs are from non-native .deb packages (see
man dh_installchangelogs). Native .deb packages name their changelog
just changelog.* . So keep them in a minimized build, too.
LP: #1943114
otherwise each and every layer above a layer with a kernel gets its own
initramfs, which is silly.
Copy/paste the cruft cleaning bit of lb_chroot_hacks to be run on
non-live layers.
for the live server build, i want to make a layer to install the kernel
into but do not want the layer itself to be published.
the implementation is a bit clunky but it works.
At this point all of the custom final_message is now obsolete.
Remove it, letting us instead use the default final_message.
Leave a note about the above.
groovy hangs during boot on ESXi when the version is greater than
10. Adding a serial port by default fixes this specific bug - increasing
the HW version will be for another branch.
This is because more investigation is needed into whether it is possible to
increment ddb.virtualHWVersion without disrupting Oracle VirtualBox images.
Some packages are in universe at release time then promoted to
the main pocket in -updates during the release lifecycle.
These packages should be considered by germinate when the root fs is
built (LP: #1921862)
Co-authored-by: Didier Roche <didrocks@ubuntu.com>
Initialize passwords from sources.list.
Use urllib everywhere.
This way authentication is added to all the required requests.
And incoming headers, are passed to the outgoing requests.
And all the response headers, are passed to the original client.
And all the TCP & HTTP errors are passed back to the client.
Thus should avoiding hanging requests upon failure.
Also rewrite the URI when requesting things.
This allows to use private-ppa.buildd outside of launchpad.
Signed-off-by: Dimitri John Ledkov <xnox@ubuntu.com>
armhf & arm64 images use grub. And despite disk-image &
disk-image-uefi installing all the grubs, some of the configuration is
done in the 999-cpc-fixes. Specifically removal of "quiet splash" is
done there, but not active on armhf & arm64. This results in arm
images to boot with "quiet splash".
Enable running the later portions of 999-cpc-fixes on armhf & arm64.
Drop duplicate call to update-grub, as update-grub2 is simply a
symlink to update-grub.
Add a guard around the call to reconfigure grub-pc, to only do that
when it is installed.
This makes armhf & arm64 uefi images consistent with amd64 uefi
images.
LP: #1925780
With that, the Dockerfile modifications[0] currently done externally
are done now here. That means that the created rootfs tarball can be
directly used within a Dockerfile to create a container from scratch:
FROM scratch
ADD livecd.ubuntu-oci.rootfs.tar.gz /
CMD ["/bin/bash"]
[0]
https://github.com/tianon/docker-brew-ubuntu-core/blob/master/update.sh
This is a copy of the ubuntu-base project.
Currently ubuntu-base is used as a base for the docker/OCI container
images. The rootfs tarball that is created with ubuntu-base is
published under [0]. That tarball is used in the FROM statement of the
Dockerfile as base and then a couple of modifications are done inside
of the Dockerfile[1].
The ubuntu-oci project will include the changes that are currently
done in the Dockerfile. With that:
1) a Dockerfile using that tarball will be just a 2 line thing:
FROM scratch
ADD ubuntu-hirsute-core-cloudimg-amd64-root.tar.gz /
CMD ["/bin/bash"]
2) Ubuntu has the full control about the build process of the
docker/OCI container. No external sources (like [1]) need to be
modified anymore.
3) Ubuntu can publish containers without depending on the official
dockerhub containers[2]. Currently the containers for the AWS ECR
registry[3] use as a base[4] the official dockerhub containers. That's
no longer needed because a container just needs a Dockerfile described
in 1)
When the ubuntu-oci project has the modifications from [1] included,
we'll also update [1] to use the ubuntu-oci rootfs tarball as a base
and drop the modifications done at [1].
Note: Creating a new ubuntu-oci project instead of using ubuntu-base
will make sure that we don't break users who are currently using
ubuntu-base rootfs tarballs for doing their own thing.
[0] https://partner-images.canonical.com/core/
[1]
https://github.com/tianon/docker-brew-ubuntu-core/blob/master/update.sh
[2] https://hub.docker.com/_/ubuntu
[3] https://gallery.ecr.aws/ubuntu/ubuntu
[4]
https://launchpad.net/~ubuntu-docker-images/ubuntu-docker-images/+oci/ubuntu/+recipe/ubuntu-20.04
One can call divert_grub; replace_kernel; undivert_grub. And
replace_kernel will call into force_boot_without_initramfs, which
under certain conditions can call divert_grub &
undivert_grub. Resulting in undivert_grub called twice in a row.
When undivert_grub is called twice in a row it wipes
systemd-detect-virt binary from disk, as the rm call is unguarded to
check that there is something to divert if systemd package is
installed. And if the systemd package is not installed, it does not
check that systemd-detect-virt file is in-fact what divert_grub has
created.
Add a guard to check that systemd-detect-virt is the placeholder one,
before removing it.
LP: #1902260
The case is for arch:subarch combo, not just arch alone even if
subarch is empty. Thus currently on adm64/arm64/armhf ubuntu-cpc
builds mbr image is created and then ignored, as the convert to qcow2
hook prefers the uefi image whenever available.
Skipping building these correctly, should speed up the build a little
bit and use slightly less disk space.
shim-signed depends on grub-efi-amd64-signed, which in turn has
alternative depends on either `grub-efi-amd64 | grub-pc`. However to
support booting with either via shim&signed-grub and BIOS, the choice
must be made to install grub-pc, not grub-efi-amd64.
This makes images consistent with Ubuntu Deskop, Live Server, buildd
bootable images; all of which already do install grub-pc and
shim-signed.
LP: #1901906
vmtools version in vmdk header (LP: #1893898)
LP: #1893898 describes missing vmtools version from the vmdk headers.
The version should be added as ddb.toolsVersion = "2147483647" however
the sed was no longer replacing a ddb.comment field with the tools
version. Rather than subbing ddb.comment with toolsVersion, this commit
deletes ddb.comment (which the comment mentions could cause errors),
and adds the correct value. There was no visibility into the descriptor
during hook creation, so debug statements were added. This allows us to
quickly verify in the logs that bad statements are removed (the possibly
offending commetns), as well as ensuring that the toolsVersion is added
MP: https://code.launchpad.net/~jchittum/livecd-rootfs/+git/livecd-rootfs/+merge/392401
When desktop-preinstalled image options were added in
38157b37487d244b27af33f7863e6b15253c8f94, for the raspi subarch, the
options listed there were not scoped for raspi subarch. This results
in those options getting also applied for the HYPERV
ubuntu:desktop-preinstalled image.
Thus scope the newly added options under raspi subarch case only.
Regression introduced in 38157b37487d244b27af33f7863e6b15253c8f94 when
desktop-preinstalled code branch was added, it dropped addint
ubuntu-desktop task. Instead it added ubuntu-desktop-raspi task, only
for the raspi subarch, which depends on ubuntu-desktop. But the hyperv
case, now ended up without ubuntu-desktop task.
It looks like introduction of "desktop-preinstalled" assumed, that it
is for raspi only, when in fact that code path now started to be used
for hyperv gallery image too.
The CPC build hooks for amd64 incorrectly attempt to install shim-signed
in addition to grub-efi-amd64 and grub-pc. These latter two packages
conflict with each other. Instead shim-signed should install whatever
packages are required.
Additionally, this will ensure that autoremove is run after installing
anything in the CPC build hooks. This is done to avoid shipping images
that include packages that are autoremovable. This will clean-up as
packages are installed and detect any breakage at build time.
Multipass on Mac OS X requires standalone kernel and initrd artifacts
to boot.
Also call update-initramfs on all installed kernels. We only have one
kernel installed, so we don't need to specify an explicit version.
There was a question on if the comment removals in the `sed` were
required. The comments (`#`) are created by vmdk-stream-converter and
seem to cause no issues. `ddb.comment` is no longer being written by the
tool anymore. Moved the check earlier to ensure the new header isn't too
large before running truncate (otherwise it may be too long, and we
remove bits we want)
LP: #1893898 describes missing vmtools version from the vmdk headers.
The version should be added as ddb.toolsVersion = "2147483647" however
the sed was no longer replacing a ddb.comment field with the tools
version. Rather than subbing ddb.comment with toolsVersion, this commit
deletes ddb.comment (which the comment mentions could cause errors),
and adds the correct value. There was no visibility into the descriptor
during hook creation, so debug statements were added. This allows us to
quickly verify in the logs that bad statements are removed (the possibly
offending commetns), as well as ensuring that the toolsVersion is added
virtualbox-guest-utils kernel modules is included in linux-modules
starting in kernel 5.4.0-33 in focal-updates. The vagrant hook also
explicit installed virtualbox-guest-utils. An error occurred with the
version installed from the archives, however, with the inclusion in
linux-modules, there's no need to explicitly install
virtualbox-guest-utils. Removes the code for the explicit install.
MOUNTPOINT_BACKUP_SOURCE_LIST is exposed when you call
setup_mountpoint. Consumers can use this variable if they need to
explicitly change something in sources.list wihout relying on the name
livecd-rootfs chooses.
Original fix proposed by Stanislav German-Evtushenko (giner)
CPC Ubuntu cloud images default to enabling a serial console connection
via the kernel commandline option `console=ttyS0`. Many clouds support
the serial connection, and utilize it for debugging purposes. Virtualbox
supports the serial connection as well. In Bionic and earlier images,
Vagrant boxes created a serial log file in the directory of the
Vagrantfile by default. However this is not standard behaviour for
Vagrant images, and so it was removed in Eoan onwards.
Starting in Eoan, there were reports of image booting slowdown (1874453
is a single example). After testing, it was determined that the serial
connection starting, without a device attached, was the cause of the
slow down. However, we did not want to revert to the old functionality
of creating a file. Much thanks to <giner> for providing the Ruby syntax
for sending to File::NULL.
This option will not create a local file, however, the default
Vagrantfile configuration is overwritable via a users Vagrantfile. The
original syntax for creating a file local to the users Vagrantfile has
been included as an example.
These introduced a regression for ppc64el and needs more time to bake.
This reverts commits 1deb0c68e8be6b06190402e32292c3c55134eee2 &
6dbb30f53bd3f2086021abdecaee04645c9ccda0.
* "ubuntu-cpc: Fix ppc64el grub console update"
* "ubuntu-cpc: Disable boot splash in all cloud images (LP: #1725358)"
The commit 6dbb30f5 (2.682) which disabled boot splash for all cloud
images introduced an error in the ppc64el hook. This patch corrects the
name of the variable that contains grub console overrides. The error
seen during testing was
'disk-image-ppc64el.binary: line 44: CONSOLES: unbound variable'
and this was due to a typo.
When trying to debug an issue on ARM64 it was reported that it was
quite difficult to debug because of control codes on the console from
the splash.
For cloud image there is a chroot customization the drops 'quiet splash'
but this is only applied to amd64. It hasn't made it into other
architectures because they don't have grub by default in the chroot.
However, when we get into binary hook for the uefi disk image and it's
derivatives grub is installed and this includes architectures that were
skipped in the chroot hook.
This patch changes the cpc-fixes chroot hook to add a cloud-images
grub config with basic overrides, including dropping the boot splash,
for all architectures. For images that never get grub installed this
addition is harmless and small while ensuring that the grub experience
is consistent for images that have grub. The configuration of console
devices as hard-coded remains arch specific.
Handle seeded lxd snap with channel name for ubuntu-cpc:minimized
The seed now specifies the lxd snap in focal as
'lxd=4.0/stable/ubuntu-20.04' which doesn't match the expectations of
the code with looks for lxd as the only snap in the seed for minimized
images. This same pattern will be used in groovy near/after release.
This patch updates the pattern to accept 'lxd' or 'lxd=*'.
MP: https://code.launchpad.net/~rcj/livecd-rootfs/+git/livecd-rootfs/+merge/388320
The seed now specifies the lxd snap in focal as
'lxd=4.0/stable/ubuntu-20.04' which doesn't match the expectations of
the code with looks for lxd as the only snap in the seed for minimized
images. This patch updates the pattern to accept 'lxd' or 'lxd=*'.
In v2.672 the default boot behavior of cloud images changed:
- Prior to v2.672, cloud images with the linux-generic kernel attempt
to boot without an initramfs, would fail, and then retry with an
initramfs.
- After v2.672, cloud images with the linux-generic kernel boot with
an initramfs on the first try.
While the behavior is different between the two, they both result in
an instance that has booted with an initramfs. To ensure the changes
in v2.672 do not regress, we need an automated way to check if we are
attempting to boot without an initramfs and failing.
With this change, when we attempt to boot with an initramfs and fail,
initrdless_boot_fallback_triggered is set to non-zero in the grubenv.
This value can be checked after boot by looking in /boot/grub/grubenv
or by using the grub-editenv list command.
Builds in LP with the Xenial kernel were happy with the recursive mount of
/sys inside the chroot while performing snap-preseeding but autopkgtests
with the groovy kernel failed. With the groovy kernel the build was
unable to unmount sys/kernel/slab/*/cgroup/* (Operation not permitted).
This patch mounts /sys and /sys/kernel/security in the chroot in the
same way we've added for binary hooks. This provides the paths under
/sys needed for snap-preseed while avoiding issues unmounting other
paths.
The snap-preseed command can do a number of things during the build
that are currently performed at first boot (apparmor profiles, systemd
unit generation, etc). This patch adds a call to reset the seeding and
apply these optimizations when adding a seeded snap. As a prerequisite
to calling snap-preseed we need to make /dev/mem available as well as
mounts from the host to perform this work, so those are also added here.
I recently pulled initramfs logic out of the base build hook, and
dropped that into the `replace_kernel` function. Any cloud image that
does not leverage the generic virtual kernel was expected to call
`replace_kernel` to pull in a custom kernel. That function will
disable initramfs boot for images that use a custom kernel.
Minimal cloud images on amd64 use the linux-kvm kernel, but the build
hook does not utilize the `replace_kernel` function. Instead, the
kernel flavor is set in `auto/config`. I pulled that logic out of
`auto/config` and am now calling `replace_kernel` in the build hook.
I also moved a call to generate the package list so that it will pick
up the change to the linux-kvm kernel.
snap_name[/classic]=track/risk/branch is now the supported snap name
specification, which allows to specify the full default track and
optional classic confinemnt.
Supporting such specification in the seedtext allows one to specify a
better default channel. For example, this will allow lxd to switch
from latest/stable/ubuntu-20.04 to 4.0/stable/ubuntu-20.04 as 4.0 is
the LTS track matching 20.04 support timeframe.
LP: #1882374
Initramfs-less boot, which is a boot optimization, should only be
applied where we know it could work for users and provide an improved
boot boot experience; images with custom kernels are candidates for
that.
Generic cloud images with the linux-generic kernel are not able to
boot without an initramfs. Previously, these images attempted to boot
without an initramfs, would fail, and then retry with an initramfs.
This slows the boot and is confusing behavior.
It was reported and confirmed in LP bug #1875400
(https://bugs.launchpad.net/cloud-images/+bug/1875400) that on the public
KVM cloud image there exists a large list of packages marked for auto-removal.
This should never be the case on a released cloud image.
These packages are marked for auto-removal because in the KVM image binary hook
we removed both initramfs-tools and busybox-initramfs packages. Due to package
dependencies this also removed:
busybox-initramfs* cloud-initramfs-copymods* cloud-initramfs-dyn-netconf*
cryptsetup-initramfs* initramfs-tools* initramfs-tools-core* multipath-tools*
overlayroot* sg3-utils-udev* ubuntu-server*
But it did not remove all the packages that the above list depended on.
This resulted in all those packages being marked for auto-removal because they
were not manually installed nor did they have any manually installed packages
that depended on them.
The removal of initramfs-tools and busybox-initramfs was to avoid the
generation of initramfs in images that should boot initramfsless.
This requirement is obsolete now because the initramfsless boot handling
is now handled via setting GRUB_FORCE_PARTUUID in /etc/default/grub.d/40-force-partuuid.cfg.
In test images I have verified that GRUB_FORCE_PARTUUID is set and that
boot speeds have not regressed.
LP: #1875400
The base image is built with packages from the release pocket; however,
we want the latest from updates and security. Those pockets are already
enabled, we just need to perform an upgrade to pull in the latest
packages.
Back in 2017 some code was added to ignore failures tearing down loop
devices. But debugging that growpart race on cloud images made me (very)
aware of a potential cause of the race: doing something like zerofree on
a device will cause udev scripts to run, and if they are still running
by the time kpartx is called, you would expect the kpartx -d to fail. So
lets see if a udevadm settle helps, and get rid of one of the "sometimes
this fails but we don't know why" comments...
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
