The UNCONFIGURED FSTAB warning was being left in the result, the discard
option wasn't included, and the fsck flag was 0 (all in marked contrast
to the preinstalled server images).
Changes in either livecd-rootfs or ubuntu-image seem to periodically
break the transfer of the pre-allocated swapfile (copying it in such a
fashion that it winds up "with holes" and thus unable to be used as a
swapfile). Rather than fight this, just use a simple systemd service to
generate the swapfile if it doesn't exist (using fallocate to keep
things snappy).
This fixes GCE shielded VM instances integrity monitoring failures on
focal and later. Our images are built with an empty /boot/grub/grubenv
file, however after the first boot `initrdless_boot_fallback_triggered`
is set to 0. This change in `grubenv` results in integrity monitoring
`lateBootReportEvent` error.
It seems that the only thing that's checking for this `grubenv` variable
is `grub-common.service`, and it is looking specifically for a `1`
value:
if grub-editenv /boot/grub/grubenv list | grep -q
initrdless_boot_fallback_triggered=1; then echo "grub:
GRUB_FORCE_PARTUUID set, initrdless boot paniced, fallback triggered.";
fi
Unsetting this variable instead of setting it to 0 would prevent issues
with integrity monitoring.
LP: 1960537 illustrates an issue where the calls to e2fsck in the
umount_partition call are failing due to an open file handle. At this
time, we are unable to find a root cause, and it's causing many builds
to fail for CPC. Adding a sleep 30 as a workaround as the file handle
releases within that timeframe. This does not address root cause.
Currently the RISC-V preinstalled server images come with partitions that
are only 1 KiB aligned. Ext4 may use 4 KiB block size. The existing
misalignment leads to decreased performance.
Decrease the size of the loader2 partition by 34 512-byte blocks. This
results in 1 MiB alignment of the EFI and root partitions.
The remaining loader2 partition size of close to 4 MiB is still large
enough for U-Boot or a future EDK II.
Fixes: a808b28d47 ("riscv64: build preinstalled riscv64 image with uboot SPL and CIDATA.")
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
livecd-rootfs creates non-private mounts. When building locally using
the auto/build script unmounting fails.
To unmount dev/pts it is insufficient to make the mount private. Its
parents must be private too. Change teardown_mountpoint() accordingly.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Current jammy builds fail with:
dpkg: error processing archive /var/cache/\
apt/archives/grub-common_2.04-1ubuntu48_armhf.deb (--unpack):
cannot copy extracted data for './usr/share/grub/unicode.pf2' \
to '/usr/share/grub/unicode.pf2.dpkg-new': \
failed to write (No space left on device)
It hangs during booting when upgrading hardware
version ESXi after deploying image in groovy.
(Current default version is 10)
It could be resolved by adding serial port in VM
when vm version is larger than 10.
Seriaol port1 has been configured as default so
we need to change setting serial0 as false.
As wsl is an image target of ubuntu-cpc, the base seed is hardcoded to
ubuntu-server instead of wsl one. For now, add it, as for the other
cpc images, in hooks.
LP: 1944004 described an issue where a libc transition caused snapd
seccomp profiles to reference a path that no longer existed, leading to
permission denied errors. The committed fix for snapd then raised an
issue where running `snapd debug seeding` would present a
preseed-system-key and seed-restart-system-key due to a mismatch
between the running kernel capabilities and the profiles being loaded by
snapd. By mounting a cgroup2 type to /sys/fs/cgroup, the capabilities
match for snapd as mounted in the chroot. This is done similarly to
live-build/functions:138-140 where apparmour and seccomp actions are
mounted after updating the buildd.
Debian changelog.Debian.* files are already keept for minimized
builds. But those changelogs are from non-native .deb packages (see
man dh_installchangelogs). Native .deb packages name their changelog
just changelog.* . So keep them in a minimized build, too.
LP: #1943114
otherwise each and every layer above a layer with a kernel gets its own
initramfs, which is silly.
Copy/paste the cruft cleaning bit of lb_chroot_hacks to be run on
non-live layers.
for the live server build, i want to make a layer to install the kernel
into but do not want the layer itself to be published.
the implementation is a bit clunky but it works.
At this point all of the custom final_message is now obsolete.
Remove it, letting us instead use the default final_message.
Leave a note about the above.
groovy hangs during boot on ESXi when the version is greater than
10. Adding a serial port by default fixes this specific bug - increasing
the HW version will be for another branch.
This is because more investigation is needed into whether it is possible to
increment ddb.virtualHWVersion without disrupting Oracle VirtualBox images.
Some packages are in universe at release time then promoted to
the main pocket in -updates during the release lifecycle.
These packages should be considered by germinate when the root fs is
built (LP: #1921862)
Co-authored-by: Didier Roche <didrocks@ubuntu.com>
Initialize passwords from sources.list.
Use urllib everywhere.
This way authentication is added to all the required requests.
And incoming headers, are passed to the outgoing requests.
And all the response headers, are passed to the original client.
And all the TCP & HTTP errors are passed back to the client.
Thus should avoiding hanging requests upon failure.
Also rewrite the URI when requesting things.
This allows to use private-ppa.buildd outside of launchpad.
Signed-off-by: Dimitri John Ledkov <xnox@ubuntu.com>
armhf & arm64 images use grub. And despite disk-image &
disk-image-uefi installing all the grubs, some of the configuration is
done in the 999-cpc-fixes. Specifically removal of "quiet splash" is
done there, but not active on armhf & arm64. This results in arm
images to boot with "quiet splash".
Enable running the later portions of 999-cpc-fixes on armhf & arm64.
Drop duplicate call to update-grub, as update-grub2 is simply a
symlink to update-grub.
Add a guard around the call to reconfigure grub-pc, to only do that
when it is installed.
This makes armhf & arm64 uefi images consistent with amd64 uefi
images.
LP: #1925780
With that, the Dockerfile modifications[0] currently done externally
are done now here. That means that the created rootfs tarball can be
directly used within a Dockerfile to create a container from scratch:
FROM scratch
ADD livecd.ubuntu-oci.rootfs.tar.gz /
CMD ["/bin/bash"]
[0]
https://github.com/tianon/docker-brew-ubuntu-core/blob/master/update.sh
This is a copy of the ubuntu-base project.
Currently ubuntu-base is used as a base for the docker/OCI container
images. The rootfs tarball that is created with ubuntu-base is
published under [0]. That tarball is used in the FROM statement of the
Dockerfile as base and then a couple of modifications are done inside
of the Dockerfile[1].
The ubuntu-oci project will include the changes that are currently
done in the Dockerfile. With that:
1) a Dockerfile using that tarball will be just a 2 line thing:
FROM scratch
ADD ubuntu-hirsute-core-cloudimg-amd64-root.tar.gz /
CMD ["/bin/bash"]
2) Ubuntu has the full control about the build process of the
docker/OCI container. No external sources (like [1]) need to be
modified anymore.
3) Ubuntu can publish containers without depending on the official
dockerhub containers[2]. Currently the containers for the AWS ECR
registry[3] use as a base[4] the official dockerhub containers. That's
no longer needed because a container just needs a Dockerfile described
in 1)
When the ubuntu-oci project has the modifications from [1] included,
we'll also update [1] to use the ubuntu-oci rootfs tarball as a base
and drop the modifications done at [1].
Note: Creating a new ubuntu-oci project instead of using ubuntu-base
will make sure that we don't break users who are currently using
ubuntu-base rootfs tarballs for doing their own thing.
[0] https://partner-images.canonical.com/core/
[1]
https://github.com/tianon/docker-brew-ubuntu-core/blob/master/update.sh
[2] https://hub.docker.com/_/ubuntu
[3] https://gallery.ecr.aws/ubuntu/ubuntu
[4]
https://launchpad.net/~ubuntu-docker-images/ubuntu-docker-images/+oci/ubuntu/+recipe/ubuntu-20.04
One can call divert_grub; replace_kernel; undivert_grub. And
replace_kernel will call into force_boot_without_initramfs, which
under certain conditions can call divert_grub &
undivert_grub. Resulting in undivert_grub called twice in a row.
When undivert_grub is called twice in a row it wipes
systemd-detect-virt binary from disk, as the rm call is unguarded to
check that there is something to divert if systemd package is
installed. And if the systemd package is not installed, it does not
check that systemd-detect-virt file is in-fact what divert_grub has
created.
Add a guard to check that systemd-detect-virt is the placeholder one,
before removing it.
LP: #1902260
The case is for arch:subarch combo, not just arch alone even if
subarch is empty. Thus currently on adm64/arm64/armhf ubuntu-cpc
builds mbr image is created and then ignored, as the convert to qcow2
hook prefers the uefi image whenever available.
Skipping building these correctly, should speed up the build a little
bit and use slightly less disk space.
shim-signed depends on grub-efi-amd64-signed, which in turn has
alternative depends on either `grub-efi-amd64 | grub-pc`. However to
support booting with either via shim&signed-grub and BIOS, the choice
must be made to install grub-pc, not grub-efi-amd64.
This makes images consistent with Ubuntu Deskop, Live Server, buildd
bootable images; all of which already do install grub-pc and
shim-signed.
LP: #1901906
vmtools version in vmdk header (LP: #1893898)
LP: #1893898 describes missing vmtools version from the vmdk headers.
The version should be added as ddb.toolsVersion = "2147483647" however
the sed was no longer replacing a ddb.comment field with the tools
version. Rather than subbing ddb.comment with toolsVersion, this commit
deletes ddb.comment (which the comment mentions could cause errors),
and adds the correct value. There was no visibility into the descriptor
during hook creation, so debug statements were added. This allows us to
quickly verify in the logs that bad statements are removed (the possibly
offending commetns), as well as ensuring that the toolsVersion is added
MP: https://code.launchpad.net/~jchittum/livecd-rootfs/+git/livecd-rootfs/+merge/392401
When desktop-preinstalled image options were added in
38157b3748, for the raspi subarch, the
options listed there were not scoped for raspi subarch. This results
in those options getting also applied for the HYPERV
ubuntu:desktop-preinstalled image.
Thus scope the newly added options under raspi subarch case only.
Regression introduced in 38157b3748 when
desktop-preinstalled code branch was added, it dropped addint
ubuntu-desktop task. Instead it added ubuntu-desktop-raspi task, only
for the raspi subarch, which depends on ubuntu-desktop. But the hyperv
case, now ended up without ubuntu-desktop task.
It looks like introduction of "desktop-preinstalled" assumed, that it
is for raspi only, when in fact that code path now started to be used
for hyperv gallery image too.
The CPC build hooks for amd64 incorrectly attempt to install shim-signed
in addition to grub-efi-amd64 and grub-pc. These latter two packages
conflict with each other. Instead shim-signed should install whatever
packages are required.
Additionally, this will ensure that autoremove is run after installing
anything in the CPC build hooks. This is done to avoid shipping images
that include packages that are autoremovable. This will clean-up as
packages are installed and detect any breakage at build time.
Multipass on Mac OS X requires standalone kernel and initrd artifacts
to boot.
Also call update-initramfs on all installed kernels. We only have one
kernel installed, so we don't need to specify an explicit version.
There was a question on if the comment removals in the `sed` were
required. The comments (`#`) are created by vmdk-stream-converter and
seem to cause no issues. `ddb.comment` is no longer being written by the
tool anymore. Moved the check earlier to ensure the new header isn't too
large before running truncate (otherwise it may be too long, and we
remove bits we want)
LP: #1893898 describes missing vmtools version from the vmdk headers.
The version should be added as ddb.toolsVersion = "2147483647" however
the sed was no longer replacing a ddb.comment field with the tools
version. Rather than subbing ddb.comment with toolsVersion, this commit
deletes ddb.comment (which the comment mentions could cause errors),
and adds the correct value. There was no visibility into the descriptor
during hook creation, so debug statements were added. This allows us to
quickly verify in the logs that bad statements are removed (the possibly
offending commetns), as well as ensuring that the toolsVersion is added
virtualbox-guest-utils kernel modules is included in linux-modules
starting in kernel 5.4.0-33 in focal-updates. The vagrant hook also
explicit installed virtualbox-guest-utils. An error occurred with the
version installed from the archives, however, with the inclusion in
linux-modules, there's no need to explicitly install
virtualbox-guest-utils. Removes the code for the explicit install.
MOUNTPOINT_BACKUP_SOURCE_LIST is exposed when you call
setup_mountpoint. Consumers can use this variable if they need to
explicitly change something in sources.list wihout relying on the name
livecd-rootfs chooses.
Original fix proposed by Stanislav German-Evtushenko (giner)
CPC Ubuntu cloud images default to enabling a serial console connection
via the kernel commandline option `console=ttyS0`. Many clouds support
the serial connection, and utilize it for debugging purposes. Virtualbox
supports the serial connection as well. In Bionic and earlier images,
Vagrant boxes created a serial log file in the directory of the
Vagrantfile by default. However this is not standard behaviour for
Vagrant images, and so it was removed in Eoan onwards.
Starting in Eoan, there were reports of image booting slowdown (1874453
is a single example). After testing, it was determined that the serial
connection starting, without a device attached, was the cause of the
slow down. However, we did not want to revert to the old functionality
of creating a file. Much thanks to <giner> for providing the Ruby syntax
for sending to File::NULL.
This option will not create a local file, however, the default
Vagrantfile configuration is overwritable via a users Vagrantfile. The
original syntax for creating a file local to the users Vagrantfile has
been included as an example.
These introduced a regression for ppc64el and needs more time to bake.
This reverts commits 1deb0c68e8 &
6dbb30f53b.
* "ubuntu-cpc: Fix ppc64el grub console update"
* "ubuntu-cpc: Disable boot splash in all cloud images (LP: #1725358)"
The commit 6dbb30f5 (2.682) which disabled boot splash for all cloud
images introduced an error in the ppc64el hook. This patch corrects the
name of the variable that contains grub console overrides. The error
seen during testing was
'disk-image-ppc64el.binary: line 44: CONSOLES: unbound variable'
and this was due to a typo.
When trying to debug an issue on ARM64 it was reported that it was
quite difficult to debug because of control codes on the console from
the splash.
For cloud image there is a chroot customization the drops 'quiet splash'
but this is only applied to amd64. It hasn't made it into other
architectures because they don't have grub by default in the chroot.
However, when we get into binary hook for the uefi disk image and it's
derivatives grub is installed and this includes architectures that were
skipped in the chroot hook.
This patch changes the cpc-fixes chroot hook to add a cloud-images
grub config with basic overrides, including dropping the boot splash,
for all architectures. For images that never get grub installed this
addition is harmless and small while ensuring that the grub experience
is consistent for images that have grub. The configuration of console
devices as hard-coded remains arch specific.
The seed now specifies the lxd snap in focal as
'lxd=4.0/stable/ubuntu-20.04' which doesn't match the expectations of
the code with looks for lxd as the only snap in the seed for minimized
images. This patch updates the pattern to accept 'lxd' or 'lxd=*'.
In v2.672 the default boot behavior of cloud images changed:
- Prior to v2.672, cloud images with the linux-generic kernel attempt
to boot without an initramfs, would fail, and then retry with an
initramfs.
- After v2.672, cloud images with the linux-generic kernel boot with
an initramfs on the first try.
While the behavior is different between the two, they both result in
an instance that has booted with an initramfs. To ensure the changes
in v2.672 do not regress, we need an automated way to check if we are
attempting to boot without an initramfs and failing.
With this change, when we attempt to boot with an initramfs and fail,
initrdless_boot_fallback_triggered is set to non-zero in the grubenv.
This value can be checked after boot by looking in /boot/grub/grubenv
or by using the grub-editenv list command.
Builds in LP with the Xenial kernel were happy with the recursive mount of
/sys inside the chroot while performing snap-preseeding but autopkgtests
with the groovy kernel failed. With the groovy kernel the build was
unable to unmount sys/kernel/slab/*/cgroup/* (Operation not permitted).
This patch mounts /sys and /sys/kernel/security in the chroot in the
same way we've added for binary hooks. This provides the paths under
/sys needed for snap-preseed while avoiding issues unmounting other
paths.
The snap-preseed command can do a number of things during the build
that are currently performed at first boot (apparmor profiles, systemd
unit generation, etc). This patch adds a call to reset the seeding and
apply these optimizations when adding a seeded snap. As a prerequisite
to calling snap-preseed we need to make /dev/mem available as well as
mounts from the host to perform this work, so those are also added here.
I recently pulled initramfs logic out of the base build hook, and
dropped that into the `replace_kernel` function. Any cloud image that
does not leverage the generic virtual kernel was expected to call
`replace_kernel` to pull in a custom kernel. That function will
disable initramfs boot for images that use a custom kernel.
Minimal cloud images on amd64 use the linux-kvm kernel, but the build
hook does not utilize the `replace_kernel` function. Instead, the
kernel flavor is set in `auto/config`. I pulled that logic out of
`auto/config` and am now calling `replace_kernel` in the build hook.
I also moved a call to generate the package list so that it will pick
up the change to the linux-kvm kernel.
snap_name[/classic]=track/risk/branch is now the supported snap name
specification, which allows to specify the full default track and
optional classic confinemnt.
Supporting such specification in the seedtext allows one to specify a
better default channel. For example, this will allow lxd to switch
from latest/stable/ubuntu-20.04 to 4.0/stable/ubuntu-20.04 as 4.0 is
the LTS track matching 20.04 support timeframe.
LP: #1882374
Initramfs-less boot, which is a boot optimization, should only be
applied where we know it could work for users and provide an improved
boot boot experience; images with custom kernels are candidates for
that.
Generic cloud images with the linux-generic kernel are not able to
boot without an initramfs. Previously, these images attempted to boot
without an initramfs, would fail, and then retry with an initramfs.
This slows the boot and is confusing behavior.
It was reported and confirmed in LP bug #1875400
(https://bugs.launchpad.net/cloud-images/+bug/1875400) that on the public
KVM cloud image there exists a large list of packages marked for auto-removal.
This should never be the case on a released cloud image.
These packages are marked for auto-removal because in the KVM image binary hook
we removed both initramfs-tools and busybox-initramfs packages. Due to package
dependencies this also removed:
busybox-initramfs* cloud-initramfs-copymods* cloud-initramfs-dyn-netconf*
cryptsetup-initramfs* initramfs-tools* initramfs-tools-core* multipath-tools*
overlayroot* sg3-utils-udev* ubuntu-server*
But it did not remove all the packages that the above list depended on.
This resulted in all those packages being marked for auto-removal because they
were not manually installed nor did they have any manually installed packages
that depended on them.
The removal of initramfs-tools and busybox-initramfs was to avoid the
generation of initramfs in images that should boot initramfsless.
This requirement is obsolete now because the initramfsless boot handling
is now handled via setting GRUB_FORCE_PARTUUID in /etc/default/grub.d/40-force-partuuid.cfg.
In test images I have verified that GRUB_FORCE_PARTUUID is set and that
boot speeds have not regressed.
LP: #1875400
Vagrant images were previously put at 10G, but this was a regression
from Trusty, in which they were 40G. This made it a tough sell for
users to upgrade if they were using a Ubuntu desktop experience.
This change does not impact disk usage as Vagrant with the virtualbox
provider dynamically allocates space with the VMDK. On a test system,
the VMDK took up 1.1G of disk space according to df, and after
creating a 2G file in Vagrant, the VMDK grew to 3.1G.
Therefore, users who are running on a system with little free space will
not see adverse effects if they upgrade to a new vagrant image
Seeing any snap via snap_preseed will evaluate the base for each snap
and seed the appropriate base. There should be no reason to explicitly
seed the 'core' snap and with snaps moving to 'core18' this will add
'core' without need.
The _snap_post_process function is meant to install snapd if core18 is the
only core snap installed or removed snapd if core is installed and snapd
was not explicitly installed. But the current logic in _snap_preseed
will never call _snap_post_process. $core_name will never be empty
with the existing logic, but even if it were that would only be for the
'core' snap and we'd miss using the 'core18' logic that pulls in snapd.
Given the case statement in _snap_post_process can handle doing the
right thing given any snap we can just call it unconditionally.
In the buildd image chroot, /etc/resolv.conf is a symbolic link to
a configuration file in the /run directory. A call to truncate will
modify that file, which we should not do. Instead, we want to remove
the symbolic link and replace it with an empty file.
The base image is built with packages from the release pocket; however,
we want the latest from updates and security. Those pockets are already
enabled, we just need to perform an upgrade to pull in the latest
packages.
Back in 2017 some code was added to ignore failures tearing down loop
devices. But debugging that growpart race on cloud images made me (very)
aware of a potential cause of the race: doing something like zerofree on
a device will cause udev scripts to run, and if they are still running
by the time kpartx is called, you would expect the kpartx -d to fail. So
lets see if a udevadm settle helps, and get rid of one of the "sometimes
this fails but we don't know why" comments...
With the removal of snap-tool failures are seen in image builds that do
not have the 'core' snap included by the seed. This is the case for the
minimized subproject of the ubuntu-cpc project where lxd/core is removed.
In that subproject, any binary hook which adds a snap that is based
on 'core' will not add 'core' and fail 'snap debug validate-seed'.
snap-tool included the following logic in the 'snap-tool info' when
determining snap bases:
# Have "base" initialized to something meaningful.
if self.is_core_snap():
snap_data["snap"]["base"] = ""
elif snap_data["snap"].get("base") is None:
snap_data["snap"]["base"] = "core"
The snap store does not return a base if the base is core which makes
this necessary. This patch looks for the base in 'snap info' output
and if none is found (and the snap is not snapd or core) it assumes the
base is 'core' and installs it. This restores the behavior lost in the
migration from snap-tool to snap cli.
snap-tool was added to support a deprecate cohort-key feature of the snap store.
Recent changes in snap assertions have added additional fields which snap-tool
is not retrieving. This resulted in snap install failures on first boot.
This patch removes snap-tool and returns to using the snap cli. This ensures
snap downloads will function without odd incompatibilities.
- drop ds-identify policy, not needed with improved cloud config
- drop disabling network, doesn't work with ip=
- fixup setting up the INSTALLER_ROOT mountpoint
- enable cloud-init
- make cloud-init handle the default/baked in networking configuration
- install and enable openssh-server for the installation only
- provide cloud.cfg that generates random installer user password
- disable subiquity on sclp_line0 line based console
Instead of injecting an empty resolv.conf with an includes.chroot, we'll
inject it in late with a hook. The empty resolv.conf breaks DNS early in
the build, and causes some binary hooks to fail.
Subiquity images mount /usr/lib/modules from a squashfs, which systemd
tries to unmount on shutdown, whilst they are still being in use. As
systemd-udevd kmod built-in's libkmod has modules.* files
memorymapped. This produces an warning on shutdown, flooding the
screen with messages as systemd switches to a more verbose
output. Specify LazyUnmount=yes on that mount, such that unmount call
succeeds without flooding the shutdown log.
When we encounter a failure in 'lb binary' the launchpad builders can
only surface the build output from stdout. If the binary hook failure
implicates the archive we can not determine fault without the apt
proxy log. This patch will dump the proxy log to stdout to aid in
debugging these failures.
As discussed and explained to Jibel and Didier on 17th of June, based
on canary image build logs, it showed clearly that lb controlled
initrd was only built once at the very first layer by
chroot_hacks. However, that is the wrong layer to build the final
initrd at, as at this point casper is not present yet and is not part
of the build.
Thus insure that chroot_hacks only runs at the live layer.
Ideally a subset of chroot_hacks should run on every layer, as each
layer should be squeaky clean, and most of layers without
initrds. However, jibel & didrocks are still implementing requested
patches to unbreak layer images and make each layer smaller. Hence
this minimal portion of the overall required work.
Signed-off-by: Dimitri John Ledkov <xnox@ubuntu.com>
When getting the list of snaps to include on an image via germinate, we
process the snaps in alphabetical order. Currently we seed several
packages `gnome-foo` and also `gtk-common-themes`, the default provider
of `gtk-3-themes`. Since `gtk-common-themes` is alphabetically after
`gnome-foo`, the `seed.yaml` we generate is invalid when we are part way
through generating.
What we really care about, though, is not that the `seed.yaml` is always
valid at every step, but it is that it is valid at the *end* of seeding.
So for the germinate case, let's defer validation to happen once at the
end. Other callsites of `snap_preseed`, where callers are careful to
seed snaps in the right order, are unaffected by this.
Chroot deconfiguration should happen in the opposite order from chroot
configuration, so, for symmetry, undivert_grub should be called between
"lb chroot_debianchroot remove" and "lb chroot_sysfs remove".
The livecd.ubuntu-cpc.ext4 that is present in each build (plus kernel
and initrd) are not renamed from /build/binary/boot/filsystem.ext4
and friends until after the binary hooks are run, so this patch moves
from trying to perform this cleanup in a binary hook. Now the cleanup
will be run at the end of live-build/binary for the ubuntu-cpc project.
In parallel builds where a list of image targets are provided the build
may produce binaries that are not part of the named set of targets but
are created by series dependencies. These implicitly created binaries
may be generated by multiple builds but are unused as our convention for
the ubuntu-cpc project is to only consume binaries from the explicitly
named image targets; this avoid overwriting the same object by multiple
parallel builds.
This patch adds support for a 'provides' keyword for series files. It can
be specified multiple times per series file. The field is used by the
make-hooks script to generate a list of output files created explicitly by
the named image targets. The list is saved to the "explicit_provides"
file in the hooks output directory. In the case of the "all" target
this list would be empty. This list is consumed by the "final.binary"
hook file.
This patch adds support for optional final.binary hooks in hooks.d/base
and/or hooks.d/extra. These final.binary hooks are always included as
the last hook(s) if either exist with the hook in "extra" running last.
The base/final.binary hook includes logic to parse the "explicit_provides"
file generated by the make-hooks script and remove any binary output not
explicitly specified.
Some series files named unnecessary dependencies, specifically
disk-image, to keep output of implicit artifacts consistent between
parallel builds. These unnecessary dependencies are removed in this
patch.
This is apparently still required; without it snapd gives an error
message like:
stateengine.go:102: state ensure error: devicemgr: cannot proceed without seeding "core"
We are creating a derivative of disk-image, replacing the kernel with
linux-kvm, removing initramfs, and creating a qcow2 image.
Source: ~patviafore/livecd-rootfs/+git/livecd-rootfs:linux_kvm_image
Modifications: fixed conflict in debian/changelog entry and bumped
version.
Signed-off-by: Tiago Stürmer Daitx <tdaitx@gmail.com>
Configure cloud-init to look for its seed in the vFAT boot partition on
raspberry pi images; the corresponding gadget is configured to place the
user-data, meta-data, and network-config files there.
Source: ~waveform/ubuntu/+source/livecd-rootfs:cloud-init-boot
This is another thing that goes unused when there is no human console user
(and we already don't have the locales themselves present on a minimal
image).
Bug-Ubuntu: https://bugs.launchpad.net/bugs/1829333
The addition of disk-image to series files in a prior commit required
some explanation. Without comment support in series files that was
not possible. This patch adds support for comments and adds those
comments as well.
The following targets have livecd.ubuntu-cpc.manifest (and
livecd.ubuntu-cpc.ext4) which differ in some way from the 'all'
target. They are all missing grub-efi and other modifications:
root-dir
squashfs
tarball
These targets do not depend on the 'disk-image' target. This means that
the ext4 produced will lack the uefi modifications (and any from the
disk-image target binary hooks).
Since the ext4 file is common to all builds there is a chance that a
parallel build from one of these targets could overwrite this artifact.
This patch ensures that all targets will produce consistent base output.
This avoids us having apt cache files in the final image.
For auto/build, we move the call to clean_debian_chroot to the end,
this also takes care of the caches generated by apt-get update
in case of a preinstalled-pool.
For layered, it's just a simple swap of the lines.
LP: #1826377
commit a993592 introduced an additional call to create_manifest
(and snap-seed-parse) to write binary/boot/filesystem.packages. This
caused duplicate snap lines in the qcow manifest. This is because the
live-build/auto/binary code assumes that after 'lb binary' is run the
filesystem.packages will only have debs and it calls snap-seed-parse to
add them to the file. The commit changed filesystem.packages in the
ubuntu-cpc uefi binary hook to include debs and snaps.
This patch keeps the intent of the prior patch, updating the
filesystem.packages file for the content of the uefi disk image, but
only writes a listing of debian packages to match the expected content
of filesystem.packages. The snaps will still be added in generic code
in live-build/auto/build.
ubuntu-cdimage/debian-cd/tools/add_live_filesystem helpfully adds a
'filesystem.' prefix to the squashfs, hence the name of the file
didn't match what's on disk.
grub-probe must not be called during image build so grub is diverted. In
multilayer image the chroot is always the tip of the filesystem, so we
divert grub at the beginning of each pass and undivdert it as the end.
For flat images, it's diverted just before building the chroot and
undiverted after.
UbuntuStudio image builds have recently begun failing as a result of adding a
grub theme customization to their flavor, which they then try to apply by
running update-grub from their maintainer scripts. This fails with:
Setting up plymouth-theme-ubuntustudio (0.57) ...
update-alternatives: using /usr/share/plymouth/themes/ubuntustudio-logo/ubuntustudio-logo.plymouth to provide /usr/share/plymouth/themes/default.plymouth (default.plymouth) in auto mode
update-alternatives: using /usr/share/plymouth/themes/ubuntustudio-text/ubuntustudio-text.plymouth to provide /usr/share/plymouth/themes/text.plymouth (text.plymouth) in auto mode
update-initramfs: diverted by livecd-rootfs (will be called later)
/usr/sbin/grub-probe: error: cannot find a device for / (is /dev mounted?).
dpkg: error processing package plymouth-theme-ubuntustudio (--configure):
installed plymouth-theme-ubuntustudio package post-installation script subprocess returned error exit status 1
Packages providing grub themes should be able to call update-grub from their
maintainer scripts, and they should be able to be installed under
livecd-rootfs. It's surprising to me that this hasn't been a problem before
now for any flavors.
We know that grub-probe should not be called as part of an image build, so
just use our standard helper function to divert it before running the chroot
stage and undivert it after.
Installing policyrcd-script-zg2 doesn't quite do this because of the way
that live-build installs its own temporary version of policy-rc.d. The
only remotely sensible way I can see to deal with this is to create the
symlink manually.
Remove wants from local-fs.target, add wants from the uuid device instead.
Tested by applying these changes from pre-pivot-root by modifying
/root files in place whilst booting with break=bottom.
Util-linux 2.33 fixed mount --rbind --make-rslave which did not pass MS_REC
with MS_BIND and livecd-rootfs did only --bind --make-rslave effectively with
prior mount versions.
While mount --rbind --rslave are properly passed the flags to mount()
unmounting did not work cleanly with --make-rslave.
To clearly stop propagation of umount, --make-private is used instead of
--make-rslave and it is always set before umounts. Umount -R is replaced
with a simple umount since submounts are tore down in teardown_mountpoint()
earlier.
LP: #1813730
When the REPO_SNAPSHOT_STAMP variable is set, the auto/build script will attempt
to launch a transparent HTTP proxy on port 8080, and insert an iptables rule to
redirect all outgoing HTTP requests to this proxy.
The proxy, contained in the `magic-proxy` Python script, examines each request
and silently overrides those pointing to InRelease files or files that are
listed in InRelease files. It will instead provide the contents of the requested
file as it was at REPO_SNAPSHOT_STAMP, by downloading the corresponding asset
"by hash".
Adds a function reduce_pass_size which removes duplicated files between
layers. A duplicate file is a file with same path, size, mode, owners
and content on 2 related layers.
We don't want to regress the ubuntu desktop traditional image when
livecd-rootfs lands uncordinated with ubuntu-cdimage & debian-cd.
Decoupling enables us to:
- land livecd-rootfs (no change for ubuntu-desktop project,
ubuntu-desktop:ubiquity-ng can be tested once the 2 others branches
lands)
- land ubuntu-cdimage & debian-cd for publishing any arbitrary squashfs
names
- finally, revert that commit and land it in livecd-rootfs to transform
the traditional ubuntu-desktop image to a layered approach.
Fixed prefix for passes to not conflict with global prefix
Set layer to none for layer without language packs
Copy size and manifest to the target directory
Make sure the diff always diffes existing manifests
lb_chroot_includes now takes the name of the pass as first argument and
will includes files for the corresponding pass from the directory
include.chroot.<pass>
Default Ubuntu desktop to the layered format
Remove the requirement on setting IMAGEFORMAT to layered to build a
layered image and relies on PASSED instead.
Don't install gpg or gpg-agent when building buildd images for series
earlier than bionic. (In earlier series, those packages didn't exist
and gnupg was build-essential.)
- Do not run apt-get update (which can bring in package lists if we are
unlikely wrt publisher schedules).
- Run apt-get clean to clear out downloaded debs of curtin/casper and
dependencies.
- Do not install user-setup.
- Use the core snap from the base filesystem if present.
This patch currently only applies to the "ubuntu-cpc" project.
More and more logic has been going into the hook scripts to decide under which conditions they should run or not. As we are moving to parallelized builds of image sets, this will get even more complicated. Base hooks will have to know which image sets they belong to and modification of the dependency chain between scripts will become more complicated and prone to errors, as the number of image sets grows.
This patch introduces explicit ordering and dependency handling for scripts through the use of `series` files and an explicit syntax for dependency specification.
2.02+dfsg1-5ubuntu5 grub will automatically fall back to booting with an
initrd if one is available, so we can now always attempt initramfsless
boot for cloud images which almost always works and causes only minimal
increase in boot speed for cases where it does not.
The generic assertion will be put in place if there are preseeded snaps.
A hook that wants to add its own model should be able to overwrite
the generic assertion. This patch allows a hook to continue unchanged
calling snap_prepare_assertions with a custom brand/model. The patch
will replace the generic brand/model if a custom brand/model is specified.
Prior to this patch the generic brand/model would remain in place.
We don't want the symlink to /run/systemd/resolve/stub-resolv.conf, and
launchpad-buildd will inject its own version if it doesn't get confused by a
symlink.
Dave Jones
Ivan Kapelyukhin
John Chittum
Brian Murray
Thomas Bechtold
Michael Hudson-Doyle
Steve Langasek
William Wilson
Łukasz 'sil2100' Zemczak
Heinrich Schuchardt
Jess Jang
Erich Eickmeyer
Didier Roche
Sebastien Bacher
Dan Bungert
Jason C. McDonald
Ethan Hsieh
Chloé Smith
Jean-Baptiste Lallement
Dimitri John Ledkov
Gauthier Jolly
Cody Shepherd
Iain Lane
Jinming Wu, Patrick
Robert C Jennings
Joshua Powers
David Krauser
Pat Viafore
Stanislav German-Evtushenko
Balint Reczey
Philip Roche
Adam Conrad
Colin Watson
David Krauser
Michael Vogt
Tobias Koch
Tiago Stürmer Daitx
Julian Andres Klode
Jean-Baptiste Lallement
Colin Watson
Steve Langasek
Aleksandr Bogdanov
Colin Watson
Julian Andres Klode
Daniel Watkins
Mathieu Trudel-Lapierre
Simon Quigley