vmtools version in vmdk header (LP: #1893898)
LP: #1893898 describes missing vmtools version from the vmdk headers.
The version should be added as ddb.toolsVersion = "2147483647" however
the sed was no longer replacing a ddb.comment field with the tools
version. Rather than subbing ddb.comment with toolsVersion, this commit
deletes ddb.comment (which the comment mentions could cause errors),
and adds the correct value. There was no visibility into the descriptor
during hook creation, so debug statements were added. This allows us to
quickly verify in the logs that bad statements are removed (the possibly
offending commetns), as well as ensuring that the toolsVersion is added
MP: https://code.launchpad.net/~jchittum/livecd-rootfs/+git/livecd-rootfs/+merge/392401
Multipass on Mac OS X requires standalone kernel and initrd artifacts
to boot.
Also call update-initramfs on all installed kernels. We only have one
kernel installed, so we don't need to specify an explicit version.
LP: #1893898 describes missing vmtools version from the vmdk headers.
The version should be added as ddb.toolsVersion = "2147483647" however
the sed was no longer replacing a ddb.comment field with the tools
version. Rather than subbing ddb.comment with toolsVersion, this commit
deletes ddb.comment (which the comment mentions could cause errors),
and adds the correct value. There was no visibility into the descriptor
during hook creation, so debug statements were added. This allows us to
quickly verify in the logs that bad statements are removed (the possibly
offending commetns), as well as ensuring that the toolsVersion is added
The seed now specifies the lxd snap in focal as
'lxd=4.0/stable/ubuntu-20.04' which doesn't match the expectations of
the code with looks for lxd as the only snap in the seed for minimized
images. This patch updates the pattern to accept 'lxd' or 'lxd=*'.
In v2.672 the default boot behavior of cloud images changed:
- Prior to v2.672, cloud images with the linux-generic kernel attempt
to boot without an initramfs, would fail, and then retry with an
initramfs.
- After v2.672, cloud images with the linux-generic kernel boot with
an initramfs on the first try.
While the behavior is different between the two, they both result in
an instance that has booted with an initramfs. To ensure the changes
in v2.672 do not regress, we need an automated way to check if we are
attempting to boot without an initramfs and failing.
With this change, when we attempt to boot with an initramfs and fail,
initrdless_boot_fallback_triggered is set to non-zero in the grubenv.
This value can be checked after boot by looking in /boot/grub/grubenv
or by using the grub-editenv list command.
Builds in LP with the Xenial kernel were happy with the recursive mount of
/sys inside the chroot while performing snap-preseeding but autopkgtests
with the groovy kernel failed. With the groovy kernel the build was
unable to unmount sys/kernel/slab/*/cgroup/* (Operation not permitted).
This patch mounts /sys and /sys/kernel/security in the chroot in the
same way we've added for binary hooks. This provides the paths under
/sys needed for snap-preseed while avoiding issues unmounting other
paths.
I recently pulled initramfs logic out of the base build hook, and
dropped that into the `replace_kernel` function. Any cloud image that
does not leverage the generic virtual kernel was expected to call
`replace_kernel` to pull in a custom kernel. That function will
disable initramfs boot for images that use a custom kernel.
Minimal cloud images on amd64 use the linux-kvm kernel, but the build
hook does not utilize the `replace_kernel` function. Instead, the
kernel flavor is set in `auto/config`. I pulled that logic out of
`auto/config` and am now calling `replace_kernel` in the build hook.
I also moved a call to generate the package list so that it will pick
up the change to the linux-kvm kernel.
It was reported and confirmed in LP bug #1875400
(https://bugs.launchpad.net/cloud-images/+bug/1875400) that on the public
KVM cloud image there exists a large list of packages marked for auto-removal.
This should never be the case on a released cloud image.
These packages are marked for auto-removal because in the KVM image binary hook
we removed both initramfs-tools and busybox-initramfs packages. Due to package
dependencies this also removed:
busybox-initramfs* cloud-initramfs-copymods* cloud-initramfs-dyn-netconf*
cryptsetup-initramfs* initramfs-tools* initramfs-tools-core* multipath-tools*
overlayroot* sg3-utils-udev* ubuntu-server*
But it did not remove all the packages that the above list depended on.
This resulted in all those packages being marked for auto-removal because they
were not manually installed nor did they have any manually installed packages
that depended on them.
The removal of initramfs-tools and busybox-initramfs was to avoid the
generation of initramfs in images that should boot initramfsless.
This requirement is obsolete now because the initramfsless boot handling
is now handled via setting GRUB_FORCE_PARTUUID in /etc/default/grub.d/40-force-partuuid.cfg.
In test images I have verified that GRUB_FORCE_PARTUUID is set and that
boot speeds have not regressed.
LP: #1875400
Vagrant images were previously put at 10G, but this was a regression
from Trusty, in which they were 40G. This made it a tough sell for
users to upgrade if they were using a Ubuntu desktop experience.
This change does not impact disk usage as Vagrant with the virtualbox
provider dynamically allocates space with the VMDK. On a test system,
the VMDK took up 1.1G of disk space according to df, and after
creating a 2G file in Vagrant, the VMDK grew to 3.1G.
Therefore, users who are running on a system with little free space will
not see adverse effects if they upgrade to a new vagrant image
With the removal of snap-tool failures are seen in image builds that do
not have the 'core' snap included by the seed. This is the case for the
minimized subproject of the ubuntu-cpc project where lxd/core is removed.
In that subproject, any binary hook which adds a snap that is based
on 'core' will not add 'core' and fail 'snap debug validate-seed'.
snap-tool included the following logic in the 'snap-tool info' when
determining snap bases:
# Have "base" initialized to something meaningful.
if self.is_core_snap():
snap_data["snap"]["base"] = ""
elif snap_data["snap"].get("base") is None:
snap_data["snap"]["base"] = "core"
The snap store does not return a base if the base is core which makes
this necessary. This patch looks for the base in 'snap info' output
and if none is found (and the snap is not snapd or core) it assumes the
base is 'core' and installs it. This restores the behavior lost in the
migration from snap-tool to snap cli.
snap-tool was added to support a deprecate cohort-key feature of the snap store.
Recent changes in snap assertions have added additional fields which snap-tool
is not retrieving. This resulted in snap install failures on first boot.
This patch removes snap-tool and returns to using the snap cli. This ensures
snap downloads will function without odd incompatibilities.
- drop ds-identify policy, not needed with improved cloud config
- drop disabling network, doesn't work with ip=
- fixup setting up the INSTALLER_ROOT mountpoint
- enable cloud-init
- make cloud-init handle the default/baked in networking configuration
- install and enable openssh-server for the installation only
- provide cloud.cfg that generates random installer user password
- disable subiquity on sclp_line0 line based console
Instead of injecting an empty resolv.conf with an includes.chroot, we'll
inject it in late with a hook. The empty resolv.conf breaks DNS early in
the build, and causes some binary hooks to fail.
Subiquity images mount /usr/lib/modules from a squashfs, which systemd
tries to unmount on shutdown, whilst they are still being in use. As
systemd-udevd kmod built-in's libkmod has modules.* files
memorymapped. This produces an warning on shutdown, flooding the
screen with messages as systemd switches to a more verbose
output. Specify LazyUnmount=yes on that mount, such that unmount call
succeeds without flooding the shutdown log.
When we encounter a failure in 'lb binary' the launchpad builders can
only surface the build output from stdout. If the binary hook failure
implicates the archive we can not determine fault without the apt
proxy log. This patch will dump the proxy log to stdout to aid in
debugging these failures.
In parallel builds where a list of image targets are provided the build
may produce binaries that are not part of the named set of targets but
are created by series dependencies. These implicitly created binaries
may be generated by multiple builds but are unused as our convention for
the ubuntu-cpc project is to only consume binaries from the explicitly
named image targets; this avoid overwriting the same object by multiple
parallel builds.
This patch adds support for a 'provides' keyword for series files. It can
be specified multiple times per series file. The field is used by the
make-hooks script to generate a list of output files created explicitly by
the named image targets. The list is saved to the "explicit_provides"
file in the hooks output directory. In the case of the "all" target
this list would be empty. This list is consumed by the "final.binary"
hook file.
This patch adds support for optional final.binary hooks in hooks.d/base
and/or hooks.d/extra. These final.binary hooks are always included as
the last hook(s) if either exist with the hook in "extra" running last.
The base/final.binary hook includes logic to parse the "explicit_provides"
file generated by the make-hooks script and remove any binary output not
explicitly specified.
Some series files named unnecessary dependencies, specifically
disk-image, to keep output of implicit artifacts consistent between
parallel builds. These unnecessary dependencies are removed in this
patch.
Source: ~davidkrauser/livecd-rootfs/+git/livecd-rootfs:hyperv-gallery-images
Modifications: update debian/changelog to merge entries and use 2.594
as the version.
Signed-off-by: Tiago Stürmer Daitx <tdaitx@gmail.com>
We are creating a derivative of disk-image, replacing the kernel with
linux-kvm, removing initramfs, and creating a qcow2 image.
Source: ~patviafore/livecd-rootfs/+git/livecd-rootfs:linux_kvm_image
Modifications: fixed conflict in debian/changelog entry and bumped
version.
Signed-off-by: Tiago Stürmer Daitx <tdaitx@gmail.com>
Configure cloud-init to look for its seed in the vFAT boot partition on
raspberry pi images; the corresponding gadget is configured to place the
user-data, meta-data, and network-config files there.
Source: ~waveform/ubuntu/+source/livecd-rootfs:cloud-init-boot
This is another thing that goes unused when there is no human console user
(and we already don't have the locales themselves present on a minimal
image).
Bug-Ubuntu: https://bugs.launchpad.net/bugs/1829333
The following targets have livecd.ubuntu-cpc.manifest (and
livecd.ubuntu-cpc.ext4) which differ in some way from the 'all'
target. They are all missing grub-efi and other modifications:
root-dir
squashfs
tarball
These targets do not depend on the 'disk-image' target. This means that
the ext4 produced will lack the uefi modifications (and any from the
disk-image target binary hooks).
Since the ext4 file is common to all builds there is a chance that a
parallel build from one of these targets could overwrite this artifact.
This patch ensures that all targets will produce consistent base output.
This avoids us having apt cache files in the final image.
For auto/build, we move the call to clean_debian_chroot to the end,
this also takes care of the caches generated by apt-get update
in case of a preinstalled-pool.
For layered, it's just a simple swap of the lines.
LP: #1826377
commit a993592 introduced an additional call to create_manifest
(and snap-seed-parse) to write binary/boot/filesystem.packages. This
caused duplicate snap lines in the qcow manifest. This is because the
live-build/auto/binary code assumes that after 'lb binary' is run the
filesystem.packages will only have debs and it calls snap-seed-parse to
add them to the file. The commit changed filesystem.packages in the
ubuntu-cpc uefi binary hook to include debs and snaps.
This patch keeps the intent of the prior patch, updating the
filesystem.packages file for the content of the uefi disk image, but
only writes a listing of debian packages to match the expected content
of filesystem.packages. The snaps will still be added in generic code
in live-build/auto/build.
When the "snap-tool" script encounters a core snap it will switch to
the "stable" channel if the channel was set to "stable/ubuntu-X.X". This
ensures that core snaps always come from one canonical source.
UbuntuStudio image builds have recently begun failing as a result of adding a
grub theme customization to their flavor, which they then try to apply by
running update-grub from their maintainer scripts. This fails with:
Setting up plymouth-theme-ubuntustudio (0.57) ...
update-alternatives: using /usr/share/plymouth/themes/ubuntustudio-logo/ubuntustudio-logo.plymouth to provide /usr/share/plymouth/themes/default.plymouth (default.plymouth) in auto mode
update-alternatives: using /usr/share/plymouth/themes/ubuntustudio-text/ubuntustudio-text.plymouth to provide /usr/share/plymouth/themes/text.plymouth (text.plymouth) in auto mode
update-initramfs: diverted by livecd-rootfs (will be called later)
/usr/sbin/grub-probe: error: cannot find a device for / (is /dev mounted?).
dpkg: error processing package plymouth-theme-ubuntustudio (--configure):
installed plymouth-theme-ubuntustudio package post-installation script subprocess returned error exit status 1
Packages providing grub themes should be able to call update-grub from their
maintainer scripts, and they should be able to be installed under
livecd-rootfs. It's surprising to me that this hasn't been a problem before
now for any flavors.
We know that grub-probe should not be called as part of an image build, so
just use our standard helper function to divert it before running the chroot
stage and undivert it after.
When the magic-proxy script could not find a valid InRelease file for the
configured timestamp, it would fall back to serving the canonical version
of it. This meant that builds would succeed, even though snap-shotting the
repository failed.
This update makes the script return HTTP 404 when an InRelease by-hash
link for a given combination of mirror, suite and timestamp cannot be
found.
Installing policyrcd-script-zg2 doesn't quite do this because of the way
that live-build installs its own temporary version of policy-rc.d. The
only remotely sensible way I can see to deal with this is to create the
symlink manually.
When the REPO_SNAPSHOT_STAMP variable is set, the auto/build script will attempt
to launch a transparent HTTP proxy on port 8080, and insert an iptables rule to
redirect all outgoing HTTP requests to this proxy.
The proxy, contained in the `magic-proxy` Python script, examines each request
and silently overrides those pointing to InRelease files or files that are
listed in InRelease files. It will instead provide the contents of the requested
file as it was at REPO_SNAPSHOT_STAMP, by downloading the corresponding asset
"by hash".
Don't install gpg or gpg-agent when building buildd images for series
earlier than bionic. (In earlier series, those packages didn't exist
and gnupg was build-essential.)
Michael Hudson-Doyle
Steve Langasek
Dimitri John Ledkov
Łukasz 'sil2100' Zemczak
Robert C Jennings
Gauthier Jolly
Cody Shepherd
Iain Lane
Joshua Powers
David Krauser
John Chittum
Pat Viafore
Dan Streetman
Philip Roche
Balint Reczey
Adam Conrad
Colin Watson
Tobias Koch
Michael Vogt
Julian Andres Klode
Tiago Stürmer Daitx
Mathieu Trudel-Lapierre
Didier Roche
Jean-Baptiste Lallement
Colin Watson