patch create_manifest to produce an sbom when called by an ubuntu-cpc
project. Patch all the ubuntu-cpc hooks and series files to include the
newly generated manifests, filelists, and sboms. Generates a number of
new artifacts in the builds. the snap utilized, cpc-sbom, is an open
source repo and a provided via a hidden snap. there is no intention of
publisizing the snap or how we generate sboms, however partners require
the ability to audit if required.
defensively checks if the snap is already installed, in the case of
multiple hooks being called in a single build (thus sharing a build
host), and only if called in an ubuntu-cpc project.
(cherry picked from commit 7c7b7df89dc96169db1f255d6bba901ebb63a43c)
feat(apparmor): Add kernel apparmor check to snap validation (LP: #2052789)
For jammy and later, snap validation verifies that the kernel
version matches the livecd-rootfs version, if available. This
change bring focal in line with that paradigm. This is necessary
due to the linux-$CLOUD-5.15 kernels requiring a different
apparmor feature set that generic.
feat: add 5.15 apparmor directory (LP: #2052789)
After the kernel roll to linux-gcp-5.15-5.15.0-1051.59_20.04.1,
basic_ubuntu::test_snap_preseed_optimized began failuring due to
a preseed mismatch. This change adds a 5.15 apparmor configuration
to the focal branch.
MP: https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/460323
For jammy and later, snap validation verifies that the kernel
version matches the livecd-rootfs version, if available. This
change bring focal in line with that paradigm. This is necessary
due to the linux-$CLOUD-5.15 kernels requiring a different
apparmor feature set that generic.
(cherry picked from commit b2f25256707373537ce6c6f37fa5d456f1958edc)
After the kernel roll to linux-gcp-5.15-5.15.0-1051.59_20.04.1,
basic_ubuntu::test_snap_preseed_optimized began failuring due to
a preseed mismatch. This change adds a 5.15 apparmor configuration
to the focal branch.
(cherry picked from commit 76628691f5e584bde009f71d05c2057a624445d5)
Commit 3b2eeb017153cbb wrongly backported a change to not modify
/etc/ssh/sshd_config . The correct fix from ubuntu/master is
3b2eeb017153c where the file is named 60-cloudimg-settings.conf
instead of 10-cloudimg-settings.conf.
This fixes problems with cloud-init which does write
50-cloud-init.conf which should have higher priority than the provided
file from the image.
(cherry picked from commit 434b21e2023c3c3e5261fbd78e2eefee9aac5b1e)
Modifying directly /etc/ssh/sshd_config creates "problems" when
upgrading eg. from Focal to Jammy because the upgrade will ask the
user what to do with the modified config. To avoid that, put the
custom configuration into /etc/ssh/sshd_config.d/ so the upgrade of
openssh-server can just replace /etc/ssh/sshd_config without asking
the user.
(cherry picked from commit b54d24ff3310f7ace00ab08e0dacfdc89e026f1c)
LP: 2034253 and LP: 2027686 both deal with buildd vm images failing to
boot when removing `--removable` and the stanzas copying EFI around. We
need to remove those stanzas for launchpad builder compatibility. even
though focal and jammy weren't failing, keeping everything aligned is
important. LP: 2034253 further showed that GRUB_DISTRIBUTOR being set by
default in grub requires lsb_release, which isn't in buildd images.
That's the root of why removing the stanzas failed. Since the only image
we know of where this bug is hit with grub is buildd (because everything
else has lsb_release), rather than adding a new dependency into buildd,
or backporting grub if we don't need to, setting GRUB_DISTRIBUTOR in the
buildd hook solves the immediate issue
This now matches the cloud images (7c760864fdcb278ca37396f06f5e3f297428d63d)
fixing bootloader updates in the buildd images, but also fixing
compatibility with using devtmpfs for losetup.
Add a file build.info on etc/cloud
with the serial information
Signed-off-by: Samir Akarioh <samir.akarioh@canonical.com>
(cherry picked from commit 105acdebc783291f740294b5c317f3e6d2da9de4)
Commit 245f7772bdb74 added code to abort the build if a snap wants to
install "core" (the 16.04 runtime). That's great but there are still
some CPC maintained image builds that use snaps based on "core". So
make it possible to continue the build if the "ALLOW_CORE_SNAP" env
variable is set.
(cherry picked from commit 34735684d5208981b2413047f67ee4c363d718d8)
Due to how `disk-image` file is structured, it builds BIOS and UEFI
images at the same time. However, certain images (e.g., GCE images)
require only UEFI image to be built, BIOS image is being simply
discarded. This results in longer build times.
Splitting out `disk-image-uefi` would allow images to use it instead of
`disk-image` and thus avoid building unused BIOS images.
`disk-image` now depends on `disk-image-uefi` for backward
compatibility.
(cherry picked from commit b40ce74fd67bbaa7f9ec94463d8da759724f9fec)
This fixes GCE shielded VM instances integrity monitoring failures on
focal and later. Our images are built with an empty /boot/grub/grubenv
file, however after the first boot `initrdless_boot_fallback_triggered`
is set to 0. This change in `grubenv` results in integrity monitoring
`lateBootReportEvent` error.
It seems that the only thing that's checking for this `grubenv` variable
is `grub-common.service`, and it is looking specifically for a `1`
value:
if grub-editenv /boot/grub/grubenv list | grep -q
initrdless_boot_fallback_triggered=1; then echo "grub:
GRUB_FORCE_PARTUUID set, initrdless boot paniced, fallback triggered.";
fi
Unsetting this variable instead of setting it to 0 would prevent issues
with integrity monitoring.
LP: 1960537 illustrates an issue where the calls to e2fsck in the
umount_partition call are failing due to an open file handle. At this
time, we are unable to find a root cause, and it's causing many builds
to fail for CPC. Adding a sleep 30 as a workaround as the file handle
releases within that timeframe. This does not address root cause.
* magic-proxy: Replace http.client with urllib calls. live-build/auto/build:
change iptables calls to query rules and quickly check that connectivity
works after transparent proxy has been installed. (LP: #1917920)
* magic-proxy: fix TypeError when trying to call get_uri() (LP: #1944906)
Currently the uri that is passed into urllib.parse.urlparse() is not
prefixed with "http(s)://" which leads urlparse() to return a wrong
scheme/netloc/path. Currently it looks like:
ParseResult(scheme='', netloc='',
path='de.archive.ubuntu.com/ubuntu/dists/impish-backports/InRelease'
, params='', query='', fragment='')
That's wrong. The path should look like
'ubuntu/dists/impish-backports/InRelease'.
Prefixing the 'host' header with 'http://' in case it's not there does
fix the problem.
This fixes:
Traceback (most recent call last):
File "/usr/lib/python3.9/socketserver.py", line 683, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.9/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.9/socketserver.py", line 747, in __init__
self.handle()
File "/usr/lib/python3.9/http/server.py", line 427, in handle
self.handle_one_request()
File "/usr/lib/python3.9/http/server.py", line 415, in handle_one_request
method()
File "/home/tom/devel/livecd-rootfs/./magic-proxy", line 787, in do_GET
File "/home/tom/devel/livecd-rootfs/./magic-proxy", line 838, in __get_request
File "/home/tom/devel/livecd-rootfs/./magic-proxy", line 84, in get_uri
TypeError: can only concatenate str (not "NoneType") to str
(cherry picked from commit 3559153c7d91dfb25e6aaf1d18152e945411d503)
Initialize passwords from sources.list.
Use urllib everywhere.
This way authentication is added to all the required requests.
And incoming headers, are passed to the outgoing requests.
And all the response headers, are passed to the original client.
And all the TCP & HTTP errors are passed back to the client.
Thus should avoiding hanging requests upon failure.
Also rewrite the URI when requesting things.
This allows to use private-ppa.buildd outside of launchpad.
Signed-off-by: Dimitri John Ledkov <xnox@ubuntu.com>
(cherry picked from commit dc2a472871907bbed3ab89d2a46d924ece80d514)
* Simplify how the subiquity client is run on the serial console in the live
server environment, breaking a unit cycle that sometimes prevents
subiquity from starting up at all. (LP: #1888497)
* Do not set the password for the installer user via cloud-init as subiquity
can now do this itself. (LP: #1933523)
With that, the Dockerfile modifications[0] currently done externally
are done now here. That means that the created rootfs tarball can be
directly used within a Dockerfile to create a container from scratch:
FROM scratch
ADD livecd.ubuntu-oci.rootfs.tar.gz /
CMD ["/bin/bash"]
[0]
https://github.com/tianon/docker-brew-ubuntu-core/blob/master/update.sh
(cherry picked from commit a81972a58b004897bf3e5c14ff371bc2f6b5e4b8)
This is a copy of the ubuntu-base project.
Currently ubuntu-base is used as a base for the docker/OCI container
images. The rootfs tarball that is created with ubuntu-base is
published under [0]. That tarball is used in the FROM statement of the
Dockerfile as base and then a couple of modifications are done inside
of the Dockerfile[1].
The ubuntu-oci project will include the changes that are currently
done in the Dockerfile. With that:
1) a Dockerfile using that tarball will be just a 2 line thing:
FROM scratch
ADD ubuntu-hirsute-core-cloudimg-amd64-root.tar.gz /
CMD ["/bin/bash"]
2) Ubuntu has the full control about the build process of the
docker/OCI container. No external sources (like [1]) need to be
modified anymore.
3) Ubuntu can publish containers without depending on the official
dockerhub containers[2]. Currently the containers for the AWS ECR
registry[3] use as a base[4] the official dockerhub containers. That's
no longer needed because a container just needs a Dockerfile described
in 1)
When the ubuntu-oci project has the modifications from [1] included,
we'll also update [1] to use the ubuntu-oci rootfs tarball as a base
and drop the modifications done at [1].
Note: Creating a new ubuntu-oci project instead of using ubuntu-base
will make sure that we don't break users who are currently using
ubuntu-base rootfs tarballs for doing their own thing.
[0] https://partner-images.canonical.com/core/
[1]
https://github.com/tianon/docker-brew-ubuntu-core/blob/master/update.sh
[2] https://hub.docker.com/_/ubuntu
[3] https://gallery.ecr.aws/ubuntu/ubuntu
[4]
https://launchpad.net/~ubuntu-docker-images/ubuntu-docker-images/+oci/ubuntu/+recipe/ubuntu-20.04
(cherry picked from commit ac4a95b9314cf1f8ce01f42016c271c0a6078372)
I recently pulled initramfs logic out of the base build hook, and
dropped that into the `replace_kernel` function. Any cloud image that
does not leverage the generic virtual kernel was expected to call
`replace_kernel` to pull in a custom kernel. That function will
disable initramfs boot for images that use a custom kernel.
Minimal cloud images on amd64 use the linux-kvm kernel, but the build
hook does not utilize the `replace_kernel` function. Instead, the
kernel flavor is set in `auto/config`. I pulled that logic out of
`auto/config` and am now calling `replace_kernel` in the build hook.
I also moved a call to generate the package list so that it will pick
up the change to the linux-kvm kernel.
Change mount option for ubuntu-cpc images from "defaults" to
"umask=0077". ESP partitions might contain sensitive data and
non-root users shouldn't have read access on it.
With this change, when we attempt to boot with an initramfs and fail,
initrdless_boot_fallback_triggered is set to non-zero in the grubenv.
This value can be checked after boot by looking in /boot/grub/grubenv
or by using the grub-editenv list command.
Addresses LP: #1870189
Initramfs-less boot, which is a boot optimization, should only be
applied where we know it could work for users and provide an improved
boot boot experience; images with custom kernels are candidates for
that.
Generic cloud images with the linux-generic kernel are not able to
boot without an initramfs. Previously, these images attempted to boot
without an initramfs, would fail, and then retry with an initramfs.
This slows the boot and is confusing behavior.
One can call divert_grub; replace_kernel; undivert_grub. And
replace_kernel will call into force_boot_without_initramfs, which
under certain conditions can call divert_grub &
undivert_grub. Resulting in undivert_grub called twice in a row.
When undivert_grub is called twice in a row it wipes
systemd-detect-virt binary from disk, as the rm call is unguarded to
check that there is something to divert if systemd package is
installed. And if the systemd package is not installed, it does not
check that systemd-detect-virt file is in-fact what divert_grub has
created.
Add a guard to check that systemd-detect-virt is the placeholder one,
before removing it.
LP: #1902260
(cherry picked from commit 096a00f40459187719840ccad99e86c7ade2ec12)
Do not use removable uefi bootloader path in the cloud-images by
default, as that prevents upgrades of the bootloader.
LP: #1912830
(cherry picked from commit 7c760864fdcb278ca37396f06f5e3f297428d63d)
Multipass on Mac OS X requires standalone kernel and initrd artifacts
to boot.
Also call update-initramfs on all installed kernels. We only have one
kernel installed, so we don't need to specify an explicit version.
Backport vmtools version in vmdk (LP: #1893898)
Backport
LP: #1893898 describes missing vmtools version from the vmdk headers.
The version should be added as ddb.toolsVersion = "2147483647" however
the sed was no longer replacing a ddb.comment field with the tools
version. Rather than subbing ddb.comment with toolsVersion, this commit
deletes ddb.comment (which the comment mentions could cause errors),
and adds the correct value. There was no visibility into the descriptor
during hook creation, so debug statements were added. This allows us to
quickly verify in the logs that bad statements are removed (the possibly
offending comments), as well as ensuring that the toolsVersion is added
MP: https://code.launchpad.net/~jchittum/livecd-rootfs/+git/livecd-rootfs/+merge/394142
Older version of vmdk-stream-converter has an incorrect header. The
original sed command replaced the incorrect "Description File" comment
with the correct "Disk DescriptorFile".
shim-signed depends on grub-efi-amd64-signed, which in turn has
alternative depends on either `grub-efi-amd64 | grub-pc`. However to
support booting with either via shim&signed-grub and BIOS, the choice
must be made to install grub-pc, not grub-efi-amd64.
This makes images consistent with Ubuntu Deskop, Live Server, buildd
bootable images; all of which already do install grub-pc and
shim-signed.
Additionally, this will ensure that autoremove is run after installing
anything in the CPC build hooks. This is done to avoid shipping images
that include packages that are autoremovable. This will clean-up as
packages are installed and detect any breakage at build time.
LP: #1901906
Backport
LP: #1893898 describes missing vmtools version from the vmdk headers.
The version should be added as ddb.toolsVersion = "2147483647" however
the sed was no longer replacing a ddb.comment field with the tools
version. Rather than subbing ddb.comment with toolsVersion, this commit
deletes ddb.comment (which the comment mentions could cause errors),
and adds the correct value. There was no visibility into the descriptor
during hook creation, so debug statements were added. This allows us to
quickly verify in the logs that bad statements are removed (the possibly
offending comments), as well as ensuring that the toolsVersion is added
Builds in LP with the Xenial kernel were happy with the recursive mount of
/sys inside the chroot while performing snap-preseeding but autopkgtests
with the groovy kernel failed. With the groovy kernel the build was
unable to unmount sys/kernel/slab/*/cgroup/* (Operation not permitted).
This patch mounts /sys and /sys/kernel/security in the chroot in the
same way we've added for binary hooks. This provides the paths under
/sys needed for snap-preseed while avoiding issues unmounting other
paths.
(cherry picked from commit 84397b50989670c2cfff01de23a5a73e67cd4088)
The snap-preseed command can do a number of things during the build
that are currently performed at first boot (apparmor profiles, systemd
unit generation, etc). This patch adds a call to reset the seeding and
apply these optimizations when adding a seeded snap. As a prerequisite
to calling snap-preseed we need to make /dev/mem available as well as
mounts from the host to perform this work, so those are also added here.
(cherry picked from commit 1ca11c979505ae1b8c4621f034d28070a2715293)
Original fix proposed by Stanislav German-Evtushenko (giner)
CPC Ubuntu cloud images default to enabling a serial console connection
via the kernel commandline option `console=ttyS0`. Many clouds support
the serial connection, and utilize it for debugging purposes. Virtualbox
supports the serial connection as well. In Bionic and earlier images,
Vagrant boxes created a serial log file in the directory of the
Vagrantfile by default. However this is not standard behaviour for
Vagrant images, and so it was removed in Eoan onwards.
Starting in Eoan, there were reports of image booting slowdown (1874453
is a single example). After testing, it was determined that the serial
connection starting, without a device attached, was the cause of the
slow down. However, we did not want to revert to the old functionality
of creating a file. Much thanks to <giner> for providing the Ruby syntax
for sending to File::NULL.
This option will not create a local file, however, the default
Vagrantfile configuration is overwritable via a users Vagrantfile. The
original syntax for creating a file local to the users Vagrantfile has
been included as an example.
- xRDP configuration changes due to the config changes in this version
compared to 18.04.
- 46-allow-update-repo.pkla inclusion to aviod "Authentication required
to refresh system repositories" bug in xRDP
- use of linux-azure, which is the optimized kernel for Hyper-V by
Microsoft
- xRDP configuration changes due to the config changes in this version
compared to 18.04.
- 46-allow-update-repo.pkla inclusion to aviod "Authentication required
to refresh system repositories" bug in xRDP
The seed now specifies the lxd snap in focal as
'lxd=4.0/stable/ubuntu-20.04' which doesn't match the expectations of
the code with looks for lxd as the only snap in the seed for minimized
images. This patch updates the pattern to accept 'lxd' or 'lxd=*'.
snap_name[/classic]=track/risk/branch is now the supported snap name
specification, which allows to specify the full default track and
optional classic confinemnt.
Supporting such specification in the seedtext allows one to specify a
better default channel. For example, this will allow lxd to switch
from latest/stable/ubuntu-20.04 to 4.0/stable/ubuntu-20.04 as 4.0 is
the LTS track matching 20.04 support timeframe.
LP: #1882374
(cherry picked from commit 7bae9201d20822d6875bcf5949e1fff839b8774c)
(cherry picked from commit 2976a99f292c500f39aace25ad08de21b37d7b31)
(cherry picked from commit d542e8e4a08467ef9b6237b9fcbd9166c8c99e8b)
ubuntustudio-default-settings in focal release has a Recommends to this
kernel, which makes it impossible to update the kernel later on, since
we would install the -updates and release kernel, which isn't allowed
and causes FTBFS. Hack out the focal-release kernel and let the rest of
the build process pull in the right one.
LP: #1884915
It was reported and confirmed in LP bug #1875400
(https://bugs.launchpad.net/cloud-images/+bug/1875400) that on the public
KVM cloud image there exists a large list of packages marked for auto-removal.
This should never be the case on a released cloud image.
These packages are marked for auto-removal because in the KVM image binary hook
we removed both initramfs-tools and busybox-initramfs packages. Due to package
dependencies this also removed:
busybox-initramfs* cloud-initramfs-copymods* cloud-initramfs-dyn-netconf*
cryptsetup-initramfs* initramfs-tools* initramfs-tools-core* multipath-tools*
overlayroot* sg3-utils-udev* ubuntu-server*
But it did not remove all the packages that the above list depended on.
This resulted in all those packages being marked for auto-removal because they
were not manually installed nor did they have any manually installed packages
that depended on them.
The removal of initramfs-tools and busybox-initramfs was to avoid the
generation of initramfs in images that should boot initramfsless.
This requirement is obsolete now because the initramfsless boot handling
is now handled via setting GRUB_FORCE_PARTUUID in /etc/default/grub.d/40-force-partuuid.cfg.
In test images I have verified that GRUB_FORCE_PARTUUID is set and that
boot speeds have not regressed.
LP: #1880170
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
