1540 Commits

Author SHA1 Message Date
Dimitri John Ledkov
838e2a6f64 magic-proxy: replace http.client with urllib calls
Initialize passwords from sources.list.
Use urllib everywhere.
This way authentication is added to all the required requests.
And incoming headers, are passed to the outgoing requests.
And all the response headers, are passed to the original client.
And all the TCP & HTTP errors are passed back to the client.
Thus should avoiding hanging requests upon failure.
Also rewrite the URI when requesting things.
This allows to use private-ppa.buildd outside of launchpad.

Signed-off-by: Dimitri John Ledkov <xnox@ubuntu.com>
(cherry picked from commit dc2a472871907bbed3ab89d2a46d924ece80d514)
2021-09-28 12:59:13 +02:00
Dimitri John Ledkov
63184732f8 Change iptables calls, to query rules and quickly check that connectivity works after transparent proxy has been installed. LP: #1917920
(cherry picked from commit 1cd8fe14b70e98220c519d679217737672ee9019)
2021-09-28 12:14:47 +02:00
Jason C. McDonald
4a994fb02e Generate manifest for HyperV desktop images
Manifests are necessary for candidate checking in the
CPC automated workflow for building and publishing
HyperV desktop images. (#1940136)
2021-08-20 09:15:03 -07:00
Thomas Bechtold
77d998829a
add configure_oci function and use it in ubuntu-oci
With that, the Dockerfile modifications[0] currently done externally
are done now here. That means that the created rootfs tarball can be
directly used within a Dockerfile to create a container from scratch:

FROM scratch
ADD livecd.ubuntu-oci.rootfs.tar.gz /
CMD ["/bin/bash"]

[0]
https://github.com/tianon/docker-brew-ubuntu-core/blob/master/update.sh

(cherry picked from commit a81972a58b004897bf3e5c14ff371bc2f6b5e4b8)
2021-04-30 13:13:34 +02:00
Thomas Bechtold
3ad6996779
Add new ubuntu-oci project
This is a copy of the ubuntu-base project.
Currently ubuntu-base is used as a base for the docker/OCI container
images. The rootfs tarball that is created with ubuntu-base is
published under [0]. That tarball is used in the FROM statement of the
Dockerfile as base and then a couple of modifications are done inside
of the Dockerfile[1].
The ubuntu-oci project will include the changes that are currently
done in the Dockerfile. With that:

1) a Dockerfile using that tarball will be just a 2 line thing:

   FROM scratch
   ADD ubuntu-hirsute-core-cloudimg-amd64-root.tar.gz /
   CMD ["/bin/bash"]

2) Ubuntu has the full control about the build process of the
docker/OCI container. No external sources (like [1]) need to be
modified anymore.
3) Ubuntu can publish containers without depending on the official
dockerhub containers[2]. Currently the containers for the AWS ECR
registry[3] use as a base[4] the official dockerhub containers. That's
no longer needed because a container just needs a Dockerfile described
in 1)

When the ubuntu-oci project has the modifications from [1] included,
we'll also update [1] to use the ubuntu-oci rootfs tarball as a base
and drop the modifications done at [1].

Note: Creating a new ubuntu-oci project instead of using ubuntu-base
will make sure that we don't break users who are currently using
ubuntu-base rootfs tarballs for doing their own thing.

[0] https://partner-images.canonical.com/core/
[1]
https://github.com/tianon/docker-brew-ubuntu-core/blob/master/update.sh
[2] https://hub.docker.com/_/ubuntu
[3] https://gallery.ecr.aws/ubuntu/ubuntu
[4]
https://launchpad.net/~ubuntu-docker-images/ubuntu-docker-images/+oci/ubuntu/+recipe/ubuntu-20.04

(cherry picked from commit ac4a95b9314cf1f8ce01f42016c271c0a6078372)
2021-04-30 13:13:18 +02:00
Łukasz 'sil2100' Zemczak
1d537bf78d Temporarily hack in ubuntu-raspi-settings for raspi images (we should be using seeds tho). 2021-04-19 21:49:43 +02:00
Michael Hudson-Doyle
e28938b482 Do not have cloud-init set the installer user's password to something random during boot, as cloud-init now hides the value it was set to much better. Instead, subiquity will reset the password itself. 2021-03-24 14:09:12 +13:00
Dimitri John Ledkov
4428dbbf8f
riscv64: generalise slightly how preinstalled server images are built, to allow using subarch to pick u-boot binaries to install. 2021-03-11 17:08:15 +00:00
Michael Hudson-Doyle
d5bc59f2fb Use the correct config keys in suppressing cloud-init output during boot of live server ISO. 2021-03-10 12:03:56 +13:00
Gauthier Jolly
76330897dc ubuntu-cpc: make /etc/fstab consistent with other ubuntu images
Modify the mount options in fstab to make the Cloud images consistent
with the default we have elsewhere.

LP: #1902103
LP: #1881006
2021-03-02 14:21:36 +00:00
Michael Hudson-Doyle
974bb5bfd1 Merge branch 'quieter-live-server-boot' into ubuntu/master 2021-02-25 15:26:23 +13:00
Łukasz 'sil2100' Zemczak
1b0212a76c Use the full name of the 'boot' flag in parted calls 2021-02-24 12:58:08 +01:00
Michael Hudson-Doyle
8ab15a145d set new cloud-init option to suppress final bit of output during boot 2021-02-23 11:06:19 +13:00
Michael Hudson-Doyle
52c8985454 Suppress cloud-init output during boot of live server ISO. 2021-02-18 13:08:09 +13:00
Cody Shepherd
4acd2e3085
Install lxd-agent-loader in bootable buildd image to improve user
experience with lxd vms
2021-02-12 13:24:54 -08:00
Cody Shepherd
5de661e137
Set grub configs to hide boot menu and enable output on ttyS0 2021-02-10 10:11:22 -08:00
Dimitri John Ledkov
82aca610dc
Merge remote-tracking branch 'xnox/fix-misisng-systemd-detect-virt' into ubuntu/master 2021-02-05 12:48:18 +00:00
Dimitri John Ledkov
096a00f404
functions: stop removing systemd-detect-virt unconditionally in undivert_grub
One can call divert_grub; replace_kernel; undivert_grub. And
replace_kernel will call into force_boot_without_initramfs, which
under certain conditions can call divert_grub &
undivert_grub. Resulting in undivert_grub called twice in a row.

When undivert_grub is called twice in a row it wipes
systemd-detect-virt binary from disk, as the rm call is unguarded to
check that there is something to divert if systemd package is
installed. And if the systemd package is not installed, it does not
check that systemd-detect-virt file is in-fact what divert_grub has
created.

Add a guard to check that systemd-detect-virt is the placeholder one,
before removing it.

LP: #1902260
2021-02-03 14:45:54 +00:00
Dimitri John Ledkov
33b7272a23
Perform fsck on all rootfs.
LP: #1912835
2021-02-01 14:45:42 +00:00
Dimitri John Ledkov
53b28d5f58
esp: perform fsck.
LP: #1912835
2021-01-22 19:16:47 +00:00
Dimitri John Ledkov
7c760864fd
esp: install grub in ubuntu bootloader id path, instead of removable.
Do not use removable uefi bootloader path in the cloud-images by
default, as that prevents upgrades of the bootloader.

LP: #1912830
2021-01-22 19:09:25 +00:00
Iain Lane
42409e7068
auto/config: Install pi-bluetooth for raspi images
These are in restricted, and so cannot be depended on by the
metapackages which are in main.
2021-01-04 10:25:09 +00:00
Sebastien Bacher
1a979b1528 Desktop is using the hwe kernel, do the same on canary 2020-12-15 16:51:32 +01:00
Dimitri John Ledkov
6746d9fbd9
disk-image-uefi: fix riscv64 subarch user-data. 2020-12-08 09:15:09 +00:00
Dimitri John Ledkov
4881f5260a
qcow2-image: unbreak builds without subarch. 2020-12-05 15:17:16 +00:00
Sebastien Bacher
ed8ee60c98 Desktop is using the hwe kernel, do the same on canary 2020-12-03 20:29:59 +01:00
Dimitri John Ledkov
3e1e76b94a
riscv64: fixup subarch build, unbound variable 2020-12-03 16:17:32 +00:00
Łukasz 'sil2100' Zemczak
c2733a55f7 Still hardcode UC20 amd64 image size, but to the same value as for UC16 and UC18. 2020-12-03 12:18:57 +01:00
Dimitri John Ledkov
2f84bc9a84
config: on hirsute build UC22 images 2020-12-02 11:05:18 +00:00
Dimitri John Ledkov
a808b28d47
riscv64: build preinstalled riscv64 image with uboot SPL and CIDATA. 2020-12-01 17:14:40 +00:00
Łukasz 'sil2100' Zemczak
dda73b5591 Do not hard-code the UC20 amd64 image size anymore. 2020-11-27 17:57:18 +01:00
Dimitri John Ledkov
3962f4ac23
disk-image-uefi: riscv64 add ubuntu:ubuntu login, without expiry. 2020-11-24 18:40:40 +00:00
Dimitri John Ledkov
da578b2c6b
disk-image-uefi: riscv64 add u-boot spl
LP: #1905274
2020-11-24 18:40:34 +00:00
Sebastien Bacher
fb0d16ea8a Correctly set the environment needed by snapd in layered
The change is similar to commit #1ca11c97 but for the canary image.
2020-11-24 11:38:10 +01:00
Sebastien Bacher
79e8f3b53e Use the target realpath to do the unmounting 2020-11-24 11:36:13 +01:00
Cody Shepherd
edaab1def3
Omit file type exensions in manifest names 2020-11-20 12:36:23 -08:00
Cody Shepherd
762124dbd7
Create manifests for buildd tarballs 2020-11-19 12:59:06 -08:00
Dimitri John Ledkov
3a20f2669c
Merge remote-tracking branch 'davidkrauser/buildd-image-perform-upgrade' into ubuntu/master 2020-11-19 17:39:26 +00:00
Dimitri John Ledkov
f53a681d8a
Merge remote-tracking branch 'patviafore/use_ftpmaster_instead_of_archive_in_binary_hooks' into ubuntu/master 2020-11-19 17:36:39 +00:00
Dimitri John Ledkov
8e986c20fd
Merge remote-tracking branch 'seb128/layered-languages-fix' into ubuntu/master 2020-11-19 17:34:02 +00:00
Dimitri John Ledkov
83c77f9c60
Merge remote-tracking branch 'davidkrauser/dk/buildd-unpack-kernel-initrd' into ubuntu/master 2020-11-19 17:28:21 +00:00
Dimitri John Ledkov
914997827d
disk-image-uefi.binary: build riscv64 image without a bootloader.
We want GPT image with ESP, however there is no bootloader for it
yet. Build without it.
2020-11-19 15:47:12 +00:00
Dimitri John Ledkov
122364df65
disk-image-uefi.binary: enable secureboot on the arm64 image. 2020-11-19 15:47:09 +00:00
Sebastien Bacher
fa5c2092c7 During the layered build generate the right content for no languages
The structure changed as part of bug #1873374
2020-11-19 16:28:11 +01:00
Dimitri John Ledkov
e8f53aabdf
disk-image.binary: however still build MBR image on amd64.
It is used to derive vagrant and ec2 images. I do wonder, if those
could be converted to derive from uefi one.
2020-11-18 00:09:08 +00:00
Dimitri John Ledkov
8e548f37d9
disk-image.binary: skip building MBR image on riscv64. 2020-11-17 23:40:53 +00:00
Dimitri John Ledkov
ff175e3f4a
disk-image.binary: actually skip building MBR image on EFI platforms.
The case is for arch:subarch combo, not just arch alone even if
subarch is empty. Thus currently on adm64/arm64/armhf ubuntu-cpc
builds mbr image is created and then ignored, as the convert to qcow2
hook prefers the uefi image whenever available.

Skipping building these correctly, should speed up the build a little
bit and use slightly less disk space.
2020-11-17 23:37:52 +00:00
Jinming Wu, Patrick
72e47a9f8b
Hyper-V hook fix 2020-11-03 16:19:01 +08:00
Dimitri John Ledkov
e2fd168f50
amd64: always install grub-pc with shim-signed
shim-signed depends on grub-efi-amd64-signed, which in turn has
alternative depends on either `grub-efi-amd64 | grub-pc`. However to
support booting with either via shim&signed-grub and BIOS, the choice
must be made to install grub-pc, not grub-efi-amd64.

This makes images consistent with Ubuntu Deskop, Live Server, buildd
bootable images; all of which already do install grub-pc and
shim-signed.

LP: #1901906
2020-10-28 22:56:31 +00:00
Robert C Jennings
df38b9ba75
Merge ensure-vmtools-in-vmdk-header into ubuntu/master [a=jchittum] [r=rcj]
vmtools version in vmdk header (LP: #1893898)

LP: #1893898 describes missing vmtools version from the vmdk headers.
The version should be added as ddb.toolsVersion = "2147483647" however
the sed was no longer replacing a ddb.comment field with the tools
version. Rather than subbing ddb.comment with toolsVersion, this commit
deletes ddb.comment (which the comment mentions could cause errors),
and adds the correct value. There was no visibility into the descriptor
during hook creation, so debug statements were added. This allows us to
quickly verify in the logs that bad statements are removed (the possibly
offending commetns), as well as ensuring that the toolsVersion is added

MP: https://code.launchpad.net/~jchittum/livecd-rootfs/+git/livecd-rootfs/+merge/392401
2020-10-28 12:00:57 -05:00