releasing package livecd-rootfs version 2.525.37

When we encounter a failure in 'lb binary' the launchpad builders can
only surface the build output from stdout.  If the binary hook failure
implicates the archive we can not determine fault without the apt
proxy log.  This patch will dump the proxy log to stdout to aid in
debugging these failures.

debian/changelog

@@ -1,3 +1,361 @@
+livecd-rootfs (2.525.37) bionic; urgency=medium
+
+  * Use the linux-raspi2-5.3 kernel flavour for the raspi3 images.  This kernel
+    is required for proper pi4 support. (LP: #1859478)
+
+ -- Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com>  Mon, 13 Jan 2020 15:33:20 +0100
+
+livecd-rootfs (2.525.36) bionic; urgency=medium
+
+  * Preserve apt preferences created by any package we install (i.e.
+    ubuntu-advantage-tools) against live-build's attempt to delete them.
+    (LP: #1855354)
+
+ -- Michael Hudson-Doyle <michael.hudson@ubuntu.com>  Thu, 19 Dec 2019 22:43:33 +1300
+
+livecd-rootfs (2.525.35) bionic; urgency=medium
+
+  * Did not expect the bionic branch being so outdated. Backport support
+    for fetching core18 model assertions on bionic hosts. This enables build
+    support for ubuntu-core 18 images. (LP: #1852332)
+
+ -- Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com>  Tue, 12 Nov 2019 21:05:29 +0100
+
+livecd-rootfs (2.525.34) bionic; urgency=medium
+
+  * Add support for HyperV Gallery Images (LP: #1837088)
+
+ -- David Krauser <david.krauser@canonical.com>  Wed, 05 Nov 2019 15:19:29 -0400
+
+livecd-rootfs (2.525.33) bionic; urgency=medium
+
+  * Add support for the 'kassel' subflavor of UC18 images. (LP: #1850674)
+
+ -- Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com>  Wed, 30 Oct 2019 18:09:02 +0100
+
+livecd-rootfs (2.525.32) bionic; urgency=medium
+
+  * Fix a terrible typo in auto/build that causes all core18 builds to fail on
+    this series (LP: #1849343).
+  * Backport 999-ubuntu-image-customization.chroot to have a common place for
+    our cloud-init datasource.  Add some modifications on top to make sure the
+    legacy raspi2 configuration stays the same (LP: #1849472).
+  * Backport enabling of adding extra snaps to ubuntu-core images via the
+    EXTRA_SNAPS environment (LP: #1849491).
+
+ -- Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com>  Tue, 22 Oct 2019 17:34:31 +0200
+
+livecd-rootfs (2.525.31) bionic; urgency=medium
+
+  * magic-proxy: dump proxy log to stdout on failure (LP: #1847300)
+
+ -- Robert C Jennings <robert.jennings@canonical.com>  Tue, 08 Oct 2019 11:00:07 -0500
+
+livecd-rootfs (2.525.30) bionic; urgency=medium
+
+  [ Robert C Jennings ]
+  * ubuntu-cpc: Only produce explicitly specified artifacts (LP: #1837254)
+
+  [ Ethan Hsieh ]
+  * Add support for the Nitrogen6x board (LP: #1838064).
+
+ -- Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com>  Tue, 10 Sep 2019 16:56:26 +0200
+
+livecd-rootfs (2.525.29) bionic; urgency=medium
+
+  * Add retry logic to snap-tool to make downloads more resilient.
+    (LP: #1837871)
+
+ -- Tobias Koch <tobias.koch@canonical.com>  Mon, 26 Aug 2019 13:41:50 +0200
+
+livecd-rootfs (2.525.28) bionic; urgency=medium
+
+  [ Tobias Koch ]
+  * Do proper error checking when calling snap-tool info to determine
+    the base of a snap. (LP: #1828500)
+
+  [ Michael Vogt ]
+  * Run "snap debug validate-seed" during preseeding to do basic validation of
+    the generated seed.yaml
+
+  [ Iain Lane ]
+  * Seed core for non minimized builds, as it is still required (LP:
+    #1836594).
+
+ -- Iain Lane <iain.lane@canonical.com>  Tue, 16 Jul 2019 13:20:18 +0100
+
+livecd-rootfs (2.525.27) bionic; urgency=medium
+
+  * Backport improvements to snap seeding from trunk.  LP: #1831675.
+
+  [ Tobias Koch ]
+  * Replace "snap download" with tool that uses snap store's coherence
+    feature.
+  * Detect which core snaps are required and install them on-the-fly.
+  * If image has core18 snaps only, automatically preseed snapd.
+
+ -- Dimitri John Ledkov <xnox@ubuntu.com>  Mon, 10 Jun 2019 18:46:16 +0100
+
+livecd-rootfs (2.525.26) bionic; urgency=medium
+
+  * Strip translation files out of the minimal images, another thing that
+    goes unused when there is no human console user (and we already don't
+    have the locales themselves present on a minimal image).  LP: #1829333.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Tue, 28 May 2019 08:54:04 -0700
+
+livecd-rootfs (2.525.25) bionic; urgency=medium
+
+  [ Robert C Jennings ]
+  * ubuntu-cpc: parallel builds (LP: #1829938)
+    - Inject a proxy into the build providing a snapshot view of the package repo.
+    - Use series files with dependency handling to generate hook symlinks dynamically
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Tue, 21 May 2019 15:35:35 -0700
+
+livecd-rootfs (2.525.24) bionic; urgency=medium
+
+  * Build WSL rootfs tarball (LP: #1827930)
+
+ -- Balint Reczey <rbalint@ubuntu.com>  Fri, 17 May 2019 15:11:42 +0200
+
+livecd-rootfs (2.525.23) bionic; urgency=medium
+
+  * Backport two minimizations for the docker images: remove apt lists that
+    are removed downstream anyway, and remove device nodes from the image.
+    (LP: #1828118)
+
+ -- Michael Hudson-Doyle <michael.hudson@ubuntu.com>  Wed, 08 May 2019 10:48:39 +1200
+
+livecd-rootfs (2.525.22) bionic; urgency=medium
+
+  * Subiquity specific changes SRU LP: #1827357
+    - subiquity: make subiquity_config.mount optional
+    - Make serial-subiquity@ use the same codepath as tty1 subiquity.
+    - Fix ubuntu-server-live images to generate initrd with casper UUID.
+
+ -- Dimitri John Ledkov <xnox@ubuntu.com>  Thu, 02 May 2019 11:34:34 +0100
+
+livecd-rootfs (2.525.21) bionic; urgency=medium
+
+  * Remove crufty files after minimize-manual (LP: #1826377)
+
+ -- Julian Andres Klode <juliank@ubuntu.com>  Fri, 26 Apr 2019 10:40:08 +0200
+
+livecd-rootfs (2.525.20) bionic; urgency=medium
+
+  [ Steve Langasek ]
+  * Drop /etc/update-motd.d/51-cloudguest from cloud images; this is not
+    consistent with current Ubuntu Advantage product language.  Any future
+    customizations to update-motd for cloud images should be done via a
+    package instead.
+
+  [ Julian Andres Klode ]
+  * Minimize the number of manually installed packages in images by marking
+    dependencies of metapackages as automatically installed. (LP: #1800610);
+    but do not mark direct dependencies of ubiquity as auto installed. This
+    caused cryptsetup to remain auto on the installed system in bionic (see
+    LP #1801629)
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Thu, 14 Mar 2019 11:04:49 -0700
+
+livecd-rootfs (2.525.19) bionic; urgency=medium
+
+  * Make sure buildd images have a /usr/sbin/policy-rc.d symlink
+    (LP: #1815251).
+
+ -- Colin Watson <cjwatson@ubuntu.com>  Mon, 18 Feb 2019 16:03:46 +0000
+
+livecd-rootfs (2.525.18) bionic; urgency=medium
+
+  [ Michael Hudson-Doyle ]
+  * Changes to kernel handling for live-server: do not include kernel/initrd
+    in filesystem.squashfs but rather install it in a throwaway layer on top
+    of installer.squashfs and fish kernel, initrd and modules out of that with
+    an initrd hook that records kernel metapackage name in /run and mounts
+    /lib/modules from a squashfs on the ISO.
+  * Do no install openssh-server in the base filsystem for the live server
+    installer.
+  * A few simple tweaks to reduce size of live servers installer.squashfs:
+    - Do not run apt-get update (which can bring in package lists if we are
+      unlucky wrt publisher schedules).
+    - Run apt-get clean to clear out downloaded debs of curtin/casper and
+      dependencies.
+    - Do not install user-setup.
+    - Use the core snap from the base filesystem if present.
+  * Do not include curtin in the live-server installer.squashfs as the
+    version of subiquity that includes it in the snap has now been released to
+    stable.
+
+ -- Adam Conrad <adconrad@ubuntu.com>  Tue, 12 Feb 2019 11:35:48 -0700
+
+livecd-rootfs (2.525.17) bionic; urgency=medium
+
+  * Add a buildd subproject (LP: #1815251).
+  * Add a LXD image to builds for the buildd subproject.
+  * Move buildd image building to binary hooks.
+
+ -- Colin Watson <cjwatson@ubuntu.com>  Fri, 08 Feb 2019 22:57:59 +0000
+
+livecd-rootfs (2.525.16) bionic; urgency=medium
+
+  * lubuntu: Select lubuntu-gtk-core task as well, so we get the HWE filter
+    applied to the explicit dependencies it has on Xorg-recommended packages.
+  * lubuntu: Manually install some packages that fall out due to the above.
+
+ -- Adam Conrad <adconrad@ubuntu.com>  Mon, 04 Feb 2019 12:48:51 -0700
+
+livecd-rootfs (2.525.15) bionic; urgency=medium
+
+  * Forward-port HWE support from xenial, make it more generic, and enable
+    it for the current LTS flavours.
+
+ -- Adam Conrad <adconrad@ubuntu.com>  Fri, 01 Feb 2019 12:41:39 -0700
+
+livecd-rootfs (2.525.14) bionic; urgency=medium
+
+  * More changes for raspi3 build support (LP: #1805668):
+    - Fix 100-purge-grub-legacy-ec2-arm.chroot to not gate on ARCH as that's
+      not defined in .chroot hooks.
+    - Use the new linux-firmware-raspi2 for boot binary blobs as
+      raspi3-firmware includes upgrade hooks that conflict with our image
+      configuration.
+
+ -- Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com>  Fri, 18 Jan 2019 16:56:58 +0100
+
+livecd-rootfs (2.525.13) bionic; urgency=medium
+
+  [ Balint Reczey ]
+  * Update Vcs-* fields in debian/control to point to git.
+
+  [ Łukasz 'sil2100' Zemczak ]
+  * Another batch of cherry-picks for raspi3 support (LP: #1805668)
+    - Add wpasupplicant to the additional packages installed for the raspi2 and
+      raspi3 targets.
+    - Default to IMAGEFORMAT=ubuntu-image for raspi3 ubuntu-cpc builds.
+    - Link the resulting raspi3 image to a filename that cdimage expects from a
+      preinstalled image build.
+  * Add the 100-purge-grub-legacy-ec2-arm.chroot hook to remove
+    grub-legacy-ec2 from any ARM based ubuntu-cpc images we create. The package
+    is no longer in the server seed of newer series anyway.
+
+ -- Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com>  Fri, 11 Jan 2019 14:53:10 +0100
+
+livecd-rootfs (2.525.12) bionic; urgency=medium
+
+  * Key netplan delegation to NetworkManager on presence of
+    /usr/sbin/NetworkManager, not on /usr/lib/NetworkManager which may have
+    hooks from other packages (i.e., wpasupplicant).
+
+  [ Cody Shepherd ]
+  * Include snaps in image manifests (LP: #1805497)
+  * Change call to add grub efi packages using new create_manifests()
+    function.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Mon, 10 Dec 2018 12:46:46 -0800
+
+livecd-rootfs (2.525.11) bionic; urgency=medium
+
+  * Backport all the required changes to enable Raspberry Pi 3 armhf and arm64
+    preinstalled image builds. (LP: #1805668)
+    - Add support for raspi3 rootfs builds (based on Ryan Finnie's changes).
+    - For ubuntu-image consumption, export the kernel and initrd to
+      image/boot/uboot for raspi*.
+    - Avoid issues of hard-linking to a symbolic vmlinuz as this can lead to a
+      dangling symlink.
+    - Add raspi3 arm64 rootfs build support.
+    - Add git to the build dependencies for the gadget tree pull.
+    - Minor fixes to raspi3 builds: add arm64+raspi3 to the supported model
+      list, pass SUITE on to the build stage and use the git:// url for the
+      gadget tree.
+
+ -- Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com>  Thu, 29 Nov 2018 16:24:23 +0100
+
+livecd-rootfs (2.525.10) bionic; urgency=medium
+
+  [ Cody Shepherd ]
+  * Include grub efi packages in manifests for uefi images.
+    (LP: #1805190)
+
+  [ Robert C Jennings ]
+  * Disable checksum generation. (LP: #1799773)
+
+  [Steve Langasek]
+  * Fix Vcs-Bzr link.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Mon, 26 Nov 2018 12:55:11 -0800
+
+livecd-rootfs (2.525.9) bionic; urgency=medium
+
+  * Ensure /lib/modules exists in root tarballs and sqashfs.
+    (LP: #1792905)
+
+ -- Tobias Koch <tobias.koch@canonical.com>  Thu, 20 Sep 2018 09:30:34 +0200
+
+livecd-rootfs (2.525.8) bionic; urgency=medium
+
+  * Disentangle enabling universe in the final image a little from having
+    PREINSTALLED=true set and enable it for a live-server build.
+    (LP: #1783129)
+  * Fix live-server journald config snippet to actually disable journald rate
+    limiting.
+
+ -- Michael Hudson-Doyle <michael.hudson@ubuntu.com>  Tue, 28 Aug 2018 11:03:37 +1200
+
+livecd-rootfs (2.525.7) bionic; urgency=medium
+
+  * Disable journald rate limiting in the live-server live session. 
+    (LP: #1776891) 
+
+ -- Michael Hudson-Doyle <michael.hudson@ubuntu.com>  Tue, 14 Aug 2018 21:41:53 -0700
+
+livecd-rootfs (2.525.6) bionic; urgency=medium
+
+  [ Steve Langasek ]
+  * generate all tar files with --xattrs.  LP: #1302192.
+
+  [ Daniel Watkins ]
+  * ubuntu-cpc: Reintroduce the -root.tar.xz artifact (LP: #1585233).
+  * ubuntu-cpc: Generate the root image contents once, and use it for both the
+    -root.tar.xz and the .squashfs.
+  * ubuntu-cpc: Generate -root.tar.xz with --xattrs.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Mon, 06 Aug 2018 14:16:04 -0700
+
+livecd-rootfs (2.525.5) bionic; urgency=medium
+
+  * live-build/auto/config: Improve linux-tools filter to work for i386 too.
+
+ -- Adam Conrad <adconrad@ubuntu.com>  Tue, 24 Jul 2018 22:21:27 -0600
+
+livecd-rootfs (2.525.4) bionic; urgency=medium
+
+  * live-build/auto/config: Dirty hack to filter linux-tools-aws from budgie.
+
+ -- Adam Conrad <adconrad@ubuntu.com>  Tue, 24 Jul 2018 21:55:49 -0600
+
+livecd-rootfs (2.525.3) bionic; urgency=medium
+
+  * live-build/auto/config: Add nasty hack to order gtk-common-themese snap
+    first in snapd's seed.yaml to work around the snapd bug in LP: #1772844
+
+ -- Adam Conrad <adconrad@ubuntu.com>  Mon, 23 Jul 2018 10:02:04 -0600
+
+livecd-rootfs (2.525.2) bionic; urgency=medium
+
+  * Update unminimize script text and install ubuntu-standard when
+    unminimizing a minimal image (LP: #1778777)
+
+ -- Francis Ginther <francis.ginther@canonical.com>  Mon, 02 Jul 2018 13:27:15 -0500
+
+livecd-rootfs (2.525.1) bionic; urgency=medium
+
+  * Symlink systemd-networkd-wait-online to /bin/true in the live installer
+    live session to avoid depending on buggy systemd behaviour. (LP: #1773719)
+  * Remove "optional: true" from installer netplan config.
+
+ -- Michael Hudson-Doyle <michael.hudson@ubuntu.com>  Wed, 23 May 2018 14:27:01 +1200
+
 livecd-rootfs (2.525) bionic; urgency=medium
 
   * Don't fail to build CPC images if /lib/modules already exists in the

debian/control

@@ -4,12 +4,13 @@ Priority: optional
 Build-Depends: debhelper (>= 7)
 Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
 Standards-Version: 3.9.6
-Vcs-Bzr: http://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk
+Vcs-Git: https://git.launchpad.net/livecd-rootfs -b ubuntu/bionic
 
 Package: livecd-rootfs
 Architecture: any
 Depends: ${misc:Depends},
          apt-utils,
+         attr,
          debootstrap,
          distro-info,
          dosfstools,
@@ -17,21 +18,25 @@ Depends: ${misc:Depends},
          gdisk,
          genisoimage,
          germinate (>= 1.25.1),
+         git,
          gnupg,
          grep-dctrl,
          kpartx,
          live-build (>= 3.0~a57-1ubuntu31~),
          lsb-release,
          lzma,
+         make,
          parted,
          procps,
          python-minimal | python,
+         python3-apt,
          python3-software-properties,
          qemu-utils,
          rsync,
-         snapd,
+         snapd (>= 2.39),
          squashfs-tools (>= 1:3.3-1),
          sudo,
+         u-boot-tools [armhf arm64],
          ubuntu-image,
          vmdk-stream-converter [amd64 i386],
          xz-utils,

debian/install

@@ -1,2 +1,6 @@
 live-build usr/share/livecd-rootfs
 get-ppa-fingerprint usr/share/livecd-rootfs
+minimize-manual usr/share/livecd-rootfs
+magic-proxy usr/share/livecd-rootfs
+lp-in-release usr/share/livecd-rootfs
+snap-tool usr/share/livecd-rootfs

debian/tests/default-bootstraps

@@ -25,6 +25,7 @@ ALL_TRIPLETS="
  mythbuntu::
  ubuntu::
  ubuntu-base::
+ ubuntu-base:buildd:
  ubuntu-budgie::
  ubuntu-budgie-desktop::
  ubuntu-budgie-live::

live-build/auto/build

@@ -17,18 +17,63 @@ fi
 
 . config/functions
 
-# Link output files somewhere BuildLiveCD will be able to find them.
+if [ -n "$REPO_SNAPSHOT_STAMP" ]; then
+    if [ "`whoami`" != "root" ]; then
+        echo "Magic repo snapshots only work when running as root." >&2
+        exit 1
+    fi
+
+    apt-get -qyy install iptables
+
+    # Redirect all outgoing traffic to port 80 to proxy instead.
+    iptables -t nat -A OUTPUT -p tcp --dport 80 -m owner ! --uid-owner daemon \
+        -j REDIRECT --to 8080
+
+    # Run proxy as "daemon" to avoid infinite loop.
+    /usr/share/livecd-rootfs/magic-proxy \
+        --address="127.0.0.1" \
+        --port=8080 \
+        --run-as=daemon \
+        --cutoff-time="$REPO_SNAPSHOT_STAMP" \
+        --log-file=/build/livecd.magic-proxy.log \
+        --pid-file=config/magic-proxy.pid \
+        --background \
+        --setsid
+fi
+
+# Link output files somewhere launchpad-buildd will be able to find them.
 PREFIX="livecd.$PROJECT${SUBARCH:+-$SUBARCH}"
 
 if [ "${IMAGEFORMAT:-}" = "ubuntu-image" ]; then
 	# Use ubuntu-image instead of live-build
 
-	CHANNEL="${CHANNEL:-edge}"
-	env SNAPPY_STORE_NO_CDN=1 \
-	ubuntu-image -c "$CHANNEL" $UBUNTU_IMAGE_ARGS \
-		-o "$PREFIX".img "$PREFIX".model-assertion
-	xz -0 -T4 "$PREFIX".img
-	mv seed.manifest "$PREFIX".manifest
+	if [ "$PROJECT" = "ubuntu-core" ]; then
+		CHANNEL="${CHANNEL:-edge}"
+		env SNAPPY_STORE_NO_CDN=1 \
+		ubuntu-image snap -c "$CHANNEL" $UBUNTU_IMAGE_ARGS \
+			-O output "$PREFIX".model-assertion
+		# XXX: currently we only have one image generated, but really
+		#  we should be supporting more than one for models that
+		#  define those.
+		mv output/*.img "$PREFIX".img
+		xz -0 -T4 "$PREFIX".img
+		mv output/seed.manifest "$PREFIX".manifest
+	else
+		# First we need to build the gadget tree
+		make -C "config/$PREFIX-gadget" ARCH=$ARCH SERIES=$SUITE
+		ubuntu-image classic $UBUNTU_IMAGE_ARGS \
+			-s $SUITE -p $PROJECT -a $ARCH --subarch $SUBARCH \
+			-O output config/$PREFIX-gadget/install
+		# XXX: currently we only have one image generated, but really
+		#  we should be supporting more than one for models that
+		#  define those.
+		mv output/*.img "$PREFIX".img
+		xz -0 -T4 "$PREFIX".img
+		# Also link the output image to a filename that cdimage expects
+		ln "$PREFIX".img.xz livecd.ubuntu-cpc.disk1.img.xz
+		mv output/filesystem.manifest "$PREFIX".manifest
+	fi
+
 	exit 0
 fi
 
@@ -63,8 +108,8 @@ Expire-Date: 0
 
 	lb bootstrap "$@"
 
-	case $PROJECT in
-		ubuntu-server|ubuntu-cpc)
+        case $PROJECT:${SUBPROJECT:-} in
+		ubuntu-server:*|ubuntu-cpc:*|ubuntu:desktop-preinstalled)
 			# Set locale to C.UTF-8 by default. We should
 			# probably do this for all images early in the
 			# 18.10 cycle but for now just do it for
@@ -86,6 +131,9 @@ Expire-Date: 0
 # Drop all man pages
 path-exclude=/usr/share/man/*
 
+# Drop all translations
+path-exclude=/usr/share/locale/*/LC_MESSAGES/*.mo
+
 # Drop all documentation ...
 path-exclude=/usr/share/doc/*
 
@@ -109,12 +157,16 @@ set -e
 echo "This system has been minimized by removing packages and content that are"
 echo "not required on a system that users do not log into."
 echo ""
-echo "This script restores the content and packages that are found on a default"
-echo "Ubuntu server system."
+echo "This script restores content and packages that are found on a default"
+echo "Ubuntu server system in order to make this system more suitable for"
+echo "interactive use."
 echo ""
 echo "Reinstallation of packages may fail due to changes to the system"
 echo "configuration, the presence of third-party packages, or for other"
 echo "reasons."
+echo ""
+echo "This operation may take some time."
+echo ""
 read -p "Would you like to continue? [y/N]" REPLY
 echo    # (optional) move to a new line
 if [ "$REPLY" != "y" ] && [ "$REPLY" != "Y" ]
@@ -142,6 +194,10 @@ if [ -f /etc/dpkg/dpkg.cfg.d/excludes ] || [ -f /etc/dpkg/dpkg.cfg.d/excludes.dp
     # This step processes the packages which still have missing documentation
     dpkg --verify --verify-format rpm | awk '/..5......   \/usr\/share\/doc/ {print $2}' | sed 's|/[^/]*$||' | sort |uniq \
          | xargs dpkg -S | sed 's|, |\n|g;s|: [^:]*$||' | uniq | DEBIAN_FRONTEND=noninteractive xargs apt-get install --reinstall -y
+    echo "Restoring system translations..."
+    # This step processes the packages which still have missing translations
+    dpkg --verify --verify-format rpm | awk '/..5......   \/usr\/share\/locale/ {print $2}' | sed 's|/[^/]*$||' | sort |uniq \
+         | xargs dpkg -S | sed 's|, |\n|g;s|: [^:]*$||' | uniq | DEBIAN_FRONTEND=noninteractive xargs apt-get install --reinstall -y
     if dpkg --verify --verify-format rpm | awk '/..5......   \/usr\/share\/doc/ {exit 1}'; then
         echo "Documentation has been restored successfully."
         rm /etc/dpkg/dpkg.cfg.d/excludes.dpkg-tmp
@@ -155,7 +211,7 @@ fi
 
 if ! dpkg-query --show --showformat='${db:Status-Status}\n' ubuntu-minimal 2> /dev/null | grep -q '^installed$'; then
     echo "Installing ubuntu-minimal package to provide the familiar Ubuntu minimal system..."
-    DEBIAN_FRONTEND=noninteractive apt-get install -y ubuntu-minimal
+    DEBIAN_FRONTEND=noninteractive apt-get install -y ubuntu-minimal ubuntu-standard
 fi
 
 if dpkg-query --show --showformat='${db:Status-Status}\n' ubuntu-server 2> /dev/null | grep -q '^installed$' \
@@ -205,6 +261,12 @@ EOF
 
 	lb chroot "$@"
 
+	if [ -d chroot/etc/apt/preferences.d.save ]; then
+		# https://twitter.com/infinite_scream
+		mv chroot/etc/apt/preferences.d.save/* chroot/etc/apt/preferences.d/
+		rmdir chroot/etc/apt/preferences.d.save
+	fi
+
 	if [ "${SUBPROJECT:-}" = minimized ]; then
 		# force removal of initramfs-tools, which we assert is not
 		# required for any minimized images but is still pulled in by
@@ -229,11 +291,7 @@ EOF
 			apt-get -y --purge autoremove"
 	fi
 
-	# remove crufty files that shouldn't be left in an image
-	rm -f chroot/var/cache/debconf/*-old chroot/var/lib/dpkg/*-old
-	Chroot chroot apt clean
-
-	if [ -f config/oem-config-preinstalled ]; then
+	if [ -f config/universe-enabled ]; then
 
 # This is cargo-culted almost verbatim (with some syntax changes for
 # preinstalled being slightly different in what it doesn't ask) from
@@ -392,7 +450,7 @@ EOF
 	# default. Installing NM on an existing system only manages wifi and wwan via
 	# /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf. When setting
 	# the global backend to NM, netplan overrides that file.
-	if [ -d chroot/usr/lib/NetworkManager ]; then
+	if [ -e chroot/usr/sbin/NetworkManager ]; then
 		echo "===== Enabling all devices in NetworkManager ===="
 		mkdir -p chroot/etc/netplan
 		cat <<EOF > chroot/etc/netplan/01-network-manager-all.yaml
@@ -410,6 +468,23 @@ EOF
 	(cd chroot && find usr/share/doc -maxdepth 1 -type d | xargs du -s | sort -nr)
 	echo END docdirs
 
+	/usr/share/livecd-rootfs/minimize-manual chroot
+
+	# remove crufty files that shouldn't be left in an image
+	rm -f chroot/var/cache/debconf/*-old chroot/var/lib/dpkg/*-old
+	Chroot chroot apt clean
+	# For the docker images we remove even more stuff.
+	if [ "${PROJECT}:${SUBPROJECT:-}" = "ubuntu-base:minimized" ]; then
+		# Remove apt lists (that are currently removed downstream
+		# anyway)
+		rm -rf chroot/var/lib/apt/lists/*
+		# Having device nodes in the docker image can cause problems
+		# (https://github.com/tianon/docker-brew-ubuntu-core/issues/62)
+		# so remove them.  We only do this for docker out of an
+		# abundance of caution.
+		rm -rf chroot/dev/*
+	fi
+
 	lb binary "$@"
 	touch binary.success
 ) 2>&1 | tee binary.log
@@ -420,6 +495,12 @@ EOF
 if [ -e binary.success ]; then
 	rm -f binary.success
 else
+	# Dump the magic-proxy log to stdout on failure to aid debugging
+	if [ -f /build/livecd.magic-proxy.log ] ; then
+		echo "================= Magic proxy log (start) ================="
+		cat /build/livecd.magic-proxy.log
+		echo "================== Magic proxy log (end) =================="
+	fi
 	exit 1
 fi
 
@@ -455,7 +536,7 @@ for ISO in binary.iso binary.hybrid.iso; do
 done
 
 if [ -e "binary/$INITFS/filesystem.dir" ]; then
-	(cd "binary/$INITFS/filesystem.dir/" && tar -c *) | \
+	(cd "binary/$INITFS/filesystem.dir/" && tar -c --xattrs *) | \
 		gzip -9 --rsyncable > "$PREFIX.rootfs.tar.gz"
 	chmod 644 "$PREFIX.rootfs.tar.gz"
 elif [ -e binary-tar.tar.gz ]; then
@@ -492,13 +573,14 @@ EOF
 fi
 
 if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-touch-custom" ]; then
-	(cd "binary/$INITFS/custom.dir/" && tar -c *) | \
+	(cd "binary/$INITFS/custom.dir/" && tar -c --xattrs *) | \
 		gzip -9 --rsyncable > "$PREFIX.custom.tar.gz"
 	chmod 644 "$PREFIX.custom.tar.gz"
 fi
 
 # '--initramfs none' produces different manifest names.
 if [ -e "binary/$INITFS/filesystem.packages" ]; then
+	./config/snap-seed-parse "chroot/" "binary/${INITFS}/filesystem.packages"
 	ln "binary/$INITFS/filesystem.packages" "$PREFIX.manifest"
 	chmod 644 "$PREFIX.manifest"
 fi
@@ -763,6 +845,8 @@ for FLAVOUR in $LB_LINUX_FLAVOURS; do
 	if [ -z "$LB_LINUX_FLAVOURS" ] || [ "$LB_LINUX_FLAVOURS" = "none" ]; then
 		continue
 	fi
+	# hwe-* kernels don't use the hwe suffix on the filesystem:
+	FLAVOUR=${FLAVOUR%%-hwe-*}
 	if [ "$FLAVOUR" = "virtual" ]; then
 		# The virtual kernel is named generic in /boot
 		FLAVOUR="generic"
@@ -796,50 +880,67 @@ done
 NUMFLAVOURS="$(set -- $LB_LINUX_FLAVOURS; echo $#)"
 if [ "$NUMFLAVOURS" = 1 ] && [ "$LB_LINUX_FLAVOURS" != "none" ]; then
 	# only one kernel flavour
-	if [ -e "binary/$INITFS/vmlinuz" ]; then
+	FLAVOUR=${LB_LINUX_FLAVOURS%%-hwe-*}
+	if [ -f "binary/$INITFS/vmlinuz" ] && ! [ -h "binary/$INITFS/vmlinuz" ]; then
 		ln "binary/$INITFS/vmlinuz" "$PREFIX.kernel"
 		chmod 644 "$PREFIX.kernel"
 	else
-		ln -sf "$PREFIX.kernel-$LB_LINUX_FLAVOURS" "$PREFIX.kernel"
+		ln -sf "$PREFIX.kernel-$FLAVOUR" "$PREFIX.kernel"
 	fi
-	if [ -e "binary/$INITFS/initrd.lz" ]; then
+	if [ -f "binary/$INITFS/initrd.lz" ] && ! [ -h "binary/$INITFS/initrd.lz" ]; then
 		ln "binary/$INITFS/initrd.lz" "$PREFIX.initrd"
 		chmod 644 "$PREFIX.initrd"
 	else
-		ln -sf "$PREFIX.initrd-$LB_LINUX_FLAVOURS" "$PREFIX.initrd"
+		ln -sf "$PREFIX.initrd-$FLAVOUR" "$PREFIX.initrd"
 	fi
 fi
 
-if [ "$SUBARCH" = "ac100" ]; then
-	# create the md5sum and size files for which we are actually doing all this
-	md5sum $PREFIX.rootfs.tar.gz >chroot/installer.md5
-	wc -c $PREFIX.rootfs.tar.gz >chroot/installer.size
+case $SUBARCH in
+	ac100)
+		# create the md5sum and size files for which we are actually doing all this
+		md5sum $PREFIX.rootfs.tar.gz >chroot/installer.md5
+		wc -c $PREFIX.rootfs.tar.gz >chroot/installer.size
 	
-	INFO_DESC="$(lsb_release -d -s)"
-	INFO_STAMP=$(date +20%y%m%d-%H:%M)
+		INFO_DESC="$(lsb_release -d -s)"
+		INFO_STAMP=$(date +20%y%m%d-%H:%M)
 	
-	echo "$INFO_DESC - $ARCH ($INFO_STAMP)" >chroot/media-info
+		echo "$INFO_DESC - $ARCH ($INFO_STAMP)" >chroot/media-info
 	
-	# make sure update-initramfs feels cosy and warm in the environment
-	lb chroot_proc install "$@"
-	lb chroot_sysfs install "$@"
-	lb chroot_devpts install "$@"
+		# make sure update-initramfs feels cosy and warm in the environment
+		lb chroot_proc install "$@"
+		lb chroot_sysfs install "$@"
+		lb chroot_devpts install "$@"
 	
-	# re-create initrd to contain the installer.md5 file
-	Chroot chroot "env FLASH_KERNEL_SKIP=1 update-initramfs -k all -t -u -v"
+		# re-create initrd to contain the installer.md5 file
+		Chroot chroot "env FLASH_KERNEL_SKIP=1 update-initramfs -k all -t -u -v"
 	
-	# create boot.img
-	Chroot chroot "abootimg --create /boot/installer-${KVERS}.img -f /boot/bootimg.cfg-$SUBARCH -r /boot/initrd.img-${KVERS} -k /boot/vmlinuz-${KVERS}"
+		# create boot.img
+		Chroot chroot "abootimg --create /boot/installer-${KVERS}.img -f /boot/bootimg.cfg-$SUBARCH -r /boot/initrd.img-${KVERS} -k /boot/vmlinuz-${KVERS}"
 	
-	# clean up
-	lb chroot_devpts remove "$@"
-	lb chroot_sysfs remove "$@"
-	lb chroot_proc remove "$@"
+		# clean up
+		lb chroot_devpts remove "$@"
+		lb chroot_sysfs remove "$@"
+		lb chroot_proc remove "$@"
 	
-	cp "chroot/boot/installer-${KVERS}.img" "$PREFIX.bootimg-$FLAVOUR"
-	ln -sf "$PREFIX.bootimg-$FLAVOUR" "$PREFIX.bootimg"
+		cp "chroot/boot/installer-${KVERS}.img" "$PREFIX.bootimg-$FLAVOUR"
+		ln -sf "$PREFIX.bootimg-$FLAVOUR" "$PREFIX.bootimg"
+		;;
 
-fi
+	raspi2|raspi3)
+		# copy the kernel and initrd to a predictable directory for
+		# ubuntu-image consumption.  In some cases, like in pi2/3
+		# u-boot, the bootloader needs to contain the kernel and initrd,
+		# so during rootfs build we copy it over to a directory that
+		# ubuntu-image looks for and shoves into the bootloader
+		# partition.
+		UBOOT_BOOT="image/boot/uboot"
+
+		mkdir -p $UBOOT_BOOT
+
+		cp $PREFIX.initrd $UBOOT_BOOT/initrd.img || true
+		cp $PREFIX.kernel $UBOOT_BOOT/vmlinuz || true
+		;;
+esac
 
 if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-touch-custom" ]; then
 	sourceslist="chroot/etc/apt/sources.list"
@@ -920,3 +1021,17 @@ case $PROJECT in
 		fi
 		;;
 esac
+
+if [ -f "config/magic-proxy.pid" ]; then
+    kill -TERM $(cat config/magic-proxy.pid)
+    rm -f config/magic-proxy.pid
+
+    # Remove previously-inserted iptables rule.
+    iptables -t nat -D OUTPUT -p tcp --dport 80 -m owner ! --uid-owner daemon \
+        -j REDIRECT --to 8080
+fi
+
+case $PROJECT in
+    ubuntu-cpc)
+        config/hooks.d/remove-implicit-artifacts
+esac

live-build/auto/config

@@ -4,6 +4,8 @@ set -e
 rm -rf config
 
 echo "Building on $(hostname --fqdn)"
+SUITENUM=$(distro-info --series="$SUITE" -r | awk '{ print $1 }')
+HWE_SUFFIX="hwe-${SUITENUM}"
 
 SEEDMIRROR=http://people.canonical.com/~ubuntu-archive/seeds/
 if [ -z "$MIRROR" ]; then
@@ -33,6 +35,7 @@ fi
 
 mkdir -p config
 cp -af /usr/share/livecd-rootfs/live-build/functions config/functions
+cp -af /usr/share/livecd-rootfs/live-build/snap-seed-parse.py config/snap-seed-parse
 
 mkdir -p config/package-lists
 
@@ -59,9 +62,19 @@ add_task ()
 	# failure.
 
 	for task; do
+		if [ "$task" = "ubuntu-budgie-desktop" ]; then
+			filter="| grep -v '^linux.*tools'"
+		else
+			filter=""
+		fi
+		if [ -n "$HWE" ]; then
+			# If HWE is set, we strip out drivers not available in the
+			# HWE stack, then sed the rest with their HWE suffixes:
+			filter="$filter | sed -e 's/xserver-xorg.*/&-${HWE_SUFFIX}/'"
+		fi
 		# We need a ridiculous number of backslashes to protect
 		# parentheses from eval.
-		echo "!chroot chroot apt-cache dumpavail | grep-dctrl -nsPackage \\\\\\( -XFArchitecture $ARCH -o -XFArchitecture all \\\\\\) -a -wFTask $task" >> "config/package-lists/livecd-rootfs.list.chroot_$pass"
+		echo "!chroot chroot apt-cache dumpavail | grep-dctrl -nsPackage \\\\\\( -XFArchitecture $ARCH -o -XFArchitecture all \\\\\\) -a -wFTask $task $filter" >> "config/package-lists/livecd-rootfs.list.chroot_$pass"
 	done
 }
 
@@ -107,8 +120,13 @@ add_binary_hook ()
 
 if [ -z "${IMAGEFORMAT:-}" ]; then
 	case $PROJECT:${SUBPROJECT:-} in
-		ubuntu-cpc:*)
-			IMAGEFORMAT=ext4
+		ubuntu-cpc:*|ubuntu:desktop-preinstalled)
+			if [ "$SUBARCH" = "raspi3" ]; then
+				# For now only raspi3, but others are soon to follow
+				IMAGEFORMAT=ubuntu-image
+			else
+				IMAGEFORMAT=ext4
+			fi
 			;;
 		ubuntu-server:live)
 			IMAGEFORMAT=plain
@@ -144,22 +162,19 @@ case $IMAGEFORMAT in
 		;;
 
 	plain)
-		INITRAMFS_TYPE=none
 		case $PROJECT:${SUBPROJECT:-} in
 			ubuntu-server:live)
-				# Stop lb installing casper into filesystem.squashfs
-				#  by skipping lb_chroot_live-packages.
-				skip_lb_stage chroot_live-packages
-				INITRAMFS_TYPE=auto
+	                        touch config/universe-enabled
 				;;
 			*)
 				PREINSTALLED=true
 				;;
 		esac
-		OPTS="${OPTS:+$OPTS }--initramfs $INITRAMFS_TYPE --chroot-filesystem $IMAGEFORMAT"
+		OPTS="${OPTS:+$OPTS }--initramfs none --chroot-filesystem $IMAGEFORMAT"
 		;;
 
 	ubuntu-image)
+		UBUNTU_IMAGE_ARGS=""
 		case "$ARCH+${SUBARCH:-}" in
 			amd64+*)
 				MODEL=pc-amd64 ;;
@@ -169,29 +184,66 @@ case $IMAGEFORMAT in
 				MODEL=dragonboard ;;
 			armhf+raspi2)
 				MODEL=pi2 ;;
-			armhf+raspi3)
+			armhf+raspi3|arm64+raspi3)
 				MODEL=pi3 ;;
 			armhf+cm3)
 				MODEL=cm3 ;;
+			armhf+imx6)
+				MODEL=nitrogen6x ;;
 			*)
 				echo "Model $ARCH+${SUBARCH:-} unknown to livecd-rootfs" >&2
 				exit 1
 				;;
 		esac
-		case $MODEL in
-			pc-amd64|pc-i386)
-				UBUNTU_IMAGE_ARGS="--image-size 3700M" ;;
-			*)
-				UBUNTU_IMAGE_ARGS="" ;;
-		esac
 
-		echo "IMAGEFORMAT=$IMAGEFORMAT" >> config/common
-		echo "UBUNTU_IMAGE_ARGS=\"$UBUNTU_IMAGE_ARGS\"" >> config/common
-		# Store model assertion in top dir to get it picked up later as a build artifact
-		env SNAPPY_STORE_NO_CDN=1 snap known --remote model series=16 model="$MODEL" brand-id=canonical > "$PREFIX".model-assertion
-		echo "Configured ubuntu-image for the following model assertion:"
-		cat "$PREFIX".model-assertion
-		echo "----------------------------------------------------------"
+		if [ $PROJECT = "ubuntu-core" ]; then
+			# snap-based core images
+
+			case $MODEL in
+				pc-amd64|pc-i386)
+					[ -z "${SUBARCH:-}" ] \
+					&& UBUNTU_IMAGE_ARGS="--image-size 3700M"
+					;;
+			esac
+			case $SUITE in
+				xenial)
+					# Ubuntu Core 16
+					;;
+				*)
+					# Ubuntu Core 18
+					MODEL="ubuntu-core-18-${MODEL#pc-}" ;;
+			esac
+
+			case "$ARCH+${SUBARCH:-}" in
+				amd64+kassel)
+					EXTRA_SNAPS="core bluez alsa-utils"
+					;;
+				*) ;;
+			esac
+			for snap in $EXTRA_SNAPS; do
+				UBUNTU_IMAGE_ARGS="$UBUNTU_IMAGE_ARGS --extra-snaps $snap"
+			done
+			echo "IMAGEFORMAT=$IMAGEFORMAT" >> config/common
+			echo "UBUNTU_IMAGE_ARGS=\"$UBUNTU_IMAGE_ARGS\"" >> config/common
+			# Store model assertion in top dir to get it picked up later as a build artifact
+			env SNAPPY_STORE_NO_CDN=1 snap known --remote model series=16 model="$MODEL" brand-id=canonical > "$PREFIX".model-assertion
+			echo "Configured ubuntu-image for the following model assertion:"
+			cat "$PREFIX".model-assertion
+			echo "----------------------------------------------------------"
+		else
+			# classic images
+
+			UBUNTU_IMAGE_ARGS="$UBUNTU_IMAGE_ARGS${PROPOSED:+ --with-proposed}"
+			UBUNTU_IMAGE_ARGS="$UBUNTU_IMAGE_ARGS${EXTRA_PPAS:+ --extra-ppas \"$EXTRA_PPAS\"}"
+
+			git clone git://git.launchpad.net/~canonical-foundations/snap-$MODEL/+git/github-mirror -b classic config/$PREFIX-gadget
+
+			echo "IMAGEFORMAT=$IMAGEFORMAT" >> config/common
+			echo "SUITE=$SUITE" >> config/common
+			echo "UBUNTU_IMAGE_ARGS=\"$UBUNTU_IMAGE_ARGS\"" >> config/common
+
+			echo "Configured ubuntu-image for the following gadget model: $MODEL"
+		fi
 		# Fake finished configuration for lb build
 		mkdir -p .build
 		touch .build/config
@@ -213,9 +265,9 @@ case $IMAGEFORMAT in
 esac
 
 if [ "$PREINSTALLED" = "true" ]; then
-	# This is an oem-config preinstalled image, touch a random file that
-	# we can refer back to during build, cause that's wildly hackish
-	touch config/oem-config-preinstalled
+	# Touch a random file that we can refer back to during build,
+	# cause that's wildly hackish
+	touch config/universe-enabled
 	case $PROJECT in
 		kubuntu*)
 			add_package live oem-config-kde ubiquity-frontend-kde
@@ -238,6 +290,13 @@ if [ "$PREINSTALLED" = "true" ]; then
 			;;
 		ubuntu-core|ubuntu-base|base|ubuntu-touch|ubuntu-touch-custom|ubuntu-cpc|ubuntu-desktop-next)
 			;;
+		ubuntu)
+			add_package live oem-config-gtk ubiquity-frontend-gtk
+			add_package live ubiquity-slideshow-ubuntu
+			if [ "$SUBPROJECT" = "desktop-preinstalled" ]; then
+				add_package live language-pack-en-base
+			fi
+			;;
 		*)
 			add_package live oem-config-gtk ubiquity-frontend-gtk
 			add_package live ubiquity-slideshow-ubuntu
@@ -254,8 +313,6 @@ case $BINARYFORMAT in
 		;;
 esac
 
-SIGNED_KERNEL_PACKAGE="linux-signed-generic"
-
 if [ "${SUBPROJECT:-}" = minimized ]; then
 	OPTS="${OPTS:+$OPTS }--bootstrap-flavour=minimal --linux-packages=linux-image"
 fi
@@ -330,10 +387,12 @@ esac
 
 case $PROJECT in
 	ubuntu|ubuntu-dvd)
+		HWE="yes"
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard ubuntu-desktop
 		LIVE_TASK='ubuntu-live'
 		case $ARCH in
-			amd64)	add_package live $SIGNED_KERNEL_PACKAGE ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		;;
 
@@ -341,9 +400,9 @@ case $PROJECT in
 		add_task install minimal standard ubuntu-desktop-next ubuntu-sdk-libs
 		COMPONENTS='main restricted universe'
 		LIVE_TASK='ubuntu-touch-live'
-		KERNEL_FLAVOURS=generic
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 
 		# system image snappy desktop next image
@@ -357,11 +416,13 @@ case $PROJECT in
 		;;
 
 	kubuntu|kubuntu-dvd)
+		HWE="yes"
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard
 		add_task install kubuntu-desktop
 		LIVE_TASK='kubuntu-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		COMPONENTS='main restricted universe'
 		add_chroot_hook remove-gnome-icon-cache
@@ -387,25 +448,25 @@ case $PROJECT in
 		;;
 
 	edubuntu|edubuntu-dvd)
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard ubuntu-desktop edubuntu-desktop-gnome
 		LIVE_TASK='edubuntu-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		COMPONENTS='main restricted universe'
 		;;
 
 	xubuntu)
+		HWE="yes"
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard xubuntu-desktop
 		add_package install xterm
 		LIVE_TASK='xubuntu-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		COMPONENTS='main restricted universe multiverse'
-		case $ARCH in
-			amd64|i386)	KERNEL_FLAVOURS=generic ;;
-		esac
 		;;
 
 	ubuntu-netbook)
@@ -414,24 +475,29 @@ case $PROJECT in
 		;;
 
 	mythbuntu)
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard mythbuntu-desktop
 		LIVE_TASK='mythbuntu-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		COMPONENTS='main restricted universe multiverse'
 		;;
 
 	lubuntu)
-		add_task install minimal standard lubuntu-desktop
+		HWE="yes"
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
+		add_task install minimal standard lubuntu-gtk-core lubuntu-desktop
+		# Installing the lubuntu-gtk-core task explicitly appears
+		# to make dbus-user-session not get installed anymore:
+		add_package install dbus-user-session
+		# These themes also get dropped, maybe worth looking closer:
+		add_package install humanity-icon-theme gnome-icon-theme adwaita-icon-theme
 		LIVE_TASK='lubuntu-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		COMPONENTS='main restricted universe multiverse'
-		case $ARCH in
-			amd64|i386)	KERNEL_FLAVOURS=generic ;;
-		esac
 
 		# The Lubuntu STRUCTURE file has "feature
 		# no-follow-recommends".  Mirror this.
@@ -439,15 +505,13 @@ case $PROJECT in
 		;;
 
 	lubuntu-next)
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard lubuntu-qt-desktop
 		LIVE_TASK='lubuntu-live-qt'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		COMPONENTS='main restricted universe multiverse'
-		case $ARCH in
-			amd64|i386)	KERNEL_FLAVOURS=generic ;;
-		esac
 
 		# The Lubuntu STRUCTURE file has "feature
 		# no-follow-recommends". Mirror this.
@@ -455,46 +519,51 @@ case $PROJECT in
 		;;
 
 	ubuntu-gnome)
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard ubuntu-gnome-desktop
 		LIVE_TASK='ubuntu-gnome-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		COMPONENTS='main restricted universe'
 		;;
 
 	ubuntu-budgie)
+		HWE="yes"
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard ubuntu-budgie-desktop
 		LIVE_TASK='ubuntu-budgie-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		COMPONENTS='main restricted universe'
 		;;
 
 	ubuntu-mate)
+		HWE="yes"
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard ubuntu-mate-core ubuntu-mate-desktop
 		LIVE_TASK='ubuntu-mate-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		COMPONENTS='main restricted universe multiverse'
 		;;
 
 	ubuntustudio-dvd)
+		KERNEL_FLAVOURS="lowlatency${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard ubuntustudio-desktop ubuntustudio-audio ubuntustudio-fonts ubuntustudio-graphics ubuntustudio-video ubuntustudio-publishing ubuntustudio-photography
 		COMPONENTS='main restricted universe multiverse'
-		case $ARCH in
-			amd64|i386)	KERNEL_FLAVOURS=lowlatency ;;
-		esac
 		;;
 
 	ubuntukylin)
+		HWE="yes"
+		KERNEL_FLAVOURS="generic${HWE:+-$HWE_SUFFIX}"
 		add_task install minimal standard ubuntukylin-desktop
 		add_package install ubuntukylin-default-settings
 		LIVE_TASK='ubuntukylin-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live linux-signed-$KERNEL_FLAVOURS ;;
 		esac
 		COMPONENTS='main restricted universe'
 		;;
@@ -509,10 +578,7 @@ case $PROJECT in
 			live)
 				add_task install standard
 				add_task install server
-				LIVE_TASK='cloud-image'
-				case $ARCH in
-					amd64)	add_package live linux-signed-generic ;;
-				esac
+				add_package install cloud-init
 				;;
 		esac
 		COMPONENTS='main'
@@ -668,6 +734,37 @@ case $PROJECT in
 		;;
 esac
 
+case $SUBPROJECT in
+	buildd)
+		OPTS="${OPTS:+$OPTS }--archive-areas main"
+		COMPONENTS='main restricted universe multiverse'
+		OPTS="${OPTS:+$OPTS }--apt-recommends false"
+		OPTS="${OPTS:+$OPTS }--apt-secure false"
+		OPTS="${OPTS:+$OPTS }--parent-mirror-binary ${MIRROR}"
+		# XXX cjwatson 2018-04-27: We need to work out how to make
+		# this conditional so that we can do things like building
+		# buildd chroots with -updates.  This probably involves
+		# either extending the PROPOSED hack or fixing the strange
+		# way that SUITE is in fact a series; in either case it's
+		# likely to involve work both here and in launchpad-buildd.
+		OPTS="${OPTS:+$OPTS }--security false --volatile false"
+
+		add_package install adduser
+		add_package install policyrcd-script-zg2
+		add_package install pkgbinarymangler
+		add_package install ca-certificates
+		add_package install gpg
+		add_package install gpg-agent
+		add_package install tzdata
+		add_package install fakeroot
+		add_package install build-essential
+		# Needed for LXD-based builds.
+		add_package install init
+
+		cp -af /usr/share/livecd-rootfs/live-build/make-lxd-metadata.py config/make-lxd-metadata
+		;;
+esac
+
 # we'll expand the base seed given here according to the STRUCTURE file, and
 # then look in all of the seeds found to see which snaps are seeded
 case $PROJECT:${SUBPROJECT:-} in
@@ -700,7 +797,12 @@ if [ -n "${BASE_SEED}" ]; then
 		seed_snaps=$(sed -rn '1,/-----/d;/-----/,$d; s/(.*) \|.*/\1/; s, \(classic\),/classic,; p' "${file}")
 		for snap in ${seed_snaps}; do
 			echo "snap: found ${snap}"
-			ALL_SNAPS="${ALL_SNAPS:+${ALL_SNAPS} }${snap}"
+			# Reorder gtk-common-themes first due to LP: #1772844
+			if [ "${snap}" = "gtk-common-themes" ]; then
+				ALL_SNAPS="${snap}${ALL_SNAPS:+ ${ALL_SNAPS}}"
+			else
+				ALL_SNAPS="${ALL_SNAPS:+${ALL_SNAPS} }${snap}"
+			fi
 		done
 	done
 	if [ -n "${ALL_SNAPS}" ] || [ -n "${HOOK_SNAPS}" ]; then
@@ -740,7 +842,7 @@ case $PROJECT in
 esac
 
 case $ARCH in
-	armel|armhf)
+	armel|armhf|arm64)
 		KERNEL_FLAVOURS="${SUBARCH:-$KERNEL_FLAVOURS}"
 		case $SUBARCH in
 			dove)
@@ -775,16 +877,27 @@ case $ARCH in
 				;;
 			raspi2)
 				COMPONENTS='main restricted universe multiverse'
-				add_package install linux-firmware-raspi2 u-boot-rpi flash-kernel u-boot-tools
+				add_package install linux-firmware-raspi2 u-boot-rpi flash-kernel u-boot-tools wpasupplicant
 				BINARY_REMOVE_LINUX=false
 				;;
+			raspi3)
+				COMPONENTS='main restricted universe multiverse'
+				KERNEL_FLAVOURS=raspi2-5.3
+				add_package install linux-firmware-raspi2 u-boot-rpi flash-kernel u-boot-tools wpasupplicant
+				BINARY_REMOVE_LINUX=false
+				;;
+			imx6)
+				COMPONENTS='main restricted universe multiverse'
+				KERNEL_FLAVOURS=generic
+				add_package install flash-kernel u-boot-tools wpasupplicant
+				BINARY_REMOVE_LINUX=false
+				;;
+
 		esac
 		;;
 esac
 
 case $PROJECT:${SUBPROJECT:-} in
-	ubuntu-server:live)
-		;;
 	ubuntu-server:*|ubuntu-base:*|ubuntu-touch:*|ubuntu-touch-custom:*)
 		OPTS="${OPTS:+$OPTS }--linux-packages=none --initramfs=none"
 		KERNEL_FLAVOURS=none
@@ -828,6 +941,7 @@ lb config noauto \
 	--initsystem none \
 	--bootloader "$BOOTLOADER" \
 	${INITRAMFS_COMPRESSION:+--initramfs-compression "$INITRAMFS_COMPRESSION"} \
+	--checksums none \
 	--cache false \
 	${BOOTAPPEND_LIVE:+--bootappend-live "$BOOTAPPEND_LIVE"} \
 	$OPTS \
@@ -840,9 +954,11 @@ echo "LB_BINARY_HOOKS=\"$BINARY_HOOKS\"" >> config/binary
 echo "BUILDSTAMP=\"$NOW\"" >> config/binary
 echo "SUBPROJECT=\"${SUBPROJECT:-}\"" >> config/binary
 echo "LB_DISTRIBUTION=\"$SUITE\"" >> config/binary
+echo "IMAGEFORMAT=\"$IMAGEFORMAT\"" >> config/chroot
+echo "SUBARCH=\"$SUBARCH\"" >> config/chroot
 
 case $ARCH+$SUBARCH in
-	armhf+raspi2)
+	armhf+raspi2|armhf+raspi3|arm64+raspi3)
 		cat > config/hooks/01-firmware-directory.chroot_early <<EOF
 #!/bin/sh -ex
 mkdir -p /boot/firmware
@@ -923,6 +1039,19 @@ rm -f /etc/fstab
 EOF
 fi
 
+if [ $PROJECT != ubuntu-cpc ]; then
+	cat > config/hooks/100-preserve-apt-prefs.chroot <<\EOF
+#! /bin/sh -ex
+
+# live-build "helpfully" removes /etc/apt/preferences.d/* so we put a
+# copy somewhere it won't touch it.
+
+if [ -n "$(ls -A /etc/apt/preferences.d)" ]; then
+    cp -a /etc/apt/preferences.d /etc/apt/preferences.d.save
+fi
+EOF
+fi
+
 if [ $PROJECT = ubuntukylin ]; then
 	cat > config/hooks/100-ubuntukylin.chroot <<EOF
 #! /bin/sh
@@ -968,15 +1097,46 @@ EOF
 		fi
 		;;
 
-	ubuntu-touch:*|ubuntu-touch-custom:*|ubuntu-core:system-image|ubuntu-desktop-next:system-image|ubuntu-cpc:*|ubuntu-server:live)
-		cp -af /usr/share/livecd-rootfs/live-build/${PROJECT}/* \
-			config/
+	ubuntu-touch:*|ubuntu-touch-custom:*|ubuntu-core:system-image|ubuntu-desktop-next:system-image|ubuntu-cpc:*|ubuntu-server:live|ubuntu:desktop-preinstalled)
+		# Ensure that most things e.g. includes.chroot are copied as is
+		for entry in /usr/share/livecd-rootfs/live-build/${PROJECT}/*; do
+			case $entry in
+				*hooks*)
+					# But hooks are shared across the projects with symlinks
+					# dereference them
+					cp -afL $entry config/
+					;;
+				*)
+					# Most places want to preserve symlinks as is
+					cp -af $entry config/
+					;;
+			esac
+		done
+
+		if [ "$PROJECT" = "ubuntu-cpc" ]; then
+			case ${IMAGE_TARGETS:-} in
+				"")
+					config/hooks.d/make-hooks --hooks-dir config/hooks all
+					;;
+				*)
+					config/hooks.d/make-hooks --hooks-dir config/hooks \
+						"$IMAGE_TARGETS"
+					;;
+			esac
+		fi
+
 		if [ "$IMAGEFORMAT" = none ]; then
 			rm -f config/hooks/*.binary*
 		fi
 		;;
 esac
 
+case $SUBPROJECT in
+	buildd)
+		cp -af /usr/share/livecd-rootfs/live-build/buildd/* config/
+		;;
+esac
+
 if [ "$EXTRA_PPAS" ]; then
 	rm -f config/archives/extra-ppas.list.chroot \
 	      config/archives/extra-ppas.pref.chroot \

live-build/buildd/hooks/00-kernel-img.chroot

@@ -0,0 +1,5 @@
+#! /bin/sh
+set -e
+
+# At one point, kernel builds needed this.
+echo do_initrd = Yes >>/etc/kernel-img.conf

live-build/buildd/hooks/00-mirror.binary

@@ -0,0 +1,12 @@
+#! /bin/sh
+set -e
+
+. config/bootstrap
+
+# Use a public-facing mirror URL, for the benefit of
+# sbuild-launchpad-chroot.  We deliberately do this only after live-build
+# has run "apt-get update" for the last time, in order that
+# /var/lib/apt/lists/ has suitable cached Packages files; this speeds up
+# builds on buildds.
+sed -i "s,${LB_PARENT_MIRROR_BINARY},${LB_MIRROR_BINARY},g" \
+	binary/etc/apt/sources.list

live-build/buildd/hooks/01-pkgbinarymangler.chroot

@@ -0,0 +1,10 @@
+#! /bin/sh
+set -e
+
+# Configure pkgbinarymangler.
+sed -i /^enable/s/false/true/ \
+	/etc/pkgbinarymangler/maintainermangler.conf \
+	/etc/pkgbinarymangler/striptranslations.conf || true
+sed -i /^invalid_current/s/ignore/fail/ \
+	/etc/pkgbinarymangler/maintainermangler.conf \
+	/etc/pkgbinarymangler/striptranslations.conf || true

live-build/buildd/hooks/01-policy-rc-d.binary

@@ -0,0 +1,13 @@
+#! /bin/sh
+set -e
+
+# Put the /usr/sbin/policy-rc.d alternatives symlink in place.  Ordinarily
+# update-alternatives ought to create this when policyrcd-script-zg2 is
+# installed, but that doesn't work because live-build has already installed
+# a dummy one at that point.  The simplest approach is to repair the
+# situation by putting it in place here.
+if [ -L binary/etc/alternatives/policy-rc.d ] && \
+   [ ! -e binary/usr/sbin/policy-rc.d ] && \
+   [ ! -L binary/usr/sbin/policy-rc.d ]; then
+	ln -s /etc/alternatives/policy-rc.d binary/usr/sbin/policy-rc.d
+fi

live-build/buildd/hooks/02-user.chroot

@@ -0,0 +1,9 @@
+#! /bin/sh
+set -e
+
+# Create the buildd user and group.
+addgroup --gid 2501 buildd
+adduser --system --disabled-password --gecos 'Build Daemon user' \
+	--ingroup buildd --uid 2001 --shell /bin/bash buildd
+mkdir -p /build/buildd
+chown buildd:buildd /build/buildd

live-build/buildd/hooks/50-buildd-tar.binary

@@ -0,0 +1,10 @@
+#! /bin/sh
+# A few things (launchpad-buildd, sbuild-launchpad-chroot) rely on the
+# top-level directory being "chroot-autobuild", so we have to do this
+# ourselves.
+set -e
+
+# gzip was chosen for fastest decompression speed: it decompresses buildd
+# chroots about twice as fast as xz and about five times as fast as bzip2.
+tar --transform='s,^binary,chroot-autobuild,' --sort=name --numeric-owner \
+	-czf "livecd.$PROJECT.rootfs.tar.gz" binary

live-build/buildd/hooks/51-buildd-lxd.binary

@@ -0,0 +1,16 @@
+#! /bin/sh
+# Some build types prefer a LXD image over a traditional chroot tarball.
+set -e
+
+. config/bootstrap
+
+TMPDIR="$(mktemp -d)"
+config/make-lxd-metadata "${LB_DISTRIBUTION%-*}" "$ARCH" \
+	>"$TMPDIR/metadata.yaml"
+tar --numeric-owner -cf "livecd.$PROJECT.lxd.tar" -C "$TMPDIR" metadata.yaml
+rm -rf "$TMPDIR"
+# When using the combined metadata/rootfs form, the rootfs must be under
+# rootfs/ rather than under chroot-autobuild/.
+tar --transform='s,^binary,rootfs,' --sort=name --numeric-owner \
+	-rf "livecd.$PROJECT.lxd.tar" binary
+gzip -9 "livecd.$PROJECT.lxd.tar"

live-build/buildd/includes.chroot/etc/apt/apt.conf.d/99buildd

@@ -0,0 +1,2 @@
+DPkg::Options {"--force-unsafe-io";};
+DPkg::Use-Pty "false";

live-build/buildd/includes.chroot/etc/apt/preferences.d/backports

@@ -0,0 +1,3 @@
+Package: *
+Pin: release a=*-backports
+Pin-Priority: 500

live-build/buildd/includes.chroot/etc/fstab

@@ -0,0 +1 @@
+/dev/root / ext2 noatime,errors=remount-ro 0 1

live-build/buildd/includes.chroot/etc/hostname

@@ -0,0 +1 @@
+INVALID

live-build/buildd/includes.chroot/etc/hosts

@@ -0,0 +1,9 @@
+127.0.0.1	localhost.localdomain	localhost
+
+# The following lines are desirable for IPv6 capable hosts
+::1     ip6-localhost ip6-loopback
+fe00::0 ip6-localnet
+ff00::0 ip6-mcastprefix
+ff02::1 ip6-allnodes
+ff02::2 ip6-allrouters
+ff02::3 ip6-allhosts

live-build/buildd/includes.chroot/usr/local/sbin/policy-rc.d

@@ -0,0 +1,13 @@
+#!/bin/sh
+
+# policy-rc.d script for chroots.
+# Copyright (c) 2007 Peter Palfrader <peter@palfrader.org>
+# License: <weasel> MIT, if you want one.
+
+while true; do
+  case "$1" in
+    -*)         shift ;;
+    makedev)    exit 0;;
+    *) echo "Not running services in chroot."; exit 101 ;;
+  esac
+done

live-build/buildd/preseed/debconf.preseed

@@ -0,0 +1,2 @@
+# We never want debconf interaction.
+debconf debconf/frontend select Noninteractive

live-build/buildd/preseed/man-db.preseed

@@ -0,0 +1,3 @@
+# Avoid unnecessary manual page database builds (see
+# https://bugs.debian.org/554914).
+man-db man-db/auto-update boolean false

live-build/buildd/preseed/sun-java6.preseed

@@ -0,0 +1,3 @@
+# Pre-accept interactive EULA prompts.
+sun-java6-bin shared/accepted-sun-dlj-v1-1 boolean true
+sun-java6-jre shared/accepted-sun-dlj-v1-1 boolean true

live-build/functions

@@ -1,7 +1,7 @@
 # vi: ts=4 expandtab syntax=sh
 
-#imagesize=${IMAGE_SIZE:-$((2252*1024**2))}  # 2.2G (the current size we ship)
-imagesize=${IMAGE_SIZE:-2361393152}  # 2.2G (the current size we ship)
+# default imagesize = 2252*1024**2 = 2.2G (the current size we ship)
+imagesize=${IMAGE_SIZE:-2361393152}
 fs_label="${FS_LABEL:-rootfs}"
 
 rootfs_dev_mapper=
@@ -43,6 +43,17 @@ create_empty_disk_image() {
     dd if=/dev/zero of="$1" bs=1 count=0 seek="${imagesize}"
 }
 
+create_manifest() {
+    local chroot_root=${1}
+    local target_file=${2}
+    echo "create_manifest chroot_root: ${chroot_root}"
+    dpkg-query --show --admindir="${chroot_root}/var/lib/dpkg" > ${target_file}
+    echo "create_manifest call to dpkg-query finished."
+    ./config/snap-seed-parse "${chroot_root}" "${target_file}"
+    echo "create_manifest call to snap_seed_parse finished."
+    echo "create_manifest finished"
+}
+
 make_ext4_partition() {
     device="$1"
     label=${fs_label:+-L "${fs_label}"}
@@ -367,6 +378,47 @@ release_ver() {
     distro-info --series="$LB_DISTRIBUTION" -r | awk '{ print $1 }'
 }
 
+_snap_post_process() {
+    # Look for the 'core' snap. If it is not present, assume that the image
+    # contains only snaps with bases >= core18. In that case snapd is
+    # preseeded. However, when 'core' is being installed and snapd has not
+    # been installed by a call to 'snap_preseed' (see below) then it is
+    # removed again.
+    local CHROOT_ROOT=$1
+    local SNAP_NAME=$2
+
+    local seed_dir="$CHROOT_ROOT/var/lib/snapd/seed"
+    local snaps_dir="$seed_dir/snaps"
+    local seed_yaml="$seed_dir/seed.yaml"
+    local assertions_dir="$seed_dir/assertions"
+    local snapd_install_stamp="$seed_dir/.snapd-explicit-install-stamp"
+
+    case $SNAP_NAME in
+        core[0-9]*)
+            # If the 'core' snap is not present, assume we are coreXX-only and
+            # install the snapd snap.
+            if [ ! -f ${snaps_dir}/core_[0-9]*.snap ]; then
+                _snap_preseed $CHROOT_ROOT snapd stable
+            fi
+            ;;
+        core)
+            # If the snapd snap has been seeded, but not marked as explicitly
+            # installed (see snap_preseed below), then remove it.
+            if [ -f ${snaps_dir}/snapd_[0-9]*.snap ] && \
+                    [ ! -f "$snapd_install_stamp" ]
+            then
+                # Remove snap, assertions and entry in seed.yaml
+                rm -f ${snaps_dir}/snapd_[0-9]*.snap
+                rm -f ${assertions_dir}/snapd_[0-9]*.assert
+                sed -i -e'N;/name: snapd/,+2d' $seed_yaml
+            fi
+            ;;
+        *)
+            # ignore
+            ;;
+    esac
+}
+
 _snap_preseed() {
     # Download the snap/assertion and add to the preseed
     local CHROOT_ROOT=$1
@@ -381,11 +433,46 @@ _snap_preseed() {
 
     # Download the snap & assertion
     local snap_download_failed=0
-    chroot $CHROOT_ROOT sh -c "
+
+    # Preseed a snap only once
+    if [ -f ${snaps_dir}/${SNAP_NAME}_[0-9]*.snap ]; then
+        return
+    fi
+
+    case $SNAP_NAME in
+        snapd)
+            # snapd is self-contained, ignore base
+            ;;
+        *)
+            # Determine if and what core snap is needed
+            local snap_info
+
+            snap_info=$(/usr/share/livecd-rootfs/snap-tool info \
+                --cohort-key="${COHORT_KEY:-}" \
+                --channel="$CHANNEL" "${SNAP_NAME}" \
+            )
+
+            if [ $? -ne 0 ]; then
+                echo "Failed to retrieve base of $SNAP_NAME!"
+                exit 1
+            fi
+
+            local core_snap=$(echo "$snap_info" | grep '^base:' | awk '{print $2}')
+
+            # If $core_snap is not the empty string then SNAP itself is not a core
+            # snap and we must additionally seed the core snap.
+            if [ -n "$core_snap" ]; then
+                _snap_preseed $CHROOT_ROOT $core_snap stable
+            fi
+            ;;
+    esac
+
+    sh -c "
         set -x;
-        cd /var/lib/snapd/seed;
-        SNAPPY_STORE_NO_CDN=1 snap download \
-            --channel=$CHANNEL \"$SNAP_NAME\"" || snap_download_failed=1
+        cd \"$CHROOT_ROOT/var/lib/snapd/seed\";
+        SNAPPY_STORE_NO_CDN=1 /usr/share/livecd-rootfs/snap-tool download \
+            --cohort-key=\"${COHORT_KEY:-}\" \
+            --channel=\"$CHANNEL\" \"$SNAP_NAME\"" || snap_download_failed=1
     if [ $snap_download_failed = 1 ] ; then
         echo "If the channel ($CHANNEL) includes '*/ubuntu-##.##' track per "
         echo "Ubuntu policy (ex. stable/ubuntu-18.04) the publisher will need "
@@ -410,6 +497,12 @@ EOF
 
     echo -n "    file: " >> $seed_yaml
     (cd $snaps_dir; ls -1 ${SNAP_NAME}_*.snap) >> $seed_yaml
+
+    # If $core_snap is the empty string then SNAP itself *may be* a core snap,
+    # and we run some post-processing logic.
+    if [ -z "$core_snap" ]; then
+        _snap_post_process $CHROOT_ROOT $SNAP_NAME
+    fi
 }
 
 snap_prepare_assertions() {
@@ -466,9 +559,10 @@ snap_prepare() {
 
     snap_prepare_assertions "$CHROOT_ROOT" "$CUSTOM_BRAND_MODEL"
 
-    # Download the core snap
-    if ! [ -f $snaps_dir/core_[0-9]*.snap ] ; then
-        _snap_preseed $CHROOT_ROOT core stable
+    # ubuntu-cpc:minimized has its own special snap handling
+    if [ "$PROJECT:${SUBPROJECT:-}" != ubuntu-cpc:minimized ]; then
+        # Download the core snap
+        _snap_preseed "$CHROOT_ROOT" core stable
     fi
 }
 
@@ -476,10 +570,37 @@ snap_preseed() {
     # Preseed a snap in the image
     local CHROOT_ROOT=$1
     local SNAP=$2
+    local SNAP_NAME=${SNAP%/*}
     # Per Ubuntu policy, all seeded snaps (with the exception of the core
     # snap) must pull from stable/ubuntu-$(release_ver) as their channel.
     local CHANNEL=${3:-"stable/ubuntu-$(release_ver)"}
 
     snap_prepare $CHROOT_ROOT
+
     _snap_preseed $CHROOT_ROOT $SNAP $CHANNEL
+
+    # Mark this image as having snapd installed explicitly.
+    case $SNAP_NAME in
+        snapd)
+            touch "$CHROOT_ROOT/var/lib/snapd/seed/.snapd-explicit-install-stamp"
+            ;;
+    esac
+
+
+    # Do basic validation of generated snapd seed.yaml, doing it here
+    # means we catch all the places(tm) that snaps are added but the
+    # downside is that each time a snap is added the seed must be valid,
+    # i.e. snaps with bases need to add bases first etc.
+    if [ -e chroot/var/lib/snapd/seed/seed.yaml ]; then
+        snap debug validate-seed "$CHROOT_ROOT/var/lib/snapd/seed/seed.yaml"
+    fi
+}
+
+is_live_layer () {
+    local pass=$1
+    for livepass in $LIVE_PASSES; do
+	    [ "$livepass" != "$pass" ] && continue
+        return 0
+    done
+    return 1
 }

live-build/make-lxd-metadata.py

@@ -0,0 +1,49 @@
+#! /usr/bin/python3
+
+"""Make a metadata.yaml file for a LXD image."""
+
+import argparse
+import json
+import sys
+import time
+
+
+# Map dpkg architecture names to LXD architecture names.
+lxd_arches = {
+    "amd64": "x86_64",
+    "arm64": "aarch64",
+    "armhf": "armv7l",
+    "i386": "i686",
+    "powerpc": "ppc",
+    "ppc64el": "ppc64le",
+    "s390x": "s390x",
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("series", help="Ubuntu series name")
+    parser.add_argument("architecture", help="Ubuntu architecture name")
+    args = parser.parse_args()
+
+    metadata = {
+        "architecture": lxd_arches[args.architecture],
+        "creation_date": int(time.time()),
+        "properties": {
+            "os": "Ubuntu",
+            "series": args.series,
+            "architecture": args.architecture,
+            "description": "Ubuntu buildd %s %s" % (
+                args.series, args.architecture),
+            },
+        }
+
+    # Encoding this as JSON is good enough, and saves pulling in a YAML
+    # library dependency.
+    json.dump(
+        metadata, sys.stdout, sort_keys=True, indent=4, separators=(",", ": "),
+        ensure_ascii=False)
+
+
+if __name__ == "__main__":
+    main()

live-build/snap-seed-parse.py

@@ -0,0 +1,68 @@
+#!/usr/bin/python3
+ 
+"""
+Usage: snap-seed-parse [${chroot_dir}] <output file>
+ 
+This script looks for a seed.yaml path in the given root directory, parsing
+it and appending the parsed lines to the given output file.
+ 
+The $chroot_dir argument is optional and will default to the empty string.
+"""
+ 
+import argparse
+import os.path
+import re
+import yaml
+ 
+ 
+def log(msg):
+    print("snap-seed-parse: {}".format(msg))
+ 
+ 
+log("Parsing seed.yaml")
+ 
+parser = argparse.ArgumentParser()
+parser.add_argument('chroot', nargs='?', default='',
+                    help='root dir for the chroot from which to generate the '
+                         'manifest')
+parser.add_argument('file', help='Output manifest to this file')
+ 
+ARGS = parser.parse_args()
+CHROOT_ROOT = ARGS.chroot
+FNAME = ARGS.file
+ 
+# Trim any trailing slashes for correct appending
+log("CHROOT_ROOT: {}".format(CHROOT_ROOT))
+if len(CHROOT_ROOT) > 0 and CHROOT_ROOT[-1] == '/':
+    CHROOT_ROOT = CHROOT_ROOT[:-1]
+ 
+# This is where we expect to find the seed.yaml file
+YAML_PATH = CHROOT_ROOT + '/var/lib/snapd/seed/seed.yaml'
+ 
+# Snaps are prepended with this string in the manifest
+LINE_PREFIX = 'snap:'
+ 
+log("yaml path: {}".format(YAML_PATH))
+if not os.path.isfile(YAML_PATH):
+    log("WARNING: yaml path not found; no seeded snaps found.")
+    exit(0)
+else:
+    log("yaml path found.")
+ 
+with open(YAML_PATH, 'r') as fh:
+    yaml_lines = yaml.safe_load(fh)['snaps']
+ 
+log('Writing manifest to {}'.format(FNAME))
+ 
+with open(FNAME, 'a+') as fh:
+    for item in yaml_lines:
+        filestring = item['file']
+        # Pull the revision number off the file name
+        revision = filestring[filestring.rindex('_')+1:]
+        revision = re.sub(r'[^0-9]', '', revision)
+        fh.write("{}{}\t{}\t{}\n".format(LINE_PREFIX,
+                                         item['name'],
+                                         item['channel'],
+                                         revision,
+                                         ))
+log('Manifest output finished.')

live-build/ubuntu-cpc/README.cpc.md

@@ -0,0 +1,68 @@
+# TL;DR
+
+In order to generate the hooks for a specific image target set, call the
+`make-hooks` script, located in `hooks.d` as
+
+    ./make-hooks --hooks-dir ../hooks <image_set>
+
+where `image_set` is the name of a series file (e.g. "vagrant") without leading
+path components. Do *not* check in the `hooks` folder, it is automatically
+generated by `auto/config` during Live Build runs.
+
+
+# Hook placement and ordering
+
+Scripts live in subfolders below the `hooks.d` folder. Currently the folders
+`chroot` and `base` exist. The folder with the name `extra` is reserved for
+private scripts, which are not included in the source of livecd-rootfs. The
+scripts are not numbered, instead the order of their execution depends on the
+order in which they are listed in a *series* file.
+
+Series files are placed in subfolders `hooks.d/base/series` or
+`hooks.d/extra/series`. Each series file contains a list of scripts to be
+executed. Empty lines and lines starting with a `#` are ignored.
+
+Series files in `extra/series` override files in `base/series` with the same
+name. For example, if a series file `base/series/cloudA` exists and a series
+file `extra/series/cloudA`, then the latter will be preferred.
+
+A series file in `extra/series` may also list scripts that are located in the
+`chroot` and `base` folders. In addition, series files can *depend* on other
+series files. For example, the series files for most custom images look similar
+to this:
+
+    depends disk-image
+    depends extra-settings
+    extra/cloudB.binary
+
+Where `disk-image` and `extra-settings` may list scripts and dependencies which
+are to be processed before the script `extra/cloudB.binary` is called.
+
+ACHTUNG: live build runs scripts with the suffix ".chroot" in a batch separate
+from scripts ending in ".binary". Even if you arrange them interleaved in your
+series files, the chroot scripts will be run before the binary scripts.
+
+# Image set selection for Live Build
+
+During a Live Build, enumerated symbolic links are generated based on the
+contents of one or more series files. The series files are selected according
+to the contents of the `IMAGE_TARGETS` environment variable. For example, in
+order to trigger the build of `squashfs` and `vagrant`, list them in the
+`IMAGE_TARGETS` variable as `squashfs,vagrant`. The separator can be a comma,
+a semi-colon or whitespace.
+
+The generation of the symbolic links is triggered from the `auto/config` script,
+from where the contents of the `IMAGE_TARGETS` environment variable are passed
+on to the `make-hooks` script.
+
+
+# Symlink generation
+
+Since Live Build itself does not know about series files, a traditional `hooks`
+folder is generated using the `make-hooks` script. The script takes as arguments
+the names of the series files to be processed.
+
+The script parses the series files and generates enumerated symbolic links for
+all entries. Per default, these are placed into a directory named `hooks` next
+to the `hooks.d` directory. This can be changed using the `--hooks-dir`
+parameter.

live-build/ubuntu-cpc/hooks.d/base/create-root-dir.binary

@@ -0,0 +1,29 @@
+#!/bin/bash -ex
+# vi: ts=4 expandtab
+#
+# Generate the root directory/manifest for rootfs.tar.xz and squashfs
+
+if [ -n "$SUBARCH" ]; then
+    echo "Skipping rootfs build for subarch flavor build"
+    exit 0
+fi
+
+. config/functions
+
+rootfs_dir=rootfs.dir
+mkdir $rootfs_dir
+cp -a chroot/* $rootfs_dir
+
+setup_mountpoint $rootfs_dir
+
+env DEBIAN_FRONTEND=noninteractive chroot $rootfs_dir apt-get --purge remove --assume-yes '^linux-.*' 'linux-base+'
+env DEBIAN_FRONTEND=noninteractive chroot $rootfs_dir apt-get --purge remove --assume-yes '^grub-.*'
+env DEBIAN_FRONTEND=noninteractive chroot $rootfs_dir apt-get autoremove --purge --assume-yes
+rm -rf $rootfs_dir/boot/grub
+
+# Keep this as some derivatives mount a tempfs here
+mkdir -p $rootfs_dir/lib/modules
+
+teardown_mountpoint $rootfs_dir
+
+create_manifest "${rootfs_dir}" "${rootfs_dir}.manifest"

live-build/ubuntu-cpc/hooks.d/base/disk-image-uefi.binary

@@ -9,8 +9,17 @@ case $ARCH in
         ;;
 esac
 
-IMAGE_STR="# CLOUD_IMG: This file was created/modified by the Cloud Image build process"
-FS_LABEL="cloudimg-rootfs"
+case ${PROJECT:-} in
+    ubuntu)
+        IMAGE_STR="# DESKTOP_IMG: This file was created/modified by the Desktop Image build process"
+        FS_LABEL="desktop-rootfs"
+        IMAGE_SIZE=12884901888 # 12G
+        ;;
+    *)
+        IMAGE_STR="# CLOUD_IMG: This file was created/modified by the Cloud Image build process"
+        FS_LABEL="cloudimg-rootfs"
+        ;;
+esac
 
 . config/binary
 
@@ -65,8 +74,8 @@ install_grub() {
     chroot mountpoint mkdir -p "${efi_boot_dir}"
 
     if [ "${SUBPROJECT:-}" = minimized ] && [ -n "$partuuid" ]; then
-        # FIXME: code duplicated between 032-disk-image.binary
-        # and 033-disk-image-uefi.binary.  We want to fix this to not
+        # FIXME: code duplicated between disk-image.binary
+        # and disk-image-uefi.binary.  We want to fix this to not
         # have initramfs-tools installed at all on these images.
         echo "partuuid found for root device; omitting initrd"
         echo "GRUB_FORCE_PARTUUID=$partuuid" >> mountpoint/etc/default/grub.d/40-force-partuuid.cfg
@@ -93,6 +102,12 @@ install_grub() {
             ;;
     esac
 
+    # This call to populate the package manifest is added here to capture
+    # grub-efi packages that otherwise would not make it into the base
+    # manifest. filesystem.packages is moved into place via symlinking to
+    # livecd.ubuntu-cpc.manifest by live-build/auto/build after lb_binary runs
+    create_manifest "mountpoint" "binary/boot/filesystem.packages"
+
     chroot mountpoint grub-install "${loop_device}" \
         --boot-directory=/boot \
         --efi-directory=/boot/efi \

live-build/ubuntu-cpc/hooks.d/base/disk-image.binary

@@ -12,6 +12,8 @@ BOOTPART_END=
 BOOT_MOUNTPOINT=
 ROOTPART_START=1
 
+my_d=$(dirname $(readlink -f ${0}))
+
 case $ARCH:$SUBARCH in
 	ppc64el:*|powerpc:*)
 		echo "POWER disk images are handled separately"
@@ -97,7 +99,7 @@ case $ARCH:$SUBARCH in
 		# not the best place for this, but neither flash-kernel nor
 		# u-boot have provisions for installing u-boot via maintainer
 		# script
-		config/hooks/raspi2/mkknlimg --dtok \
+		${my_d}/raspi2/mkknlimg --dtok \
 			mountpoint/usr/lib/u-boot/rpi_2/u-boot.bin \
 			mountpoint/boot/firmware/uboot.bin
 		;;

live-build/ubuntu-cpc/hooks.d/base/qcow2-image.binary

@@ -1,14 +1,5 @@
 #!/bin/bash -ex
 
-case $IMAGE_TARGETS in
-	""|*qcow2*)
-		;;
-	*)
-		echo "Skipping qcow2 image build"
-		exit 0
-		;;
-esac
-
 case $ARCH:$SUBARCH in
 	# Not sure if any other cloud images use subarch for something that
 	# should take qcow2 format, so only skipping this on raspi2 for now.

live-build/ubuntu-cpc/hooks.d/base/root-squashfs.binary

@@ -0,0 +1,20 @@
+#!/bin/bash -ex
+# vi: ts=4 noexpandtab
+#
+# Generate a squashfs root and manifest
+
+if [ -n "$SUBARCH" ]; then
+	echo "Skipping rootfs build for subarch flavor build"
+	exit 0
+fi
+
+# This is the directory created by create-root-dir.binary
+rootfs_dir=rootfs.dir
+
+squashfs_f="$PWD/livecd.ubuntu-cpc.squashfs"
+
+cp $rootfs_dir.manifest $squashfs_f.manifest
+
+(cd $rootfs_dir &&
+      mksquashfs . $squashfs_f \
+        -no-progress -xattrs -comp xz )

live-build/ubuntu-cpc/hooks.d/base/root-xz.binary

@@ -0,0 +1,15 @@
+#!/bin/bash -ex
+# vi: ts=4 expandtab
+#
+# Generate the rootfs.tar.xz and manifest
+
+if [ -n "$SUBARCH" ]; then
+    echo "Skipping rootfs build for subarch flavor build"
+    exit 0
+fi
+
+# This is the directory created by create-root-dir.binary
+rootfs_dir=rootfs.dir
+
+cp $rootfs_dir.manifest livecd.ubuntu-cpc.rootfs.manifest
+(cd $rootfs_dir/ && tar -c --xattrs *) | xz > livecd.ubuntu-cpc.rootfs.tar.xz

live-build/ubuntu-cpc/hooks.d/base/series/all

@@ -0,0 +1 @@
+base

live-build/ubuntu-cpc/hooks.d/base/series/base

@@ -0,0 +1,8 @@
+depends root-dir
+depends tarball
+depends squashfs
+depends disk-image
+depends qcow2
+depends vmdk
+depends vagrant
+depends wsl

live-build/ubuntu-cpc/hooks.d/base/series/disk-image

@@ -0,0 +1,10 @@
+base/disk-image.binary
+base/disk-image-uefi.binary
+base/disk-image-ppc64el.binary
+provides livecd.ubuntu-cpc.ext4
+provides livecd.ubuntu-cpc.initrd-generic
+provides livecd.ubuntu-cpc.initrd-generic-lpae
+provides livecd.ubuntu-cpc.kernel-generic
+provides livecd.ubuntu-cpc.kernel-generic-lpae
+provides livecd.ubuntu-cpc.kernel-kvm
+provides livecd.ubuntu-cpc.manifest

live-build/ubuntu-cpc/hooks.d/base/series/qcow2

@@ -0,0 +1,3 @@
+depends disk-image
+base/qcow2-image.binary
+provides livecd.ubuntu-cpc.img

live-build/ubuntu-cpc/hooks.d/base/series/root-dir

@@ -0,0 +1 @@
+base/create-root-dir.binary

live-build/ubuntu-cpc/hooks.d/base/series/squashfs

@@ -0,0 +1,4 @@
+depends root-dir
+base/root-squashfs.binary
+provides livecd.ubuntu-cpc.squashfs
+provides livecd.ubuntu-cpc.squashfs.manifest

live-build/ubuntu-cpc/hooks.d/base/series/tarball

@@ -0,0 +1,4 @@
+depends root-dir
+base/root-xz.binary
+provides livecd.ubuntu-cpc.rootfs.tar.xz
+provides livecd.ubuntu-cpc.rootfs.manifest

live-build/ubuntu-cpc/hooks.d/base/series/vagrant

@@ -0,0 +1,3 @@
+depends disk-image
+base/vagrant.binary
+provides livecd.ubuntu-cpc.vagrant.box

live-build/ubuntu-cpc/hooks.d/base/series/vmdk

@@ -0,0 +1,5 @@
+depends disk-image
+base/vmdk-image.binary
+base/vmdk-ova-image.binary
+provides livecd.ubuntu-cpc.vmdk
+provides livecd.ubuntu-cpc.ova

live-build/ubuntu-cpc/hooks.d/base/series/wsl

@@ -0,0 +1,4 @@
+depends root-dir
+base/wsl-gz.binary
+provides livecd.ubuntu-cpc.wsl.rootfs.tar.gz
+provides livecd.ubuntu-cpc.wsl.rootfs.manifest

live-build/ubuntu-cpc/hooks.d/base/vagrant.binary

@@ -24,15 +24,6 @@ case ${SUBPROJECT:-} in
         ;;
 esac
 
-case $IMAGE_TARGETS in
-	""|*vagrant*)
-		;;
-	*)
-		echo "Skipping Vagrant image build"
-		exit 0
-		;;
-esac
-
 cur_d=${PWD}
 my_d=$(dirname $(readlink -f ${0}))
 

live-build/ubuntu-cpc/hooks.d/base/vmdk-image.binary

@@ -18,15 +18,6 @@ case $ARCH in
                      exit 0;;
 esac
 
-case ${IMAGE_TARGETS:-} in
-	""|*vmdk*)
-		;;
-	*)
-		echo "Skipping VMDK image build"
-		exit 0
-		;;
-esac
-
 . config/functions
 
 if [ -e binary/boot/disk-uefi.ext4 ]; then

live-build/ubuntu-cpc/hooks.d/base/vmdk-ova-image.binary

@@ -7,7 +7,7 @@
 # and checksums. This step produces an OVA that is suitable for use with
 # Cloud's that support the OVF specification.
 #
-# For this step, we re-use the VMDK's made in 040-vmdk-image.binary
+# For this step, we re-use the VMDK's made in vmdk-image.binary
 
 case ${SUBPROJECT:-} in
     minimized)
@@ -32,15 +32,6 @@ case $ARCH in
             exit 0;;
 esac
 
-case ${IMAGE_TARGETS:-} in
-	""|*vmdk*)
-		;;
-	*)
-		echo "Skipping OVA image build"
-		exit 0
-		;;
-esac
-
 cur_d=${PWD}
 my_d=$(dirname $(readlink -f ${0}))
 

live-build/ubuntu-cpc/hooks.d/base/wsl-gz.binary

@@ -0,0 +1,56 @@
+#!/bin/bash -eux
+# vi: ts=4 expandtab
+#
+# Generate the compressed root directory for WSL
+
+case ${SUBPROJECT:-} in
+    minimized)
+        echo "Skipping minimized $0 build as WSL systems are designed to be interactive"
+        exit 0
+        ;;
+    *)
+        ;;
+esac
+
+case $ARCH in
+    amd64|arm64)
+        ;;
+    *)
+        echo "WSL root tarballs are not generated for $ARCH."
+        exit 0;;
+esac
+
+if [ -n "${SUBARCH:-}" ]; then
+    echo "Skipping rootfs build for subarch flavor build"
+    exit 0
+fi
+
+. config/functions
+
+rootfs_dir=wslroot.dir
+
+# This is the directory created by create-root-dir.binary
+cp -a rootfs.dir $rootfs_dir
+
+setup_mountpoint $rootfs_dir
+
+env DEBIAN_FRONTEND=noninteractive chroot $rootfs_dir apt-get -y -qq install ubuntu-wsl
+
+create_manifest $rootfs_dir livecd.ubuntu-cpc.wsl.rootfs.manifest
+teardown_mountpoint $rootfs_dir
+
+# remove attributes not supported by WSL's tar
+if [ -d $rootfs_dir/var/log/journal ]; then
+    setfattr -x system.posix_acl_access $rootfs_dir/var/log/journal
+    setfattr -x system.posix_acl_default $rootfs_dir/var/log/journal
+fi
+
+# The reason not using just tar .. -C $rootfs_dir . is that using '.' was found
+# not working once and checking if using the simpler command is safe needs
+# verification of the app installation on all Windows 10 builds we support
+# with WSL.
+cd $rootfs_dir
+tar --xattrs --sort=name -czf ../livecd.ubuntu-cpc.wsl.rootfs.tar.gz *
+cd ..
+
+rm -rf $rootfs_dir

live-build/ubuntu-cpc/hooks.d/chroot/100-preserve-apt-prefs.chroot

@@ -0,0 +1,8 @@
+#! /bin/sh -ex
+
+# live-build "helpfully" removes /etc/apt/preferences.d/* so we put a
+# copy somewhere it won't touch it.
+
+if [ -n "$(ls -A /etc/apt/preferences.d)" ]; then
+    cp -a /etc/apt/preferences.d /etc/apt/preferences.d.save
+fi

live-build/ubuntu-cpc/hooks.d/chroot/100-purge-grub-legacy-ec2-arm.chroot

@@ -0,0 +1,13 @@
+#!/bin/sh -eux
+
+# Only execute the hack for ARM images
+# ARCH is not available in .chroot hooks so we need to get the architecture
+# manually.
+arch=$(dpkg --print-architecture)
+if [ "$arch" != "armhf" ] && [ "$arch" != "arm64" ]; then
+	exit 0
+fi
+
+export DEBIAN_FRONTEND=noninteractive
+# Why is grub-legacy-ec2 even on the image?
+apt-get remove  --yes --purge grub-legacy-ec2 || true

live-build/ubuntu-cpc/hooks.d/chroot/999-cpc-fixes.chroot

@@ -102,24 +102,6 @@ cat > /etc/fstab << EOM
 LABEL=cloudimg-rootfs	/	 ext4	defaults	0 0
 EOM
 
-## Make sure that the update-motd.d directory exists
-[ ! -e "${rootd}/etc/update-motd.d" ] &&
-   mkdir -p "${rootd}/etc/update-motd.d"
-
-## write a MOTD file advertising support for images
-cat > "${rootd}/etc/update-motd.d/51-cloudguest" << EOF
-#!/bin/sh
-#
-${CLOUD_IMG_STR}
-# This file is not managed by a package.  If you no longer want to
-# see this message you can safely remove the file.
-echo ""
-echo "  Get cloud support with Ubuntu Advantage Cloud Guest:"
-echo "    http://www.ubuntu.com/business/services/cloud"
-EOF
-
-chmod +x "${rootd}/etc/update-motd.d/51-cloudguest"
-
 # for quantal and newer, add /etc/overlayroot.local.conf
 # but do not overwrite anything that somehow got there
 if [ -f "${rootd}/etc/overlayroot.conf" ] &&
@@ -160,22 +142,17 @@ fi
 #### END COMMON ARCH FUNCTIONS
 
 
-case $arch in
-	# ARM images are special
-	armhf|arm64)
-		# Dirty hack because SUBARCH doesn't exist when running chroot hooks,
-		# and we don't want raspi2 images to depend on a cloud data source:
-		if _xchroot "${rootd}" dpkg -l linux-image-raspi2 2>/dev/null | grep -q '^.i'; then
-			fake_cloud_init
-		fi
+case $arch+${SUBARCH:-} in
+    # raspi2 images are special
+    armhf+raspi2)
+        fake_cloud_init
+        echo "Image architecture is ARM. Existing vmbuilder-fixups"
+        exit 0
+        ;;
 
-		echo "Image architecture is ARM. Existing vmbuilder-fixups"
-
-		exit 0
-		;;
-	powerpc|ppc64el|s390x)
-		exit 0
-		;;
+    armhf+*|arm64+*|powerpc+*|ppc64el+*|s390x+*)
+        exit 0
+        ;;
 esac
 
 psuedo_grub_probe() {

live-build/ubuntu-cpc/hooks.d/chroot/999-ubuntu-image-customization.chroot

@@ -0,0 +1,23 @@
+#!/bin/bash -ex
+
+. /root/config/chroot
+
+# Specific ubuntu-image chroot configuration goes here.
+# Currently the none IMAGEFORMAT is equivalent to building a rootfs for an image
+# for a physical device, not a cloud instance. For such images we want some
+# special cloud-init configuration that should not be happening for cloud
+# cases. This check should be changed to a per-MODEL/per-SUBARCH check once
+# we support building cloud images via ubuntu-image.
+if [ "$IMAGEFORMAT" == "none" ]; then
+    # If the cloud does not provide a meta-data service this should be run
+    # This will setup a nocloud datasource on the boot partition.
+    # This is the default behavior for our ubuntu-image built preinstalled
+    # images.
+    cat << EOF > /etc/cloud/cloud.cfg.d/99-fake_cloud.cfg
+# configure cloud-init for NoCloud
+datasource_list: [ NoCloud, None ]
+datasource:
+  NoCloud:
+    fs_label: system-boot
+EOF
+fi

live-build/ubuntu-cpc/hooks.d/make-hooks

@@ -0,0 +1,280 @@
+#!/usr/bin/env python3
+#-*- encoding: utf-8 -*-
+"""
+This script parses a series file and its dependencies and generates a hooks
+folder containing symbolic links to the scripts that need to be invoked for
+a given image target set.
+
+For example, if you wish to build the image target sets "vmdk" and "vagrant",
+you would call this script as
+
+./make-hooks --hooks-dir hooks vmdk vagrant
+
+Scripts live in subfolders below the "hooks.d" folder. Currently the folders
+"chroot" and "base" exist. The folder with the name "extra" is reserved for
+private scripts, which are not included in the source of livecd-rootfs. The
+scripts are not numbered, instead the order of their execution depends on the
+order in which they are listed in a series file.
+
+Series files are placed into the subfolders "base/series" or "extra/series".
+Each series file contains a list of scripts to be executed. Empty lines and
+lines starting with a '#' are ignored. Series files in "extra/series" override
+files in "base/series" with the same name. For example, if a series file
+"base/series/cloudA" exists and a series file "extra/series/cloudA", then the
+latter will be preferred.
+
+A series file in "extra/series" may also list scripts that are located in the
+"chroot" and "base" folders. In addition, series files can depend on other
+series files. For example, the series files for most custom images look similar
+to this:
+
+    depends disk-image
+    depends extra-settings
+    extra/cloudB.binary
+    provides livecd.ubuntu-cpc.disk-kvm.img
+    provides livecd.ubuntu-cpc.disk-kvm.manifest
+
+Where "disk-image" and "extra-settings" may list scripts and dependencies which
+are to be processed before the script "extra/cloudB.binary" is called.
+
+The "provides" directive defines a file that the hook creates; it can be
+specified multiple times.  The field is used by this script to generate a list
+of output files created explicitly by the named image targets.  The list is
+saved to the "explicit_provides" file in the hooks output directory. In
+the case of the "all" target this list would be empty.  This list is
+consumed by the "remove-implicit-artifacts" which is run at the end of the build.
+
+ACHTUNG: live build runs scripts with the suffix ".chroot" in a batch separate
+from scripts ending in ".binary". Even if you arrange them interleaved in your
+series files, the chroot scripts will be run before the binary scripts.
+"""
+
+import argparse
+import os
+import re
+import shutil
+import sys
+import yaml
+
+SCRIPT_DIR = os.path.normpath(os.path.dirname(os.path.realpath(sys.argv[0])))
+HOOKS_DIR = os.path.normpath(os.path.join(SCRIPT_DIR, "..", "hooks"))
+
+EXIT_OK = 0
+EXIT_ERR = 1
+
+class MakeHooksError(Exception):
+    pass
+
+class MakeHooks:
+    """This class provides series file parsing and symlink generator
+    functionality."""
+
+    def __init__(self, hooks_dir=None, quiet=False):
+        """The hooks_dir parameter can be used to specify the path to the
+        directory, into which the hook symlinks to the actual script files
+        should be placed.
+
+        If quiet is set to True, info messages during symlink creation will
+        be suppressed. Use this if your build is not private, but you would
+        like to hide which scripts are being run.
+        """
+        self._script_dir = SCRIPT_DIR
+        self._hooks_dir  = hooks_dir or HOOKS_DIR
+        self._quiet      = quiet
+        self._hooks_list = []
+        self._included   = set()
+        self._provides   = []
+
+    def reset(self):
+        """Reset the internal state allowing instance to be reused for
+        another run."""
+        self._hooks_list.clear()
+        self._included.clear()
+
+    def print_usage(self):
+        print(
+            "CPC live build hook generator script                               \n"
+            "                                                                   \n"
+            "Usage: ./make-hooks.sh [OPTIONS] <image_set>                       \n"
+            "                                                                   \n"
+            "Options:                                                           \n"
+            "                                                                   \n"
+            " --help, -h              Show this message and exit.               \n"
+            " --quiet, -q             Only show warnings and error messages.    \n"
+            " --hooks-dir, -d <dir>   The directory where to write the symlinks.\n"
+        )
+
+    def find_series_file(self, image_set):
+        """Search for the series file requested in the image_set parameter.
+
+        The image_set parameter should be a string containing the name of an
+        image target set represented by a series file. First the "extra/series"
+        folder is searched followed by the "base/series" folder.
+        
+        When a file with the given name is found, the search stops and the
+        full path to the file is returned.
+        """
+        for subdir in ["extra", "base"]:
+            series_file = os.path.join(self._script_dir, subdir, "series",
+                    image_set)
+            if os.path.isfile(series_file):
+                return series_file
+        return None
+
+    def make_hooks(self, image_sets):
+        """Entry point for parsing series files and their dependencies and
+        for generating the symlinks in the hooks folder.
+
+        The image_sets parameter must be an iterable containing the names of
+        the series files representing the corresponding image target sets,
+        e.g. "vmdk" or "vagrant".
+        """
+        self.collect_chroot_hooks()
+        self.collect_binary_hooks(image_sets, explicit_sets=True)
+        self.create_symlinks()
+        self.create_explicit_provides()
+
+    def collect_chroot_hooks(self):
+        """Chroot hooks are numbered and not explicitly mentioned in series
+        files. Collect them, sort them and add them to the internal list of
+        paths to hook sripts.
+        """
+        chroot_hooks_dir = os.path.join(self._script_dir, "chroot")
+
+        chroot_entries = os.listdir(chroot_hooks_dir)
+        chroot_entries.sort()
+
+        for entry in chroot_entries:
+            if not (entry.endswith(".chroot_early") or 
+                    entry.endswith(".chroot")):
+                continue
+            self._hooks_list.append(os.path.join("chroot", entry))
+
+    def collect_binary_hooks(self, image_sets, explicit_sets=False):
+        """Search the series files for the given image_sets and parse them
+        and their dependencies to generate a list of hook scripts to be run
+        during image build.
+
+        The image_sets parameter must be an iterable containing the names of
+        the series files representing the corresponding image target sets,
+        e.g. "vmdk" or "vagrant".
+
+        Populates the internal list of paths to hook scripts in the order in
+        which the scripts are to be run.
+
+        If "explicit_sets" is True, the files specified on lines starting
+        with "provides" will be added to self._provides to track explicit
+        output artifacts.  This is only True for the initial images_sets
+        list, dependent image sets should set this to False.
+        """
+        for image_set in image_sets:
+            series_file = self.find_series_file(image_set)
+
+            if not series_file:
+                raise MakeHooksError(
+                    "Series file for image set '%s' not found." % image_set)
+
+            with open(series_file, "r", encoding="utf-8") as fp:
+                for line in fp:
+                    line = line.strip()
+                    if not line or line.startswith("#"):
+                        continue
+
+                    m = re.match(r"^\s*depends\s+(\S+.*)$", line)
+                    if m:
+                        include_set = m.group(1)
+                        if include_set in self._included:
+                            continue
+                        self._included.add(include_set)
+                        self.collect_binary_hooks([include_set,])
+                        continue
+
+                    m = re.match(r"^\s*provides\s+(\S+.*)$", line)
+                    if m:
+                        if explicit_sets:
+                            self._provides.append(m.group(1))
+                        continue
+
+                    if not line in self._hooks_list:
+                        self._hooks_list.append(line)
+
+    def create_symlinks(self):
+        """Once the internal list of hooks scripts has been populated by a
+        call to collect_?_hooks, this method is used to populate the hooks
+        folder with enumerated symbolic links to the hooks scripts. If the
+        folder does not exist, it will be created. If it exists, it must be
+        empty or a MakeHooksError will be thrown.
+        """
+        if os.path.isdir(self._hooks_dir) and os.listdir(self._hooks_dir):
+            # Only print a warning, because directory might have been created
+            # by auto/config voodoo.
+            sys.stderr.write("WARNING: Hooks directory exists and is not empty.\n")
+        os.makedirs(self._hooks_dir, exist_ok=True)
+
+        for counter, hook in enumerate(self._hooks_list, start=1):
+            hook_basename = os.path.basename(hook)
+
+            m = re.match(r"^\d+-(?:\d+-)?(?P<basename>.*)$", hook_basename)
+            if m:
+                hook_basename = m.group("basename")
+
+            linkname = ("%03d-" % counter) + hook_basename
+            linkdest = os.path.join(self._hooks_dir, linkname)
+            linksrc  = os.path.relpath(os.path.join(self._script_dir, hook),
+                    self._hooks_dir)
+
+            if not self._quiet:
+                print("[HOOK] %s => %s" % (linkname, hook))
+            os.symlink(linksrc, linkdest)
+
+    def create_explicit_provides(self):
+        """
+        Create a file named "explicit_provides" in self._script_dir
+        listing all files named on "provides" in the series files of
+        targets explicitly named by the user.  The file is created but
+        left empty if there are no explict "provides" keywords in the
+        targets (this is the case for 'all')
+        """
+        with open(os.path.join(self._script_dir, "explicit_provides"), "w",
+                  encoding="utf-8") as fp:
+            empty = True
+            for provides in self._provides:
+                if not self._quiet:
+                    print("[PROVIDES] %s" % provides)
+                fp.write("%s\n" % provides)
+                empty = False
+            if not empty:
+                fp.write('livecd.magic-proxy.log\n')
+
+    def cli(self, args):
+        """Command line interface to the hooks generator."""
+        parser = argparse.ArgumentParser()
+
+        parser.add_argument("-q", "--quiet", dest="quiet", type=bool,
+                help="Only show warnings and error messages.")
+        parser.add_argument("-d", "--hooks-dir", dest="hooks_dir", type=str,
+                help="The directory where to create the symlinks.")
+        parser.add_argument("image_target", nargs="+", type=str,
+                help="")
+
+        self.reset()
+        options = parser.parse_args(args)
+
+        # Copy options to object attributes.
+        for key, value in vars(options).items():
+            if value and hasattr(self, "_" + key):
+                setattr(self, "_" + key, value)
+
+        # Take remaining command line arguments, sanitize and turn into list.
+        image_sets = re.sub(r";|,", " ", " ".join(options.image_target))\
+                .split()
+
+        self.make_hooks(image_sets)
+
+
+if __name__ == "__main__":
+    try:
+        MakeHooks().cli(sys.argv[1:])
+    except MakeHooksError as e:
+        sys.stderr.write("%s: %s\n" % (os.path.basename(sys.argv[0]), str(e)))
+        sys.exit(EXIT_ERR)

live-build/ubuntu-cpc/hooks.d/remove-implicit-artifacts

@@ -0,0 +1,41 @@
+#!/usr/bin/env python3
+#-*- encoding: utf-8 -*-
+"""
+Remove output files not created by explicitly specified image targets
+
+This uses the 'explicit_provides' file generated by the 'make-hooks'
+script.  If the file is empty, all output will be saved.
+"""
+import glob
+import os
+import sys
+
+if __name__ == "__main__":
+    print('Running {}'.format(__file__))
+    scriptname = os.path.basename(__file__)
+    explicit = set()
+    with open('./config/hooks.d/explicit_provides', 'r',
+              encoding='utf-8') as fp:
+        for filename in fp:
+            explicit.add(filename.rstrip())
+
+    if not explicit:
+        print('{}: explicit_provides is empty. '
+              'All binary output will be included'.format(scriptname))
+        sys.exit(0)
+
+    all = set(glob.glob('livecd.ubuntu-cpc.*'))
+    implicit = all - explicit
+
+    print('{}: all artifacts considered: {}'.format(scriptname, all))
+    print('{}: explict artifacts to keep: {}'.format(scriptname, explicit))
+    print('{}: implicit artifacts to remove: {}'.format(scriptname, implicit))
+
+    for file in implicit:
+        if os.path.islink(file):
+            print('{}: unlinking {}'.format(scriptname, file))
+            os.unlink(file)
+        elif os.path.isfile(file):
+            print('{}: removing {} '
+                  '{} bytes'.format(scriptname, file, os.stat(file).st_size))
+            os.remove(file)

live-build/ubuntu-cpc/hooks/032-root-squashfs.binary

@@ -1,49 +0,0 @@
-#!/bin/bash -ex
-# vi: ts=4 noexpandtab
-#
-# Generate a squashfs root and manifest
-
-case $IMAGE_TARGETS in
-	""|*squashfs*)
-		;;
-	*)
-		echo "Skipping squashfs build"
-		exit 0
-		;;
-esac
-
-if [ -n "$SUBARCH" ]; then
-	echo "Skipping rootfs build for subarch flavor build"
-	exit 0
-fi
-
-. config/functions
-
-mkdir binary/boot/squashfs.dir
-cp -a chroot/* binary/boot/squashfs.dir
-
-setup_mountpoint binary/boot/squashfs.dir
-
-chroot binary/boot/squashfs.dir dpkg-divert --local --rename /usr/sbin/grub-probe
-chroot binary/boot/squashfs.dir touch /usr/sbin/grub-probe
-chroot binary/boot/squashfs.dir chmod +x /usr/sbin/grub-probe
-
-env DEBIAN_FRONTEND=noninteractive chroot binary/boot/squashfs.dir apt-get --purge remove --assume-yes '^linux-.*' 'linux-base+'
-env DEBIAN_FRONTEND=noninteractive chroot binary/boot/squashfs.dir apt-get --purge remove --assume-yes '^grub-.*'
-env DEBIAN_FRONTEND=noninteractive chroot binary/boot/squashfs.dir apt-get autoremove --purge --assume-yes
-rm -rf binary/boot/squashfs.dir/boot/grub
-chroot binary/boot/squashfs.dir mkdir -p /lib/modules
-
-chroot binary/boot/squashfs.dir rm /usr/sbin/grub-probe
-chroot binary/boot/squashfs.dir dpkg-divert --remove --local --rename /usr/sbin/grub-probe
-
-teardown_mountpoint binary/boot/squashfs.dir
-
-squashfs_f="${PWD}/livecd.ubuntu-cpc.squashfs"
-squashfs_f_manifest="${squashfs_f}.manifest"
-
-dpkg-query --admindir=binary/boot/squashfs.dir/var/lib/dpkg -W > ${squashfs_f_manifest}
-
-(cd "binary/boot/squashfs.dir/" &&
-      mksquashfs . ${squashfs_f} \
-        -no-progress -xattrs -comp xz )

live-build/ubuntu-cpc/hooks/999-extras.binary

@@ -1,16 +0,0 @@
-#!/bin/bash
-# Execute extra binary hooks.
-
-my_dir=$(dirname $(readlink -f ${0}))
-extra_d=${my_dir}/extra
-
-if [ ! -d ${my_dir}/extra ]; then
-    exit 0
-fi
-
-export IMAGE_STR="# CLOUD_IMG: This file was created/modified by the Cloud Image build process"
-export CLOUD_IMG_STR="$IMAGE_STR"
-export FS_LABEL="cloudimg-rootfs"
-
-# Cleaner execution
-/bin/run-parts --exit-on-error --regex ".*\.binary" "${extra_d}"

live-build/ubuntu-server/hooks/030-root-squashfs.binary

@@ -28,7 +28,7 @@ cp -a chroot/* binary/boot/squashfs.dir
 squashfs_f="${PWD}/livecd.${PROJECT}.squashfs"
 squashfs_f_manifest="${squashfs_f}.manifest"
 
-dpkg-query --admindir=binary/boot/squashfs.dir/var/lib/dpkg -W > ${squashfs_f_manifest}
+create_manifest "binary/boot/squashfs.dir" "${squashfs_f_manifest}"
 
 (cd "binary/boot/squashfs.dir/" &&
       mksquashfs . ${squashfs_f} \

live-build/ubuntu-server/hooks/032-installer-squashfs.binary

@@ -24,13 +24,14 @@ fi
 . config/functions
 . config/common
 
-SQUASH_ROOT=binary/boot/squashfs.dir
+FILESYSTEM_ROOT=binary/boot/squashfs.dir
+INSTALLER_ROOT=binary/boot/installer.squashfs.dir
 OVERLAY_ROOT=binary/overlay
 
-mkdir -p "$OVERLAY_ROOT"
+mkdir -p "$INSTALLER_ROOT" "$OVERLAY_ROOT"
 
 # Create an installer squashfs layer
-mount_overlay "$SQUASH_ROOT/" "$OVERLAY_ROOT/" "$SQUASH_ROOT/"
+mount_overlay "$FILESYSTEM_ROOT/" "$OVERLAY_ROOT/" "$INSTALLER_ROOT/"
 
 setup_mountpoint binary/boot/squashfs.dir
 
@@ -40,42 +41,41 @@ setup_mountpoint binary/boot/squashfs.dir
 # It would be better to have this in ../includes.binary/overlay but
 # you can't have backslashes in filenames in bzr branches!
 DEVICE_UNIT_NAME='dev-disk-by\x2duuid-00c629d6\x2d06ab\x2d4dfd\x2db21e\x2dc3186f34105d.device'
-mkdir -p "$SQUASH_ROOT/etc/systemd/system/$DEVICE_UNIT_NAME.d"
-cat > "$SQUASH_ROOT/etc/systemd/system/$DEVICE_UNIT_NAME.d/override.conf" <<EOF
+mkdir -p "$INSTALLER_ROOT/etc/systemd/system/$DEVICE_UNIT_NAME.d"
+cat > "$INSTALLER_ROOT/etc/systemd/system/$DEVICE_UNIT_NAME.d/override.conf" <<EOF
 [Unit]
 JobRunningTimeoutSec=0s
+Wants=subiquity_config.mount
 EOF
 
 # Prepare installer layer.
 
-# Install any requirements for the installer, for things we don't want
-# to see on the installed system
-chroot $SQUASH_ROOT apt-get update
-chroot $SQUASH_ROOT apt-get -y install user-setup curtin lupin-casper
+# Install casper for live session magic.
+chroot $INSTALLER_ROOT apt-get -y install lupin-casper
+chroot $INSTALLER_ROOT apt-get clean
 
 # For bug #1743643 "Install to dirty disk with swap fails" remove the
 # "helpful" casper script that mounts any swap partitions it finds.
-rm -f $SQUASH_ROOT/usr/share/initramfs-tools/scripts/casper-bottom/*swap
-
-# Installing casper means we need a new initramfs
-UPDATE_INITRAMFS_OPTIONS=CASPER_GENERATE_UUID=1 recreate_initramfs $SQUASH_ROOT
+rm -f $INSTALLER_ROOT/usr/share/initramfs-tools/scripts/casper-bottom/*swap
 
 # Don't let cloud-init run in the live session.
-touch $SQUASH_ROOT/etc/cloud/cloud-init.disabled
+touch $INSTALLER_ROOT/etc/cloud/cloud-init.disabled
 
 # Do the snap seeding dance.
-chroot $SQUASH_ROOT mkdir -p /var/lib/snapd/seed/snaps /var/lib/snapd/seed/assertions
-chroot $SQUASH_ROOT sh -c '
+chroot $INSTALLER_ROOT mkdir -p /var/lib/snapd/seed/snaps /var/lib/snapd/seed/assertions
+chroot $INSTALLER_ROOT sh -c '
 set -x;
-cd /var/lib/snapd/seed;
-sudo SNAPPY_STORE_NO_CDN=1 snap download core;
-sudo SNAPPY_STORE_NO_CDN=1 snap download subiquity;
+mkdir -p /var/lib/snapd/seed/snaps/
+cd /var/lib/snapd/seed/snaps/;
+if [ ! -e core_*.snap ]; then
+	SNAPPY_STORE_NO_CDN=1 snap download core
+fi
+SNAPPY_STORE_NO_CDN=1 snap download subiquity;
 
-CORE_SNAP=$(ls -1 core*.snap);
+CORE_SNAP=$(ls -1 core_*.snap);
 SUBIQUITY_SNAP=$(ls -1 subiquity*.snap);
 
 mv *.assert /var/lib/snapd/seed/assertions/;
-mv *.snap /var/lib/snapd/seed/snaps/;
 
 cat <<EOF > /var/lib/snapd/seed/seed.yaml
 snaps:
@@ -89,10 +89,7 @@ snaps:
 EOF
 '
 
-teardown_mountpoint "$SQUASH_ROOT"
-
-# Then unmount the overlay
-umount "$SQUASH_ROOT"
+teardown_mountpoint "$INSTALLER_ROOT"
 
 squashfs_f="${PWD}/livecd.${PROJECT}.installer.squashfs"
 

live-build/ubuntu-server/hooks/033-kernel-bits.binary

@@ -0,0 +1,90 @@
+#!/bin/bash -eux
+# vi: ts=4 noexpandtab
+#
+# Generate a squashfs root and manifest
+
+echo "033-kernel-bits.binary"
+
+case ${IMAGE_TARGETS-} in
+	""|*squashfs*)
+		;;
+	*)
+		echo "Skipping squashfs build"
+		exit 0
+		;;
+esac
+
+if [ -n "${SUBARCH-}" ]; then
+	echo "Skipping rootfs build for subarch flavor build"
+	exit 0
+fi
+
+. config/functions
+. config/common
+
+INSTALLER_ROOT=binary/boot/installer.squashfs.dir
+
+KERNEL_BITS_ROOT=binary/boot/kernel-bits.dir
+KERNEL_BITS_OVERLAY=binary/boot/overlay-kernel-bits
+
+variants='ga hwe'
+
+for variant in $variants; do
+	if [ "$variant" = "ga" ]; then
+		kernel_metapkg=linux-generic
+		flavor=generic
+	elif [ "$variant" = "hwe" ]; then
+		kernel_metapkg=linux-generic-hwe-$(lsb_release -sr)
+		flavor=generic-hwe
+	else
+		echo "bogus variant: $variant"
+		exit 1
+	fi
+
+	# Make preparations
+	mkdir -p $KERNEL_BITS_ROOT $KERNEL_BITS_OVERLAY
+	mount_overlay "$INSTALLER_ROOT/" "$KERNEL_BITS_OVERLAY/" "$KERNEL_BITS_ROOT/"
+	setup_mountpoint $KERNEL_BITS_ROOT
+
+	# Our initramfs hook implements a kind of extreme version of
+	# cloud-initramfs-copymods, so remove that and prevent duelling hooks
+	env DEBIAN_FRONTEND=noninteractive chroot $KERNEL_BITS_ROOT apt-get -y remove cloud-initramfs-copymods ubuntu-server || true
+
+	# Configure initramfs creation
+	mkdir -p "$KERNEL_BITS_ROOT"/etc/initramfs-tools/conf.d/
+	if [ -n "$LB_INITRAMFS_COMPRESSION" ]; then
+		echo "COMPRESS=$LB_INITRAMFS_COMPRESSION" > "$KERNEL_BITS_ROOT"/etc/initramfs-tools/conf.d/livecd-rootfs.conf
+	fi
+	echo "export CASPER_GENERATE_UUID=1" > "$KERNEL_BITS_ROOT"/etc/initramfs-tools/conf.d/casper.conf
+
+	# Add a hook to record which kernel was booted and mount the
+	# modules.squashfs created below.
+	cat <<EOF > "$KERNEL_BITS_ROOT"/etc/initramfs-tools/scripts/init-bottom/live-server
+#!/bin/sh
+case \$1 in
+prereqs) exit 0;;
+esac
+
+echo ${kernel_metapkg} > /run/kernel-meta-package
+mkdir -p \$rootmnt/lib/modules
+mount \$rootmnt/cdrom/casper/extras/modules.squashfs-$flavor \$rootmnt/lib/modules
+EOF
+	chmod +x "$KERNEL_BITS_ROOT"/etc/initramfs-tools/scripts/init-bottom/live-server
+
+	# Install the kernel!
+	env DEBIAN_FRONTEND=noninteractive chroot $KERNEL_BITS_ROOT apt-get -y install ${kernel_metapkg}
+
+	# Fish out generated kernel image and initrd
+	mv "$KERNEL_BITS_ROOT"/boot/initrd.img-* ${PWD}/livecd.${PROJECT}.initrd-$flavor
+	mv "$KERNEL_BITS_ROOT"/boot/vmlinu?-* ${PWD}/livecd.${PROJECT}.kernel-$flavor
+
+	# Create squashfs containing all the modules
+	modules_squashfs_path="${PWD}/livecd.${PROJECT}.modules.squashfs-$flavor"
+	(cd "$KERNEL_BITS_ROOT/lib/modules" &&
+		mksquashfs . $modules_squashfs_path -no-progress -xattrs -comp xz)
+
+	# And clean up
+	teardown_mountpoint $KERNEL_BITS_ROOT
+	umount $KERNEL_BITS_ROOT
+	rm -rf $KERNEL_BITS_ROOT $KERNEL_BITS_OVERLAY
+done

live-build/ubuntu-server/includes.binary/overlay/etc/netplan/00-installer-config.yaml

@@ -7,9 +7,7 @@ network:
             match:
                 name: "en*"
             dhcp4: true
-            optional: true
         all-eth:
             match:
                 name: "eth*"
             dhcp4: true
-            optional: true

live-build/ubuntu-server/includes.binary/overlay/etc/systemd/journald.conf.d/no-rate-limit.conf

@@ -0,0 +1,2 @@
+[Journal]
+RateLimitIntervalSec=0

live-build/ubuntu-server/includes.binary/overlay/lib/systemd/system/local-fs.target.wants/subiquity_config.mount

@@ -1 +0,0 @@
-../subiquity_config.mount

live-build/ubuntu-server/includes.binary/overlay/lib/systemd/system/serial-subiquity@.service

@@ -7,9 +7,9 @@ ConditionPathExists=!/run/subiquity/complete
 StartLimitInterval=0
 
 [Service]
-Environment=PYTHONPATH=/usr/share/subiquity
+Environment=SNAP_REEXEC=0
 ExecStartPre=/bin/systemctl stop serial-getty@%I
-ExecStart=/sbin/agetty -n --keep-baud -l /snap/bin/subiquity --login-options "--serial" 115200,38400,9600 %I $TERM
+ExecStart=/usr/bin/snap run subiquity.subiquity-service %I
 ExecStopPost=/bin/systemctl start serial-getty@%I
 Type=idle
 Restart=always

live-build/ubuntu-server/includes.binary/overlay/lib/systemd/systemd-networkd-wait-online

@@ -0,0 +1 @@
+/bin/true

live-build/ubuntu/hooks/033-disk-image-uefi.binary

@@ -0,0 +1 @@
+../../ubuntu-cpc/hooks.d/base/disk-image-uefi.binary

live-build/ubuntu/hooks/040-hyperv-desktop-images.binary

@@ -0,0 +1,128 @@
+#!/bin/bash -eux
+
+echo "Creating Hyper-V image with Desktop..."
+
+case ${SUBPROJECT:-} in
+    minimized)
+        echo "We don't create minimized images for $0."
+        exit 0
+        ;;
+    *)
+        ;;
+esac
+
+case "${ARCH}" in
+    amd64)
+        ;;
+    *)
+        echo "Hyper-V only supports amd64";
+        exit 0
+        ;;
+esac
+
+IMAGE_STR="# DESKTOP_IMG: This file was created/modified by the Desktop Image build process"
+FS_LABEL="desktop-rootfs"
+
+. config/functions
+
+export DEBIAN_FRONTEND=noninteractive
+
+create_derivative uefi hyperv
+scratch_d=$(mktemp -d)
+mount_disk_image "${derivative_img}" "${scratch_d}"
+
+cleanup_hyperv() {
+    umount_disk_image ${scratch_d}
+    rm -rf ${scratch_d}
+}
+trap cleanup_hyperv EXIT
+
+# Perform customisations
+
+chroot "${scratch_d}" apt-get update -y
+chroot "${scratch_d}" apt-get -y install xrdp linux-tools-virtual linux-cloud-tools-virtual
+
+cat > ${scratch_d}/etc/modules-load.d/hyperv.conf << EOF
+${IMAGE_STR}
+hv_sock
+EOF
+
+cat << EOF >> "${scratch_d}/etc/fstab"
+LABEL=$FS_LABEL	/		ext4	defaults	0 0
+EOF
+
+# Customise xrdp
+
+CHANGED_FILE_SUFFIX=.replaced-by-desktop-img-build
+
+# use vsock transport.
+sed -i${CHANGED_FILE_SUFFIX} -e 's/use_vsock=false/use_vsock=true/g' "${scratch_d}/etc/xrdp/xrdp.ini"
+# use rdp security.
+sed -i${CHANGED_FILE_SUFFIX} -e 's/security_layer=negotiate/security_layer=rdp/g' "${scratch_d}/etc/xrdp/xrdp.ini"
+# remove encryption validation.
+sed -i${CHANGED_FILE_SUFFIX} -e 's/crypt_level=high/crypt_level=none/g' "${scratch_d}/etc/xrdp/xrdp.ini"
+# disable bitmap compression since its local its much faster
+sed -i${CHANGED_FILE_SUFFIX} -e 's/bitmap_compression=true/bitmap_compression=false/g' "${scratch_d}/etc/xrdp/xrdp.ini"
+
+# Add script to setup the ubuntu session properly
+cat > "${scratch_d}/etc/xrdp/startubuntu.sh" << EOF
+#!/bin/sh
+${IMAGE_STR}
+export GNOME_SHELL_SESSION_MODE=ubuntu
+export XDG_CURRENT_DESKTOP=ubuntu:GNOME
+exec /etc/xrdp/startwm.sh
+EOF
+chmod a+x "${scratch_d}/etc/xrdp/startubuntu.sh"
+
+# use the script to setup the ubuntu session
+sed -i${CHANGED_FILE_SUFFIX} -e 's/startwm/startubuntu/g' "${scratch_d}/etc/xrdp/sesman.ini"
+
+# rename the redirected drives to 'shared-drives'
+sed -i${CHANGED_FILE_SUFFIX} -e 's/FuseMountName=thinclient_drives/FuseMountName=shared-drives/g' "${scratch_d}/etc/xrdp/sesman.ini"
+
+# Changed the allowed_users
+sed -i${CHANGED_FILE_SUFFIX} -e 's/allowed_users=console/allowed_users=anybody/g' "${scratch_d}/etc/X11/Xwrapper.config"
+
+# Blacklist the vmw module
+cat > "${scratch_d}/etc/modprobe.d/blacklist_vmw_vsock_vmci_transport.conf" << EOF
+${IMAGE_STR}
+blacklist vmw_vsock_vmci_transport
+EOF
+
+# Configure the policy xrdp session
+cat > ${scratch_d}/etc/polkit-1/localauthority/50-local.d/45-allow-colord.pkla << EOF
+${IMAGE_STR}
+[Allow Colord all Users]
+Identity=unix-user:*
+Action=org.freedesktop.color-manager.create-device;org.freedesktop.color-manager.create-profile;org.freedesktop.color-manager.delete-device;org.freedesktop.color-manager.delete-profile;org.freedesktop.color-manager.modify-device;org.freedesktop.color-manager.modify-profile
+ResultAny=no
+ResultInactive=no
+ResultActive=yes
+EOF
+
+sed -i${CHANGED_FILE_SUFFIX} -e 's|After=|ConditionPathExists=!/var/lib/oem-config/run\nAfter=|g' "${scratch_d}/lib/systemd/system/xrdp.service"
+
+# End xrdp customisation
+
+# Don't run gnome-initial-setup from gdm
+sed -i${CHANGED_FILE_SUFFIX} "s|#WaylandEnable=false|#WaylandEnable=false\nInitialSetupEnable=false|" "${scratch_d}/etc/gdm3/custom.conf"
+chroot "${scratch_d}" /usr/sbin/useradd -d /home/oem -m -N -u 29999 oem
+chroot "${scratch_d}" /usr/sbin/oem-config-prepare --quiet
+touch "${scratch_d}/var/lib/oem-config/run"
+
+chroot "${scratch_d}" apt-get clean
+
+# End customisations
+
+cleanup_hyperv
+trap - EXIT
+
+raw_img=binary/boot/disk-hyperv-uefi.ext4
+vhd_img=livecd.ubuntu-desktop-hyperv.vhdx
+
+qemu-img convert -O vhdx "$raw_img" "$vhd_img"
+rm "$raw_img"
+
+apt-get install -y zip
+zip "$vhd_img.zip" "$vhd_img"
+rm "$vhd_img"

live-build/ubuntu/includes.chroot/etc/hosts

@@ -0,0 +1,9 @@
+127.0.0.1	localhost.localdomain	localhost
+::1		localhost6.localdomain6	localhost6
+
+# The following lines are desirable for IPv6 capable hosts
+::1     localhost ip6-localhost ip6-loopback
+fe00::0 ip6-localnet
+ff02::1 ip6-allnodes
+ff02::2 ip6-allrouters
+ff02::3 ip6-allhosts

lp-in-release

@@ -0,0 +1 @@
+magic-proxy

magic-proxy

@@ -0,0 +1,971 @@
+#!/usr/bin/python3 -u
+#-*- encoding: utf-8 -*-
+"""
+This script can be called as "lp-in-release" or as "magic-proxy". When called
+under the former name, it acts as a CLI tool, when called under the latter name
+it will act as a transparent HTTP proxy.
+
+The CLI tool parses the directory listing of
+
+    http://<mirror>/dists/suite/by-hash/SHA256
+
+and figures out which hashes belong to an InRelease file. For example, to list
+all available hashes for "cosmic" run
+
+    ./lp-in-release list --suite cosmic
+
+Per default the script scans archive.ubuntu.com, but you can tell it to use a
+different mirror with the --mirror-url command line parameter. Analogously, you
+can list the hashes for "cosmic-updates" or "cosmic-security". The script can
+also find the hash that was valid at a given timestamp via
+
+    ./lp-in-release select --suite cosmic --cutoff-time <timestamp>
+
+Finally, you can use the script to inject inrelease-path settings into a
+sources.list file via
+
+    ./lp-in-release inject --cutoff-time <timestamp> /etc/apt/sources.list
+
+The proxy is just an extension to this functionality. Whenever a URL points at
+an InRelease file or a path listed in an InRelease file, the proxy will
+automatically inject the by hash URL for the resource according to the timestamp
+it was configured for. The proxy works in transparent and non-transparent mode.
+"""
+from datetime import datetime, timedelta, tzinfo
+
+import argparse
+import copy
+import fcntl
+import getopt
+import hashlib
+import http.client
+import http.server
+import json
+import os
+import pwd
+import re
+import shutil
+import socketserver
+import sys
+import threading
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+
+EXIT_OK  = 0
+EXIT_ERR = 1
+
+class LPInReleaseBaseError(Exception):
+    pass
+
+class LPInReleaseIndexError(LPInReleaseBaseError):
+    pass
+
+class LPInReleaseCacheError(LPInReleaseBaseError):
+    pass
+
+class LPInReleaseProxyError(LPInReleaseBaseError):
+    pass
+
+class InRelease:
+    """This class represents an InRelease file."""
+
+    def __init__(self, mirror, suite, data, hash_=None, last_modified=None):
+        """mirror must contain the proper URL of the package repository up to
+        the "dists" folder, e.g.
+
+        http://archive.ubuntu.com/ubuntu
+
+        suite is the name of the suite this InRelease file belongs to, e.g.
+        <release>, <release>-updates or <release>-security.
+
+        data must contain the full contents of the InReleaes file as a unicode
+        string.
+
+        If supplied, then hash_ will be used as the sha256 hexdigest of the
+        binary encoding of the InRelease file. If not supplied, the hash will
+        be calculated. This is just used as a time-saver, when cache contents
+        are read back in.
+
+        last_modified must be a string of format
+
+        Thu, 26 Apr 2018 23:37:48 UTC
+
+        representing the publication time of the InRelease file. If not given,
+        the generation time stored in the InRelease file will be used. Below,
+        this is set explicitly to correspond to the Last-Modified header spat
+        out by the Web server.
+        """
+        self.mirror = mirror
+        self.suite  = suite
+        self.data   = data
+        self.dict   = {}
+
+        if hash_:
+            self.hash = hash_
+        else:
+            h = hashlib.sha256()
+            h.update(data.encode("utf-8"))
+            self.hash = h.hexdigest()
+
+        if last_modified:
+            self.published = self._parse_datetime(last_modified)
+        else:
+            self.published = self._extract_timestamp(data)
+
+    @property
+    def datetime(self):
+        """Return the publication time of this InRelease file as a string in
+        YYYY-MM-DD HH:MM:SS ISO format. The result is always in GMT."""
+        return datetime \
+                .utcfromtimestamp(self.published) \
+                .strftime('%Y-%m-%d %H:%M:%S')
+
+    @property
+    def normalized_address(self):
+        """Return the "normalized" address of the mirror URL, consisting of
+        only the hostname and the path. This may be used as an index into an
+        InReleaseCache."""
+        result  = urllib.parse.urlparse(self.mirror)
+        address = result.hostname + result.path.rstrip("/")
+        return address
+
+    @property
+    def contents(self):
+        """Return the pure contents of the InRelease file with the signature
+        stripped off."""
+        return self._split_release_and_sig(self.data)[0]
+
+    @property
+    def signature(self):
+        """Return the ASCII-armored PGP signature of the InRelease file."""
+        return self._split_release_and_sig(self.data)[1]
+
+    def serialize(self):
+        """Serializes the InRelease object into Python structures to be stored
+        in an InReleaseCache."""
+        month_names = [ "_ignore_",
+            "Jan", "Feb", "Mar", "Apr", "May", "Jun",
+            "Jul", "Aug", "Sep", "Oct", "Nov", "Dec",
+        ]
+
+        wkday_names = [
+            "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun",
+        ]
+
+        dt = datetime.utcfromtimestamp(self.published)
+
+        published = "{}, {:02} {} {} {:02}:{:02}:{:02} GMT".format(
+            wkday_names[dt.weekday()],
+            dt.day,
+            month_names[dt.month],
+            dt.year,
+            dt.hour,
+            dt.minute,
+            dt.second
+        )
+
+        return {
+            "mirror":    self.mirror,
+            "suite":     self.suite,
+            "hash":      self.hash,
+            "published": published,
+            "data":      self.data,
+        }
+
+    def get_hash_for(self, path):
+        """Check if the given path is listed in this InRelease file and if so
+        return the corresponding hash in hexdigest format. If the path is not
+        listed, None is returned."""
+        if not self.dict:
+            self._parse_contents()
+        return self.dict.get(path)
+
+    def _parse_contents(self):
+        """This method parses out all lines containing SHA256 hashes and creates
+        an internal dict, mapping resources to hashes."""
+        regex = re.compile(
+            r" (?P<hash>[0-9a-f]{64})\s+(?P<size>\d+)\s+(?P<path>\S+)")
+
+        for line in self.contents.splitlines():
+            m = regex.match(line)
+            if not m:
+                continue
+            self.dict[m.group("path")] = m.group("hash")
+
+    def _parse_datetime(self, datetime_string):
+        """Because the behavior of Python's strptime's would be
+        locale-dependent, we parse datetime strings of the format found in
+        Last-Modified HTTP headers ourselves. This returns an integer
+        representing a posix timestamp or None, if the parsing failed."""
+        class UTC(tzinfo):
+            def utcoffset(self, dt):
+                return timedelta(0)
+
+        # we need a map, because strptime would be locale-dependent
+        month_name_to_number = {
+            "Jan":  1, "Feb":  2, "Mar":  3, "Apr":  4, "May":  5, "Jun":  6,
+            "Jul":  7, "Aug":  8, "Sep":  9, "Oct": 10, "Nov": 11, "Dec": 12
+        }
+
+        rexpr = r"""^\s*\w+,\s+
+                    (?P<day>\d+)   \s+
+                    (?P<month>\w+) \s+
+                    (?P<year>\d+)  \s+
+                    (?P<hour>\d+)  :
+                    (?P<min>\d+)   :
+                    (?P<sec>\d+)   .*$"""
+
+        m = re.match(rexpr, datetime_string, flags=re.VERBOSE)
+        if not m:
+            return None
+
+        parts    = list(m.group("year", "month", "day", "hour", "min", "sec"))
+        parts[1] = month_name_to_number[m.group("month")]
+        parts    = [int(s) for s in parts]
+        dt       = datetime(*parts, tzinfo=UTC())
+        epoch    = datetime(1970, 1, 1, tzinfo=UTC())
+        posix    = (dt - epoch).total_seconds()
+
+        return int(posix)
+
+    def _extract_timestamp(self, data):
+        """Parse the contents of the InRelease file to find the time it was
+        generated. Returns a POSIX timestamp if found or None otherwise."""
+        for line in data.splitlines():
+            if line.startswith("Date:"):
+                return self._parse_datetime(line.split(":", 1)[1])
+
+        return None
+
+    def _split_release_and_sig(self, data):
+        """Split the InRelease file into content and signature parts and return
+        a tuple of unicode strings (content, signature)."""
+        rexpr = re.escape("-----BEGIN PGP SIGNED MESSAGE-----") + r"\r?\n|" + \
+                re.escape("-----BEGIN PGP SIGNATURE-----"     ) + r"\r?\n|" + \
+                re.escape("-----END PGP SIGNATURE-----"       )
+
+        # returns content and signature
+        return re.split(rexpr, data)[1:3]
+
+
+class LPInReleaseCache:
+    """A cache for InRelease files that can optionally be saved to and
+    loaded from disk."""
+
+    def __init__(self, filename=None):
+        """If filename is given, it is the name of the file that cache contents
+        will be saved to or loaded from when the save and load methods are
+        called, respectively."""
+        self._filename = filename
+        self._data     = {}
+        self._lock     = threading.Lock()
+
+        self.load()
+
+    def load(self):
+        """Load the cache contents from disk performing some rudimentary file
+        locking to prevent corruption."""
+        if not self._filename:
+            return
+
+        buf = []
+        fd  = None
+        try:
+            fd = os.open(self._filename, os.O_CREAT | os.O_RDWR)
+
+            fcntl.flock(fd, fcntl.LOCK_EX)
+
+            while True:
+                tmp = os.read(fd, 4096)
+                if not tmp:
+                    break
+                buf.append(tmp)
+
+            fcntl.flock(fd, fcntl.LOCK_UN)
+        except OSError as e:
+            raise LPInReleaseCacheError("Failed to load cache file: {}"
+                    .format(str(e)))
+        finally:
+            if fd:
+                os.close(fd)
+
+        cache_data = {} if not buf else json.loads(
+                b"".join(buf).decode("utf-8"))
+
+        with self._lock:
+            self._data = cache_data
+
+    def save(self):
+        """Save the cache contents to disk performing some rudimentary file
+        locking to prevent corruption."""
+        if not self._filename:
+            return
+
+        with self._lock:
+            buf = json \
+                .dumps(self._data, ensure_ascii=False, indent=4,
+                        sort_keys=True) \
+                .encode("utf-8")
+
+        fd = None
+        try:
+            fd = os.open(self._filename, os.O_CREAT | os.O_RDWR)
+
+            fcntl.flock(fd, fcntl.LOCK_EX)
+
+            os.ftruncate(fd, 0)
+            os.write(fd, buf)
+
+            fcntl.flock(fd, fcntl.LOCK_UN)
+        except OSError as e:
+            raise LPInReleaseCacheError("Failed to store cache file: {}"
+                    .format(str(e)))
+        finally:
+            if fd:
+                os.close(fd)
+
+    def add(self, inrelease):
+        """Add the given InRelease object to the cache."""
+        with self._lock:
+            self._data \
+                .setdefault(inrelease.normalized_address, {}) \
+                .setdefault(inrelease.suite, {}) \
+                .setdefault(inrelease.hash, inrelease.serialize())
+
+    def get_one(self, mirror, suite, hash_):
+        """Return a single InRelease object for the given mirror and suite,
+        corresponding to the hash or None if such an entry does not exist."""
+        with self._lock:
+            url_obj = urllib.parse.urlparse(mirror)
+            address = url_obj.hostname + url_obj.path.rstrip("/")
+
+            inrel = self._data\
+                .get(address, {})\
+                .get(suite, {})\
+                .get(hash_)
+
+            if not inrel:
+                return None
+
+            return InRelease(
+                inrel["mirror"],
+                inrel["suite"],
+                inrel["data"],
+                hash_=inrel["hash"],
+                last_modified=inrel["published"]
+            )
+
+    def get_all(self, mirror, suite):
+        """Retrieve a list of InRelease objects for the given mirror and suite.
+        Return a list of all known InRelease objects for the given mirror and
+        suite."""
+        with self._lock:
+            url_obj = urllib.parse.urlparse(mirror)
+            address = url_obj.hostname + url_obj.path.rstrip("/")
+
+            inrel_by_hash = self._data\
+                .get(address, {})\
+                .get(suite, {})
+
+            inrelease_list = []
+
+            for hash_, inrel in inrel_by_hash.items():
+                inrelease_list.append(
+                    InRelease(
+                        inrel["mirror"],
+                        inrel["suite"],
+                        inrel["data"],
+                        hash_=inrel["hash"],
+                        last_modified=inrel["published"]
+                    )
+                )
+
+            return inrelease_list
+
+
+class LPInReleaseIndex:
+    """Abstraction to the build system's view of the "by hash" database.
+    Currently, that interface is the by-hash directory listing of the Web
+    server."""
+
+    def __init__(self, mirror, suite, cache=None):
+        """The mirror is the base URL of the repository up to the "dists"
+        folder, e.g.
+
+        http://archive.ubuntu.com/ubuntu
+
+        suite is the name of the suite this InReleaseIndex object operates on,
+        e.g. <release>, <release>-updates or <release>-security.
+
+        Optionally, cache can be initialized to a LPInReleaseCache object, in
+        which case all look-ups will first go to the cache and only cache
+        misses will result in requests to the Web server.
+        """
+        self._mirror = mirror
+        self._suite  = suite
+        self._cache  = cache
+
+        self._base_url = "/".join([self._mirror, "dists", self._suite,
+            "by-hash/SHA256"])
+
+    def inrelease_files(self):
+        """Iterate over all InRelease files found in the archive for the mirror
+        and suite this index has been configured to operate on."""
+        hashes = self._retrieve_hashes()
+
+        for h in hashes:
+            inrelease = None
+
+            if self._cache:
+                inrelease = self._cache.get_one(self._mirror,
+                                self._suite, hash_=h)
+            if not inrelease:
+                inrelease = self._retrieve_inrelease(h)
+            if not inrelease:
+                continue
+
+            yield inrelease
+
+    def get_inrelease_for_timestamp(self, time_gmt):
+        """Find and return the InRelease file that was valid at the given Posix
+        timestamp."""
+        candidate = None
+
+        for inrelease in self.inrelease_files():
+            if inrelease.published > time_gmt:
+                continue
+            if not candidate or inrelease.published > candidate.published:
+                candidate = inrelease
+
+        return candidate
+
+    def _retrieve_inrelease(self, hash_):
+        """Retrieve the contents of the file identified by hash_. Check if the
+        file is an InRelease file and return a corresponding InRelease object.
+        If the hash_ does not belong to an InRelease file, None is returned."""
+        _500KB = 500 * 1024
+
+        buf = b""
+        inrelease = None
+        url = self._base_url + "/" + hash_
+
+        try:
+            with urllib.request.urlopen(url) as response:
+
+                # InRelease files seem to be around 200-300KB
+
+                content_length = response.headers.get("Content-Length")
+                last_modified  = response.headers.get("Last-Modified")
+
+                if not content_length:
+                    buf = response.read(_500KB + 1)
+                    content_length = len(buf)
+                else:
+                    content_length = int(content_length)
+
+                # Slightly silly heuristic, but does the job
+
+                if content_length > _500KB or content_length < 1024:
+                    return None
+
+                buf += response.read()
+
+                content_encoding = self \
+                    ._guess_content_encoding_for_response(response)
+
+                # few additional checks to see if this is an InRelease file
+
+                try:
+                    buf = buf.decode(content_encoding)
+                except UnicodeError:
+                    return None
+
+                if not buf.startswith("-----BEGIN PGP SIGNED MESSAGE-----"):
+                    return None
+
+                for kw in ["Origin:", "Label:", "Suite:", "Acquire-By-Hash:"]:
+                    if not kw in buf:
+                        return None
+
+                inrelease = InRelease(self._mirror, self._suite, buf,
+                        hash_=hash_, last_modified=last_modified)
+
+                if self._cache:
+                    self._cache.add(inrelease)
+        except urllib.error.HTTPError as e:
+            if not e.code in [404,]:
+                raise LPInReleaseIndexError("Error retrieving {}: {}"
+                    .format(url, str(e)))
+
+        return inrelease
+
+    def _guess_content_encoding_for_response(self, response):
+        """Guess the content encoding of the given HTTPResponse object."""
+        content_encoding = response.headers.get("Content-Encoding")
+        content_type     = response.headers.get("Content-Type",
+                "text/html;charset=UTF-8")
+
+        if not content_encoding:
+            m = re.match(r"^.*charset=(\S+)$", content_type)
+
+            if m:
+                content_encoding = m.group(1)
+            else:
+                content_encoding = "UTF-8"
+
+        return content_encoding
+
+    def _retrieve_hashes(self):
+        """Retrieve all available by-hashes for the mirror and suite that this
+        index is configured to operate on."""
+        hashes = []
+
+        if self._cache:
+            cache_entry = self._cache.get_all(self._mirror, self._suite)
+            if cache_entry:
+                return [inrel.hash for inrel in cache_entry]
+
+        try:
+            with urllib.request.urlopen(self._base_url) as response:
+                content_encoding = self._guess_content_encoding_for_response(
+                        response)
+
+                body   = response.read().decode(content_encoding)
+                hashes = list(set(re.findall(r"[a-z0-9]{64}", body)))
+        except urllib.error.URLError as e:
+            raise LPInReleaseIndexError("Could not retrieve hash listing: {}"
+                    .format(str(e)))
+
+        return hashes
+
+
+class LPInReleaseIndexCli:
+    """A CLI interface for LPInReleaseIndex."""
+
+    def __init__(self, name):
+        self._name      = name
+        self._mirror    = None
+        self._suite     = None
+        self._timestamp = None
+        self._cachefile = None
+        self._cache     = None
+        self._infile    = None
+        self._outfile   = None
+
+    def __call__(self, args):
+        options = vars(self._parse_opts(args))
+
+        # Copy settings to object attributes
+        for key, value in options.items():
+            if hasattr(self, "_" + key):
+                setattr(self, "_" + key, value)
+
+        if self._cachefile:
+            self._cache = LPInReleaseCache(self._cachefile)
+
+        try:
+            options["func"]()
+        except LPInReleaseIndexError as e:
+            sys.stderr.write("{}: {}\n".format(self._name, str(e)))
+            sys.exit(EXIT_ERR)
+
+        if self._cache:
+            self._cache.save()
+
+    def list(self):
+        """List all InRelease hashes for a given mirror and suite."""
+        for inrelease in self._list(self._mirror, self._suite):
+            if self._timestamp and inrelease.published > self._timestamp:
+                continue
+
+            print("{} {} ({})".format(
+                inrelease.hash,
+                inrelease.datetime,
+                inrelease.published,
+            ))
+
+    def select(self):
+        """Find the hash of the InRelease file valid at a given timestamp."""
+        candidate = self._select(self._mirror, self._suite)
+
+        if candidate:
+            print("{} {} ({})".format(
+                candidate.hash,
+                candidate.datetime,
+                candidate.published,
+            ))
+
+    def inject(self):
+        """Inject by-hash and inrelease-path settings into a sources.list."""
+        sources_list = self._infile
+
+        if not os.path.exists(sources_list):
+            sys.stderr.write("{}: No such file: {}.\n"
+                    .format(self._name, sources_list))
+            sys.exit(EXIT_ERR)
+
+        with open(sources_list, "r", encoding="utf-8") as fp:
+            buf = fp.read()
+
+        rexpr = re.compile(r"""^
+            (?P<type>deb(?:-src)?)\s+
+            (?P<opts>\[[^\]]+\]\s+)?
+            (?P<mirror>(?P<scheme>\S+):\S+)\s+
+            (?P<suite>\S+)\s+
+            (?P<comps>.*)$""", flags=re.VERBOSE)
+
+        lines = buf.splitlines(True)
+
+        for i, line in enumerate(lines):
+            line = lines[i]
+            m = rexpr.match(line)
+
+            if not m:
+                continue
+            if m.group("scheme") not in ["http", "https", "ftp"]:
+                continue
+
+            opts = {}
+            if m.group("opts"):
+                for entry in m.group("opts").strip().strip("[]").split():
+                    k, v = entry.split("=")
+                    opts[k] = v
+
+            inrelease = self._select(m.group("mirror"), m.group("suite"))
+            if inrelease:
+                opts["by-hash"]        = "yes"
+                opts["inrelease-path"] = "by-hash/SHA256/" + inrelease.hash
+
+                groupdict = m.groupdict()
+                groupdict["opts"] = " ".join(["{0}={1}".format(*o) for o in
+                    opts.items()])
+
+                lines[i] = "{type} [{opts}] {mirror} {suite} {comps}\n"\
+			.format(**groupdict)
+
+        outfile = None
+        try:
+            if not self._outfile or self._outfile == "-":
+                outfile = sys.stdout
+            else:
+                outfile = open(self._outfile, "w+", encoding="utf-8")
+            outfile.write("".join(lines))
+        finally:
+            if outfile and outfile != sys.stdout:
+                outfile.close()
+
+    def _parse_opts(self, args):
+        """Parse command line arguments and initialize the CLI object."""
+        main_parser = argparse.ArgumentParser()
+        subparsers = main_parser.add_subparsers(dest="command")
+
+        parser_inject = subparsers.add_parser("inject",
+            help="Rewrite a sources.list file injecting appropriate hashes.")
+        parser_list = subparsers.add_parser("list",
+            help="List InRelease hashes for a given release and suite.")
+        parser_select = subparsers.add_parser("select",
+            help="Select hash to use for a given timestamp, release, suite.")
+
+        parser_inject.set_defaults(func=self.inject)
+        parser_list.set_defaults(func=self.list)
+        parser_select.set_defaults(func=self.select)
+
+        # Options common to all commands
+        for parser in [parser_inject, parser_list, parser_select]:
+            cutoff_time_required = True if parser != parser_list else False
+
+            parser.add_argument("-t", "--cutoff-time", dest="timestamp",
+                type=int, required=cutoff_time_required,
+                help="A POSIX timestamp to pin the repo to.")
+            parser.add_argument("--cache-file", dest="cachefile", type=str,
+                help="A file where to cache intermediate results (optional).")
+
+        mirror = "http://archive.ubuntu.com/ubuntu"
+
+        # Options common to list, select commands
+        for parser in [parser_list, parser_select]:
+            parser.add_argument("-m", "--mirror", dest="mirror", type=str,
+                default=mirror, help="The URL of the mirror to use.")
+            parser.add_argument("-s", "--suite",
+                dest="suite", type=str, required=True,
+                help="The suite to scan (e.g. 'bionic', 'bionic-updates').")
+
+        # Extra option for inject command
+        parser_inject.add_argument("-o", "--output-file", dest="outfile",
+            type=str, help="")
+        parser_inject.add_argument("infile", type=str,
+            help="The sources.list file to modify.")
+
+        if not args:
+            main_parser.print_help()
+            sys.exit(EXIT_ERR)
+
+        return main_parser.parse_args(args)
+
+    def _list(self, mirror, suite):
+        """Internal helper for the list command. This is also used
+        implicitly by the _select method."""
+        index = LPInReleaseIndex(mirror, suite, cache=self._cache)
+
+        inrelease_files = \
+            reversed(
+                sorted(
+                    list(index.inrelease_files()),
+                    key=lambda x: x.published
+                )
+            )
+
+        return inrelease_files
+
+    def _select(self, mirror, suite):
+        """Internal helper for the select command."""
+        candidate = None
+
+        for inrelease in self._list(mirror, suite):
+            if inrelease.published > self._timestamp:
+                continue
+            if not candidate or inrelease.published > candidate.published:
+                candidate = inrelease
+
+        return candidate
+
+
+class ProxyingHTTPRequestHandler(http.server.BaseHTTPRequestHandler):
+    """Request handler providing a virtual snapshot of the package
+    repositories."""
+
+    def do_HEAD(self):
+        """Process a HEAD request."""
+        self.__get_request(verb="HEAD")
+
+    def do_GET(self):
+        """Process a GET request."""
+        self.__get_request()
+
+    def __get_request(self, verb="GET"):
+        """Pass all requests on to the destination server 1:1 except when the
+        target is an InRelease file or a resource listed in an InRelease files.
+
+        In that case we silently download the resource via the by-hash URL
+        which was most recent at the cutoff (or repo snapshot) time and inject
+        it into the response.
+
+        It is important to understand that there is no status 3xx HTTP redirect
+        happening here, the client does not know that what it receives is not
+        exactly what it requested."""
+
+        host, path = self.__get_host_path()
+
+        m = re.match(
+            r"^(?P<base>.*?)/dists/(?P<suite>[^/]+)/(?P<target>.*)$",
+            path
+        )
+
+        if m:
+            mirror = "http://" + host + m.group("base")
+            base   = m.group("base")
+            suite  = m.group("suite")
+            target = m.group("target")
+
+            index = LPInReleaseIndex(mirror, suite,
+                        cache=self.server.inrelease_cache)
+            inrelease = index.get_inrelease_for_timestamp(
+                    self.server.snapshot_stamp)
+
+            if inrelease is None:
+                self.__send_error(404, "No InRelease file found for given "
+                                       "mirror, suite and timestamp.")
+                return
+
+            if target == "InRelease":
+                # If target is InRelease, send back contents directly.
+                data = inrelease.data.encode("utf-8")
+
+                self.log_message(
+                    "Inject InRelease '{}'".format(inrelease.hash))
+
+                self.send_response(200)
+                self.send_header("Content-Length", len(data))
+                self.end_headers()
+
+                if verb == "GET":
+                    self.wfile.write(data)
+
+                return
+            else:
+                # If target hash is listed, then redirect to by-hash URL.
+                hash_ = inrelease.get_hash_for(target)
+
+                if hash_:
+                    self.log_message(
+                        "Inject {} for {}".format(hash_, target))
+
+                    target_path = target.rsplit("/", 1)[0]
+
+                    path = "{}/dists/{}/{}/by-hash/SHA256/{}"\
+                            .format(base, suite, target_path, hash_)
+
+        try:
+            client = http.client.HTTPConnection(host)
+            client.request(verb, path)
+        except Exception as e:
+            self.log_error("Failed to retrieve http://{}{}: {}"
+                    .format(host, path, str(e)))
+            return
+
+        try:
+            self.__send_response(client.getresponse())
+        except Exception as e:
+            self.log_error("Error delivering response: {}".format(str(e)))
+
+    def __get_host_path(self):
+        """Figure out the host to contact and the path of the resource that is
+        being requested."""
+        host = self.headers.get("host")
+        url  = urllib.parse.urlparse(self.path)
+        path = url.path
+
+        return host, path
+
+    def __send_response(self, response):
+        """Pass on upstream response headers and body to the client."""
+        self.send_response(response.status)
+
+        for name, value in response.getheaders():
+            self.send_header(name, value)
+
+        self.end_headers()
+        shutil.copyfileobj(response, self.wfile)
+
+    def __send_error(self, status, message):
+        """Return an HTTP error status and a message in the response body."""
+        self.send_response(status)
+        self.send_header("Content-Type", "text/plain; charset=utf-8")
+        self.end_headers()
+        self.wfile.write(message.encode("utf-8"))
+
+
+class MagicHTTPProxy(socketserver.ThreadingMixIn, http.server.HTTPServer):
+    """Tiny HTTP server using ProxyingHTTPRequestHandler instances to provide
+    a snapshot view of the package repositories."""
+
+    def __init__(self, server_address, server_port, cache_file=None,
+            repo_snapshot_stamp=time.time(), run_as=None):
+
+        try:
+            super(http.server.HTTPServer, self).__init__(
+                (server_address, server_port), ProxyingHTTPRequestHandler)
+        except OSError as e:
+            raise LPInReleaseProxyError(
+                "Could not initialize proxy: {}".format(str(e)))
+
+        self.inrelease_cache = LPInReleaseCache(filename=cache_file)
+        self.snapshot_stamp  = repo_snapshot_stamp
+
+
+class MagicHTTPProxyCli:
+    """A CLI interface for the MagicHTTPProxy."""
+
+    def __init__(self, name):
+        self._name = name
+        self._address = "127.0.0.1"
+        self._port = 8080
+        self._timestamp = time.time()
+        self._run_as = None
+        self._pid_file = None
+        self._log_file = None
+        self._background = False
+        self._setsid = False
+
+    def __call__(self, args):
+        options = self._parse_opts(args)
+
+        proxy = MagicHTTPProxy(
+            options.address,
+            options.port,
+            cache_file=None,
+            repo_snapshot_stamp=options.timestamp
+        )
+
+        # Detach, but keep all streams open.
+        if options.background:
+            pid = os.fork()
+            if pid:
+                os._exit(EXIT_OK)
+
+        if options.log_file:
+            fd = open(options.log_file, "wb+")
+            os.dup2(fd.fileno(), sys.stdout.fileno())
+            os.dup2(fd.fileno(), sys.stderr.fileno())
+
+        # Become session leader and give up controlling terminal.
+        if options.setsid:
+            if not options.log_file:
+                fd = open(os.devnull, "wb+")
+                os.dup2(fd.fileno(), sys.stdout.fileno())
+                os.dup2(fd.fileno(), sys.stderr.fileno())
+            os.setsid()
+
+        if options.pid_file:
+            with open(options.pid_file, "w+", encoding="utf-8") as fp:
+                fp.write(str(os.getpid()))
+
+        if options.run_as is not None:
+            try:
+                uid = pwd.getpwnam(options.run_as).pw_uid
+                os.setuid(uid)
+            except KeyError as e:
+                sys.stderr.write("Failed to lookup {}: {}\n"
+                        .format(options.run_as, str(e)))
+                sys.exit(EXIT_ERR)
+            except PermissionError as e:
+                sys.stderr.write("Cannot setuid: {}\n".format(str(e)))
+                sys.exit(EXIT_ERR)
+
+        proxy.serve_forever()
+
+    def _parse_opts(self, args):
+        """Parse command line arguments and initialize the CLI object."""
+        parser = argparse.ArgumentParser()
+
+        parser.add_argument("--address", dest="address", type=str,
+                default="127.0.0.1", help="The address of the interface to "
+                    "bind to (default: 127.0.0.1)")
+        parser.add_argument("--port", dest="port", type=int, default=8080,
+                help="The port to listen on (default: 8080)")
+        parser.add_argument("-t", "--cutoff-time", dest="timestamp", type=int,
+                required=True, help="A POSIX timestamp to pin the repo to.")
+        parser.add_argument("--run-as", dest="run_as", type=str,
+                help="Drop privileges and run as this user.")
+        parser.add_argument("--pid-file", dest="pid_file", type=str,
+                help="Store the PID to this file.")
+        parser.add_argument("--log-file", dest="log_file", type=str,
+                help="Re-direct all streams to this file.")
+        parser.add_argument("--background", dest="background",
+                action="store_true",
+                help="Whether to go into the background.")
+        parser.add_argument("--setsid", dest="setsid",
+                action="store_true",
+                help="Become session leader and drop controlling TTY.")
+
+        return parser.parse_args(args)
+
+if __name__ == "__main__":
+    name = os.path.basename(sys.argv[0])
+
+    try:
+        if name == "lp-in-release":
+            cli = LPInReleaseIndexCli(name)
+        else:
+            cli = MagicHTTPProxyCli(name)
+
+        cli(sys.argv[1:])
+    except LPInReleaseBaseError as e:
+        sys.stderr.write("{}: {}\n".format(name, str(e)))
+        sys.exit(EXIT_ERR)
+    except KeyboardInterrupt:
+        sys.stderr.write("{}: Caught keyboard interrupt, exiting...\n"
+                .format(name))
+        sys.exit(EXIT_ERR)

minimize-manual

@@ -0,0 +1,63 @@
+#!/usr/bin/python3
+"""Minimize the number of manually installed packages in the image.
+
+Finds all manually installed meta packages, and marks their dependencies
+as automatically installed.
+"""
+import sys
+
+import apt
+
+
+def is_root(pkg):
+    """Check if the package is a root package (manually inst. meta)"""
+    return (pkg.is_installed and
+            not pkg.is_auto_installed and
+            (pkg.section == "metapackages" or
+             pkg.section.endswith("/metapackages")))
+
+
+def main():
+    """Main function"""
+    cache = apt.Cache(rootdir=sys.argv[1] if len(sys.argv) > 1 else None)
+    roots = set(pkg for pkg in cache if is_root(pkg))
+    workset = set(roots)
+    seen = set()
+    ubiquity_depends = set()
+
+    with cache.actiongroup():
+        while True:
+            print("Iteration", file=sys.stderr)
+            to_proc = workset - seen
+            if not to_proc:
+                break
+            for pkg in sorted(to_proc):
+                print("    Visiting", pkg, file=sys.stderr)
+
+                if pkg not in roots and pkg not in ubiquity_depends:
+                    pkg.mark_auto()
+
+                for dep in (pkg.installed.dependencies +
+                            pkg.installed.recommends):
+                    for bdep in dep.or_dependencies:
+                        for ver in bdep.target_versions:
+                            if ver.package.is_installed:
+                                if pkg.name == "ubiquity":
+                                    ubiquity_depends.add(ver.package)
+                                if pkg.name != "ubiquity":
+                                    # Reprocess this package again, as we did not mark it when we visited it from ubiquity
+                                    try:
+                                        ubiquity_depends.remove(ver.package)
+                                        # This will raise the KeyError here if ubiquity did not depend on it
+                                        seen.remove(ver.package)
+                                    except KeyError:
+                                        pass
+                                workset.add(ver.package)
+
+                seen.add(pkg)
+
+        cache.commit()
+
+
+if __name__ == '__main__':
+    main()

snap-tool

@@ -0,0 +1,603 @@
+#!/usr/bin/python3
+#-*- encoding: utf-8 -*-
+"""
+This script can be used instead of the traditional `snap` command to download
+snaps and accompanying assertions. It uses the new store API (v2) which allows
+creating temporary snapshots of the channel map.
+
+To create such a snapshot run
+
+    snap-tool cohort-create
+
+This will print a "cohort-key" to stdout, which can then be passed to future
+invocations of `snap-tool download`. Whenever a cohort key is provided, the
+store will provide a view of the channel map as it existed when the key was
+created.
+"""
+
+from textwrap import dedent
+
+import argparse
+import base64
+import binascii
+import getopt
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+
+EXIT_OK  = 0
+EXIT_ERR = 1
+
+
+class SnapError(Exception):
+    """Generic error thrown by the Snap class."""
+    pass
+
+
+class SnapCraftError(SnapError):
+    """Error thrown on problems with the snapcraft APIs."""
+    pass
+
+
+class SnapAssertionError(SnapError):
+    """Error thrown on problems with the assertions API."""
+    pass
+
+
+class ExpBackoffHTTPClient:
+    """This class is an abstraction layer on top of urllib with additional
+    retry logic for more reliable downloads."""
+
+    class Request:
+        """This is a convenience wrapper around urllib.request."""
+
+        def __init__(self, request, do_retry, base_interval, num_tries):
+            """
+            :param request:
+                An urllib.request.Request instance.
+            :param do_retry:
+                Whether to enable the exponential backoff and retry logic.
+            :param base_interval:
+                The initial interval to sleep after a failed attempt.
+            :param num_tries:
+                How many attempts to make.
+            """
+            self._request       = request
+            self._do_retry      = do_retry
+            self._base_interval = base_interval
+            self._num_tries     = num_tries
+            self._response      = None
+
+        def open(self):
+            """Open the connection."""
+            if not self._response:
+                self._response = self._retry_urlopen()
+
+        def close(self):
+            """Close the connection."""
+            if self._response:
+                self._response.close()
+                self._response = None
+
+        def data(self):
+            """Return the raw response body."""
+            with self:
+                return self.read()
+
+        def json(self):
+            """Return the deserialized response body interpreted as JSON."""
+            return json.loads(self.data(), encoding="utf-8")
+
+        def text(self):
+            """Return the response body as a unicode string."""
+            encoding = "utf-8"
+
+            with self:
+                content_type = self._response.getheader("Content-Type", "")
+
+                if content_type == "application/json":
+                    encoding = "utf-8"
+                else:
+                    m = re.match(r"text/\S+;\s*charset=(?P<charset>\S+)",
+                            content_type)
+                    if m:
+                        encoding=m.group("charset")
+
+                return self.read().decode(encoding)
+
+        def read(self, size=None):
+            """Read size bytes from the response. If size if not set, the
+            complete response body is read in."""
+            return self._response.read(size)
+
+        def __enter__(self):
+            """Make this class a context manager."""
+            self.open()
+            return self
+
+        def __exit__(self, type, value, traceback):
+            """Make this class a context manager."""
+            self.close()
+
+        def _retry_urlopen(self):
+            """Try to open the HTTP connection as many times as configured
+            through the constructor. Every time an error occurs, double the
+            time to wait until the next attempt."""
+            for attempt in range(self._num_tries):
+                try:
+                    return urllib.request.urlopen(self._request)
+                except Exception as e:
+                    if isinstance(e, urllib.error.HTTPError) and e.code < 500:
+                        raise
+                    if attempt >= self._num_tries - 1:
+                        raise
+                    sys.stderr.write(
+                        "WARNING: failed to open URL '{}': {}\n"
+                        .format(self._request.full_url, str(e))
+                    )
+                else:
+                    break
+
+                sleep_interval = self._base_interval * 2**attempt
+                sys.stderr.write(
+                    "Retrying HTTP request in {} seconds...\n"
+                    .format(sleep_interval)
+                )
+                time.sleep(sleep_interval)
+
+
+    def __init__(self, do_retry=True, base_interval=2, num_tries=8):
+        """
+        :param do_retry:
+            Whether to enable the retry logic.
+        :param base_interval:
+            The initial interval to sleep after a failed attempt.
+        :param num_tries:
+            How many attempts to make.
+        """
+        self._do_retry      = do_retry
+        self._base_interval = base_interval
+        self._num_tries     = num_tries if do_retry else 1
+
+    def get(self, url, headers=None):
+        """Create a GET request that can be used to retrieve the resource
+        at the given URL.
+
+        :param url:
+            An HTTP URL.
+        :param headers:
+            A dictionary of extra headers to send along.
+        :return:
+            An ExpBackoffHTTPClient.Request instance.
+        """
+        return self._prepare_request(url, headers=headers)
+
+    def post(self, url, data=None, json=None, headers=None):
+        """Create a POST request that can be used to submit data to the
+        endpoint at the given URL."""
+        return self._prepare_request(
+            url, data=data, json_data=json, headers=headers
+        )
+
+    def _prepare_request(self, url, data=None, json_data=None, headers=None):
+        """Prepare a Request instance that can be used to retrieve data from
+        and/or send data to the endpoint at the given URL.
+
+        :param url:
+            An HTTP URL.
+        :param data:
+            Raw binary data to send along in the request body.
+        :param json_data:
+            A Python data structure to be serialized and sent out in JSON
+            format.
+        :param headers:
+            A dictionary of extra headers to send along.
+        :return:
+            An ExpBackoffHTTPClient.Request instance.
+        """
+        if data is not None and json_data is not None:
+            raise ValueError(
+                "Parameters 'data' and 'json_data' are mutually exclusive."
+            )
+
+        if json_data:
+            data = json.dumps(json_data, ensure_ascii=False)
+            if headers is None:
+                headers = {}
+            headers["Content-Type"] = "application/json"
+        if isinstance(data, str):
+            data = data.encode("utf-8")
+
+        return ExpBackoffHTTPClient.Request(
+            urllib.request.Request(url, data=data, headers=headers or {}),
+            self._do_retry,
+            self._base_interval,
+            self._num_tries
+        )
+
+
+class Snap:
+    """This class provides methods to retrieve information about a snap and
+    download it together with its assertions."""
+
+    def __init__(self, name, channel="stable", arch="amd64", series=16,
+            cohort_key=None, assertion_url="https://assertions.ubuntu.com",
+            snapcraft_url="https://api.snapcraft.io", **kwargs):
+        """
+        :param name:
+            The name of the snap.
+        :param channel:
+            The channel to operate on.
+        :param arch:
+            The Debian architecture of the snap (e.g. amd64, armhf, arm64, ...).
+        :param series:
+            The device series. This should always be 16.
+        :param cohort_key:
+            A cohort key to access a snapshot of the channel map.
+        """
+        self._name          = name
+        self._channel       = channel
+        self._arch          = arch
+        self._series        = series
+        self._cohort_key    = cohort_key
+        self._assertion_url = assertion_url
+        self._snapcraft_url = snapcraft_url
+        self._details       = None
+        self._assertions    = {}
+
+    @classmethod
+    def cohort_create(cls):
+        """Get a cohort key for the current moment. A cohort key is valid
+        across all snaps, channels and architectures."""
+        return Snap("core")\
+            .get_details(cohort_create=True)\
+            .get("cohort-key")
+
+    def download(self, download_assertions=True):
+        """Download the snap container. If download_assertions is True, the
+        corresponding assertions will be downloaded, as well."""
+        snap = self.get_details()
+
+        snap_name         = snap["name"]
+        snap_revision     = snap["revision"]
+        publisher_id      = snap["publisher"]["id"]
+        snap_download_url = snap["download"]["url"]
+        snap_byte_size    = snap["download"]["size"]
+        filename          = snap_name + "_" + str(snap_revision)
+        snap_filename     = filename + ".snap"
+        assert_filename   = filename + ".assert"
+
+        skip_snap_download = False
+
+        if os.path.exists(snap_filename) and os.path.getsize(snap_filename) \
+                == snap_byte_size:
+            skip_snap_download = True
+
+        headers = {}
+
+        if os.environ.get("SNAPPY_STORE_NO_CDN", "0") != "0":
+            headers.update({
+                "X-Ubuntu-No-Cdn": "true",
+                "Snap-CDN": "none",
+            })
+
+        if not skip_snap_download:
+            http_client = ExpBackoffHTTPClient()
+            response = http_client.get(snap_download_url, headers=headers)
+            with response, open(snap_filename, "wb+") as fp:
+                shutil.copyfileobj(response, fp)
+
+        if os.path.getsize(snap_filename) != snap_byte_size:
+            raise SnapError(
+                "The downloaded snap does not have the expected size."
+            )
+
+        if not download_assertions:
+            return
+
+        required_assertions = [
+            "account-key",
+            "account",
+            "snap-declaration",
+            "snap-revision",
+        ]
+
+        if publisher_id == "canonical":
+            required_assertions.remove("account")
+
+        for assertion_name in required_assertions:
+            attr_name = "get_assertion_" + assertion_name.replace("-", "_")
+            # This will populate self._assertions[<assertion_name>].
+            getattr(self, attr_name)()
+
+        with open(assert_filename, "w+", encoding="utf-8") as fp:
+            fp.write("\n".join(self._assertions[a] for a in
+                required_assertions))
+
+    def get_details(self, cohort_create=False):
+        """Get details about the snap. On subsequent calls, the cached results
+        are returned. If cohort_create is set to True, a cohort key will be
+        created and included in the result."""
+        if self._details and not cohort_create:
+            return self._details
+
+        if self.is_core_snap() and self._channel.startswith("stable/ubuntu-"):
+            sys.stderr.write(
+                "WARNING: switching channel from '{}' to 'stable' for '{}' "
+                "snap.\n".format(self._channel, self._name)
+            )
+            self._channel = "stable"
+
+        path = "/v2/snaps/refresh"
+
+        data = {
+            "context": [],
+            "actions": [
+                {
+                    "action":       "download",
+                    "instance-key": "0",
+                    "name":         self._name,
+                    "channel":      self._channel,
+                }
+            ],
+            "fields": [
+                "base",
+                "created-at",
+                "download",
+                "license",
+                "name",
+                "prices",
+                "publisher",
+                "revision",
+                "snap-id",
+                "summary",
+                "title",
+                "type",
+                "version",
+            ],
+        }
+
+        # These are mutually exclusive.
+        if cohort_create:
+            data["actions"][0]["cohort-create"] = True
+        elif self._cohort_key:
+            data["actions"][0]["cohort-key"] = self._cohort_key
+
+        try:
+            response_dict = self._do_snapcraft_request(path, json_data=data)
+        except SnapCraftError as e:
+            raise SnapError("failed to get details for '{}': {}"
+                    .format(self._name, str(e)))
+
+        snap_data = response_dict["results"][0]
+
+        if snap_data.get("result") == "error":
+            raise SnapError(
+                "failed to get details for '{}': {}"
+                .format(self._name, snap_data.get("error", {}).get("message"))
+            )
+
+        # Have "base" initialized to something meaningful.
+        if self.is_core_snap():
+            snap_data["snap"]["base"] = ""
+        elif snap_data["snap"].get("base") is None:
+            snap_data["snap"]["base"] = "core"
+
+        # Copy the key into the snap details.
+        if "cohort-key" in snap_data:
+            snap_data["snap"]["cohort-key"] = snap_data["cohort-key"]
+
+        if "error" in snap_data:
+            raise SnapError(
+                "failed to get details for '{}' in '{}' on '{}': {}"
+                    .format(self._name, self._channel, self._arch,
+                        snap_data["error"]["message"])
+            )
+
+        self._details = snap_data["snap"]
+        return self._details
+
+    def get_assertion_snap_revision(self):
+        """Download the snap-revision assertion associated with this snap. The
+        assertion is returned as a string."""
+        if "snap-revision" in self._assertions:
+            return self._assertions["snap-revision"]
+        snap = self.get_details()
+
+        snap_sha3_384 = base64.urlsafe_b64encode(
+            binascii.a2b_hex(snap["download"]["sha3-384"])
+        ).decode("us-ascii")
+
+        data = self._do_assertion_request("/v1/assertions/snap-revision/{}"
+                .format(snap_sha3_384))
+        self._assertions["snap-revision"] = data
+        return data
+
+    def get_assertion_snap_declaration(self):
+        """Download the snap-declaration assertion associated with this snap.
+        The assertion is returned as a string."""
+        if "snap-declaration" in self._assertions:
+            return self._assertions["snap-declaration"]
+        snap = self.get_details()
+        series = self._series
+        snap_id = snap["snap-id"]
+
+        data = self._do_assertion_request(
+                "/v1/assertions/snap-declaration/{}/{}"
+                    .format(series, snap_id))
+
+        self._assertions["snap-declaration"] = data
+        return data
+
+    def get_assertion_account(self):
+        """Download the account assertion associated with this snap. The
+        assertion is returned as a string."""
+        if "account" in self._assertions:
+            return self._assertions["account"]
+        snap = self.get_details()
+        publisher_id = snap["publisher"]["id"]
+        data = self._do_assertion_request("/v1/assertions/account/{}"
+                .format(publisher_id))
+        self._assertions["account"] = data
+        return data
+
+    def get_assertion_account_key(self):
+        """Download the account-key assertion associated with this snap. The
+        assertion will be returned as a string."""
+        if "account-key" in self._assertions:
+            return self._assertions["account-key"]
+
+        declaration_data = self.get_assertion_snap_declaration()
+        sign_key_sha3 = None
+
+        for line in declaration_data.splitlines():
+            if line.startswith("sign-key-sha3-384:"):
+                sign_key_sha3 = line.split(":")[1].strip()
+
+        data = self._do_assertion_request("/v1/assertions/account-key/{}"
+                .format(sign_key_sha3))
+
+        self._assertions["account-key"] = data
+        return data
+
+    def is_core_snap(self):
+        return re.match(r"^core\d*$", self._name) != None
+
+    def _do_assertion_request(self, path):
+        url = self._assertion_url + path
+
+        headers = {
+            "Accept": "application/x.ubuntu.assertion",
+        }
+
+        http_client = ExpBackoffHTTPClient()
+        try:
+            with http_client.get(url, headers=headers) as response:
+                return response.text()
+        except urllib.error.HTTPError as e:
+            raise SnapAssertionError(str(e))
+
+    def _do_snapcraft_request(self, path, json_data=None):
+        url = self._snapcraft_url + "/" + path
+
+        headers = {
+            "Snap-Device-Series": str(self._series),
+            "Snap-Device-Architecture": self._arch,
+        }
+
+        http_client = ExpBackoffHTTPClient()
+        try:
+            response = http_client.post(url, json=json_data, headers=headers)
+            with response:
+                return response.json()
+        except urllib.error.HTTPError as e:
+            raise SnapCraftError(str(e))
+
+
+class SnapCli:
+
+    def __call__(self, args):
+        """Parse the command line arguments and execute the selected command."""
+        options = self._parse_opts(args)
+
+        try:
+            options.func(getattr(options, "snap", None), **vars(options))
+        except SnapError as e:
+            sys.stderr.write("snap-tool {}: {}\n".format(
+                options.command, str(e)))
+            return EXIT_ERR
+        return EXIT_OK
+
+    @staticmethod
+    def _get_host_deb_arch():
+        result = subprocess.run(["dpkg", "--print-architecture"],
+                stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+                universal_newlines=True, check=True)
+
+        return result.stdout.strip()
+
+    def _parse_opts(self, args):
+        main_parser = argparse.ArgumentParser()
+        subparsers = main_parser.add_subparsers(dest="command")
+
+        parser_cohort_create = subparsers.add_parser("cohort-create",
+                help="Create a cohort key for the snap store channel map.")
+        parser_cohort_create.set_defaults(func=self._cohort_create)
+
+        parser_download = subparsers.add_parser("download",
+                help="Download a snap from the store.")
+        parser_download.set_defaults(func=self._download)
+
+        parser_info = subparsers.add_parser("info",
+                help="Retrieve information about a snap.")
+        parser_info.set_defaults(func=self._info)
+
+        # Add common parameters.
+        for parser in [parser_download, parser_info]:
+            parser.add_argument("--cohort-key", dest="cohort_key",
+                help="A cohort key to pin the channel map to.", type=str)
+            parser.add_argument("--channel", dest="channel",
+                help="The publication channel to query (default: stable).",
+                type=str, default="stable")
+            parser.add_argument("--series", dest="series",
+                help="The device series (default: 16)",
+                type=int, default=16)
+            parser.add_argument("--arch", dest="arch",
+                help="The Debian architecture (default: amd64).",
+                type=str, default=self._get_host_deb_arch())
+            parser.add_argument("snap", help="The name of the snap.")
+
+        if not args:
+            main_parser.print_help()
+            sys.exit(EXIT_ERR)
+
+        return main_parser.parse_args(args)
+
+    def _cohort_create(self, _, **kwargs):
+        print(Snap.cohort_create())
+
+    def _download(self, snap_name, **kwargs):
+        Snap(snap_name, **kwargs).download()
+
+    def _info(self, snap_name, **kwargs):
+        snap = Snap(snap_name, **kwargs)
+        info = snap.get_details()
+
+        print(dedent("""\
+            name:      {}
+            summary:   {}
+            arch:      {}
+            base:      {}
+            channel:   {}
+            publisher: {}
+            license:   {}
+            snap-id:   {}
+            revision:  {}"""
+            .format(
+                snap_name,
+                info.get("summary", ""),
+                snap._arch,
+                info.get("base"),
+                snap._channel,
+                info.get("publisher", {}).get("display-name", ""),
+                info.get("license", ""),
+                info.get("snap-id", ""),
+                info.get("revision", "")
+            ))
+        )
+
+
+if __name__ == "__main__":
+    try:
+        rval = SnapCli()(sys.argv[1:])
+    except KeyboardInterrupt:
+        sys.stderr.write("snap-tool: caught keyboard interrupt, exiting.\n")
+        sys.exit(EXIT_ERR)
+    sys.exit(rval)