releasing package livecd-rootfs version 2.408.24

all our livefses.
* Pass --cache false to lb config; otherwise we copy around caches of
  .debs that are never used properly, and which prevent us from emptying
  /var/cache/apt in images.
* When building minimized cloud images, remove various packages that we
  don't want installed by default.  Some are tools that aren't needed for
  non-interactive use; some are libraries whose reverse-dependencies
  will have already been removed; and one, open-vm-tools, should only be
  included in images that are targeted to VMWare (which is not the case
  for any of the current minimal images), rather than being included
  directly in the cloud-image seed.

(expanded changelog)

debian/changelog

@@ -1,3 +1,297 @@
+livecd-rootfs (2.408.24) xenial; urgency=medium
+
+  * For the IMAGEFORMAT=none parts to work for ubuntu-image classic builds, we
+    need to make sure IMAGEFORMAT is preserved and not overwritten if defined.
+    Backporting from artful.
+
+ -- Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com>  Tue, 21 Nov 2017 17:01:14 +0100
+
+livecd-rootfs (2.408.23) xenial; urgency=medium
+
+  [ Gary Wang ]
+  * Add the IMAGEFORMAT=none to support generating a single rootfs
+    (LP: #1730642)
+
+  [ Christopher Glass (Ubuntu) ]
+  * Backport "minimized round 2" changes from trunk to Xenial (LP: #1731492)
+  * Remove apt, debconf, dpkg cruft files from /var/cache and /var/lib in
+    all our livefses.
+  * Pass --cache false to lb config; otherwise we copy around caches of
+    .debs that are never used properly, and which prevent us from emptying
+    /var/cache/apt in images.
+  * When building minimized cloud images, remove various packages that we
+    don't want installed by default.  Some are tools that aren't needed for
+    non-interactive use; some are libraries whose reverse-dependencies
+    will have already been removed; and one, open-vm-tools, should only be
+    included in images that are targeted to VMWare (which is not the case
+    for any of the current minimal images), rather than being included
+    directly in the cloud-image seed.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Tue, 14 Nov 2017 11:21:48 -0800
+
+livecd-rootfs (2.408.22) xenial; urgency=medium
+
+  * Make sure to remove any initramfs that was generated in a minimized
+    image.
+  * divert /usr/bin/systemd-detect-virt as part of the grub diversions, so
+    that the grub kernel postinst hook is operational when we're building
+    in a container.  LP: #1729034.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Sat, 21 Oct 2017 20:53:02 -0700
+
+livecd-rootfs (2.408.21) xenial; urgency=medium
+
+  [ Balint Reczey ]
+  * Use kvm kernel only on amd64.
+  * Make non-x86 minimized images consistent with x86 by not explicitly
+    installing the server task.
+  * Clean up dangling /boot/initrd.img symlink left behind on minimized
+    builds.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Fri, 20 Oct 2017 23:10:29 -0700
+
+livecd-rootfs (2.408.20) xenial; urgency=medium
+
+  * Now that grub-related diversions have been factored out in 2.466
+    instead of having bogus root=stuff arg generated in grub.cfg, it is
+    actually empty. Therefore update the sed command to make the arg in
+    the root= token optional. This should resolve non-booting livecd cpc
+    images.
+
+ -- Dimitri John Ledkov <xnox@ubuntu.com>  Tue, 10 Oct 2017 13:49:49 +0100
+
+livecd-rootfs (2.408.19) xenial; urgency=medium
+
+  [ Nishanth Aravamudan ]
+  * live-build/ubuntu-cpc/hooks/061-open-iscsi.chroot: generate iSCSI
+    Initiator Name at first iscsid run for cloud images to ensure it is
+    unique (LP: #1444992).
+
+  [ Steve Langasek ]
+  * Improve teardown_mountpoint to recursively find all submounts and
+    unmount them, instead of working from a hard-coded list.  This makes
+    the code resilient against other submounts being added later, including
+    downstream.  LP: #1721279.
+  * Also nuke the sleep / udevadm settle calls in the process, which should
+    never be required and slow down the builds.
+  * Fix a reference to an undefined variable in a script that's set -u.
+  * Use /bin/sh, not /bin/bash, for autopkgtest.
+  * debian/tests/default-bootstraps: minor adjustments to shell syntax,
+    syncing with artful where this originated.
+
+  [ Steve Langasek, Balint Reczey ]
+  * Introduce a new project-independent 'minimized' subproject
+    (LP: #1721261):
+    - omit ubuntu-minimal in favor of using only the minbase package set.
+    - boot directly by partuuid, avoiding the use of an initramfs.
+    - Bump needed live-build version which can build images without initrd
+    - drop man pages and most of the documentation from minimized images
+      (/usr/share/doc/*/copyright and changelog.Debian.gz files are still
+      kept)
+    - Add unminimize script for reverting minimization on a running system
+    - Mention unminimize script in motd
+    - Run autopkgtest for SUBPROJECT=minimized
+    - If we're using SUBPROJECT=minimized, and tzdata is not installed,
+      remove files that have been left behind.  This is a workaround for a
+      bug that should be fixed in tzdata.
+  * Factor out grub-related diversions and use them consistently, so we
+    don't end up with wrong os-probe output in our grub.cfg.
+
+  [ Balint Reczey ]
+  * Mount using --make-rslave to ensure safe unmounts for rbind mounts
+  * When SUBPROJECT environment variable is not set assume it to be ""
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Wed, 04 Oct 2017 18:43:48 +0000
+
+livecd-rootfs (2.408.18) xenial; urgency=medium
+
+  [ Robert C Jennings ]
+  * Install udev before calls to udevadm
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Wed, 13 Sep 2017 22:10:11 -0700
+
+livecd-rootfs (2.408.17) xenial; urgency=medium
+
+  [ Robert C Jennings ]
+  * live-build/ubuntu-cpc/functions: Add a function, teardown_mountpoint,
+    to reverse the work done in setup_mountpoint.  Lack of this function
+    has forced users of setup_mountpoint to implement this separately
+    and the implementations have diverged. (LP: #1716992)
+  * live-build/ubuntu-cpc/functions: Remove umount_settle function.
+    The was only used where teardown_mountpoint was lacking.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Wed, 13 Sep 2017 12:21:30 -0700
+
+livecd-rootfs (2.408.16) xenial; urgency=medium
+
+  * live-build/ubuntu-cpc/functions: call apt-get update in the chroot
+    after mounting a blank /var/lib/apt, so that further operations work
+    as expected; otherwise, 'apt-get purge ^grub-.*' fails on s390x because
+    no such packages are known to apt.
+  * live-build/ubuntu-cpc/hooks/030-root-tarball.binary: correct a missing
+    unmount of /var/{lib,cache}/apt on cleanup, detected via autopkgtests.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Tue, 12 Sep 2017 13:25:58 -0700
+
+livecd-rootfs (2.408.15) xenial; urgency=medium
+
+  [ Mathieu Trudel-Lapierre ]
+  * Drop preloading of grub modules that are built into the grub signed
+    image.  This is functionally a no-op, changed only to clean up the code.
+
+  [ Steve Langasek ]
+  * live-build/ubuntu-cpc/functions: mount tmpfs on /var/cache/apt and
+    /var/lib/apt, so we don't have to leave empty space in our derivative
+    images for packages that have been downloaded/installed/removed.  This
+    normally isn't relevant for the installed system, since the root
+    filesystem will auto-expand in place on the target disk, but lets us
+    ship smaller images.
+  * live-build/ubuntu-cpc/hooks/033-disk-image-uefi.binary: call apt-get
+    update *before* installing packages, not after.
+
+  [ Colin Watson ]
+  * Mount and unmount /dev recursively, to cope with setups where there are
+    interesting bind-mounts under /dev (e.g. loop devices bind-mounted by
+    LXD).  LP: #1716465.
+
+  [ Balint Reczey ]
+  * Fix suppression of kpartx error.  LP: #1684090.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Mon, 11 Sep 2017 14:38:42 -0700
+
+livecd-rootfs (2.408.14) xenial; urgency=medium
+
+  * live-build/auto/config: Filter libgles1-mesa out of tasks, as it is not a
+    dependency of anything in updates anymore, fixing mate/myth (LP: #1704013)
+
+ -- Adam Conrad <adconrad@ubuntu.com>  Wed, 12 Jul 2017 16:28:13 -0600
+
+livecd-rootfs (2.408.13) xenial; urgency=medium
+
+  [ Balint Reczey ]
+  * Source ubuntu-cpc functions from the right place
+  * Use all config hooks from the proper place, not from /build/
+  * Add basic but configurable autopkgtest (LP: #1690440)
+  * sync before calling kpartx to let writing to loop devices finish
+  * wrap kpartx and trap spurious errors, to work around kpartx
+    unreliability.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Tue, 06 Jun 2017 21:52:21 -0700
+
+livecd-rootfs (2.408.12) xenial; urgency=medium
+
+  * live-build/ubuntu-cpc/hooks/999-extras.binary: Exit on first failure.
+    (LP: #1687752)
+
+ -- Robert C Jennings <robert.jennings@canonical.com>  Tue, 09 May 2017 13:57:44 -0700
+
+livecd-rootfs (2.408.11) xenial; urgency=medium
+
+  * Add `apt-get update` to ubuntu-cpc ppc64el builds so they use the new
+    sources.
+
+ -- Daniel Watkins <daniel.watkins@canonical.com>  Fri, 07 Apr 2017 16:12:53 -0400
+
+livecd-rootfs (2.408.10) xenial; urgency=medium
+
+  [ Adam Conrad ]
+  * Fix security mirror sources.list entries for non-x86 architectures.
+    (LP: #1679252)
+
+ -- Daniel Watkins <daniel.watkins@canonical.com>  Mon, 03 Apr 2017 14:05:25 -0400
+
+livecd-rootfs (2.408.9) xenial; urgency=medium
+
+  [ Daniel Watkins ]
+  * Don't overwrite the default sources.list in cloud images.
+  * Replace sources.list generated using COMPONENTS with the sources.list from
+    an Ubuntu Server installation (i.e. with all components enabled, and all
+    deb-src lines commented).  LP: #1513529.
+
+  [ Chris Glass ]
+  * Fix the manifest generation in OVA files so that ovf files don't have
+    double extensions.  (LP: #1627931)
+  * Fix the OVF's metadata to include Ubuntu specific identifiers and
+    descriptions instead of the generic Linux ones.  (LP: #1656293) 
+
+  [ Daniel Watkins ]
+  * Add replace_grub_root_with_label function thereby consolidating multiple
+    uses of the same calls to sed.
+
+  [ Robert C Jennings ]
+  * ubuntu-cpc: Remove redundant copy of grub files. (LP: #1637290)
+
+ -- Robert C Jennings <robert.jennings@canonical.com>  Thu, 23 Mar 2017 14:40:59 -0400
+
+livecd-rootfs (2.408.8) xenial; urgency=medium
+
+  * Set device_tree_address for the new kernel in the RPi bootloader config.txt
+
+ -- Adam Conrad <adconrad@ubuntu.com>  Thu, 16 Feb 2017 10:21:51 -0700
+
+livecd-rootfs (2.408.7) xenial; urgency=medium
+
+  [ Łukasz 'sil2100' Zemczak ]
+  * Add additional hints to ubuntu-touch* i386/amd64 image builds to pull in
+    the gles version of the UITK.
+  * Now that we have have systemd working on ubuntu-touch, remove the
+    systemd-sysv- in add_package config to make sure we can build an image with
+    systemd as the init system.
+
+  [ Adam Conrad ]
+  * Forward-port hardware enablement delta from trusty to xenial, with changes:
+    - Implement hwe stack swapping as task filters instead of install hints
+    - Revert ubuntu-desktop and ubuntu-gnome to use tasks again for the above
+
+ -- Adam Conrad <adconrad@ubuntu.com>  Wed, 08 Feb 2017 11:50:52 -0700
+
+livecd-rootfs (2.408.5) xenial; urgency=medium
+ 
+  * Temporarily change the 60-install-click.chroot script to pull in clicks for
+    arm64 builds from a different location for ubuntu-touch.
+  * Add the new dhcpd user to touch hooks introduced by latest archive changes.
+  * Remove the ubuntu-pd project logic.
+  * Synced ubuntu-touch-custom changes from the vivid branch to xenial. This
+    also adds a new symlink called ubuntu-touch-custom to the ubuntu-touch hooks
+    dir.
+  * Stop using the overlay PPA for touch livecd-rootfs changes (LP: #1628085).
+
+ -- Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com>  Tue, 27 Sep 2016 13:27:28 +0200
+
+livecd-rootfs (2.408.4) xenial; urgency=medium
+
+  * live-build/ubuntu-cpc/hooks/042-vagrant.binary: fix unmount handling
+    so that the teardown is done properly /before/ we try to make an
+    image from our filesystem, since otherwise /etc/resolv.conf is broken.
+    LP: #1621393.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Fri, 09 Sep 2016 17:04:54 -0700
+
+livecd-rootfs (2.408.3) xenial-proposed; urgency=medium
+
+  [ Louis Zuckerman ]
+  * Fixes for vagrant box builder in ubuntu-cpc LP: #1565985
+    - Install virtualbox-guest-utils
+    - Don't disable default synced folder
+    - Don't set vm name
+    - Add cloud-init config to manage /etc/hosts LP: #1561250
+
+ -- Brian Murray <brian@ubuntu.com>  Tue, 30 Aug 2016 13:17:55 -0700
+
+livecd-rootfs (2.408.2) xenial; urgency=medium
+
+  * Switch ubuntu-gnome from using tasks to metapackages (LP: #1602035)
+
+ -- Adam Conrad <adconrad@ubuntu.com>  Wed, 20 Jul 2016 01:46:41 -0600
+
+livecd-rootfs (2.408.1) xenial; urgency=medium
+
+  * Rename old ubuntu-core tarballs to ubuntu-base, for consistency with
+    the changes already made to yakkety and the ubuntu-cdimage project.
+    LP: #1579950.
+
+ -- Steve Langasek <steve.langasek@ubuntu.com>  Mon, 09 May 2016 17:00:19 -0700
+
 livecd-rootfs (2.408) xenial; urgency=medium
 
   * drop linux-firmware-raspi2, it does not actually contain driver

debian/control

@@ -4,11 +4,11 @@ Priority: optional
 Build-Depends: debhelper (>= 7)
 Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
 Standards-Version: 3.9.6
-Vcs-Bzr: http://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk
+Vcs-Bzr: lp:~ubuntu-core-dev/livecd-rootfs/xenial-proposed/
 
 Package: livecd-rootfs
 Architecture: any
-Depends: ${misc:Depends}, debootstrap, rsync, python-minimal | python, procps, squashfs-tools (>= 1:3.3-1), grep-dctrl, lsb-release, lzma, e2fsprogs, germinate (>= 1.25.1), apt-utils, gnupg, live-build (>= 3.0~a57-1ubuntu12~), android-tools-fsutils [armhf], python3-software-properties
+Depends: ${misc:Depends}, debootstrap, rsync, python-minimal | python, procps, squashfs-tools (>= 1:3.3-1), grep-dctrl, lsb-release, lzma, e2fsprogs, germinate (>= 1.25.1), apt-utils, gnupg, live-build (>= 3.0~a57-1ubuntu25.5~), android-tools-fsutils [armhf], python3-software-properties
 Suggests: partimage
 Breaks: ubuntu-defaults-builder (<< 0.32)
 Description: construction script for the livecd rootfs

debian/tests/control

@@ -0,0 +1,7 @@
+Tests: default-bootstraps
+Depends: @, lsb-release
+Restrictions: needs-root isolation-machine
+
+Tests: minimized
+Depends: @, lsb-release
+Restrictions: needs-root isolation-machine

debian/tests/default-bootstraps

@@ -0,0 +1,96 @@
+#!/bin/sh
+# autopkgtest check: Build default rootfs for all supported project:subproject pairs
+# (C) 2017 Canonical Ltd.
+# Author: Balint Reczey <balint.reczey@canonical.com>
+
+set -e
+
+if [ -z "$SUITE" ]; then
+    SUITE=$(lsb_release -c -s)
+fi
+
+# Known project:subproject:template combinations.
+# Listed subprojects can be combined with other projects as well,
+# but this list gives reasonable coverage.
+ALL_TRIPLETS="
+ base::
+ edubuntu::
+ edubuntu-dvd::
+ kubuntu::
+ kubuntu-active::
+ kubuntu-dvd::
+ kubuntu-plasma5::
+ lubuntu::
+ lubuntu-next::
+ mythbuntu::
+ ubuntu::
+ ubuntu-base::
+ ubuntu-budgie::
+ ubuntu-budgie-desktop::
+ ubuntu-budgie-live::
+ ubuntu-core:system-image:ubuntu-core
+ ubuntu-cpc::ubuntu-cpc
+ ubuntu-cpc:minimized:ubuntu-cpc
+ ubuntu-desktop-next:system-image:ubuntu-desktop-next
+ ubuntu-desktop-next::ubuntu-desktop-next
+ ubuntu-dvd::
+ ubuntu-gnome::
+ ubuntukylin::
+ ubuntu-mate::
+ ubuntu-mate-core::
+ ubuntu-mate-desktop::
+ ubuntu-mate-live::
+ ubuntu-netbook::
+ ubuntu-server::
+ ubuntu-server:ubuntu-rtm:
+ ubuntu-server:ubuntu-rtm/foo:
+ ubuntu-server:wubi:
+ ubuntu-touch-custom::ubuntu-touch-custom
+ ubuntu-touch::ubuntu-touch
+ xubuntu::"
+
+if [ -z "$SELECTED_TRIPLETS" ]; then
+    SELECTED_TRIPLETS="
+ ubuntu-base::
+ ubuntu-cpc::ubuntu-cpc
+"
+fi
+
+live_build_rootfs() {
+    PROJECT=${1%%:*}
+    local SUBPROJECT_TMP=${1%:*}
+    SUBPROJECT=${SUBPROJECT_TMP#*:}
+    TEMPLATE=${1##*:}
+    ARCH=$(dpkg --print-architecture)
+    echo "Building rootfs for project: '$PROJECT' subproject: '$SUBPROJECT' template: '$TEMPLATE' in $PWD"
+    cp -a /usr/share/livecd-rootfs/live-build/auto .
+    if [ -n "$TEMPLATE" ]; then
+        cp -a /usr/share/livecd-rootfs/live-build/$TEMPLATE .
+    fi
+    env PROJECT=$PROJECT \
+         SUBPROJECT=$SUBPROJECT \
+         SUITE=$SUITE \
+         ARCH=$ARCH \
+         lb config
+    mkdir chroot
+    # this part needs root rights, but right now the whole script ran as root by autopkgtest
+    env PROJECT=$PROJECT \
+         SUBPROJECT=$SUBPROJECT \
+         ARCH=$ARCH \
+         lb build
+    echo "Build results for project: '$PROJECT' subproject: '$SUBPROJECT' template: '$TEMPLATE' in $PWD"
+    du -sh *
+    echo ""
+}
+
+WORKDIR=$(mktemp -d)
+trap "RET=\$?; rm -rf $WORKDIR; exit \$RET" 0 INT QUIT ABRT PIPE TERM
+cd $WORKDIR
+
+for i in $SELECTED_TRIPLETS; do
+    mkdir $i
+    (cd $i && live_build_rootfs $i)
+    # clean up after build to avoid filling the disk, needs root rights
+    rm -rf $i
+done
+

debian/tests/minimized

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+env SELECTED_TRIPLETS=ubuntu-cpc:minimized:ubuntu-cpc debian/tests/default-bootstraps

live-build/auto/build

@@ -31,6 +31,112 @@ Expire-Date: 0
 
 	lb bootstrap "$@"
 
+	if [ "${SUBPROJECT:-}" = minimized ] \
+	   && ! Chroot chroot dpkg -l tzdata 2>&1 |grep -q ^ii; then
+		# workaround for tzdata purge not removing these files
+		rm -f chroot/etc/localtime chroot/etc/timezone
+	fi
+
+	if [ "${SUBPROJECT:-}" = minimized ]; then
+		# set up dpkg filters to skip installing docs on minimized system
+		mkdir -p chroot/etc/dpkg/dpkg.cfg.d
+		cat > chroot/etc/dpkg/dpkg.cfg.d/excludes <<EOF
+# Drop all man pages
+path-exclude=/usr/share/man/*
+
+# Drop all documentation ...
+path-exclude=/usr/share/doc/*
+
+# ... except copyright files ...
+path-include=/usr/share/doc/*/copyright
+
+# ... and Debian changelogs
+path-include=/usr/share/doc/*/changelog.Debian.*
+EOF
+
+		# Remove docs installed by bootstrap
+		Chroot chroot dpkg-query -f '${binary:Package}\n' -W | Chroot chroot xargs apt-get install --reinstall
+
+                # Add unminimizer script which restores default image behavior
+                mkdir -p chroot/usr/local/sbin
+                cat > chroot/usr/local/sbin/unminimize <<'EOF'
+#!/bin/sh
+
+set -e
+
+echo "This system has been minimized by removing packages and content that are"
+echo "not required on a system that users do not log into."
+echo ""
+echo "This script restores the content and packages that are found on a default"
+echo "Ubuntu server system."
+echo ""
+echo "Reinstallation of packages may fail due to changes to the system"
+echo "configuration, the presence of third-party packages, or for other"
+echo "reasons."
+read -p "Would you like to continue? [y/N]" REPLY
+echo    # (optional) move to a new line
+if [ "$REPLY" != "y" ] && [ "$REPLY" != "Y" ]
+then
+    exit 1
+fi
+
+if [ -f /etc/dpkg/dpkg.cfg.d/excludes ] || [ -f /etc/dpkg/dpkg.cfg.d/excludes.dpkg-tmp ]; then
+    echo "Re-enabling installation of all documentation in dpkg..."
+    if [ -f /etc/dpkg/dpkg.cfg.d/excludes ]; then
+        mv /etc/dpkg/dpkg.cfg.d/excludes /etc/dpkg/dpkg.cfg.d/excludes.dpkg-tmp
+    fi
+    echo "Updating package list and upgrading packages..."
+    apt-get update
+    # apt-get upgrade asks for confirmation before upgrading packages to let the user stop here
+    apt-get upgrade
+    echo "Restoring system documentation..."
+    echo "Reinstalling packages with files in /usr/share/man/ ..."
+    # Reinstallation takes place in two steps because a single dpkg --verified
+    # command generates very long parameter list for "xargs dpkg -S" and may go
+    # over ARG_MAX. Since many packages have man pages the second download
+    # handles a much smaller amount of packages.
+    dpkg -S /usr/share/man/ |sed 's|, |\n|g;s|: [^:]*$||' | DEBIAN_FRONTEND=noninteractive xargs apt-get install --reinstall -y
+    echo "Reinstalling packages with system documentation in /usr/share/doc/ .."
+    # This step processes the packages which still have missing documentation
+    dpkg --verify --verify-format rpm | awk '/..5......   \/usr\/share\/doc/ {print $2}' | sed 's|/[^/]*$||' | sort |uniq \
+         | xargs dpkg -S | sed 's|, |\n|g;s|: [^:]*$||' | uniq | DEBIAN_FRONTEND=noninteractive xargs apt-get install --reinstall -y
+    if dpkg --verify --verify-format rpm | awk '/..5......   \/usr\/share\/doc/ {exit 1}'; then
+        echo "Documentation has been restored successfully."
+        rm /etc/dpkg/dpkg.cfg.d/excludes.dpkg-tmp
+    else
+        echo "There are still files missing from /usr/share/doc/:"
+        dpkg --verify --verify-format rpm | awk '/..5......   \/usr\/share\/doc/ {print " " $2}'
+        echo "You may want to try running this script again or you can remove"
+        echo "/etc/dpkg/dpkg.cfg.d/excludes.dpkg-tmp and restore the files manually."
+    fi
+fi
+
+if ! dpkg-query --show --showformat='${db:Status-Status}\n' ubuntu-minimal 2> /dev/null | grep -q '^installed$'; then
+    echo "Installing ubuntu-minimal package to provide the familiar Ubuntu minimal system..."
+    DEBIAN_FRONTEND=noninteractive apt-get install -y ubuntu-minimal
+fi
+
+# unminimization succeeded, there is no need to mention it in motd
+rm -f /etc/update-motd.d/60-unminimize
+
+EOF
+                chmod +x chroot/usr/local/sbin/unminimize
+
+		# inform users about the unminimize script
+		cat > "chroot/etc/update-motd.d/60-unminimize" << EOF
+#!/bin/sh
+#
+# This file is not managed by a package.  If you no longer want to
+# see this message you can safely remove the file.
+echo "This system has been minimized by removing packages and content that are"
+echo "not required on a system that users do not log into."
+echo ""
+echo "To restore this content, you can run the 'unminimize' command."
+EOF
+
+		chmod +x chroot/etc/update-motd.d/60-unminimize
+        fi
+
 	Chroot chroot "dpkg-divert --quiet --add \
 		--divert /usr/sbin/update-initramfs.REAL --rename \
 		/usr/sbin/update-initramfs"
@@ -52,122 +158,74 @@ EOF
 
 	lb chroot "$@"
 
+	if [ "${SUBPROJECT:-}" = minimized ]; then
+		# and if initramfs-tools was configured before our kernel,
+		# /etc/kernel/postinst.d/initramfs-tools will have created
+		# an initramfs despite the generic dpkg-divert; so remove it
+		# here.
+		rm -f chroot/boot/initrd.img-*
+	fi
+
+	# remove crufty files that shouldn't be left in an image
+	rm -f chroot/var/cache/debconf/*-old chroot/var/lib/dpkg/*-old
+	Chroot chroot apt clean
+
 	if [ -f config/oem-config-preinstalled ]; then
 
 # This is cargo-culted almost verbatim (with some syntax changes for
 # preinstalled being slightly different in what it doesn't ask) from
 # debian-installer's apt-setup:
 
-codename=$LB_DISTRIBUTION
-file="chroot/etc/apt/sources.list"
-dists="main"
-alldists="main"
-if echo "$LB_PARENT_ARCHIVE_AREAS" | grep -q restricted; then
-	dists="$dists restricted"
-	alldists="$alldists restricted"
-fi
-if echo "$LB_PARENT_ARCHIVE_AREAS" | grep -q universe; then
-	UNIVERSE=true
-else
-	UNIVERSE=false
-fi
-if echo "$LB_PARENT_ARCHIVE_AREAS" | grep -q multiverse; then
-	MULTIVERSE=true
-else
-	MULTIVERSE=false
-fi
-
-cat > $file <<EOF
+cat > chroot/etc/apt/sources.list << EOF
 # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
 # newer versions of the distribution.
-
-deb $LB_PARENT_MIRROR_BINARY $codename $dists
-deb-src $LB_PARENT_MIRROR_BINARY $codename $dists
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION main restricted
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION main restricted
 
 ## Major bug fix updates produced after the final release of the
 ## distribution.
-deb $LB_PARENT_MIRROR_BINARY $codename-updates $dists
-deb-src $LB_PARENT_MIRROR_BINARY $codename-updates $dists
-EOF
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates main restricted
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates main restricted
 
-# Even if universe isn't enabled, we write example lines for it.
-echo >> $file
-if [ "$UNIVERSE" = true ]; then
-	alldists="$alldists universe"
-	COMMENT=
-else
-	cat >> $file <<EOF
-## Uncomment the following two lines to add software from the 'universe'
-## repository.
-EOF
-	COMMENT='# '
-fi
-cat >> $file <<EOF
 ## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
 ## team. Also, please note that software in universe WILL NOT receive any
 ## review or updates from the Ubuntu security team.
-${COMMENT}deb $LB_PARENT_MIRROR_BINARY $codename universe
-${COMMENT}deb-src $LB_PARENT_MIRROR_BINARY $codename universe
-${COMMENT}deb $LB_PARENT_MIRROR_BINARY $codename-updates universe
-${COMMENT}deb-src $LB_PARENT_MIRROR_BINARY $codename-updates universe
-EOF
-
-# Multiverse is different, don't write anything unless enabled.
-if [ "$MULTIVERSE" = true ]; then
-	alldists="$alldists multiverse"
-	cat >> $file <<EOF
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION universe
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION universe
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates universe
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates universe
 
 ## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
 ## team, and may not be under a free licence. Please satisfy yourself as to
 ## your rights to use the software. Also, please note that software in
 ## multiverse WILL NOT receive any review or updates from the Ubuntu
 ## security team.
-deb $LB_PARENT_MIRROR_BINARY $codename multiverse
-deb-src $LB_PARENT_MIRROR_BINARY $codename multiverse
-deb $LB_PARENT_MIRROR_BINARY $codename-updates multiverse
-deb-src $LB_PARENT_MIRROR_BINARY $codename-updates multiverse
-EOF
-fi
-
-cat >> $file <<EOF
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION multiverse
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION multiverse
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates multiverse
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates multiverse
 
 ## N.B. software from this repository may not have been tested as
 ## extensively as that contained in the main release, although it includes
 ## newer versions of some applications which may provide useful features.
 ## Also, please note that software in backports WILL NOT receive any review
 ## or updates from the Ubuntu security team.
-# deb $LB_PARENT_MIRROR_BINARY $codename-backports $alldists
-# deb-src $LB_PARENT_MIRROR_BINARY $codename-backports $alldists
-EOF
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-backports main restricted universe multiverse
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-backports main restricted universe multiverse
 
-cat >> $file <<EOF
+## Uncomment the following two lines to add software from Canonical's
+## 'partner' repository.
+## This software is not part of Ubuntu, but is offered by Canonical and the
+## respective vendors as a service to Ubuntu users.
+# deb http://archive.canonical.com/ubuntu $LB_DISTRIBUTION partner
+# deb-src http://archive.canonical.com/ubuntu $LB_DISTRIBUTION partner
 
-deb $LB_PARENT_MIRROR_BINARY $codename-security $dists
-deb-src $LB_PARENT_MIRROR_BINARY $codename-security $dists
-EOF
-
-# Security sources for Ubuntu universe; not used much, but e.g. unsupported
-# binary packages from a supported source package will end up here.
-if [ "$UNIVERSE" = true ]; then
-        COMMENT=
-else
-        COMMENT='# '
-fi
-cat >> $file <<EOF
-${COMMENT}deb $LB_PARENT_MIRROR_BINARY $codename-security universe
-${COMMENT}deb-src $LB_PARENT_MIRROR_BINARY $codename-security universe
-EOF
-
-# Security sources for Ubuntu multiverse, with the same caveats as for
-# universe.
-if [ "$MULTIVERSE" = true ]; then
-        COMMENT=
-else
-        COMMENT='# '
-fi
-cat >> $file <<EOF
-${COMMENT}deb $LB_PARENT_MIRROR_BINARY $codename-security multiverse
-${COMMENT}deb-src $LB_PARENT_MIRROR_BINARY $codename-security multiverse
+deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security main restricted
+# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security main restricted
+deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security universe
+# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security universe
+deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security multiverse
+# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security multiverse
 EOF
 
 	fi
@@ -238,7 +296,7 @@ deb file:/var/lib/preinstalled-pool/ $LB_DISTRIBUTION $LB_PARENT_ARCHIVE_AREAS
 			> chroot/etc/apt/sources.list
 		rm chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list.orig
 	fi
-	if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-pd" ]; then
+	if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-touch-custom" ]; then
 		if [ "$ARCH" = "armhf" ]; then
 			INFO_DESC="$(lsb_release -d -s)"
 			echo "$INFO_DESC - $ARCH ($BUILDSTAMP)" >chroot/etc/media-info
@@ -251,16 +309,6 @@ deb file:/var/lib/preinstalled-pool/ $LB_DISTRIBUTION $LB_PARENT_ARCHIVE_AREAS
 build_name: server
 serial: $BUILDSTAMP
 EOF
-		cat > chroot/etc/apt/sources.list << EOF
-deb ${LB_PARENT_MIRROR_BINARY} ${LB_DISTRIBUTION} main restricted universe multiverse
-deb ${LB_PARENT_MIRROR_BINARY} ${LB_DISTRIBUTION}-updates main restricted universe multiverse
-deb ${LB_PARENT_MIRROR_BINARY_SECURITY} ${LB_DISTRIBUTION}-security main restricted universe multiverse
-EOF
-		lb chroot_hosts install
-		lb chroot_resolv install
-		Chroot chroot "apt-get update"
-		lb chroot_resolv remove
-		lb chroot_hosts remove
 	fi
 
 	echo "===== Checking size of /usr/share/doc ====="
@@ -319,7 +367,7 @@ elif [ -e binary-tar.tar.gz ]; then
 	cp -a binary-tar.tar.gz "$PREFIX.rootfs.tar.gz"
 fi
 
-if [ "$PROJECT:$SUBPROJECT" = "ubuntu-core:system-image" ]; then
+if [ "$PROJECT:${SUBPROJECT:-}" = "ubuntu-core:system-image" ]; then
     if [ -e "binary/$INITFS/filesystem.dir" ]; then
         rootfs="binary/$INITFS/filesystem.dir"
 
@@ -345,7 +393,7 @@ EOF
     fi
 fi
 
-if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-pd" ]; then
+if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-touch-custom" ]; then
 	(cd "binary/$INITFS/custom.dir/" && tar -c *) | \
 		gzip -9 --rsyncable > "$PREFIX.custom.tar.gz"
 	chmod 644 "$PREFIX.custom.tar.gz"
@@ -367,7 +415,7 @@ fi
 # ubuntu-core and ubuntu-desktop-next splits kernel stuff into a "device" tarball so
 # at this point we reset it to "none" as all the work to extract it was done already
 # in a binary hook
-case $PROJECT:$SUBPROJECT in
+case $PROJECT:${SUBPROJECT:-} in
         ubuntu-core:system-image|ubuntu-desktop-next:system-image)
 
     # create device tarball (for snappy only atm)
@@ -602,6 +650,7 @@ for FLAVOUR in $LB_LINUX_FLAVOURS; do
 	if [ -z "$LB_LINUX_FLAVOURS" ] || [ "$LB_LINUX_FLAVOURS" = "none" ]; then
 		continue
 	fi
+	FLAVOUR=${FLAVOUR%%-hwe-*}
 	if [ "$FLAVOUR" = "virtual" ]; then
 		# The virtual kernel is named generic in /boot
 		FLAVOUR="generic"
@@ -635,17 +684,18 @@ done
 NUMFLAVOURS="$(set -- $LB_LINUX_FLAVOURS; echo $#)"
 if [ "$NUMFLAVOURS" = 1 ] && [ "$LB_LINUX_FLAVOURS" != "none" ]; then
 	# only one kernel flavour
+	FLAVOUR=${LB_LINUX_FLAVOURS%%-hwe-*}
 	if [ -e "binary/$INITFS/vmlinuz" ]; then
 		ln "binary/$INITFS/vmlinuz" "$PREFIX.kernel"
 		chmod 644 "$PREFIX.kernel"
 	else
-		ln -sf "$PREFIX.kernel-$LB_LINUX_FLAVOURS" "$PREFIX.kernel"
+		ln -sf "$PREFIX.kernel-$FLAVOUR" "$PREFIX.kernel"
 	fi
 	if [ -e "binary/$INITFS/initrd.lz" ]; then
 		ln "binary/$INITFS/initrd.lz" "$PREFIX.initrd"
 		chmod 644 "$PREFIX.initrd"
 	else
-		ln -sf "$PREFIX.initrd-$LB_LINUX_FLAVOURS" "$PREFIX.initrd"
+		ln -sf "$PREFIX.initrd-$FLAVOUR" "$PREFIX.initrd"
 	fi
 fi
 
@@ -689,7 +739,7 @@ if [ "$SUBARCH" = "ac100" ] || [ "$SUBARCH" = "nexus7" ]; then
 
 fi
 
-if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-pd" ]; then
+if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-touch-custom" ]; then
 	sourceslist="chroot/etc/apt/sources.list"
 
 	lb chroot_proc install "$@"

live-build/auto/config

@@ -58,9 +58,15 @@ add_task ()
 	# failure.
 
 	for task; do
+	    if [ -z "$HWE_BUILD" ]; then
 		# We need a ridiculous number of backslashes to protect
 		# parentheses from eval.
 		echo "!chroot chroot apt-cache dumpavail | grep-dctrl -nsPackage \\\\\\( -XFArchitecture $ARCH -o -XFArchitecture all \\\\\\) -a -wFTask $task" >> "config/package-lists/livecd-rootfs.list.chroot_$pass"
+	    else
+		# If HWE_BUILD is set, we strip out drivers not available
+		# in the HWE stack, then sed the rest with their HWE suffixes
+		echo "!chroot chroot apt-cache dumpavail | grep-dctrl -nsPackage \\\\\\( -XFArchitecture $ARCH -o -XFArchitecture all \\\\\\) -a -wFTask $task | grep -v xserver-xorg-input-vmmouse | grep -v libgles1-mesa | sed -e 's/xserver-xorg.*/&-hwe-16.04/'" >> "config/package-lists/livecd-rootfs.list.chroot_$pass"
+	    fi
 	done
 }
 
@@ -101,17 +107,19 @@ add_binary_hook ()
 	BINARY_HOOKS="${BINARY_HOOKS:+$BINARY_HOOKS }$1"
 }
 
-case $PROJECT in
+if [ -z "${IMAGEFORMAT:-}" ]; then
+	case $PROJECT in
 		ubuntu-cpc)
 			IMAGEFORMAT=ext4
 			;;
-esac
+	esac
+fi
 
 case $IMAGEFORMAT in
 	ext2|ext3|ext4)
 		OPTS="${OPTS:+$OPTS }--initramfs none --chroot-filesystem $IMAGEFORMAT"
 		PREINSTALLED=true
-		case $SUBPROJECT in
+		case ${SUBPROJECT:-} in
 			wubi)
 				add_package install lupin-support
 				COMPONENTS='main restricted universe multiverse'
@@ -133,9 +141,12 @@ case $IMAGEFORMAT in
 		PREINSTALLED=true
 		;;
 
+	none)
+		OPTS="${OPTS:+$OPTS }--chroot-filesystem $IMAGEFORMAT"
+		;;
 	*)
 		case $PROJECT in
-			ubuntu-server|ubuntu-touch|ubuntu-pd)
+			ubuntu-server|ubuntu-touch|ubuntu-touch-custom)
 				;;
 			*)
 				add_package live lupin-casper
@@ -168,7 +179,7 @@ if [ "$PREINSTALLED" = "true" ] && [ "$SUBPROJECT" != "wubi" ]; then
 		ubuntu-server)
 			add_package live oem-config-debconf ubiquity-frontend-debconf
 			;;
-		ubuntu-core|base|ubuntu-touch|ubuntu-pd|ubuntu-cpc|ubuntu-desktop-next)
+		ubuntu-core|ubuntu-base|base|ubuntu-touch|ubuntu-touch-custom|ubuntu-cpc|ubuntu-desktop-next)
 			;;
 		*)
 			add_package live oem-config-gtk ubiquity-frontend-gtk
@@ -186,16 +197,24 @@ case $BINARYFORMAT in
 		;;
 esac
 
-SIGNED_KERNEL_PACKAGE="linux-signed-generic"
-UNITY_HWE_HINTS="unity-settings-daemon notify-osd libqt4-sql-sqlite unity gnome-terminal"
+HWE_X_PACKAGES="xserver-xorg-hwe-16.04 xserver-xorg-video-all-hwe-16.04 xserver-xorg-input-all-hwe-16.04"
+HWE_KERNEL_FLAVOUR="generic-hwe-16.04"
+HWE_SIGNED_KERNEL_PACKAGE="linux-signed-$HWE_KERNEL_FLAVOUR"
+
+if [ "${SUBPROJECT:-}" = minimized ]; then
+	OPTS="${OPTS:+$OPTS }--bootstrap-flavour=minimal --linux-packages=linux-image"
+fi
 
 case $PROJECT in
 	ubuntu|ubuntu-dvd)
-		add_package install ubuntu-minimal ubuntu-standard
-		add_package install ubuntu-desktop $UNITY_HWE_HINTS
+		HWE_BUILD="yes"
+		add_task install minimal standard ubuntu-desktop
+		add_package install $HWE_X_PACKAGES
 		LIVE_TASK='ubuntu-live'
+		LIVE_TASK_FILTER='--not -Pe ^linux-\(headers\|image\|signed\)'
+		KERNEL_FLAVOURS="$HWE_KERNEL_FLAVOUR"
 		case $ARCH in
-			amd64)	add_package live $SIGNED_KERNEL_PACKAGE ;;
+			amd64)	add_package live $HWE_SIGNED_KERNEL_PACKAGE ;;
 		esac
 		;;
 
@@ -213,17 +232,21 @@ case $PROJECT in
 		#      CDIMAGE_PREINSTALLED is not passed from build.py
 		#      and PREINSTALLED means something different. So
 		#      we use SUBPROJECT to pass on the information
-		if [ "$SUBPROJECT" = "system-image" ]; then
+		if [ "${SUBPROJECT:-}" = "system-image" ]; then
 			OPTS="${OPTS:+$OPTS }--linux-packages=linux-image"
 		fi
 		;;
 
 	kubuntu|kubuntu-dvd)
+		HWE_BUILD="yes"
 		add_task install minimal standard
 		add_task install kubuntu-desktop
+		add_package install $HWE_X_PACKAGES
 		LIVE_TASK='kubuntu-live'
+		LIVE_TASK_FILTER='--not -Pe ^linux-\(headers\|image\|signed\)'
+		KERNEL_FLAVOURS="$HWE_KERNEL_FLAVOUR"
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live $HWE_SIGNED_KERNEL_PACKAGE ;;
 		esac
 		COMPONENTS='main restricted universe'
 		add_chroot_hook remove-gnome-icon-cache
@@ -258,16 +281,17 @@ case $PROJECT in
 		;;
 
 	xubuntu)
+		HWE_BUILD="yes"
 		add_task install minimal standard xubuntu-desktop
+		add_package install $HWE_X_PACKAGES
 		add_package install xterm
 		LIVE_TASK='xubuntu-live'
+		LIVE_TASK_FILTER='--not -Pe ^linux-\(headers\|image\|signed\)'
+		KERNEL_FLAVOURS="$HWE_KERNEL_FLAVOUR"
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live $HWE_SIGNED_KERNEL_PACKAGE ;;
 		esac
 		COMPONENTS='main restricted universe multiverse'
-		case $ARCH in
-			amd64|i386)	KERNEL_FLAVOURS=generic ;;
-		esac
 		;;
 
 	ubuntu-netbook)
@@ -276,44 +300,62 @@ case $PROJECT in
 		;;
 
 	mythbuntu)
+		HWE_BUILD="yes"
 		add_task install minimal standard mythbuntu-desktop
+		add_package install $HWE_X_PACKAGES
 		LIVE_TASK='mythbuntu-live'
+		LIVE_TASK_FILTER='--not -Pe ^linux-\(headers\|image\|signed\)'
+		KERNEL_FLAVOURS="$HWE_KERNEL_FLAVOUR"
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live $HWE_SIGNED_KERNEL_PACKAGE ;;
 		esac
 		COMPONENTS='main restricted universe multiverse'
 		;;
 
 	lubuntu)
+		HWE_BUILD="yes"
 		add_task install minimal standard lubuntu-desktop
-		LIVE_TASK='lubuntu-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64|i386)	add_package install thermald ;;
+		esac
+		add_package install $HWE_X_PACKAGES
+		LIVE_TASK='lubuntu-live'
+		LIVE_TASK_FILTER='--not -Pe ^linux-\(headers\|image\|signed\)'
+		KERNEL_FLAVOURS="$HWE_KERNEL_FLAVOUR"
+		case $ARCH in
+			amd64)	add_package live $HWE_SIGNED_KERNEL_PACKAGE ;;
 		esac
 		COMPONENTS='main restricted universe multiverse'
-		case $ARCH in
-			amd64|i386)	KERNEL_FLAVOURS=generic ;;
-		esac
-
 		# The Lubuntu STRUCTURE file has "feature
 		# no-follow-recommends".  Mirror this.
 		export APT_OPTIONS="--yes --no-install-recommends"
 		;;
 
 	ubuntu-gnome)
+		HWE_BUILD="yes"
 		add_task install minimal standard ubuntu-gnome-desktop
+		add_package install $HWE_X_PACKAGES
 		LIVE_TASK='ubuntu-gnome-live'
+		LIVE_TASK_FILTER='--not -Pe ^linux-\(headers\|image\|signed\)'
+		KERNEL_FLAVOURS="$HWE_KERNEL_FLAVOUR"
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live $HWE_SIGNED_KERNEL_PACKAGE ;;
 		esac
 		COMPONENTS='main restricted universe'
 		;;
 
 	ubuntu-mate)
+		HWE_BUILD="yes"
 		add_task install minimal standard ubuntu-mate-core ubuntu-mate-desktop
-		LIVE_TASK='ubuntu-mate-live'
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64|i386)	add_package install thermald ;;
+		esac
+		add_package install $HWE_X_PACKAGES
+		LIVE_TASK='ubuntu-mate-live'
+		LIVE_TASK_FILTER='--not -Pe ^linux-\(headers\|image\|signed\)'
+		KERNEL_FLAVOURS="$HWE_KERNEL_FLAVOUR"
+		case $ARCH in
+			amd64)	add_package live $HWE_SIGNED_KERNEL_PACKAGE ;;
 		esac
 		COMPONENTS='main restricted universe multiverse'
 		# The Ubuntu MATE STRUCTURE file has "feature
@@ -322,19 +364,25 @@ case $PROJECT in
 		;;
 
 	ubuntustudio-dvd)
+		HWE_BUILD="yes"
 		add_task install minimal standard ubuntustudio-desktop ubuntustudio-audio ubuntustudio-fonts ubuntustudio-graphics ubuntustudio-video ubuntustudio-publishing ubuntustudio-photography
+		add_package install $HWE_X_PACKAGES linux-generic-hwe-16.04-
 		COMPONENTS='main restricted universe multiverse'
 		case $ARCH in
-			amd64|i386)	KERNEL_FLAVOURS=lowlatency ;;
+			amd64|i386)	KERNEL_FLAVOURS=lowlatency-hwe-16.04 ;;
 		esac
 		;;
 
 	ubuntukylin)
+		HWE_BUILD="yes"
 		add_task install minimal standard ubuntukylin-desktop
+		add_package install $HWE_X_PACKAGES
 		add_package install ubuntukylin-default-settings
 		LIVE_TASK='ubuntukylin-live'
+		LIVE_TASK_FILTER='--not -Pe ^linux-\(headers\|image\|signed\)'
+		KERNEL_FLAVOURS="$HWE_KERNEL_FLAVOUR"
 		case $ARCH in
-			amd64)	add_package live linux-signed-generic ;;
+			amd64)	add_package live $HWE_SIGNED_KERNEL_PACKAGE ;;
 		esac
 		COMPONENTS='main restricted universe'
 		;;
@@ -413,13 +461,18 @@ case $PROJECT in
 		OPTS="${OPTS:+$OPTS }--bootstrap-flavour=minimal"
 		;;
 
-	ubuntu-touch|ubuntu-pd)
-		if [ "$PROJECT" = "ubuntu-touch" ]; then
-			meta_package=ubuntu-touch
-		else
-			meta_package=ubuntu-pocket-desktop
-		fi
-		add_package install ubuntu-minimal $meta_package systemd-sysv- packagekit ubuntu-system-settings-online-accounts
+	ubuntu-base)
+		OPTS="${OPTS:+$OPTS }--bootstrap-flavour=minimal"
+		;;
+
+	ubuntu-touch|ubuntu-touch-custom)
+		HINTS="packagekit ubuntu-system-settings-online-accounts"
+		case $ARCH in
+			amd64|i386)
+				HINTS="$HINTS qml-module-ubuntu-components-gles unity8"
+				;;
+		esac
+		add_package install ubuntu-minimal ubuntu-touch $HINTS
 
 		COMPONENTS='main restricted universe'
 		BOOTAPPEND_LIVE='hostname=ubuntu-phablet username=ubuntu'
@@ -433,7 +486,7 @@ case $PROJECT in
 		# SUBPROJECT, but it's a handy thing that launchpad-buildd
 		# already passes through to us that we weren't otherwise
 		# using here.
-		case $SUBPROJECT in
+		case ${SUBPROJECT:-} in
 		    ubuntu-rtm/dogfood)
 			MIRROR=http://derived-archive.dogfood.content.paddev.net/ubuntu-rtm/
 			OPTS="${OPTS:+$OPTS }--apt-secure false"
@@ -451,29 +504,47 @@ case $PROJECT in
 		;;
 
 	ubuntu-cpc)
+		if [ "${SUBPROJECT:-}" = minimized ]; then
+			add_task install cloud-image
+			add_package install sudo
+			# linux-kvm currently only exists in xenial, not in
+			# non-LTS suites.  Fall back to virtual flavor, which
+			# may or may not boot initramfsless but enables us to
+			# test building and possibly build derivative images
+			# using other kernel flavors.
+			# If you enable an extra ppa, it is assumed that
+			# linux-kvm is available since you control the
+			# archive and can provide this metapackage as
+			# necessary.
+			if [ "$ARCH" != "amd64" ] || ([ -z "$EXTRA_PPAS" ] && [ "$SUITE" != xenial ]); then
+				KERNEL_FLAVOURS=virtual
+			else
+				KERNEL_FLAVOURS=kvm
+			fi
+		else
 			add_task install minimal standard cloud-image
 			add_package install ubuntu-minimal
+			KERNEL_FLAVOURS=virtual
+		        case $ARCH in
+				armhf|arm64|ppc64el|powerpc)
+					add_task install server
+					;;
+			esac
+		fi
 
 		BINARY_REMOVE_LINUX=false
 		OPTS="${OPTS:+$OPTS }--initramfs=none"
-		KERNEL_FLAVOURS=virtual
 		case $ARCH in
 			armhf)
 				KERNEL_FLAVOURS=generic-lpae
 				add_package install flash-kernel
-				add_task install server
 				;;
 			arm64)
 				KERNEL_FLAVOURS=generic
 				add_package install flash-kernel
-				add_task install server
-				;;
-			ppc64el)
-				add_task install server
 				;;
 			powerpc)
 				KERNEL_FLAVOURS=powerpc64-smp
-				add_task install server
 				;;
 		esac
 		OPTS="${OPTS:+$OPTS }--system=normal"
@@ -557,7 +628,7 @@ case $ARCH in
 esac
 
 case $PROJECT in
-	ubuntu-server|ubuntu-core|ubuntu-touch|ubuntu-pd)
+	ubuntu-server|ubuntu-core|ubuntu-base|ubuntu-touch|ubuntu-touch-custom)
 		case $SUBPROJECT in
 			system-image)
 				# keep the kernel for the system-image build
@@ -586,7 +657,7 @@ case $PROJECT in
 		;;
 esac
 
-case $SUBPROJECT in
+case ${SUBPROJECT:-} in
 	wubi)
 		add_binary_hook build-wubildr
 		;;
@@ -607,13 +678,16 @@ lb config noauto \
 	--initsystem none \
 	--bootloader "$BOOTLOADER" \
 	--initramfs-compression lzma \
+	--cache false \
 	${BOOTAPPEND_LIVE:+--bootappend-live "$BOOTAPPEND_LIVE"} \
 	$OPTS \
 	"$@"
 
 echo "LB_CHROOT_HOOKS=\"$CHROOT_HOOKS\"" >> config/chroot
+echo "SUBPROJECT=\"${SUBPROJECT:-}\"" >> config/chroot
 echo "LB_BINARY_HOOKS=\"$BINARY_HOOKS\"" >> config/binary
 echo "BUILDSTAMP=\"$NOW\"" >> config/binary
+echo "SUBPROJECT=\"${SUBPROJECT:-}\"" >> config/binary
 
 case $ARCH+$SUBARCH in
 	armhf+raspi2)
@@ -637,6 +711,7 @@ cat > /boot/firmware/config.txt << EOM
 # Some settings may impact device functionality. See link above for details
 
 kernel=uboot.bin
+device_tree_address=0x02000000
 
 # enable i2c
 dtparam=i2c_arm=on
@@ -727,7 +802,7 @@ EOF
 	      config/archives/proposed.list.binary
 fi
 
-case $PROJECT:$SUBPROJECT in
+case $PROJECT:${SUBPROJECT:-} in
 	*-dvd:*)
 		. config/bootstrap
 
@@ -743,9 +818,12 @@ EOF
 		fi
 		;;
 
-	ubuntu-touch:*|ubuntu-pd:*|ubuntu-core:system-image|ubuntu-desktop-next:system-image|ubuntu-cpc:*)
+	ubuntu-touch:*|ubuntu-touch-custom:*|ubuntu-core:system-image|ubuntu-desktop-next:system-image|ubuntu-cpc:*)
 		cp -af /usr/share/livecd-rootfs/live-build/${PROJECT}/* \
 			config/
+		if [ "$IMAGEFORMAT" = none ]; then
+			rm -f config/hooks/*.binary*
+		fi
 		;;
 esac
 
@@ -813,7 +891,7 @@ EOF
 	fi
 fi
 
-case $SUBPROJECT in
+case ${SUBPROJECT:-} in
     ubuntu-rtm|ubuntu-rtm/*)
 	# debootstrap doesn't know about ubuntu-rtm series directly.  Rather
 	# than having to teach it, we employ a few hacks to make it use the

live-build/ubuntu-cpc/functions

@@ -11,9 +11,22 @@ backing_img=
 apt-get -qqy install dosfstools gdisk
 
 clean_loops() {
+    local kpartx_ret
+    local kpartx_stdout
 
     if [ -n "${backing_img}" ]; then
-        kpartx -v -d "${backing_img}"
+        # sync before removing loop to avoid "Device or resource busy" errors
+        sync
+        kpartx_ret=""
+        kpartx_stdout=$(kpartx -v -d "${backing_img}") || kpartx_ret=$?
+        echo "$kpartx_stdout"
+        if [ -n "$kpartx_ret" ]; then
+            if echo "$kpartx_stdout" | grep -q "loop deleted"; then
+                echo "Suppressing kpartx returning error (#860894)"
+            else
+                exit $kpartx_ret
+            fi
+        fi
         unset backing_img
     fi
 
@@ -69,16 +82,32 @@ mount_image() {
 setup_mountpoint() {
     local mountpoint="$1"
 
-    mount --bind /dev "$mountpoint/dev"
-    mount devpts-live -t proc "$mountpoint/dev/pts"
+    mount --rbind --make-rslave /dev "$mountpoint/dev"
     mount proc-live -t proc "$mountpoint/proc"
     mount sysfs-live -t sysfs "$mountpoint/sys"
     mount -t tmpfs none "$mountpoint/tmp"
+    mount -t tmpfs none "$mountpoint/var/lib/apt"
+    mount -t tmpfs none "$mountpoint/var/cache/apt"
     mv "$mountpoint/etc/resolv.conf" resolv.conf.tmp
     cp /etc/resolv.conf "$mountpoint/etc/resolv.conf"
+    chroot "$mountpoint" apt-get update
 
 }
 
+teardown_mountpoint() {
+    # Reverse the operations from setup_mountpoint
+    local mountpoint="$1"
+
+    # ensure we have exactly one trailing slash, and escape all slashes for awk
+    mountpoint_match=$(echo "$mountpoint" | sed -e's,/$,,; s,/,\\/,g;')'\/'
+    # sort -r ensures that deeper mountpoints are unmounted first
+    for submount in $(awk </proc/self/mounts "\$2 ~ /$mountpoint_match/ \
+                      { print \$2 }" | LC_ALL=C sort -r); do
+        umount $submount
+    done
+    mv resolv.conf.tmp "$mountpoint/etc/resolv.conf"
+}
+
 mount_partition() {
     partition="$1"
     mountpoint="$2"
@@ -111,21 +140,11 @@ EOF
 
 }
 
-umount_settle() {
-    # Unmount device, and let it settle
-    umount $1
-    udevadm settle
-    sleep 3
-}
-
 umount_partition() {
     local mountpoint=${1}
-    mv resolv.conf.tmp "$mountpoint/etc/resolv.conf"
-    for submnt in proc sys dev/pts dev tmp;
-    do
-        umount_settle $mountpoint/$submnt
-    done
-    umount_settle $mountpoint
+    teardown_mountpoint $mountpoint
+    umount -R $mountpoint
+    udevadm settle
 
     if [ -n "${rootfs_dev_mapper}" -a -b "${rootfs_dev_mapper}" ]; then
         # buildd's don't have /etc/mtab symlinked
@@ -249,4 +268,59 @@ convert_to_qcow2() {
     qemu-img info "$destination"
 }
 
+replace_grub_root_with_label() {
+    # When update-grub is run, it will detect the disks in the build system.
+    # Instead, we want grub to use the cloudimg-rootfs labelled disk
+    CHROOT_ROOT="$1"
 
+    # If boot by partuuid has been requested, don't override.
+    if [ -f $CHROOT_ROOT/etc/default/grub.d/40-force-partuuid.cfg ] && \
+           grep -q ^GRUB_FORCE_PARTUUID= $CHROOT_ROOT/etc/default/grub.d/40-force-partuuid.cfg
+    then
+        return 0
+    fi
+    sed -i -e "s,root=[^ ]*,root=LABEL=cloudimg-rootfs," \
+        "$CHROOT_ROOT/boot/grub/grub.cfg"
+}
+
+
+# When running update-grub in a chroot on a build host, we don't want it to
+# probe for disks or probe for other installed OSes.  Extract common
+# diversion wrappers, so this isn't reinvented differently for each image.
+divert_grub() {
+	CHROOT_ROOT="$1"
+
+	chroot "$CHROOT_ROOT" dpkg-divert --local \
+		--rename /usr/sbin/grub-probe
+	chroot "$CHROOT_ROOT" touch /usr/sbin/grub-probe
+	chroot "$CHROOT_ROOT" chmod +x /usr/sbin/grub-probe
+
+	chroot "$CHROOT_ROOT" dpkg-divert --local \
+		--divert /etc/grub.d/30_os-prober.dpkg-divert \
+		--rename /etc/grub.d/30_os-prober
+
+	# Divert systemd-detect-virt; /etc/kernel/postinst.d/zz-update-grub
+	# no-ops if we are in a container, and the launchpad farm runs builds
+	# in lxd.  We therefore pretend that we're never in a container (by
+	# exiting 1).
+	chroot "$CHROOT_ROOT" dpkg-divert --local \
+		--rename /usr/bin/systemd-detect-virt
+	echo "exit 1" > "$CHROOT_ROOT"/usr/bin/systemd-detect-virt
+	chmod +x "$CHROOT_ROOT"/usr/bin/systemd-detect-virt
+}
+
+undivert_grub() {
+	CHROOT_ROOT="$1"
+
+	chroot "$CHROOT_ROOT" rm /usr/sbin/grub-probe
+	chroot "$CHROOT_ROOT" dpkg-divert --remove --local \
+		--rename /usr/sbin/grub-probe
+
+	chroot "$CHROOT_ROOT" dpkg-divert --remove --local \
+		--divert /etc/grub.d/30_os-prober.dpkg-divert \
+		--rename /etc/grub.d/30_os-prober
+
+	rm "$CHROOT_ROOT"/usr/bin/systemd-detect-virt
+	chroot "$CHROOT_ROOT" dpkg-divert --remove --local \
+		--rename /usr/bin/systemd-detect-virt
+}

live-build/ubuntu-cpc/hooks/030-root-tarball.binary

@@ -5,7 +5,7 @@ if [ -n "$SUBARCH" ]; then
 	exit 0
 fi
 
-. /build/config/functions
+. config/functions
 
 mkdir binary/boot/filesystem.dir
 cp -a chroot/* binary/boot/filesystem.dir
@@ -24,9 +24,4 @@ chroot binary/boot/filesystem.dir mkdir /lib/modules
 chroot binary/boot/filesystem.dir rm /usr/sbin/grub-probe
 chroot binary/boot/filesystem.dir dpkg-divert --remove --local --rename /usr/sbin/grub-probe
 
-mv resolv.conf.tmp "binary/boot/filesystem.dir/etc/resolv.conf"
-umount "binary/boot/filesystem.dir/proc"
-umount "binary/boot/filesystem.dir/sys"
-umount "binary/boot/filesystem.dir/dev/pts"
-umount "binary/boot/filesystem.dir/dev"
-umount "binary/boot/filesystem.dir/tmp"
+teardown_mountpoint binary/boot/filesystem.dir

live-build/ubuntu-cpc/hooks/032-disk-image.binary

@@ -1,6 +1,8 @@
 #!/bin/bash -ex
 
-. /build/config/functions
+. config/functions
+
+. config/binary
 
 BOOTPART_START=
 BOOTPART_END=
@@ -64,6 +66,8 @@ create_empty_partition "${disk_image}" "$ROOTPART" "$ROOTPART_START" -1 ext2 "$R
 
 mount_image "${disk_image}" "$ROOTPART"
 
+partuuid=$(blkid -s PARTUUID -o value "$rootfs_dev_mapper")
+
 # Copy the chroot in to the disk
 make_ext4_partition "${rootfs_dev_mapper}"
 mkdir mountpoint
@@ -88,7 +92,7 @@ case $ARCH:$SUBARCH in
 		# not the best place for this, but neither flash-kernel nor
 		# u-boot have provisions for installing u-boot via maintainer
 		# script
-		/build/config/hooks/raspi2/mkknlimg --dtok \
+		config/hooks/raspi2/mkknlimg --dtok \
 			mountpoint/usr/lib/u-boot/rpi_2/u-boot.bin \
 			mountpoint/boot/firmware/uboot.bin
 		;;
@@ -111,6 +115,15 @@ if [ "${should_install_grub}" -eq 1 ]; then
         ${loop_device}
 
     rm mountpoint/tmp/device.map
+
+	if [ "${SUBPROJECT:-}" = minimized ] && [ -n "$partuuid" ]; then
+		echo "partuuid found for root device; forcing it in Grub"
+		mkdir -p mountpoint/etc/default/grub.d
+		echo "GRUB_FORCE_PARTUUID=$partuuid" >> mountpoint/etc/default/grub.d/40-force-partuuid.cfg
+		divert_grub mountpoint
+		chroot mountpoint update-grub
+		undivert_grub mountpoint
+	fi
 fi
 
 if [ "$ARCH" = "s390x" ]; then

live-build/ubuntu-cpc/hooks/033-disk-image-uefi.binary

@@ -9,7 +9,7 @@ case $ARCH in
         ;;
 esac
 
-. /build/config/functions
+. config/functions
 
 apt-get -qqy install dosfstools gdisk
 
@@ -61,18 +61,30 @@ install_grub() {
     efi_boot_dir="/boot/efi/EFI/BOOT"
     chroot mountpoint mkdir -p "${efi_boot_dir}"
 
+    if [ "${SUBPROJECT:-}" = minimized ] && [ -n "$partuuid" ]; then
+        # FIXME: code duplicated between 032-disk-image.binary
+        # and 033-disk-image-uefi.binary.  We want to fix this to not
+        # have initramfs-tools installed at all on these images.
+        echo "partuuid found for root device; omitting initrd"
+        echo "GRUB_FORCE_PARTUUID=$partuuid" >> mountpoint/etc/default/grub.d/40-force-partuuid.cfg
+    fi
+
+    chroot mountpoint apt-get -y update
+
+    # The modules below only make sense on non-Secure Boot UEFI systems.
+    # Otherwise, with Secure Boot enabled GRUB will refuse to load them.
+    # Any modules already in debian/build-efi-images do not need to be listed.
+    # Furthermore, other modules such as terminal, video_* and efi_* are all
+    # already available.
     case $ARCH in
         arm64)
             chroot mountpoint apt-get -qqy install --no-install-recommends grub-efi-arm64 grub-efi-arm64-bin
-            grub_modules="part_gpt fat gzio ext2 normal chain boot configfile linux search_fs_uuid search_label terminal serial video video_fb efi_gop"
+            grub_modules="serial"
             efi_target=arm64-efi
             ;;
         amd64)
             chroot mountpoint apt-get install -qqy grub-efi-amd64-signed grub-efi-amd64 shim-signed
-            grub_modules="part_gpt fat ext2 normal chain boot configfile linux multiboot search_fs_uuid search_label terminal serial video video_fb video_bochs usb usb_keyboard efi_gop efi_uga"
-            chroot mountpoint cp /usr/lib/shim/shim.efi.signed "${efi_boot_dir}/shimx64.efi"
-            chroot mountpoint cp /usr/lib/shim/MokManager.efi.signed "${efi_boot_dir}/MokManager.efi"
-            chroot mountpoint cp /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed "${efi_boot_dir}/grubx64.efi"
+            grub_modules="multiboot serial usb usb_keyboard"
             efi_target=x86_64-efi
             ;;
     esac
@@ -80,8 +92,9 @@ install_grub() {
     cat << EOF >> mountpoint/etc/default/grub.d/50-cloudimg-settings.cfg
 ${CLOUD_IMG_STR}
 # For Cloud Image compatability
-GRUB_PRELOAD_MODULES="${grub_modules}"
+GRUB_PRELOAD_MODULES="${GRUB_PRELOAD_MODULES:-$grub_modules}"
 EOF
+
     chroot mountpoint grub-install "${loop_device}" \
         --boot-directory=/boot \
         --efi-directory=/boot/efi \
@@ -106,19 +119,16 @@ EOF
         chroot mountpoint grub-install --target=i386-pc "${loop_device}"
     fi
 
-    chroot mountpoint dpkg-divert --local --rename /etc/grub.d/30_os-prober
+    divert_grub mountpoint
     chroot mountpoint update-grub
-    sed -i "s,root=.* ,root=LABEL=cloudimg-rootfs ,g" mountpoint/boot/grub/grub.cfg
-    chroot mountpoint dpkg-divert --remove --local --rename /etc/grub.d/30_os-prober
+    replace_grub_root_with_label mountpoint
+    undivert_grub mountpoint
 
     chroot mountpoint apt-get -y clean
-    chroot mountpoint apt-get -y update
 
     rm mountpoint/tmp/device.map
-    sync
     umount mountpoint/boot/efi
-    sleep 5
-    udevadm settle
+    apt-get install -qqy udev
     mount
     umount_partition mountpoint
     rmdir mountpoint
@@ -130,6 +140,8 @@ create_empty_disk_image "${disk_image}"
 create_partitions "${disk_image}"
 mount_image "${disk_image}" 1
 
+partuuid=$(blkid -s PARTUUID -o value "$rootfs_dev_mapper")
+
 # Copy the chroot in to the disk
 make_ext4_partition "${rootfs_dev_mapper}"
 mkdir mountpoint

live-build/ubuntu-cpc/hooks/034-disk-image-ppc64el.binary

@@ -7,7 +7,7 @@ case $ARCH in
         ;;
 esac
 
-. /build/config/functions
+. config/functions
 
 create_partitions() {
     disk_image="$1"
@@ -26,6 +26,7 @@ install_grub() {
     mkdir mountpoint
     mount_partition "${rootfs_dev_mapper}" mountpoint
 
+    chroot mountpoint apt-get -qqy update
     chroot mountpoint apt-get -qqy install grub2
     chroot mountpoint apt-get -qqy remove --purge grub-legacy-ec2
 
@@ -49,10 +50,10 @@ EOF
         --boot-directory=/boot \
         --target=powerpc-ieee1275
 
-    chroot mountpoint dpkg-divert --local --rename /etc/grub.d/30_os-prober
+    divert_grub mountpoint
     chroot mountpoint update-grub
-    sed -i "s,root=.* ,root=LABEL=cloudimg-rootfs ,g" mountpoint/boot/grub/grub.cfg
-    chroot mountpoint dpkg-divert --remove --local --rename /etc/grub.d/30_os-prober
+    replace_grub_root_with_label mountpoint
+    undivert_grub mountpoint
 
     umount_partition mountpoint
     rmdir mountpoint

live-build/ubuntu-cpc/hooks/040-qcow2-image.binary

@@ -12,7 +12,7 @@ esac
 
 apt-get install -qqy qemu-utils
 
-. /build/config/functions
+. config/functions
 
 if [ -f binary/boot/disk.ext4 ]; then
     convert_to_qcow2 binary/boot/disk.ext4 livecd.ubuntu-cpc.disk1.img

live-build/ubuntu-cpc/hooks/040-vmdk-image.binary

@@ -11,7 +11,7 @@ case $ARCH in
                      exit 0;;
 esac
 
-. /build/config/functions
+. config/functions
 
 if [ -e binary/boot/disk.ext4 ]; then
     create_vmdk binary/boot/disk.ext4 livecd.ubuntu-cpc.disk1.vmdk

live-build/ubuntu-cpc/hooks/041-vmdk-ova-image.binary

@@ -9,11 +9,24 @@
 #
 # For this step, we re-use the VMDK's made in 040-vmdk-image.binary
 
+
+# Switch on $ARCH to determine which ID and description to use in the produced
+# OVF. We have fancy Ubuntu-specific IDs in the OVF specification, we might as
+# well use them.
 case $ARCH in
-        amd64|i386) ;;
-                 *) echo "OVA images are not supported for $ARCH yet.";
+        amd64)
+            ovf_id=94
+            ovf_os_type="ubuntu64Guest"
+            ovf_desc_bits=64 ;;
+        i386)
+            ovf_id=93
+            ovf_os_type="ubuntu32Guest"
+            ovf_desc_bits=32 ;;
+        *)
+            echo "OVA images are not supported for $ARCH yet.";
             exit 0;;
 esac
+
 cur_d=${PWD}
 my_d=$(dirname $(readlink -f ${0}))
 
@@ -57,7 +70,10 @@ sed -i "${ovf}" \
     -e "s/@@NUM_CPUS@@/2/g" \
     -e "s/@@VERSION@@/${version}/g" \
     -e "s/@@DATE@@/${serial_stamp}/g" \
-    -e "s/@@MEM_SIZE@@/1024/g"
+    -e "s/@@MEM_SIZE@@/1024/g" \
+    -e "s/@@OVF_ID@@/${ovf_id}/g" \
+    -e "s/@@OVF_OS_TYPE@@/${ovf_os_type}/g" \
+    -e "s/@@OVF_DESC_BITS@@/${ovf_desc_bits}/g"
 
 # Get the checksums
 vmdk_sha256=$(sha256sum ${vmdk_f} | cut -d' ' -f1)
@@ -67,7 +83,7 @@ ovf_sha256=$(sha256sum ${ovf} | cut -d' ' -f1)
 manifest="${scratch_d}/${prefix}.mf"
 cat > "${manifest}" <<EOF
 SHA256(${vmdk_f##*/})= ${vmdk_sha256}
-SHA256(${ovf##*/}.ovf)= ${ovf_sha256}
+SHA256(${ovf##*/})= ${ovf_sha256}
 EOF
 
 # Now create the OVA

live-build/ubuntu-cpc/hooks/042-vagrant.binary

@@ -1,35 +1,41 @@
-#!/bin/bash -eux
+#!/bin/bash -ex
 # vi: ts=4 noexpandtab
 #
 # Generate a generic Vagrant Box.
 #
-# Vagrant images are essentially nothing more than OVA's with extra-metadata.
+# Vagrant images are essentially nothing more than OVA's with extra-metadata
+# and some preinstalled packages.
 #
-# We can't use the OVA's for Vagrant since Vagrant uses SSH to modify the instance.
-# This build step creates a cloud-config ISO so that Cloud-Init will configure
-# the initial user, creates meta-data that tells Vagrant how to interact with
-# the cloud-init created users, and finally create the OVA.
+# We can't use the OVA's for Vagrant since Vagrant uses SSH to modify the
+# instance. This build step creates a cloud-config ISO so that Cloud-Init
+# will configure the initial user, creates meta-data that tells Vagrant how
+# to interact with the cloud-init created users, and finally create the OVA.
 #
-# For this step, we re-use the VMDK's made in 040-vmdk-image.binary
+# For this step, we make a deriviative of binary/boot/disk.ext4 and install
+# some packages in it, convert it to a vmdk, and then assemble the vagrant
+# box.
 
 cur_d=${PWD}
 my_d=$(dirname $(readlink -f ${0}))
 
-base_vmdk="livecd.ubuntu-cpc.disk1.vmdk"
-
+# Switch on $ARCH to determine which ID and description to use in the produced
+# OVF. We have fancy Ubuntu-specific IDs in the OVF specification, we might as
+# well use them.
 case $ARCH in
-	amd64|i386) ;;
+        amd64)
+           ovf_id=94
+           ovf_os_type="ubuntu64Guest"
+           ovf_desc_bits=64 ;;
+        i386)
+           ovf_id=93
+           ovf_os_type="ubuntu32Guest"
+           ovf_desc_bits=32 ;;
         *)
-		echo "Vagrant images are not supported for $ARCH"
-		exit 0
+           echo "Vagrant images are not supported for $ARCH yet."
+           exit 0;;
 esac
 
-if [ ! -e ${base_vmdk} ]; then
-	echo "Did not find VMDK to produce Vagrant images."
-    exit 0
-fi
-
-. /build/config/functions
+. config/functions
 
 # Virtualbox is needed for making a small VMDK
 apt-get -qqy install genisoimage qemu-utils
@@ -37,7 +43,26 @@ apt-get -qqy install genisoimage qemu-utils
 # Lets be safe about this
 box_d=$(mktemp -d)
 seed_d=$(mktemp -d)
-trap "rm -rf ${box_d} ${seed_d}" EXIT
+mount_d=$(mktemp -d)
+
+create_derivative "disk" "vagrant" #sets ${derivative_img}
+mount_disk_image ${derivative_img} ${mount_d}
+
+cleanup_vagrant() {
+    if [ -d "$mount_d" ]; then
+        umount_disk_image "$mount_d"
+    fi
+    rm -rf ${box_d} ${seed_d} ${mount_d} ${derivative_img}
+}
+trap cleanup_vagrant EXIT
+
+chroot ${mount_d} apt-get update
+# virtualbox-guest-utils Recommends: virtualbox-guest-x11, which we want to
+# avoid pulling into a cloud image.
+chroot ${mount_d} apt-get install --no-install-recommends -y virtualbox-guest-utils
+chroot ${mount_d} apt-get clean
+umount_disk_image "$mount_d"
+rmdir "$mount_d"
 
 # Used to identify bits
 suite=$(chroot chroot lsb_release -c -s)
@@ -47,7 +72,7 @@ distro=$(chroot chroot lsb_release --id --short | tr [:upper:] [:lower:])
 # Get the VMDK in place
 prefix="${distro}-${suite}-${version}-cloudimg"
 vmdk_f="${box_d}/${prefix}.vmdk"
-cp ${base_vmdk} ${vmdk_f}
+create_vmdk ${derivative_img} ${vmdk_f}
 
 # Vagrant needs a base user. We either inject the well-known SSH key
 # or use password authentication. Both are ugly. So we'll use a password
@@ -70,6 +95,7 @@ cat > ${seed_d}/user-data <<END
 password: ${ubuntu_user_pass}
 chpasswd: { expire: False }
 ssh_pwauth: True
+manage_etc_hosts: localhost
 END
 
 # Create the fake meta-data
@@ -111,15 +137,11 @@ Vagrant.configure("2") do |config|
   config.vm.base_mac = "${macaddr}"
   config.ssh.username = "ubuntu"
   config.ssh.password = "${ubuntu_user_pass}"
-  config.vm.synced_folder '.', '/vagrant', disabled: true
 
   config.vm.provider "virtualbox" do |vb|
-     vb.name = "${prefix}"
      vb.customize [ "modifyvm", :id, "--uart1", "0x3F8", "4" ]
-     vb.customize [ "modifyvm", :id, "--uartmode1", "file", File.join(Dir.pwd, "%s-console.log" % vb.name) ]
+     vb.customize [ "modifyvm", :id, "--uartmode1", "file", File.join(Dir.pwd, "${prefix}-console.log") ]
   end
-
-
 end
 EOF
 
@@ -160,7 +182,10 @@ sed -i "${ovf}" \
     -e "s/@@NUM_CPUS@@/2/g" \
     -e "s/@@VERSION@@/${version}/g" \
     -e "s/@@DATE@@/${serial_stamp}/g" \
-    -e "s/@@MEM_SIZE@@/1024/g"
+    -e "s/@@MEM_SIZE@@/1024/g" \
+    -e "s/@@OVF_ID@@/${ovf_id}/g" \
+    -e "s/@@OVF_OS_TYPE@@/${ovf_os_type}/g" \
+    -e "s/@@OVF_DESC_BITS@@/${ovf_desc_bits}/g"
 
 ovf_sha256=$(sha256sum ${ovf} | cut -d' ' -f1)
 
@@ -169,7 +194,7 @@ manifest="${box_d}/${prefix}.mf"
 cat > "${manifest}" <<EOF
 SHA256(${vmdk_f##*/})= ${vmdk_sha256}
 SHA256(${cdrom_vmdk_f##*/})= ${cdrom_sha256}
-SHA256(${ovf##*/}.ovf)= ${ovf_sha256}
+SHA256(${ovf##*/})= ${ovf_sha256}
 EOF
 
 # Now create the box

live-build/ubuntu-cpc/hooks/061-open-iscsi.chroot

@@ -0,0 +1,9 @@
+#!/bin/bash
+#
+# Set InitiatorName to be runtime generated when iscsid first starts, so
+# that each cloud image gets a unique value
+#
+
+if [ -f /etc/iscsi/initiatorname.iscsi ]; then
+    echo "GenerateName=yes" > /etc/iscsi/initiatorname.iscsi
+fi

live-build/ubuntu-cpc/hooks/999-cpc-fixes.chroot

@@ -3,6 +3,8 @@ rootd="${1:-/}"
 root_fs_label=cloudimg-rootfs
 set -ex
 
+. /root/config/chroot
+
 CLOUD_IMG_STR="# CLOUD_IMG: This file was created/modified by the Cloud Image build process"
 
 LANG=C
@@ -87,7 +89,9 @@ _xchroot "${rootd}" sh -c 'sed -i "/^127.0.1.1/d" /etc/hosts'
 _xchroot "${rootd}" sh -c 'rm -f /etc/ssh/ssh_host_[rd]sa_key*'
 
 ## --------------
-_xchroot "${rootd}" locale-gen en_US.utf8
+if [ "${SUBPROJECT:-}" != minimized ]; then
+	_xchroot "${rootd}" locale-gen en_US.utf8
+fi
 
 ## --------------
 # set cloud-init to be on
@@ -150,6 +154,32 @@ if [ -f "${rootd}/etc/overlayroot.conf" ] &&
    } > "${rootd}/etc/overlayroot.local.conf"
 fi
 
+# previous steps may have left a dangling symlink here with
+# SUBPROJECT=minimized and that breaks lb_chroot_hacks step
+if [ -L "${rootd}/boot/initrd.img" ] && [ ! -e "${rootd}/boot/initrd.img" ]; then
+	rm "${rootd}/boot/initrd.img"
+fi
+
+if [ "${SUBPROJECT:-}" = minimized ]; then
+	# Remove various packages that we don't want in the minimized images.
+	# Some of these are tools that don't make sense by default
+	# non-interactively; some are libraries whose reverse-dependencies
+	# will have already been removed; open-vm-tools, it's a bug that this
+	# is in the common cloud seed because this should only be included
+	# in VMWare guest images, and we know none of the minimized images
+	# are targeted at VMWare.
+	_xchroot "${rootd}" env DEBIAN_FRONTEND=noninteractive \
+		apt-mark auto '^lib.*' '^python*' vim-runtime 2>/dev/null
+	# FIXME: iso-codes is a dep of software-properties and shouldn't be
+	_xchroot "${rootd}" env DEBIAN_FRONTEND=noninteractive \
+		apt-get -y autoremove --purge iso-codes xauth pastebinit \
+		plymouth open-vm-tools git shared-mime-info vim vim-common \
+		console-setup ncurses-term tmux screen policykit-1 \
+		xdg-user-dirs less run-one apport-symptoms \
+		ubuntu-cloudimage-keyring file
+
+	_xchroot "${rootd}" apt clean
+fi
 
 #### END COMMON ARCH FUNCTIONS
 

live-build/ubuntu-cpc/hooks/999-extras.binary

@@ -9,7 +9,7 @@ if [ ! -d ${my_dir}/extra ]; then
 fi
 
 # Export the common functions to the extras
-. /build/config/functions
+. config/functions
 
 # Cleaner execution
-/bin/run-parts --regex ".*\.binary" "${extra_d}"
+/bin/run-parts --exit-on-error --regex ".*\.binary" "${extra_d}"

live-build/ubuntu-cpc/hooks/ovf/ubuntu-ova-v1-cloudcfg-vmdk.tmpl

@@ -18,8 +18,9 @@
   <VirtualSystem ovf:id="@@NAME@@">
     <Info>A virtual machine</Info>
     <Name>@@NAME@@</Name>
-    <OperatingSystemSection ovf:id="100" vmw:osType="other3xLinux64Guest">
+    <OperatingSystemSection ovf:id="@@OVF_ID@@" vmw:osType="@@OVF_OS_TYPE@@">
       <Info>The kind of installed guest operating system</Info>
+      <Description>Ubuntu Linux (@@OVF_DESC_BITS@@-bit)</Description>
     </OperatingSystemSection>
 
     <ProductSection ovf:required="false">

live-build/ubuntu-cpc/hooks/ovf/ubuntu-ova-v1-vmdk.tmpl

@@ -16,8 +16,9 @@
   <VirtualSystem ovf:id="@@NAME@@">
     <Info>A virtual machine</Info>
     <Name>@@NAME@@</Name>
-    <OperatingSystemSection ovf:id="100" vmw:osType="other3xLinux64Guest">
+    <OperatingSystemSection ovf:id="@@OVF_ID@@" vmw:osType="@@OVF_OS_TYPE@@">
       <Info>The kind of installed guest operating system</Info>
+      <Description>Ubuntu Linux (@@OVF_DESC_BITS@@-bit)</Description>
     </OperatingSystemSection>
 
     <ProductSection ovf:required="false">

live-build/ubuntu-touch/hooks/00-uid-gid-fix.chroot_early

@@ -53,6 +53,7 @@ systemd-network:x:112:117:systemd Network Management,,,:/run/systemd/netif:/bin/
 systemd-resolve:x:113:118:systemd Resolver,,,:/run/systemd/resolve:/bin/false
 systemd-bus-proxy:x:114:119:systemd Bus Proxy,,,:/run/systemd:/bin/false
 nm-openvpn:x:115:120:NetworkManager OpenVPN,,,:/var/lib/openvpn/chroot:/bin/false
+dhcpd:x:116:122::/var/run:/bin/false
 EOF
 else
     echo "/etc/passwd post-debootstrap hash doesn't match record" >&2
@@ -101,6 +102,7 @@ systemd-network:*:16372:0:99999:7:::
 systemd-resolve:*:16372:0:99999:7:::
 systemd-bus-proxy:*:16372:0:99999:7:::
 nm-openvpn:*:16909:0:99999:7:::
+dhcpd:*:16925:0:99999:7:::
 EOF
 else
     echo "/etc/shadow post-debootstrap hash doesn't match record" >&2
@@ -186,6 +188,7 @@ systemd-resolve:x:118:
 systemd-bus-proxy:x:119:
 input:x:121:
 nm-openvpn:x:120:
+dhcpd:x:122:
 EOF
 else
     echo "/etc/group post-debootstrap hash doesn't match record" >&2
@@ -271,6 +274,7 @@ systemd-resolve:!::
 systemd-bus-proxy:!::
 input:!::
 nm-openvpn:!::
+dhcpd:!::
 EOF
 else
     echo "/etc/gshadow post-debootstrap hash doesn't match record" >&2

live-build/ubuntu-touch/hooks/60-install-click.chroot

@@ -4,8 +4,19 @@ set -e
 
 echo "Setting up click packages"
 
+CLICKARCH=$(dpkg --print-architecture)
+
 click_uri=http://archive-team.internal/click_packages
-click_list=$click_uri/click_list
+if [ "$CLICKARCH" = "arm64" ]; then
+    # FIXME: this is temporary. Since right now we can't have arm64 clicks in the store
+    # (before implementing fat-packages), we need to fetch the arm64 click list from a
+    # different place 
+    click_list=$click_uri/click_list.arm64
+    click_install_flags="--allow-unauthenticated"
+else
+    click_list=$click_uri/click_list
+    click_install_flags=""
+fi
 click_db=/usr/share/click/preinstalled
 click_db_custom=/custom/click
 
@@ -20,8 +31,6 @@ tmpdir="$(mktemp -d)"
 cleanup () { rm -rf "$tmpdir"; }
 trap cleanup EXIT
 
-CLICKARCH=$(dpkg --print-architecture)
-
 wget --no-verbose -O "$tmpdir/click_list" "$click_list"
 for package in $(cat "$tmpdir/click_list")
 do
@@ -61,7 +70,7 @@ do
                 mv /etc/click/databases/10_core.conf \
                    /etc/click/databases/10_core.conf.tmp
             fi
-            click install --force-missing-framework --root="$root" --all-users \
+            click install --force-missing-framework --root="$root" --all-users $click_install_flags \
                 "$tmpdir/$package"
             if [ "$root" = "$click_db_custom" ]; then
                 mv /etc/click/databases/10_core.conf.tmp \