livecd-rootfs

Commit Graph

Author	SHA1	Message	Date
Robert C Jennings	9bea8296ee	ubuntu-cpc: Only produce explicitly named artifacts (LP: #1837254 ) In parallel builds where a list of image targets are provided the build may produce binaries that are not part of the named set of targets but are created by series dependencies. These implicitly created binaries may be generated by multiple builds but are unused as our convention for the ubuntu-cpc project is to only consume binaries from the explicitly named image targets; this avoid overwriting the same object by multiple parallel builds. This patch adds support for a 'provides' keyword for series files. It can be specified multiple times per series file. The field is used by the make-hooks script to generate a list of output files created explicitly by the named image targets. The list is saved to the "explicit_provides" file in the hooks output directory. In the case of the "all" target this list would be empty. This list is consumed by the "final.binary" hook file. This patch adds support for optional final.binary hooks in hooks.d/base and/or hooks.d/extra. These final.binary hooks are always included as the last hook(s) if either exist with the hook in "extra" running last. The base/final.binary hook includes logic to parse the "explicit_provides" file generated by the make-hooks script and remove any binary output not explicitly specified. Some series files named unnecessary dependencies, specifically disk-image, to keep output of implicit artifacts consistent between parallel builds. These unnecessary dependencies are removed in this patch.	6 years ago
Robert C Jennings	996d7e20d5	ubuntu-cpc: Ensure base disk image is the same between all build targets The following targets have livecd.ubuntu-cpc.manifest (and livecd.ubuntu-cpc.ext4) which differ in some way from the 'all' target. They are all missing grub-efi and other modifications: root-dir squashfs tarball These targets do not depend on the 'disk-image' target. This means that the ext4 produced will lack the uefi modifications (and any from the disk-image target binary hooks). Since the ext4 file is common to all builds there is a chance that a parallel build from one of these targets could overwrite this artifact. This patch ensures that all targets will produce consistent base output.	6 years ago
Robert C Jennings	77ae8d704f	ubuntu-cpc: parallel builds * Replace "snap download" with tool that uses snap store's coherence feature This is important for parallel image builds to ensure all pre-seeded snaps have the same versions across image variants. * Inject a proxy into the build providing a snapshot view of the package repo. When the REPO_SNAPSHOT_STAMP variable is set, the auto/build script will attempt to launch a transparent HTTP proxy on port 8080, and insert an iptables rule to redirect all outgoing HTTP requests to this proxy. The proxy, contained in the `magic-proxy` Python script, examines each request and silently overrides those pointing to InRelease files or files that are listed in InRelease files. It will instead provide the contents of the requested file as it was at REPO_SNAPSHOT_STAMP, by downloading the corresponding asset "by hash". * Use series files with dependency handling to generate hook symlinks dynamically This patch currently only applies to the "ubuntu-cpc" project. More and more logic has been going into the hook scripts to decide under which conditions they should run or not. As we are moving to parallelized builds of image sets, this will get even more complicated. Base hooks will have to know which image sets they belong to and modification of the dependency chain between scripts will become more complicated and prone to errors, as the number of image sets grows. This patch introduces explicit ordering and dependency handling for scripts through the use of `series` files and an explicit syntax for dependency specification.	6 years ago

Author

SHA1

Message

Date

Robert C Jennings

9bea8296ee

ubuntu-cpc: Only produce explicitly named artifacts (LP: #1837254 )

In parallel builds where a list of image targets are provided the build
may produce binaries that are not part of the named set of targets but
are created by series dependencies.  These implicitly created binaries
may be generated by multiple builds but are unused as our convention for
the ubuntu-cpc project is to only consume binaries from the explicitly
named image targets; this avoid overwriting the same object by multiple
parallel builds.

This patch adds support for a 'provides' keyword for series files. It can
be specified multiple times per series file.  The field is used by the
make-hooks script to generate a list of output files created explicitly by
the named image targets.  The list is saved to the "explicit_provides"
file in the hooks output directory. In the case of the "all" target
this list would be empty.  This list is consumed by the "final.binary"
hook file.

This patch adds support for optional final.binary hooks in hooks.d/base
and/or hooks.d/extra.  These final.binary hooks are always included as
the last hook(s) if either exist with the hook in "extra" running last.

The base/final.binary hook includes logic to parse the "explicit_provides"
file generated by the make-hooks script and remove any binary output not
explicitly specified.

Some series files named unnecessary dependencies, specifically
disk-image, to keep output of implicit artifacts consistent between
parallel builds.  These unnecessary dependencies are removed in this
patch.

Robert C Jennings

996d7e20d5

ubuntu-cpc: Ensure base disk image is the same between all build targets

The following targets have livecd.ubuntu-cpc.manifest (and
livecd.ubuntu-cpc.ext4) which differ in some way from the 'all'
target. They are all missing grub-efi and other modifications:

root-dir
squashfs
tarball

These targets do not depend on the 'disk-image' target. This means that
the ext4 produced will lack the uefi modifications (and any from the
disk-image target binary hooks).

Since the ext4 file is common to all builds there is a chance that a
parallel build from one of these targets could overwrite this artifact.
This patch ensures that all targets will produce consistent base output.

Robert C Jennings

77ae8d704f

ubuntu-cpc: parallel builds

* Replace "snap download" with tool that uses snap store's coherence feature

    This is important for parallel image builds to ensure all pre-seeded snaps
    have the same versions across image variants.

* Inject a proxy into the build providing a snapshot view of the package repo.

    When the REPO_SNAPSHOT_STAMP variable is set, the auto/build script will attempt
    to launch a transparent HTTP proxy on port 8080, and insert an iptables rule to
    redirect all outgoing HTTP requests to this proxy.

    The proxy, contained in the `magic-proxy` Python script, examines each request
    and silently overrides those pointing to InRelease files or files that are
    listed in InRelease files. It will instead provide the contents of the requested
    file as it was at REPO_SNAPSHOT_STAMP, by downloading the corresponding asset
    "by hash".

* Use series files with dependency handling to generate hook symlinks dynamically

    This patch currently only applies to the "ubuntu-cpc" project.

    More and more logic has been going into the hook scripts to decide
    under which conditions they should run or not. As we are moving
    to parallelized builds of image sets, this will get even more
    complicated. Base hooks will have to know which image sets they
    belong to and modification of the dependency chain between scripts
    will become more complicated and prone to errors, as the number of
    image sets grows.

    This patch introduces explicit ordering and dependency handling for
    scripts through the use of `series` files and an explicit syntax
    for dependency specification.

3 Commits (515bc1430b7253db5f3990d92fa37dff2a5e7351)