Robert C Jennings 77ae8d704f
ubuntu-cpc: parallel builds
* Replace "snap download" with tool that uses snap store's coherence feature

    This is important for parallel image builds to ensure all pre-seeded snaps
    have the same versions across image variants.

* Inject a proxy into the build providing a snapshot view of the package repo.

    When the REPO_SNAPSHOT_STAMP variable is set, the auto/build script will attempt
    to launch a transparent HTTP proxy on port 8080, and insert an iptables rule to
    redirect all outgoing HTTP requests to this proxy.

    The proxy, contained in the `magic-proxy` Python script, examines each request
    and silently overrides those pointing to InRelease files or files that are
    listed in InRelease files. It will instead provide the contents of the requested
    file as it was at REPO_SNAPSHOT_STAMP, by downloading the corresponding asset
    "by hash".

* Use series files with dependency handling to generate hook symlinks dynamically

    This patch currently only applies to the "ubuntu-cpc" project.

    More and more logic has been going into the hook scripts to decide
    under which conditions they should run or not. As we are moving
    to parallelized builds of image sets, this will get even more
    complicated. Base hooks will have to know which image sets they
    belong to and modification of the dependency chain between scripts
    will become more complicated and prone to errors, as the number of
    image sets grows.

    This patch introduces explicit ordering and dependency handling for
    scripts through the use of `series` files and an explicit syntax
    for dependency specification.
2019-05-21 17:06:59 -05:00

69 lines
2.9 KiB
Markdown

# TL;DR
In order to generate the hooks for a specific image target set, call the
`make-hooks` script, located in `hooks.d` as
./make-hooks --hooks-dir ../hooks <image_set>
where `image_set` is the name of a series file (e.g. "vagrant") without leading
path components. Do *not* check in the `hooks` folder, it is automatically
generated by `auto/config` during Live Build runs.
# Hook placement and ordering
Scripts live in subfolders below the `hooks.d` folder. Currently the folders
`chroot` and `base` exist. The folder with the name `extra` is reserved for
private scripts, which are not included in the source of livecd-rootfs. The
scripts are not numbered, instead the order of their execution depends on the
order in which they are listed in a *series* file.
Series files are placed in subfolders `hooks.d/base/series` or
`hooks.d/extra/series`. Each series file contains a list of scripts to be
executed. Empty lines and lines starting with a `#` are ignored.
Series files in `extra/series` override files in `base/series` with the same
name. For example, if a series file `base/series/cloudA` exists and a series
file `extra/series/cloudA`, then the latter will be preferred.
A series file in `extra/series` may also list scripts that are located in the
`chroot` and `base` folders. In addition, series files can *depend* on other
series files. For example, the series files for most custom images look similar
to this:
depends disk-image
depends extra-settings
extra/cloudB.binary
Where `disk-image` and `extra-settings` may list scripts and dependencies which
are to be processed before the script `extra/cloudB.binary` is called.
ACHTUNG: live build runs scripts with the suffix ".chroot" in a batch separate
from scripts ending in ".binary". Even if you arrange them interleaved in your
series files, the chroot scripts will be run before the binary scripts.
# Image set selection for Live Build
During a Live Build, enumerated symbolic links are generated based on the
contents of one or more series files. The series files are selected according
to the contents of the `IMAGE_TARGETS` environment variable. For example, in
order to trigger the build of `squashfs` and `vagrant`, list them in the
`IMAGE_TARGETS` variable as `squashfs,vagrant`. The separator can be a comma,
a semi-colon or whitespace.
The generation of the symbolic links is triggered from the `auto/config` script,
from where the contents of the `IMAGE_TARGETS` environment variable are passed
on to the `make-hooks` script.
# Symlink generation
Since Live Build itself does not know about series files, a traditional `hooks`
folder is generated using the `make-hooks` script. The script takes as arguments
the names of the series files to be processed.
The script parses the series files and generates enumerated symbolic links for
all entries. Per default, these are placed into a directory named `hooks` next
to the `hooks.d` directory. This can be changed using the `--hooks-dir`
parameter.