Add new "huge" AMQP queues
Whenever a package like glibc or perl gets uploaded, these trigger thousands of
tests that fill the queues for several days. This can't (solely) be addressed
by adding more capacity due to the sheer size. The main annoyance of this is
that this blocks propagation of every other package in -proposed during that
time.
AMQP has no concept of priorities within a queue (they are strictly FIFO), so
for packages that trigger a huge number of tests, send these to the new
debci-$release-huge-$arch queues instead.
LP: #1647948
We're still seeing britney runs failing due to this exception. The
timeout has already been raised to 30 seconds - maybe retrying the
requests a few times will work better.
Doing this means that you can't use the hint tester for packages with
uppercase characters in the version, e.g.
Version mismatch, dreamchess 0.2.1-rc2-2build1 != 0.2.1-RC2-2build1
The original commit had an unreliable policy info (the peer PPA group), as
correct updating of friends' excuses depends on the processing order.
- Also add the currently investigated package to the policy info.
- Add a HTML excuse too, so that excuses.html actually shows the reason for
blocking.
- Don't invalidate other excuses multiple times.
- Change the policy_info to always carry the friend information with it, as
that's useful to know for accepted packages too. Adjust test_approve_ppa()
accordingly.
- Verify "reason" field in test_sourceppa_policy()
- Add another integration test to ensure that this also applies to unbuilt
packages, not just to rejection from other Policies.
- Avoid FileNotFoundError, this does not yet exist in Python 3.2.
Add new autopkgtest policy: it determines the autopkgtests for a
source package (its own, direct reverse binary dependencies, and
Testsuite-Triggers), requests tests via AMQP, fetches results from swift, and
keeps track of pending tests between run. This also caches the downloaded
results from swift, as re-dowloading them all is very expensive.
This introduces two new hints:
* force-badtest pkg/ver[/arch]: Failing results for that package will be
ignored. This is useful to deal with broken tests that get imported from
Debian or are from under-maintained packages, or broke due to some
infrastructure changes. These are long-lived usually.
* force-skiptest pkg/ver: Test results *triggered by* that package (i. e.
reverse dependencies) will be ignored. This is mostly useful for landing
packages that trigger a huge amount of tests (glibc, perl) where some tests
are just too flaky to get them all passing, and one just wants to land it
after the remaining failures have been checked. This should be used rarely
and the hints should be removed immediately again.
Add integration tests that call britney in various scenarios on constructed
fake archives, with mocked AMQP and Swift results.
We want to treat linux-$flavor and linux-meta-$flavor as one set in britney
which goes in together or not at all. We never want to promote linux-$flavor
without the accompanying linux-meta-$flavor.
Introduce a synthetic linux* → linux-meta* dependency to enforce this grouping.
- use relative paths
- set Ubuntu architectures
- make all output files series specific
- mark -proposed as a partial suite
- set mindays to 0 for all urgencies
- drop tpu and pu
- disable smooth updates
- disable removals of obsolete source packages
- disable components, using old merged package lists for now
We don't use os.makedirs(dir, exist_ok=True) as that is too strict: it fails if
the directory already exists with different permissions (e. g. with 775). Thus
introduce a helper function ensuredir().
Partially revert commit ac66e311 which caused packages with unsatisfiable
dependencies to only get rejected if they were not in testing. In Ubuntu we
always want to block those.
Strip of Multi-Arch qualifiers ":any" and ":native" when building the
dependency fields, as they are not part of the package name.
This will fix cases like
Package: ipython3
Depends: python3:any (>= 3)
and include ipython3 in python3's reverse dependencies.
Closes: #794194
This gives Policies the opportunity to see if a previous check
(build/installability) or earlier policies already invalidated the update. This
allows writing policies that work on groups of packages, or skipping expensive
checks (such as triggering autopkgtests while the package is not built or
installable yet).
It is unwieldy to have one half of output data generation in the policy but not
the other half of updating the excuse. Now that apply_policy() gets the excuse
object as argument we can move everything there.
This allows tests to check whether there are any missing builds or old
binaries, so that expensive actions such as "trigger an autopkgtest" are not
done too early/in vain.
For future policies such as running autopkgtests it is important to know
whether a package has built, so that expensive actions such as "trigger an
autopkgtest" are not done too early/in vain.
This requires dropping the "age != 0" check for adding the out-of-date-ness to
the Excuse, as the policies now run later. But this check only applied to an
infinitesimal age, and even with age == 0 it is still a valid excuse that there
are missing binaries.
Martin Pitt
Iain Lane
Robert Bruce Park
Martin Pitt
Dimitri John Ledkov
Colin Watson
Niels Thykier
Jonathan Wiltshire
Emilio Pozuelo Monfort