Testsuite regression from 4c27eadffa.
submit-time is the time the request was submitted to amqp. For tests
it's the time the testcase was run. Many of our testcases test the exact
parameters that were passed to amqp, so they now fail because
submit-time is in there but it's not being checked for. It would be
possible to either verify the submit-time too, or switch to fuzzy
matching - but it's actually not very valuable to test submit-time,
since it is only for human readers of autopkgtest-cloud log files or
autopkgtest.ubuntu.com/running.
Instead strip the field from the amqp parameters before any tests see
them.
The apt version comparison sorts 'blacklisted' greater than most version
numbers, which means that we accidentally apply force hints for version
'blacklisted' to all uploads. Since this is the only case of a hacked
version number, let's special case it so that 'blacklisted' hints only
match packages with 'blacklisted' version.
Currently we re-trigger all reverse binary dependencies of a package,
including binary packages built from the same source. We already
explicity trigger the source's own tests if they still exist in unstable
- don't also consider the source when looking at reverse dependencies.
Add new autopkgtest policy: it determines the autopkgtests for a
source package (its own, direct reverse binary dependencies, and
Testsuite-Triggers), requests tests via AMQP, fetches results from swift, and
keeps track of pending tests between run. This also caches the downloaded
results from swift, as re-dowloading them all is very expensive.
This introduces two new hints:
* force-badtest pkg/ver[/arch]: Failing results for that package will be
ignored. This is useful to deal with broken tests that get imported from
Debian or are from under-maintained packages, or broke due to some
infrastructure changes. These are long-lived usually.
* force-skiptest pkg/ver: Test results *triggered by* that package (i. e.
reverse dependencies) will be ignored. This is mostly useful for landing
packages that trigger a huge amount of tests (glibc, perl) where some tests
are just too flaky to get them all passing, and one just wants to land it
after the remaining failures have been checked. This should be used rarely
and the hints should be removed immediately again.
Add integration tests that call britney in various scenarios on constructed
fake archives, with mocked AMQP and Swift results.
Julian Andres Klode
Steve Langasek
Iain Lane
Robert Bruce Park
Andy Whitcroft
Martin Pitt