python3-amqplib already exists in trusty, but python3-kombu does not. This
makes it possible to run britney on a standard trusty without manual backports.
Stop special-casing the kernel and move to one test run per trigger. This
allows us to only install the triggering package from unstable and run the rest
out of testing, which gives much better isolation.
For linux* themselves we don't want to trigger tests -- these should all come
from linux-meta*. A new kernel ABI without a corresponding -meta won't be
installed and thus we can't sensibly run tests against it. This caused
unnecessary and wrong regressions, and unnecessary runs (like linux-goldfish
being triggered by linux).
Test requesting: Don't re-request a test if we already have a result for it for
that trigger (for a relevant version), but there is a new version of the tested
package. First this unnecessarily delays propagation as the test will go back
to "in progress", and second if it fails in the next run this isn't the fault
of the original trigger, but the new version of the tested package.
Result finding: Don't limit acceptable results to the latest version of the
tested package. It's perfectly fine if an earlier version (like the one in
testing, or an earlier upload) was ran and gave a result for the requesting
trigger. If it's PASS, then we are definitively done, and if it's a failure
there is the "Checking for new results for failed test..." logic in collect()
logic which will also accept newer results.
In addition to not reading ever_passed for kernel triggers, we must also not
write it for those. Otherwise we introduce false regressions for e. g. "dkms"
when some DKMS package always failed on the main kernel but succeeds on one
flavour.
We trigger independent tests for every linux/linux-meta* reverse dependencies,
as they run under the triggering kernel. Thus "ever passed" is rather
meaningless for these as we don't want to track this on a per-trigger basis (as
it would be wrong for everything else but kernels). This led to a lot of false
regressions, as some DKMS modules only work on some kernel flavours.
The kernel team is doing per-kernel regression analysis of the test results, so
we don't need to duplicate this logic in britney. Thus effectively disable the
"Regression" state for kernel reverse dependencies, and rely on the kernel
test machinery to untag the tracking bug only if there are no actual
regressions.
When fetching a result with explicit triggers, always update self.results, not
just when we have a pending trigger for it. Otherwise satisfied_triggers will
be empty after reading the first result, and we clobber test results for all
triggers with the latest result.
When we need to blow away and rebuild results.cache we want to avoid
re-triggering all tests. Thus collect already existing results for requested
tests before submitting new requests.
This is rather hackish now, as fetch_one_result() now has to deal with both
self.requested_tests and self.pending_tests. The code should be refactored to
eliminate one of these maps.
When downloading results, check the ADT_TEST_TRIGGERS var in testinfo.json to
get back the original trigger that previous britney runs requested the test run
for. This allows us to much more precisely map a test result to an original
test request.
In order to tell apart test results which pass/fail depending on the package
which triggers them (in particular, if that is a kernel), we must keep track of
pass/fail results on a per-trigger granularity. Rearrange the results map
accordingly.
Keep the old "latest test result applies to all triggers of this pkg/version"
logic around as long as we still have existing test results without
testinfo.json.
ATTENTION: This breaks the format of results.cache, so this needs to be removed
when rolling this out.
gccgo-5 exists in Ubuntu 15.04 only and builds all binary packages of gcc-5.
Triggering all tests is pointless and a big waste of test resources, so trim
down the list to actually useful ones. This can be dropped when 15.04 goes EOL.
We often get "tmpfail" results (repeated failure to start cloud instance, etc.)
with no package/version at all. Stop attributing them to the latest pending
request for that package, as that has already messed up some results. With
moving to tracking test triggers in testinfo.jar and running multiple test
requests for each triggering kernel version it becomes completely impossible to
interpret anything into a tmpfail result without testpkg-version, so just
ignore them.
This will leave some orphaned entries in pending.txt and thus require manual
retries after fixing the tmpfail reason. But this needs to happen anyway, so
this does not complicate operation but instead shows those as "in progress"
instead of "regression".
So far we only added the triggering test name. Add the version as well, so that
we'll retain the complete trigger information in result.tar's testinfo.json in
swift. This will allow us to completely reconstruct our results.cache from
scratch without losing any trigger information.
This isn't significantly harder to parse from shell either (in tests): You can
still iterate over $ADT_TEST_TRIGGERS with a "for" loop and split package and
version on '/'.
In collect(), check if there are new results for failed tests on a
per-architecture level. This updates results while tests for other
architectures are still in progress (i. e. in self.pending_tests).
So far we've only calculated the reverse dependencies on amd64. This breaks
when triggering packages which do not exist on some architectures, like
bcmwl-kernel-source. It also makes it impossible to e. g. trigger DKMS tests on
armhf only for an ARM-only kernel like linux-ti-omap4.
Through the usual reverse dependency triggering, gcc-* usually triggers many
hundreds of (mostly universe) tests via libgccN. But:
- This does not help to prevent compiler regressions: as all packages are
built in -proposed anyway, the new compiler is being used immediately, so we
can't hold it back in -proposed.
- It does not trigger toolchain tests which actually are affected, most
importantly binutils and linux.
- This puts enormous stress onto our test infrastructure.
So special case gcc by triggering binutils and linux, and fglrx-installer as a
typical (and important) example of a DKMS package which also needs a compiler,
and libreoffice as our favourite tool chain stress test to cover libgccN.
If a package gets triggered by several sources, we can ordinarily run just one
test for all triggers. But for proposed kernels we want to run a separate test
for each, so that the test can run under that particular kernel.
With this, tests can do special things when they get triggered by a particular
package. E. g. "linux" or "gcc" could skip their "rebuild myself" test if they
were triggered by a new version of themselves (as opposed to a new binutils).
This is particularly aimed at DKMS tests which need to install the triggering
kernel (e. g. -generic vs. -generic-lts-backport-XXX).
New kernels are prone to break LXC. In https://bugs.debian.org/779559 there is
a proposal for a flexible approach to add extra "reverse test dependencies".
Hardcode this trigger until this gets implemented.
By the kernel team's request we want to trigger DKMS package tests on new
kernel uploads, to ensure that we don't regress them with newer kernels.
Pretend that linux-meta builds the "dkms" binary, so that the existing reverse
dependency magic takes care of the actual triggering.
Note that this needs to be "linux-meta", not "linux", so that tests will
actually use the new kernel (via dist-upgrade).
Add Excuse.addtest() for adding a test type/package/arch/result, so that the
excuses YAML will get structured test results instead of pre-formatted HTML.
Move the HTML rendering into Excuse.html() instead.
This supports a "test type" whose only value is "autopkgtest" right now, but
we will have "bootest", perhaps "piuparts" and other tests in the future.
Drop the "(<ver> is unbuilt/uninstallable)" note from excuses.html as this is
really a per-architecture property, not a per-tested-source one. This needs to
be re-thought and generalized.
r472 added >= version matching to the results evaluation. But we also must do
this in add_test_request() so that we avoid requesting a test for the testing
version over and over again if we get results for the unstable version only.
But here it is enough to only check the requested version and the unstable
version (if that's higher).
If a result.tar does not contain a testpkg-version, we must still match it
against pending.txt, but we must not add it to the results cache. This ends up
being a "null" version key (JSON's serialization of None) which becomes an
actual version string once this is read back.
There are scenarios when britney requests a package test for a particular
version but we actually get a result for a later version:
* When britney runs the later version is not built yet and thus it is in
excludes; but at the time when the test actually runs the package is built.
* We don't support running tests for a given older (source) version yet, tests
always get run from the latest unstable source even if that isn't built yet.
Thus we need to consider results >= the requested version. However, we prefer a
succesful result for the originally requested version so that we can continue
to remove a broken version from unstable. This is already covered by
TestAutoPkgTest.test_remove_from_unstable.
Disabling AMQP requests with "ADT_ENABLE = yes" but ADT_AMQP unset made sense
while we still supported adt-britney. But as that's gone now, let's use the
ADT_ENABLE switch only, and if it's on, require ADT_AMQP and ADT_SWIFT_URL be
set.
This simplifies the code a bit and is less confusing.
We already handle the exclusions in tests_for_source() (and run the testing
version if appropriate), so don't unconditionally skip requests for those.
Adjust the TestAutoPkgTest.test_rdepends_unbuilt case to catch that: The "run
britney once to pick up previous results" was a thinko as this already
satisfies all tests for green 2.
The previous commit introduced a KeyError crash in tests_for_source() for
packages which are unbuilt/uninstallable and only present in unstable.
Ignore these in tests_for_source() as they can't possibly be a regression for
their dependencies, and there is no sensible way to run a test for them.
Commit 463 ("Don't promote packages with unbuilt reverse dependencies") turned
out to be too strict: This holds up too many innocent packages in -proposed.
If unstable has an unbuilt/uninstallable reverse dependency D of a package P,
trigger a test anyway (which will then most likely run against the testing
version of D). If that succeeds, the unstable P did not break D and can be
accepted. If it fails, D needs to be fixed.
Ideally we would set up some clever apt pinning to force installation of
testing-D, to avoid running into the uninstallability of unstable-D, but this
is tricky and error prone.
Drop the temporary "UNINST" state from commit 466 again. Instead, excuses.html
will now show a test against the testing version of D together with a note that
the unstable version is unbuilt/uninstallable.
This should ideally clear up all cases where a requested result is neither
present or pending. Log an error if that still happens (will be checked in the
next couple of runs), and ensure in the tests that we don't trigger any
outstanding "FIXME" log messages.
Commit 463 introduced waiting on reverse dependencies which are not built or⎵
installable yet, but set their status as "RUNNING". This is confusing as there
is no actual test in progress yet.
Instead, set their status to a new UNINST value, displaying as⎵
"Unbuilt/uninstallable"
If a reverse dependency D of a package P is not built yet, then D will be in
"exclusions" as we can't sensibly run D's tests at that time. In that case,
don't just ignore the missing test result but consider D's test as "in
progress".
Note that this might lead to stalling an innocent P if a broken (FTBFS) D gets
uploaded at the same time. This can/should be handled by overrides if fixing
D isn't appropriate, but this is better than allowing P to break D in that
situation.
Swift results were considered for older versions of triggers instead of waiting
for results for the actual package/version that triggered a new test.
This broke due to two reasons:
* When evaluating the test results we need to check whether we have a result
for the tested package/version that got triggered by the current excuse, not
just for any older excuse.
* AutoPackageTest.fetch_swift_results() re-downloaded all results for a
package due to a wrong "marker" value: The marker needs to be the
complete object path, not just the timestamp suffix. This caused old test
results to be considered as "newer than the given marker".
Now that we look at autopkgtest results from swift we can drop the
adt-britney/lp:auto-package-testing code from autopkgtest.py.
Note that we still need it for boottest.py.
Adjust TestBoottestEnd2End.test_with_adt() for cloud results.
Change AutoPackageTest.results() to evaluate the Swift results instead of the
adt-britney ones.
TODO:
- Add more tests (like for adt-britney)
- Drop triggering of adt-britney tests
- Drop adt-britney tests (which fail now)
- Adjust TestBoottestEnd2End.test_with_adt
Martin Pitt
Iain Lane
Didier Roche