Cloud test framework can run into errors (like SSH connection drops)
which cause it to error out and not report test results. To handle these
kinds of errors the policy now checks the return code and stores
std_err.
Also updates the test already run check to not count states of zero
failures and non-zero errors; since this means the test framework
encountered an error external to the tests.
Cloud policy now uses the ADT_PPAS field as the source but will only
take private PPAS due to requiring a fingerprint. If ADT_PPAS is empty
then the 'proposed' archive is used.
Adds logic to try and retrieve the install source of a package from the
logs. This is useful if multiple PPAs are defined since the policy won't
explicitly know when contains the package under test.
Changes made
- Multiple hardcoded fields moved to config
- Series is now retrieved from options
- Pocket is now called source and retrieved from config
- Adds source type config which can be either archive or ppa
- Returns REJECTED_PERMANENTLY policy verdict when test failures or
errors occur. Adds verdict info the the excuse.
The cloud policy is currently used to test whether the proposed
migration will break networking for an image in the Azure cloud. Cloud
testing will likely increase in scope in the future.
britney currently spends a majority of its runtime querying for baseline
test results that it won't find, and that it doesn't need. Refactor to
eliminate many of these excess queries.
@canonical.com is now DKIM signed and SPF published which means emails
from proposed-migration running on snakefruit sending direct would
likely be caught out. Since we're here, the project is Ubuntu related
so switch to using an @ubuntu.com address instead.
When querying swift there is no way to take results only newer than a
specified point, you can only query newer than or equal to. But for sqlite
we can absolutely use > instead of >= and avoid re-processing results we've
already seen.
Logging all force-reset-test hints for every package causes
about 850 MB of logs in the last run of 880 MB of logs in total,
let's only log ones matching the package instead, as we do for
force-badtest.
In Ubuntu, we only fetch results on demand, so we might not
have seen the results yet.
Debian always fetches results at the beginning so has all the
data ready.
Due to the number of hints in standing use in Ubuntu, hints.search() is an
expensive operation, and we call it once for *every single test* referenced
from -proposed. Since force-reset-test are a small proportion of the hints
in use, searching once for all the hints of this type and only searching
this subset for each autopkgtest improves performance (with 23000
autopkgtests referenced in -proposed, this saves roughly 1 minute of
runtime, or 11% on a 9-minute britney run; the number of packages in
-proposed is typically much higher at other points in the release cycle,
therefore the absolute improvement in performance is expected to be
greater.)
The force-reset-test hints are an Ubuntu delta so this is not expected to be
upstreamed; and it could eventually be dropped if and when baseline
retesting is implemented in Ubuntu and the number of hints required drops.
This could be implemented with a more generic, elegant solution in
HintsCollection, but again, the scalability problem of hints is hopefully
short-lived so I didn't consider it worth the investment here.
urlopen() supports non-http URLs, but when called on them, http-related
features are absent - such as getcode(). Make the code work with file:///
URLs.
Since we've never used remove hints in Ubuntu and never will, we shouldn't
pay the cost for them in britney runs. Remove references to them entirely,
to speed up britney runs.
This is an Ubuntu delta not expected to be upstreamed, and can be dropped at
some future point when we don't have 11,000 active hints. (This count of
hints is expected to drop dramatically once we have autopkgtest baseline
retesting.)
This reverts commit 7ff150ced7.
Sadly we can't use the X-Auth-Token approach due to implementational details (the token is valid only for an hour). So we need to switch back to using swiftclient.
This way there's less secrets that need to be shared and less new code to introduce. We also modified the test tooling to be able to check for authentication tokens in the queries.
We currently skip the ALWAYSFAIL/REGRESSION handling for kernels. This
can lead to us missing genuine regressions in kernel uploads. The
idea is that results from one kernel flavour shouldn't influence
another.
We can keep this idea but do better and actually check for regressions:
when looking at results, if we're considering a kernel, only look at
results which were triggered by this kernel.
Aleksa Svitlica
Łukasz 'sil2100' Zemczak
Steve Langasek
Brian Murray
Julian Andres Klode
Haw Loeung
Michael Hudson-Doyle
Iain Lane
Iain Lane