autopkgtest-cloud will now serve:
autopkgtest.ubuntu.com/static/autopkgtest.db.sha256
Britney now calculates the sha256 of the newly downloaded db locally and
checks that it matches the sha256 file served by autopkgtest-cloud,
instead of checking that the content-length header matches the
size of the new downloaded database.
Since the most recent apache2 security update in focal [1], the
content-length header isn't served by default, and it seems that when
it is served it's not entirely accurate. This check has become
brittle, and so we have implemented this new mechanism.
[1] https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/2061816
We don't know why the autopkgtest webserver has stopped providing a
Content-Length header but the current code doesn't handle its absence, so
detect this rather than throwing an exception.
Should be reverted once the header is back, we don't want to be in the dark
about short reads of the db.
As seen in
https://ubuntu-archive-team.ubuntu.com/proposed-migration/log/noble/2024-04-01/12:52:27.log
Having an entire britney run bail because of a connection reset is a bad
outcome!
Instead, catch this exception and avoid adding the test in question to the
list of queued tests (we can pick it up on the next run).
Possibly we should do more clever handling of a ConnectionResetError such as
reconnecting, but this is a minimum fix that will stop britney from aborting.
This avoids endlessly requeuing the test if the test produces
an older result.
This will make tests "disappear" if the infrastructure returns
old results for newer triggers but avoids the problem right
now where we end up queuing the same tests every run.
For rolling out britney on a new machine, we want to generate update_excuses
and update_output to confirm it's working correctly all the way through, so
we don't want to use the global --dry-run option; but we *do* want to
disable queuing tests and instead let the production instance of britney
queue the tests while we simply query the results. Add support for
ADT_ENABLE=dry-run in britney.conf, parallelling the behavior of other
policies.
britney currently spends a majority of its runtime querying for baseline
test results that it won't find, and that it doesn't need. Refactor to
eliminate many of these excess queries.
@canonical.com is now DKIM signed and SPF published which means emails
from proposed-migration running on snakefruit sending direct would
likely be caught out. Since we're here, the project is Ubuntu related
so switch to using an @ubuntu.com address instead.
When querying swift there is no way to take results only newer than a
specified point, you can only query newer than or equal to. But for sqlite
we can absolutely use > instead of >= and avoid re-processing results we've
already seen.
Logging all force-reset-test hints for every package causes
about 850 MB of logs in the last run of 880 MB of logs in total,
let's only log ones matching the package instead, as we do for
force-badtest.
In Ubuntu, we only fetch results on demand, so we might not
have seen the results yet.
Debian always fetches results at the beginning so has all the
data ready.
Due to the number of hints in standing use in Ubuntu, hints.search() is an
expensive operation, and we call it once for *every single test* referenced
from -proposed. Since force-reset-test are a small proportion of the hints
in use, searching once for all the hints of this type and only searching
this subset for each autopkgtest improves performance (with 23000
autopkgtests referenced in -proposed, this saves roughly 1 minute of
runtime, or 11% on a 9-minute britney run; the number of packages in
-proposed is typically much higher at other points in the release cycle,
therefore the absolute improvement in performance is expected to be
greater.)
The force-reset-test hints are an Ubuntu delta so this is not expected to be
upstreamed; and it could eventually be dropped if and when baseline
retesting is implemented in Ubuntu and the number of hints required drops.
This could be implemented with a more generic, elegant solution in
HintsCollection, but again, the scalability problem of hints is hopefully
short-lived so I didn't consider it worth the investment here.
urlopen() supports non-http URLs, but when called on them, http-related
features are absent - such as getcode(). Make the code work with file:///
URLs.
Since we've never used remove hints in Ubuntu and never will, we shouldn't
pay the cost for them in britney runs. Remove references to them entirely,
to speed up britney runs.
This is an Ubuntu delta not expected to be upstreamed, and can be dropped at
some future point when we don't have 11,000 active hints. (This count of
hints is expected to drop dramatically once we have autopkgtest baseline
retesting.)
We currently skip the ALWAYSFAIL/REGRESSION handling for kernels. This
can lead to us missing genuine regressions in kernel uploads. The
idea is that results from one kernel flavour shouldn't influence
another.
We can keep this idea but do better and actually check for regressions:
when looking at results, if we're considering a kernel, only look at
results which were triggered by this kernel.
We often introduce new kernels post-release, and we still want to not
trigger tests for the kernel image packages. Check for -meta in the
*source* suite as well as the target.
Fix the -meta name calculation for linux-signed-foo.
We currently skip running autopkgtests where there is an installability
problem, but in a few cases the depends policy notes these only, but
otherwise doesn't block migration on them.
In these cases, let's try to run the autopkgtests anyway. There will be
a few instances of uninstallability here, but since we migrate the
packages we should give them a chance to be tested.
We currently concatenate all triggers together into a string, but the
AMQP consumer expects this to be a list.
When using AMQP, keep the triggers as a list. Ensure that the "real"
trigger (the package being tested) is kept first, as before.
On some distros (Ubuntu), arch:all packages are built along with one of
the architectures. We shouldn't be listing 'all' as its own arch in this
case. Instead we filter out the binaries except for on the
'all_buildarch'.
We just had the autopkgtest queues DoSed because britney was crashing
after requesting each reverse dependency for a perl upload, but before
it had written pending.json out so it knew what not to request again.
This was 25,000 requests per arch...
Let's write pending.json straight after sending each request, so that
the next run - even after a crash - won't re-request the same things
again.
Tim Andersson
Steve Langasek
Julian Andres Klode
Michael Hudson-Doyle
Brian Murray
Juerg Haefliger
Łukasz 'sil2100' Zemczak
Haw Loeung
Iain Lane
Iain Lane