Rework the dependencies between excuses.
Dependencies are specified based on packages (source or binary). Based
on these packages, dependencies on other excuses (that have these
packages) are calculated.
As with package dependencies, dependencies between excuses can contain
alternatives.
Each alternative can satisfy the dependency, so the excuse only becomes
invalid if all of the alternatives of a specific dependency are
invalidated.
Tracking of the alternatives and their validity is moved to separate
objects.
Because autopkgtest failure in a package is now a serious bug (RC), packages
that don't opt-in should not be tested. Due to the way autopkgtest and autodep8
work together, package that don't declare they have an autopkgtest can still be
tested. However, maintainers should not be force to say their package doesn't
work with autodep8 by introducing a fake test.
When there is no test in testing, and a failing test in unstable, don't
allow the package to migrate. Doing so would make it instantly RC-buggy.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
When the arch: all packages were uploaded by the maintainer, a binnmu (built
on the buildds) shouldn't be blocked, because it doesn't contain the arch: all
binaries anyway.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
The update of gcc to gcc-9 introduced a regression in buildability of
anything relying on kernel headers. This could have been caught by the
kernel's standard rebuild autopkgtest, but we currently only trigger the
linux autopkgtest for source packages named gcc-N, which excludes
gcc-defaults.
Include gcc-defaults in the list of packages that trigger a linux rebuild
test.
Bug-Ubuntu: https://bugs.launchpad.net/bugs/1836100
This avoid messages like
Required age reduced by 0 days because of autopkgtest
that is hardly useful.
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
The intialise method is already complex enough and this was a trivial
snippet to extract to reduce the complexity a bit.
Signed-off-by: Niels Thykier <niels@thykier.net>
When we convert legacy results in the autopkgtest-results.cache file,
we are only touching leaf results. By moving the loops into a
function, we can remove 2-3 levels of ("redundant") nesting. This in
turn makes it more clear what is relevant in the conversion.
This same also holds in save_state when we convert an Enum to a
string.
Signed-off-by: Niels Thykier <niels@thykier.net>
Move the logic of apply_src_policy and apply_srcarch_policy into PolicyEngine.
This fixes an issue with the excuses.yaml output introduced in commit
15e5228669: only the last verdict was added to the excuse info for that
policy.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
When a source has only arch: all binaries, the Build-Depends had no relevant
architectures, so the check was skipped. Instead check it on any architecture,
just like Build-Depends-Indep.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
The src_policy defines wether, for source items, the source policy should be
run (RUN_SRC, the default), the arch policy should be run on every arch
(RUN_ON_EVERY_ARCH_ONLY), or both (RUN_SRC_AND_EVERY_ARCH).
Signed-off-by: Ivo De Decker <ivodd@debian.org>
The recent code changes made use remove from the "binaries" field in
SourcePackages. Lists are not particularly optimized for this kind
of removal and we have a few source packages with a lot of binary
packages (e.g. libreoffice, gcc-X-cross{,-ports}) that might trip
poor performance.
Signed-off-by: Niels Thykier <niels@thykier.net>
Regex compilation is often rather expensive and in this case, we can
do it once instad of once per migration item.
Signed-off-by: Niels Thykier <niels@thykier.net>
Note that this now also applies to binNMUs. A source block also blocks
binaries. Binaries on a specific architecture can be unblock with an
architecture-specific unblock hint.
This also means all binaries from non-primary suites
(testing-proposed-updates, etc) need approval.
Closes: #916209
Currently no policy implements this.
A policy can now implement apply_src_policy_impl or apply_srcarch_policy_impl
(or both), so apply_src_policy_impl is no longer an abstractmethod.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
apply_src_policy expects an excuse with a new source and binaries. It doesn't
apply to srcarch excuses, which only have new binaries for an existing source.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
This is a step towards making migration unit-testable. This step
reduces the need for global state (in the MigrationItem class as class
fields) and with another step we can remove the global state entirely
and enable unit tests to create migration items without having to
worry about other unit tests.
Signed-off-by: Niels Thykier <niels@thykier.net>
All types of dependencies between excuses (Depends, Build-Depends,
Build-Depends-Indep, ...) are handled by the same code. The DependencyType is
used to distinguish between the types where needed.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
This is useful to run tests with the data files from a specific point in time,
without changes due to ageing when the test runs later.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
Currently autopkgtest tries to install our trigger from unstable and the rest
from testing. If that fails, than autopkgtest has a fall-back to allow all
packages from unstable to be installed. This has two severe issues:
1) the version of the test and the package it came from may be out-of-sync
2) too much from unstable may be installed, even stuff that should not/is not
allowed to migrate as it breaks stuff.
Make sure that test depends also get added to triggers if they are broken.
E.g. imagine the following scenario: trigger X changes (breaks) the output
generated by Y. Package Z has Y in the test dependencies and compares the
output in the autopkgtest. We want to have the opportunity that a new version
is automatically fixing the situation.
Two use cases are currently unsupported: needs-build (autopkgtest restriction)
and test dependencies generated by autodep8.
In case autopkgtest triggering is delayed because the required builds aren't
ready yet or the package is not installable, currently there is only the
message that autopktest delays the migration, but no hint why. This commit adds
these hints.
The initial idea was to do this to bootstrap the baseline, but it turns out
that this has the drawback it triggers runs for a package that has a new
autopkgtest where it didn't have it in the version in the target suite. It was
considered harmless (as it would just have a failing reference), but due to
autodep8, package can have a passing result in the target suite while the new
autopkgtest is actually broken. Such a package should not be blocked / getting
a penalty.
The alternative is to make the check here smarter, but as this is only for
bootstrapping, lets do that outside of britney proper.
When determining whether a policy applies to a given item, use the
suite class rather than the suite name.
Signed-off-by: Niels Thykier <niels@thykier.net>
1) the update didn't happen for all but the first
2) we don't want a package that fixes a regression in unstable to influence the
reference for another package until it actually migrates, so this updating
is flawed.
Given that only one value is defined ("reference"), it is a better
option to allow the config to be unset when one does not want to have
adt_baseline set to "reference".
Signed-off-by: Niels Thykier <niels@thykier.net>
Notable omissions are "pending_tests" and "tests_results". This is
omission is due to these (some times) being initialized from the
output of "json.load" (so we cannot assume defaultdict semantics
without manually imported the data into one).
Signed-off-by: Niels Thykier <niels@thykier.net>
The first case is to avoid a creating a list, which is then converted
to a set only to throw away the list again. Here we can just create
the set right away without a list inbetween.
The second case is "if x in [...]:" is better written as "if x in
{...}:" as sets provides faster "__contains__" (assuming you are on a
"recent enough python3", which britney is).
Signed-off-by: Niels Thykier <niels@thykier.net>
Arguable, this is not a problem in the code as the failure case
invokes sys.exit. However, this is more future proof as the sys.exit
may be replaced (or we may later catch another exception that is
"recoverable").
Signed-off-by: Niels Thykier <niels@thykier.net>
- revert most of commit adbe6d5 as checking the version in testing doesn't work
when other packages migrate and cause regressions
- Alternative way of determining if a package is regressing, by comparison to a
reference set. The reference set is to be created by a holy trigger that
doesn't take packages from the base suite, but instead tests in the testing
suite. This reference needs a retry when a package causing regression
migrates nevertheless, e.g. due to hints or to bounty/penalty policy.
This commit rewrites the make-shift "log" methods to use the logging
framework without requiring changes to the callers. This will be done
in a latter commit to keep things reviewable.
Signed-off-by: Niels Thykier <niels@thykier.net>
The other item includes bug numbers so this one is less useful.
Closes: https://bugs.debian.org/886509
Signed-off-by: Niels Thykier <niels@thykier.net>
The PiupartsPolicy does not use the report URL for testing, so we do
not need to store it in memory. Unfortunately, the logic was broken
and the discard did not happen.
Signed-off-by: Niels Thykier <niels@thykier.net>
If a package had:
Build-Depends: foo [i386], uninstallable-pkg
Then the excuses for amd64 would point to "foo [i386]" rather than
"uninstallable-pkg".
Signed-off-by: Niels Thykier <niels@thykier.net>
Add a new "BuildDependsPolicy" that will check the satisfiability of
the build-dependencies listed in the Build-Depends and
Build-Depends-Arch fields. This enables gating of packages based on
missing / broken build-dependencies.
There are some limitations:
* Build-Depends-Indep is ignored for now. Missing or broken packages
listed in Build-Depends-Indep will be continue to be silently
ignored.
* Being a policy check, it does not enforce "self-containedness" as
a package can still migrate before a build-dependency. However,
this can only happen if the build-dependency is ready to migrate
itself. If the build-dependency is not ready (e.g. new RC bugs),
then packages build-depending on it cannot migrate either (unless
the version in testing satisfies there requirements).
Signed-off-by: Niels Thykier <niels@thykier.net>
Ivo De Decker
Paul Gevers
Steve Langasek
Niels Thykier
Mattia Rizzolo
Iain Lane
Paul Wise