Currently no policy implements this.
A policy can now implement apply_src_policy_impl or apply_srcarch_policy_impl
(or both), so apply_src_policy_impl is no longer an abstractmethod.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
apply_src_policy expects an excuse with a new source and binaries. It doesn't
apply to srcarch excuses, which only have new binaries for an existing source.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
This is a step towards making migration unit-testable. This step
reduces the need for global state (in the MigrationItem class as class
fields) and with another step we can remove the global state entirely
and enable unit tests to create migration items without having to
worry about other unit tests.
Signed-off-by: Niels Thykier <niels@thykier.net>
All types of dependencies between excuses (Depends, Build-Depends,
Build-Depends-Indep, ...) are handled by the same code. The DependencyType is
used to distinguish between the types where needed.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
The other fields related to invalidation are currently not deterministic and
depend on the order in which the excuses are invalidated.
The new invalidated-by-other-package field is just a boolean which shows the
excuse was invalidated by some other package for some reason (without
specifying the reason). It does not depend on the order of processing and
should be deterministic.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
In Debian, the same override is applied to both suites and they are
always consistent. It's not the case in Kali, we keep the value from
Debian when we import the package in britney's source distribution, but
if the same version is already present in the target distribution, it
keeps its original section (dating back to its initial import). In those
situations, the code will fail with an error like this one:
E: [2018-12-28T19:57:57+0000] - Mismatch found coinor-libdylp0 1.6.0-1.1 amd64 differs
I: [2018-12-28T19:57:57+0000] - ... section libs != science
[...]
ValueError: Inconsistent / Unsupported data set
Commit 7efa865a04 which was supposed to
move code around introduced the check on this field. Prior to this, the
section was not checked. Since the section only has an impact on which
packages take part to the smooth updates feature, the impact of such a
mismatch is negligible so I simply dropped that check.
This is useful to run tests with the data files from a specific point in time,
without changes due to ageing when the test runs later.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
Have the transaction code verify that there is at most one active
child at the time and no one is using the parent while child is
active. This is how the code is intended to be used and also the
code almost certainly does not work otherwise.
The new code does not cover commiting/rolling back a parent before a
child but that is already covered by the existing code (it will
trigger when child transaction is rolled back/committed or when
leaving the contextmanager from start_transaction).
This would have caught 7d758760d1
immediately with an assertion error.
Signed-off-by: Niels Thykier <niels@thykier.net>
Correct the return value of current_transaction that treated the
_transaction field incorrectly as a queue rather than a stack like
everything else. This completely broke the ability to commit and
rollback child transactions (correctly). Fortunately, it could only
trigger on a "hint"-hint.
Signed-off-by: Niels Thykier <niels@thykier.net>
When we try to compute_groups for a group which has a source or a binary with
a lower version than testing, throw an exception. In the cases where this can
happen, the exception is caught. In other cases, it is not and it serves as an
assert.
This can only happen when there are multiple candidates (from multiple suites)
changing the same source or binary.
This should fix the ordering issues tested in these tests:
- tpu-unstable-binnmu
- binnmu-tpu
- tpu-with-unstable-binnmu
With this change, it should be possible to accept binNMUs from *pu again.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
This isolates the undo handling in the new transaction object and in
doop_source, which currently generates the undo items. This commit
will be a stepping stone to rewriting the undo handling.
Signed-off-by: Niels Thykier <niels@thykier.net>
When calculating smooth updateable binaries, filter out cruft binaries from
unstable, because they will not be part of the set of packages that britney
will try to migrate to testing.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
The parsing of migration items should also look for the suite name in the
architecture part. This fixes the parsing for migration items like
some-src/amd64_tpu and some-src/amd64_tpu/1.0-1
Signed-off-by: Ivo De Decker <ivodd@debian.org>
This commit updates the test suite to use the BinaryPackageUniverse
instead of the InstallabilityTester where that makes sense. The rest
of Britney has yet to be updated except where absolutely necessary (as
that will come in a later commit).
Signed-off-by: Niels Thykier <niels@thykier.net>
The InstallabilityTester is suffering from a lack of clear purpose
because it serves multiple. This commit extracts most of one of these
purposes into the BinaryPackageUniverse class while retaining the
original API of the InstallabilityTester.
Signed-off-by: Niels Thykier <niels@thykier.net>
Currently autopkgtest tries to install our trigger from unstable and the rest
from testing. If that fails, than autopkgtest has a fall-back to allow all
packages from unstable to be installed. This has two severe issues:
1) the version of the test and the package it came from may be out-of-sync
2) too much from unstable may be installed, even stuff that should not/is not
allowed to migrate as it breaks stuff.
Make sure that test depends also get added to triggers if they are broken.
E.g. imagine the following scenario: trigger X changes (breaks) the output
generated by Y. Package Z has Y in the test dependencies and compares the
output in the autopkgtest. We want to have the opportunity that a new version
is automatically fixing the situation.
Two use cases are currently unsupported: needs-build (autopkgtest restriction)
and test dependencies generated by autodep8.
Admittedly, no policy adds them yet so this is currently no-op code.
However, future commits can start to rely on this infrastructure code.
Signed-off-by: Niels Thykier <niels@thykier.net>
Cherry-Pick: 80bf9060de
Cherry-Pick: f32907acea
Cherry-Pick: 9ef496177f86b18d9f910da1360dd773b82f1fb7
Cherry-Pick: b16530a37d
Signed-off-by: Niels Thykier <niels@thykier.net>
Flatten the defaultdict(set) for unsat_deps into a standard dict for output
In case autopkgtest triggering is delayed because the required builds aren't
ready yet or the package is not installable, currently there is only the
message that autopktest delays the migration, but no hint why. This commit adds
these hints.
The initial idea was to do this to bootstrap the baseline, but it turns out
that this has the drawback it triggers runs for a package that has a new
autopkgtest where it didn't have it in the version in the target suite. It was
considered harmless (as it would just have a failing reference), but due to
autodep8, package can have a passing result in the target suite while the new
autopkgtest is actually broken. Such a package should not be blocked / getting
a penalty.
The alternative is to make the check here smarter, but as this is only for
bootstrapping, lets do that outside of britney proper.
When determining whether a policy applies to a given item, use the
suite class rather than the suite name.
Signed-off-by: Niels Thykier <niels@thykier.net>
Into 3 categories:
* target suite ("testing")
* primary source suite ("unstable")
* additional source suites ("pu" and "tpu")
This will be useful for implementing logic working with suites without
basing it on the name of the suite.
Signed-off-by: Niels Thykier <niels@thykier.net>
At the moment, it is just a glorified dict. However, we will
eventually use it to get rid of the hardcoded references to "testing"
etc. all over the code.
Signed-off-by: Niels Thykier <niels@thykier.net>
1) the update didn't happen for all but the first
2) we don't want a package that fixes a regression in unstable to influence the
reference for another package until it actually migrates, so this updating
is flawed.
Given that only one value is defined ("reference"), it is a better
option to allow the config to be unset when one does not want to have
adt_baseline set to "reference".
Signed-off-by: Niels Thykier <niels@thykier.net>
Notable omissions are "pending_tests" and "tests_results". This is
omission is due to these (some times) being initialized from the
output of "json.load" (so we cannot assume defaultdict semantics
without manually imported the data into one).
Signed-off-by: Niels Thykier <niels@thykier.net>
The first case is to avoid a creating a list, which is then converted
to a set only to throw away the list again. Here we can just create
the set right away without a list inbetween.
The second case is "if x in [...]:" is better written as "if x in
{...}:" as sets provides faster "__contains__" (assuming you are on a
"recent enough python3", which britney is).
Signed-off-by: Niels Thykier <niels@thykier.net>
Arguable, this is not a problem in the code as the failure case
invokes sys.exit. However, this is more future proof as the sys.exit
may be replaced (or we may later catch another exception that is
"recoverable").
Signed-off-by: Niels Thykier <niels@thykier.net>
- revert most of commit adbe6d5 as checking the version in testing doesn't work
when other packages migrate and cause regressions
- Alternative way of determining if a package is regressing, by comparison to a
reference set. The reference set is to be created by a holy trigger that
doesn't take packages from the base suite, but instead tests in the testing
suite. This reference needs a retry when a package causing regression
migrates nevertheless, e.g. due to hints or to bounty/penalty policy.
This commit rewrites the make-shift "log" methods to use the logging
framework without requiring changes to the callers. This will be done
in a latter commit to keep things reviewable.
Signed-off-by: Niels Thykier <niels@thykier.net>
The other item includes bug numbers so this one is less useful.
Closes: https://bugs.debian.org/886509
Signed-off-by: Niels Thykier <niels@thykier.net>
The PiupartsPolicy does not use the report URL for testing, so we do
not need to store it in memory. Unfortunately, the logic was broken
and the discard did not happen.
Signed-off-by: Niels Thykier <niels@thykier.net>
If a package had:
Build-Depends: foo [i386], uninstallable-pkg
Then the excuses for amd64 would point to "foo [i386]" rather than
"uninstallable-pkg".
Signed-off-by: Niels Thykier <niels@thykier.net>
Add a new "BuildDependsPolicy" that will check the satisfiability of
the build-dependencies listed in the Build-Depends and
Build-Depends-Arch fields. This enables gating of packages based on
missing / broken build-dependencies.
There are some limitations:
* Build-Depends-Indep is ignored for now. Missing or broken packages
listed in Build-Depends-Indep will be continue to be silently
ignored.
* Being a policy check, it does not enforce "self-containedness" as
a package can still migrate before a build-dependency. However,
this can only happen if the build-dependency is ready to migrate
itself. If the build-dependency is not ready (e.g. new RC bugs),
then packages build-depending on it cannot migrate either (unless
the version in testing satisfies there requirements).
Signed-off-by: Niels Thykier <niels@thykier.net>
We basically use them as sets and do not need to rely on the ordering,
so we might as well just turn them into proper sets.
Signed-off-by: Niels Thykier <niels@thykier.net>
Rewrite _compute_scc to be iterative to avoid call recursion limit for
graphs with long dependency chains.
Signed-off-by: Niels Thykier <niels@thykier.net>
Sometimes the solver would correctly compute which items are grouped
together, but then fail to emit some groups. This is trivially
fixable by forcing an update of "before" + "after" relations.
Admittedly, this looks unnecessary, so it may just be hiding the bug.
Nonetheless, the change makes the new test_solver_simple_scc test
produce the expected result.
Signed-off-by: Niels Thykier <niels@thykier.net>
devscripts' grep-excuses expects each <li> to be on its own line. When
d7a676d074 added the verdict summary,
without an EOL, it broke grep-excuses' ability to search by maintainer.
Signed-off-by: James McCoy <jamessan@debian.org>
Signed-off-by: Niels Thykier <niels@thykier.net>
With this change, Britney can now provide a very brief summary of the
migration via one single value (YAML) or line (HTML). This solves two
issues:
* It provides an aggregated version of the policy decision without
having to loop over all policies (and even those would not give
a full verdict on their own as not all rejections come from
policies)
* It enables a simple way to inform readers of the HTML excuses of
whether a rejection is permanent or not. This should hopefully
make it easier for contributors to understand Britney and react
more pro-actively.
Signed-off-by: Niels Thykier <niels@thykier.net>
Britney has a special case for essential packages to ensure that any
package that with essential packages are not installable. This check
did not account for a case, where a package is not co-installable with
two or more pseudo-essential package part of the same OR dependency.
A contrived example based on real world data:
Package: foo
# Conflict with all providers of "awk"
Conflicts: mawk | gawk | original-awk
This alone is actually not sufficient to trigger the bug, as
_get_min_pseudo_ess_set is in theory some times smart enough to pick
an "obvious" solution between the pseudo-essential option.
When it does, one of the above ends up in the (de-facto) essential set
and then the installability tester correctly rejects "foo".
Though, even with the fix above, the handling for this is probably not
correct if the essential set is not (fully co-)installable. However,
that basically only happens if we are bootstrapping an architecture
(or testing is royally broken, in which case this is the least of our
worries).
Signed-off-by: Niels Thykier <niels@thykier.net>
Add some "no cover" to some unrecoverable exceptions
(e.g. misconfiguration) or base-class methods that are not intended to
be invoked.
Signed-off-by: Niels Thykier <niels@thykier.net>
doop_source never (no longer?) creates an element in "virtual" where
the package name is prefixed with "-", so we can drop that branch.
Signed-off-by: Niels Thykier <niels@thykier.net>
The doop_source method does not generate any /binary/ undo items,
where the package name starts with "-". This fact allows us to remove
a branch in the undo code. This also happens to be fortunate because
the call to "inst_tester.remove_testing_binary" in that branch happens
to be wrong.
Signed-off-by: Niels Thykier <niels@thykier.net>
As a side effect, remove mips64el from NEW_ARCHES as we no longer need
that as a work around.
Closes: Debian/britney2#12
Signed-off-by: Niels Thykier <niels@thykier.net>
Ivo De Decker
Niels Thykier
Raphaël Hertzog
Iain Lane
Paul Gevers
Mathieu Trudel-Lapierre
Paul Wise
James McCoy
Jonathan Wiltshire