Cleanly split doop_source into a (small) part about source packages
and a (longer) part about binary packages.
Signed-off-by: Niels Thykier <niels@thykier.net>
Coverage suggests that the conditions are always true. If so, we can
replace the "elif" with a regular "if" - but for now, lets keep an
assert.
Signed-off-by: Niels Thykier <niels@thykier.net>
Add a check to ensure we do not create broken test data (as happened
recently, where the architecture values were swapped for a single
binary). A cold hard failure like this is easier to debug
compared to Britney playing "garbage in, garbage out".
Signed-off-by: Niels Thykier <niels@thykier.net>
Add some "no cover" to some unrecoverable exceptions
(e.g. misconfiguration) or base-class methods that are not intended to
be invoked.
Signed-off-by: Niels Thykier <niels@thykier.net>
This gives Policies the opportunity to see if a previous check
(build/installability) or earlier policies already invalidated the update. This
allows writing policies that work on groups of packages, or skipping expensive
checks (such as triggering autopkgtests while the package is not built or
installable yet).
Signed-off-by: Niels Thykier <niels@thykier.net>
Why duplicate that in the configuration when Britney can just pull it
in the Release file for testing? :)
Closes: Debian/britney2#11
Signed-off-by: Niels Thykier <niels@thykier.net>
As a side effect, remove mips64el from NEW_ARCHES as we no longer need
that as a work around.
Closes: Debian/britney2#12
Signed-off-by: Niels Thykier <niels@thykier.net>
It is unwieldy to have one half of output data generation in the policy but not
the other half of updating the excuse. Now that apply_policy() gets the excuse
object as argument we can move everything there.
Signed-off-by: Niels Thykier <niels@thykier.net>
This allows tests to check whether there are any missing builds or old
binaries, so that expensive actions such as "trigger an autopkgtest" are not
done too early/in vain.
Signed-off-by: Niels Thykier <niels@thykier.net>
For future policies such as running autopkgtests it is important to know
whether a package has built, so that expensive actions such as "trigger an
autopkgtest" are not done too early/in vain.
This requires dropping the "age != 0" check for adding the out-of-date-ness to
the Excuse, as the policies now run later. But this check only applied to an
infinitesimal age, and even with age == 0 it is still a valid excuse that there
are missing binaries.
Signed-off-by: Niels Thykier <niels@thykier.net>
FAILED/SUCCESS lines would be separated by a whitespace from the list
of architectures, but not itself followed by whitespace. This is slightly
confusing, as one could interpret it as being a heading for the following
block of tested packages, rather that the final result of the previous
block.
This includes refining "HINTS_ALL" to cover all hints added at
runtime.
Currently, it is not very useful. However, a later commit will allow
policies to use this feature.
Signed-off-by: Niels Thykier <niels@thykier.net>
For items not having an age requirements (e.g. urgency=critical)
always list the "missing build" note if present.
Signed-off-by: Niels Thykier <niels@thykier.net>
This hint will block all "new" source migrations. Source migrations
for packages already in testing will be affected by this. As usual,
this hint can be overruled by an unblock hint.
Closes: GH#8
Signed-off-by: Niels Thykier <niels@thykier.net>
Reduce a "loop over all valid items" to a "loop over an item's
dependencies + reverse dependencies". For sparse graphs, this
is much more efficient.
Signed-off-by: Niels Thykier <niels@thykier.net>
python-apt's "TagFile" seems to cut the field short if it meets a
comment. Therefore allow comments in the actual field value to avoid
this nasty behaviour.
Signed-off-by: Niels Thykier <niels@thykier.net>
Packages in main should not need them. Presuming we eventually make
Britney enforce separation between components, "non-free" seems like a
better default.
Signed-off-by: Niels Thykier <niels@thykier.net>
If there is a regression in "present-and-installable" constraints (on
non-break architectures), then discard the item even if the nuninst
counters have improved.
Signed-off-by: Niels Thykier <niels@thykier.net>
For (non-hint) migrations, split the set of affected into two
parts; one for reverse dependencies and one "negative dependencies"
(e.g. Conflicts).
If there are only regressions in the nuninst after checking the first
set, then there is no reason to continue with the second set (as
"negative dependencies" can only make it worse at that point).
Signed-off-by: Niels Thykier <niels@thykier.net>
Ideally we would reject all items with known unsatisfiable
dependencies as they would not be installable in testing. However,
there are a few known corner cases where we still want to migrate them
(notably when they are already broken in testing).
This commit is an attempt to weed out some of the "obviously" broken
items that will not successfully migrate.
Signed-off-by: Niels Thykier <niels@thykier.net>
In the next commit, the binary packages will be turned into
namedtuples and will become immutable objects.
Signed-off-by: Niels Thykier <niels@thykier.net>
In an architecture only migration, we currently see two notes for
every arch:all packages:
* Ignoring "new" arch:all package
* Ignoring "removal" of arch:all package
But a closer look at the situation is that the arch:all packages is
generally the same in both testing and the source suite.
This commit removes these notes when the arch:all is the same in both
suites.
Signed-off-by: Niels Thykier <niels@thykier.net>
There was an implicit assumption in Britney that was violated recently
(presumbly in 12d9ae8). This assumption is that a source package will
reference the *latest* version of an arch:all package; even if that
version is not built by that source (version).
Image the following packages:
* p-all/1 and p-all/2 are arch:all packages.
- p-all/1 is generally superseded by p-all/2 except it has not
been removed yet (regardless of why that happens).
* src-a/2 builds p-all/2.
When Britney loads the packages (assuming packages are sorted[1]):
* p-all/1 is read and associated with src-a/2 (despite not being
built from that source)
* p-all/2 is read afterwards. It would replace p-all/1 and be
assoicated with src-a/2 as well.
- Prior to 12d9ae8, the assoication would be "pkg, arch" and
therefore the same.
- In 12d9ae8a and after, the association would include version
and both would be associated at the same time.
* Since p-all/1 is discarded, it is never added to the
installability tester (nor the solver). It then promptly throws
an exception when asked to deal with a package it believe does
not exist.
- This can trivially be solved by removing the association
between p-all/1 and src-a/2 when p-all/2 is read.
So far so good. Unfortunately, this adds another another problem!
When Britney migrates a binNMU, she currently removes *all* binary
packages on that architecture - *including* the arch:all packages.
This works out in the end because the arch:all packages be re-added
since they associated with the source package.
At first glance, this appears to be a non-issue until you add
hijacking to the mix. This is *exactly* the case behind #709460.
However, now we end up in a situation where the arch:all package is
removed during binNMU and not re-added (because the arch:all package
is not associated with *that* source any more).
This commit fixes all of the above by:
1) Correctly disassociating superseded arch:all packages from their
source packages
2) Falsely associating the "new" arch:all package to the source that
provided the superseded arch:all package (to avoid issues with
binNMUs and hijacked packages).
- This should probably be undone at some point when Britney does
not need this "hack" any more.
This fix is currently relying on the packages being sorted. But so
did the fix for #709460 (see [1] again). It might be prudent to fix
that as well (or at least reject the packages file rather than produce
"random" results).
[1] Which is another assumption we have been relying and that is
currently satisfied.
Signed-off-by: Niels Thykier <niels@thykier.net>
* A package being removed is *not* affected
- It will just be filtered out later in "check_packages"
* Since all transitive reverse dependencies will be added to
"affected" at the end of the method, there is no reason to
find the immediate set of reverse dependencies of a package
if the package is added itself as well.
Signed-off-by: Niels Thykier <niels@thykier.net>
Adds a --components command line argument (and corresponding config file
option). If specified, package info is expected to be in the usual Debian
mirror layout, ie:
testing/source/Sources
testing/binary-${ARCH}/Packages
(nthykier: Squashed, rebased and did some porting to Python3)
Signed-off-by: Niels Thykier <niels@thykier.net>
A package can have old cruft no longer in testing, which can't
migrate because it depends on old libraries or packages that
aren't in testing anymore, preventing migration.
There's no point in trying to migrate old cruft that has already
been removed in testing anyway, so don't do that.
Signed-off-by: Emilio Pozuelo Monfort <pochu@debian.org>
Signed-off-by: Niels Thykier <niels@thykier.net>
The "include_hijacked" parameter of "_compute_groups" was always
false, so there was little point in having it.
Signed-off-by: Niels Thykier <niels@thykier.net>
The same_source is supposed to compare two versions and (if needed)
"massage" a binNMU version into a source version. This extra feature
of same_source happens to be unused (and not generally applicable):
1) We always compare two source versions, so there is never a
binNMU version in the first place.
2) binary versions are *not* always equal to their source version
(even with the binNMU suffix stripped). This happens when
packages use "dpkg-gencontrol -v<version>".
Note this causes results from some live-data tests to change, because
there has been a sourceful upload with a binNMU version. It was
intended as a binNMU, but was uploaded with the source as well. As
Britney no longer works around this issue, it makes her remove the
affected packages in the end (as their source version does not match
the version in testing).
Signed-off-by: Niels Thykier <niels@thykier.net>
This bug involves a corner case that involves:
* source orig providing liborig1 and orig-doc in testing
* source orig providing liborig2 and orig-doc in unstable
* source hijack providing liborig2 and orig-doc in both testing and unstable,
where the versions of hijack's binaries has a higher version than those of
"orig".
The arch:all packages are needed to trigger this, because Britney
flags an arch:any package as "out of date" and stops the migration
there. However, she is more lenient with arch:all packages.
What happens is that Britney realises that src:orig need to be updated
in testing (to remove liborig1). This leaves src:orig with no
binaries left in testing (as the orig-doc from hijack is used) and it
is therefore removed as an obsolete source. The obsolete removal then
exploded because Britney was also trying to remove the liborig1
package, which is no longer there.
Signed-off-by: Niels Thykier <niels@thykier.net>
With this patch, Britney will correctly parse (and deparse) a
versioned Provides. Furthermore, she will allow it to satisfy any
unversioned dependency on the provided package.
This is the easy half of #786803.
Signed-off-by: Niels Thykier <niels@thykier.net>
It can be used to A) to make the mismatch check more efficient and B)
share identical binaries between suites.
Signed-off-by: Niels Thykier <niels@thykier.net>
Previously whether such packages received excuses, and the specific
content of such excuses, was dependent on the order in which their
binary packages were considered.
Signed-off-by: Adam D. Barratt <adam@adam-barratt.org.uk>
This made iter_packages_hint a thin wrapper around try_migration, so
it was inlined into its only caller "do_all".
Signed-off-by: Niels Thykier <niels@thykier.net>
The callers of get_dependency_solvers need to do those table lookups
anyway. By moving it out, it is now possible to reuse the results.
Signed-off-by: Niels Thykier <niels@thykier.net>
Rely on the Installability tester to locate all of the affected
packages plus their transitive reverse dependencies. As the
InstallabilityTester is suite agnostic, the set of affected packages
now includes (versions of) packages not in testing, which is filtered
out during the check.
Signed-off-by: Niels Thykier <niels@thykier.net>
If we have an up-to-date arch all package available for this architecture,
that doesn't mean this architecture has an up-to-date build. We need an
architecture specific up-to-date package for that.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
For fucked architectures, binaries from older versions are allowed to be in
testing, so we only remove them if they are gone from unstable.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
Move nuninst cloning out of the check loop and always populate the new
nuninst entirely.
This will allow some simplifications in other places.
Signed-off-by: Niels Thykier <niels@thykier.net>
Use a set to filter out seen items to avoid doing O(n^2)
de-duplication. For very large hints, this can take considerable
time.
Using "seen_items" to build the actual hints on the (unverified)
assumption that Python can do something "smart" to turn a set into a
frozenset faster than it can with a list.
Signed-off-by: Niels Thykier <niels@thykier.net>
Britney is now smart enough to produce the same result from hints
regardless of the order of the items in the hint. With this in mind,
we can have the original auto-hinter produce hints as sets and filter
out duplicates as we produce them.
Note that the hints are sorted to produce deterministic output (to
make it easier to compare the hints between runs and changes).
Signed-off-by: Niels Thykier <niels@thykier.net>
Avoid some cases of O(n^2) behaviour in sort_actions and reduce the
size of n for the remaining O(n^2)-ish behaviour by filtering out
removals early on.
Signed-off-by: Niels Thykier <niels@thykier.net>
It's more natural to say "check this package if the current arch is in
this list" than "do not check this package if the current arch is not
in this list"
Signed-off-by: Adam D. Barratt <adam@adam-barratt.org.uk>
This updates the doop_source and _compute_groups functions so
that binary packages that are built from a different source
aren't included as part of an update to the original source.
In the event that it's a binary-only update, also don't remove
the hijacked packages from testing.
This change also removes an obsolete comment regardarding pre-conditions
for the _compute_groups function.
The live-data tests rely on an inconsistency, since they were before
Britney started to record the Essential field.
Signed-off-by: Niels Thykier <niels@thykier.net>
britney assumes that a package build is uniquely described by its
name, version and architecture. Particularly when constructing
Packages files by hand for testing purposes this assumption can be
violated, leading to confusing behaviour. This change makes britney
look for such mismatches, and report if any are found.
Notably:
* Avoid repeated calls frozenset(X), where we can trivially do
without.
* Skip the inner loop, when "i" is in "to_skip".
* Use a set rather than a list for "to_skip" as we do more
membership tests.
Signed-off-by: Niels Thykier <niels@thykier.net>
The get_dependency_solvers method returns a (boolean, list)-tuple, but
the boolean can always be implied from the list (in boolean context).
Signed-off-by: Niels Thykier <niels@thykier.net>
cmp is gone in python3. Also add a sorting method to Excuse that is
compatible with its __eq__/__hash__ methods.
Signed-off-by: Julien Cristau <jcristau@debian.org>
It doesn't exist in python3, but 1000 days should be safe enough as a
fallback for a package without urgency.
Signed-off-by: Julien Cristau <jcristau@debian.org>
- split the one-liner into a for and an if
- use open() as a context manager
- don't use string.strip which is gone in python3
Signed-off-by: Julien Cristau <jcristau@debian.org>
The "remark" hint is only intended for showing up in the output of "d"
(or via hint grep). It has no effect on Britney's behaviour.
Admittedly, the original code would have ignored it as well. But this
change makes it explicit and not simply a "ignored due to insufficient
permissions".
Signed-off-by: Niels Thykier <niels@thykier.net>
For out-of-date binaries, generate different excuses when the build is
missing, or when old (cruft) binaries for previous version are still around.
Signed-off-by: Ivo De Decker <ivodd@debian.org>
As part of a migration, we remove all the existing binaries built by
the source (possibly on a particular architecture) from testing; this
includes architecture-independent binary packages. However, when a
binNMU is in *pu, only the arch-dependent binary pakcages are present.
As a result, after the migration the architecture-independent packages
are no longer present in testing. This usually isn't a practical
problem, as dak will re-add them when it generates the packages files.
It is, however, wrong and will break if a source migration is tempted
during the same run as (and after) the *pu binary migration happened.
The simple fix is to not remove the architecture-independent packages
when performing such migrations.
Signed-off-by: Adam D. Barratt <adam@adam-barratt.org.uk>
In the rare case that a hint removed an uninstallable binary, the
binary could still be included in the nuninst counter.
Regression introduced in a46dd88.
Signed-off-by: Niels Thykier <niels@thykier.net>
sort_actions() can be quite expensive and it is wasteful to resort
actions after each successful "easy"-hint.
Signed-off-by: Niels Thykier <niels@thykier.net>
There are no uses of "lundo" left for a non-hint recurse run (i.e.
the "main run"), so there is no point in building it.
The "lundo"-list is still used in the recurse run of a "hint"-hint.
Signed-off-by: Niels Thykier <niels@thykier.net>
The "do_all"-method now checks the architectures of all changes
applied. If they entirely consist of items from "break archs", then
"do_all" will disregard the current "break archs" setting when
comparing nuninst counters.
This change avoids unintended installability regressions on break
arches when a hint (manual or automatic) apply only to packages on
break arches.
Signed-off-by: Niels Thykier <niels@thykier.net>
Rename local variables and avoid repeated chained lookups. In
particular, avoid confusing cases like:
[...]
version = binaries[parch][0][binary][VERSION]
[...]
binaries[parch][0][binary] = self.binaries[item.suite][parch][0][binary]
version = binaries[parch][0][binary][VERSION]
Where "version" here will refer to two different versions. The former
the version from testing of a hijacked binary and the latter the
version from the source suite (despite the look up using the "testing"
table, due to the testing copy being updated).
Notable renamings:
* binaries => packages_t (a.k.a. self.binaries['testing'])
* binaries[parch][0] => binaries_t_a
* binaries[parch][1] => provides_t_a
* Similar naming used for "item.suite" instead of "testing"
The naming is based on the following logic:
* self.binaries from "packages" files
(by this logic, it ought to be "self.packages", but that is
for later)
* The "_X_a" is short for "[<suite>][<parch>]" look ups.
* binaries_X_a and provides_X_a are the specialised parts of
packages_X_a that deal with (real) binary packages and
provides (i.e. virtual packages) respectively.
Signed-off-by: Niels Thykier <niels@thykier.net>
Extract a specialised iter_packages_hint from iter_packages that only
deals with applying hints. This simplifies iter_packages AND avoids
having to re-compute the uninstallability counters after each single
item in the hint.
This means that a hint can now avoid triggering expontential runtime
provided only that the "post-hint" stage does not trigger expontential
runtime. Previously the hint had to be ordered such that none of the
items in the hint caused such behaviour (if at all possible).
Signed-off-by: Niels Thykier <niels@thykier.net>
Avoid creating two dependency clauses for dependencies emulating a
"version range" a la:
Depends: pkg-a (>= 2), pkg-a (<< 3~)
Previously this would create two clauses a la:
- (pkg-a, 2, arch), (pkg-a, 3, arch)
- (pkg-a, 1, arch), (pkg-a, 2, arch)
However, it is plain to see that only (pkg-a, 2, arch) is a valid
solution and the other options are just noise. This patch makes
Britney merge these two claues into a single clause containing exactly
(pkg-a, 2, arch).
Signed-off-by: Niels Thykier <niels@thykier.net>
The "new" auto hinter relies on partial ordering to determine, when
what can migrate (and what needs to migrate at the same time). At the
same time, it leverages on "_compute_groups" to allow it to include
"removals" in its hints.
Signed-off-by: Niels Thykier <niels@thykier.net>
Avoid smooth-updating libraries in hints, when all of their reverse
dependencies will certainly disappear in the same hint.
Note that in "hint"-hint, reverse dependencies removed in the
following "full run" will not cause the smooth-updated library to be
removed. Instead these will still be removed in the end as usual, but
in some cases that is too late.
Signed-off-by: Niels Thykier <niels@thykier.net>
Rename find_upgraded_binaries into _compute_groups. The new method
will also compute what binaries will be updated in or added to testing
after migration.
Signed-off-by: Niels Thykier <niels@thykier.net>
Based on Colin Watson's code to do the same from the "britney2-ubuntu"
repository[1] revision 306, 308 and 309.
Notable differences include:
* output include version of source package being removed
* output prefix removals with a "-" (otherwise it would be identical to
a upgrade/new source with the change above).
[1] http://bazaar.launchpad.net/~ubuntu-release/britney/britney2-ubuntu/revision/306
Signed-off-by: Niels Thykier <niels@thykier.net>
This introduces a new variable HINTSDIR, which overrides the location of the
Hints dir (normally it is read from the UNSTABLE dir).
Please note that this is the location of the dir that contains the Hints dir.
Signed-off-by: Niels Thykier <niels@thykier.net>
This introduces a new variable OUTPUTDIR, which overrides the location where
the new dates file is written. This allow to run britney against a read-only
copy of the data.
Signed-off-by: Niels Thykier <niels@thykier.net>
Set the default to maxint until we've read something.
Reported-by: Ivo De Decker <ivo.dedecker@ugent.be>
Signed-off-by: Julien Cristau <jcristau@debian.org>
The new Installability Tester (IT) module replaces the remaining
C-parts. Unlike C-implementation, it does not give up and emit an
"AIEEE" half-way through.
In order to determine installability, it uses two sets "musts" and
"never". As the names suggest, the sets represents the packages that
must be (co-)installable with the package being tested and those that
can never be co-installable. For a package to be installable, "musts"
and "never" have remain disjoint.
These sets are also used to reduce the number of alternatives that are
available to satisfy a given dependency. When these sets are unable
to remove the choice completely, the new IT defers the choice to later.
This occasionally reduces backtracking as a later package may conflict
or unconditionally depend on one of the remaining alternatives.
Signed-off-by: Niels Thykier <niels@thykier.net>
We stopped populating the element with real data some time ago, it's
time to drop it entirely.
Signed-off-by: Adam D. Barratt <adam@adam-barratt.org.uk>
Multiarch adds a Depends: foo:any syntax, permitted only if the
target of the dependency is "Multi-Arch: allowed". This has
been supported by dpkg and apt for some time and is now safe to
use in unstable.
[Adam D. Barratt: adjusted to use consts.py]
Signed-off-by: Adam D. Barratt <adam@adam-barratt.org.uk>
As HintItem is now redundant, also replace it with a new class -
UnversionnedMigrationItem - and migrate users of the classes to use
the new versions.
Signed-off-by: Adam D. Barratt <adam@adam-barratt.org.uk>
Where possible, avoid creating a list only to discard immediately
afterwards. Example:
"""
for x in sorted([x for x in ...]):
...
"""
Creates a list, passes it to sorted, which generates a new list and
sorts that copy. Since sorted accepts an iterable, we can avoid the
"inner" list and just pass it a generator expression instead.
Signed-off-by: Niels Thykier <niels@thykier.net>
Niels Thykier
Martin Pitt
Colin Watson
Robert Bruce Park
Mathieu Trudel-Lapierre
Julien Cristau
Anthony Towns
Emilio Pozuelo Monfort
Paul Wise
Adam D. Barratt
Ivo De Decker
Cyril Brulebois
Ivo De Decker
Colin Watson