1107 Commits

Author SHA1 Message Date
Niels Thykier
e92bbf386f britney_util: Replace broken if with an assert
Replace a "if" with an unconditionally false condition with a negative
assert of the "correct" condition.  The bug has been present for 2½
years with no known adverse effects.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:38 +00:00
Niels Thykier
ec37bd06aa Optimise checking of affected packages
For (non-hint) migrations, split the set of affected into two
parts; one for reverse dependencies and one "negative dependencies"
(e.g. Conflicts).

If there are only regressions in the nuninst after checking the first
set, then there is no reason to continue with the second set (as
"negative dependencies" can only make it worse at that point).

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:38 +00:00
Niels Thykier
a4daee0866 britney.py: Optimise scheduling a bit
Avoid rescheduling the tail of packages if we can avoid it.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:38 +00:00
Niels Thykier
ac66e3118d Reject some excuses with unsatisfiable depends
Ideally we would reject all items with known unsatisfiable
dependencies as they would not be installable in testing.  However,
there are a few known corner cases where we still want to migrate them
(notably when they are already broken in testing).

This commit is an attempt to weed out some of the "obviously" broken
items that will not successfully migrate.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:38 +00:00
Niels Thykier
8991889fad Prefer pkg.foo to pkg[FOO] for binary packages
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:38 +00:00
Niels Thykier
4eebb5180f Make binary packages a namedtuple
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:37 +00:00
Niels Thykier
7230cd29c0 Re-order parsing of suites to avoid changing binary pkgs
In the next commit, the binary packages will be turned into
namedtuples and will become immutable objects.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:37 +00:00
Niels Thykier
c4e7596837 read_pkgs: Parse Provides a bit earlier
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:37 +00:00
Niels Thykier
946d705e2b read_pkgs: Refactor source+src_version parsing
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:37 +00:00
Niels Thykier
71a29aa264 excuse: present dependency info in a machine parseable way
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:37 +00:00
Niels Thykier
eb2c6aad27 Remove redundant notes about arch:all packages
In an architecture only migration, we currently see two notes for
every arch:all packages:

 * Ignoring "new" arch:all package
 * Ignoring "removal" of arch:all package

But a closer look at the situation is that the arch:all packages is
generally the same in both testing and the source suite.

This commit removes these notes when the arch:all is the same in both
suites.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:37 +00:00
Niels Thykier
5d31eee805 Add block/unblock-hint info to excuses.yaml
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:37 +00:00
Niels Thykier
1028860c53 Move missing-builds/cruft to excuses
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:37 +00:00
Niels Thykier
7b8b9e8258 excuses.yaml: Distinguish between source and item names
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:37 +00:00
Niels Thykier
1400fa1384 Remove "Not considered" note from excuses.yaml
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:36 +00:00
Niels Thykier
a7e493c64d excuses.yaml: Make maint + component info machine parsable
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:36 +00:00
Niels Thykier
26c6f3eb87 Add policy-info to YAML excuses
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:36 +00:00
Niels Thykier
63b08f8278 Optimise a few hints.search calls
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:36 +00:00
Niels Thykier
ce5b893a19 Prefer bugs files from the state-dir
Closes GH#2.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:36 +00:00
Niels Thykier
7e94e5567c Prefer Urgencies file from the state-dir
Partly solves GH#2

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:36 +00:00
Niels Thykier
bcff800040 Move Dates into a new state-dir
Partly solves GH#2.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:36 +00:00
Niels Thykier
6d6a7ac529 Move RC bug handing into a policy
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:36 +00:00
Niels Thykier
5ec3aea43a Move age-handling into a separate file
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-05-21 12:43:36 +00:00
Niels Thykier
2e81e55c56 undo_items: Gracefully handle skipped cruft items in undo
In 203466b, we stopped migrating cruft items if they were not in
testing already.  However, if the migration failed we would still
attempt to undo it from testing, which (unsurprisingly) does not work
very well.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-27 17:11:34 +00:00
Niels Thykier
4ced8a3d41 britney.py: Fix crash with arch:all packages being shadowed
There was an implicit assumption in Britney that was violated recently
(presumbly in 12d9ae8).  This assumption is that a source package will
reference the *latest* version of an arch:all package; even if that
version is not built by that source (version).

Image the following packages:

  * p-all/1 and p-all/2 are arch:all packages.
    - p-all/1 is generally superseded by p-all/2 except it has not
      been removed yet (regardless of why that happens).
  * src-a/2 builds p-all/2.

When Britney loads the packages (assuming packages are sorted[1]):

 * p-all/1 is read and associated with src-a/2 (despite not being
   built from that source)
 * p-all/2 is read afterwards.  It would replace p-all/1 and be
   assoicated with src-a/2 as well.
   - Prior to 12d9ae8, the assoication would be "pkg, arch" and
     therefore the same.
   - In 12d9ae8a and after, the association would include version
     and both would be associated at the same time.
 * Since p-all/1 is discarded, it is never added to the
   installability tester (nor the solver).  It then promptly throws
   an exception when asked to deal with a package it believe does
   not exist.
   - This can trivially be solved by removing the association
     between p-all/1 and src-a/2 when p-all/2 is read.

So far so good.  Unfortunately, this adds another another problem!

When Britney migrates a binNMU, she currently removes *all* binary
packages on that architecture - *including* the arch:all packages.
This works out in the end because the arch:all packages be re-added
since they associated with the source package.

  At first glance, this appears to be a non-issue until you add
hijacking to the mix.  This is *exactly* the case behind #709460.
However, now we end up in a situation where the arch:all package is
removed during binNMU and not re-added (because the arch:all package
is not associated with *that* source any more).

This commit fixes all of the above by:

 1) Correctly disassociating superseded arch:all packages from their
    source packages
 2) Falsely associating the "new" arch:all package to the source that
    provided the superseded arch:all package (to avoid issues with
    binNMUs and hijacked packages).
    - This should probably be undone at some point when Britney does
      not need this "hack" any more.

This fix is currently relying on the packages being sorted.  But so
did the fix for #709460 (see [1] again).  It might be prudent to fix
that as well (or at least reject the packages file rather than produce
"random" results).

[1] Which is another assumption we have been relying and that is
currently satisfied.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-12 18:18:16 +00:00
Niels Thykier
8ab7fc0e89 Avoid some unnecessary effort in compuing "affected"
* A package being removed is *not* affected
   - It will just be filtered out later in "check_packages"
 * Since all transitive reverse dependencies will be added to
   "affected" at the end of the method, there is no reason to
   find the immediate set of reverse dependencies of a package
   if the package is added itself as well.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:28 +00:00
Niels Thykier
37b6536cfd Remove two redundant ifs duplicating an earlier if
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
2b5c48ad2f Remove redundant assert
We asserted that a list of packages was a subset of the "affected"
set.  This was a given, since the list was seeded *only* with packages
from affected (and only if their installability changed).

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
eb24cbcd67 Remove unused named parameter
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
38a198e8b2 Replace some string concat+split with tuples
Replace some of the:

 p = binary + "/" + parch
 binary, parch = p.split("/")

with regular tuples.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
1ddf63e14b britney.py: Remove uncommented printf debugging
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
28f69fa505 britney.py: Log when old libs are present but not removed
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
b5db57715b Move two installability testing functions to britney_util
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
05de911090 Make excuses a dict rather than a list
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
9b9d44db3a Move some of the hint parsing into hints.py
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
5083311ec2 britney.py: Rename __log to log
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
7b0d4023d0 britney.py: Remove redundant open before TagFile
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
5bbca73391 Make components and --control-files mutually exclusive
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Anthony Towns
496e48c9a7 britney.py: Add support for multiple components
Adds a --components command line argument (and corresponding config file
option). If specified, package info is expected to be in the usual Debian
mirror layout, ie:

   testing/source/Sources
   testing/binary-${ARCH}/Packages

(nthykier: Squashed, rebased and did some porting to Python3)
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
e2b98872cf Refactor some local expressions
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
7e475cd9f9 britney.py: assert provides table is up to date
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
3ff01b5ec9 britney.py: Avoid creating empty lists for provides
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Emilio Pozuelo Monfort
203466bb51 Fix migration for sources with old cruft that isn't in testing
A package can have old cruft no longer in testing, which can't
migrate because it depends on old libraries or packages that
aren't in testing anymore, preventing migration.

There's no point in trying to migrate old cruft that has already
been removed in testing anyway, so don't do that.

Signed-off-by: Emilio Pozuelo Monfort <pochu@debian.org>
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:27 +00:00
Niels Thykier
842a97adbd britney_util: Remove some unused imports
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:26 +00:00
Niels Thykier
efd3359aba britney.py: Remove unused named parameter
The "include_hijacked" parameter of "_compute_groups" was always
false, so there was little point in having it.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:26 +00:00
Niels Thykier
22cef671a5 Refactor some code to avoid unnecessary table lookups/checks
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:26 +00:00
Niels Thykier
12d9ae8fa3 Use pkg_id instead pkg/arch in BINARIES
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:26 +00:00
Niels Thykier
aa34a47f8a Remove all calls to "same_source" - they were overkill
The same_source is supposed to compare two versions and (if needed)
"massage" a binNMU version into a source version.  This extra feature
of same_source happens to be unused (and not generally applicable):

 1) We always compare two source versions, so there is never a
    binNMU version in the first place.
 2) binary versions are *not* always equal to their source version
    (even with the binNMU suffix stripped).  This happens when
    packages use "dpkg-gencontrol -v<version>".

Note this causes results from some live-data tests to change, because
there has been a sourceful upload with a binNMU version.  It was
intended as a binNMU, but was uploaded with the source as well.  As
Britney no longer works around this issue, it makes her remove the
affected packages in the end (as their source version does not match
the version in testing).

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:26 +00:00
Niels Thykier
179ced304d britney: Work around bug 815995
This bug involves a corner case that involves:

 * source orig providing liborig1 and orig-doc in testing
 * source orig providing liborig2 and orig-doc in unstable
 * source hijack providing liborig2 and orig-doc in both testing and unstable,
   where the versions of hijack's binaries has a higher version than those of
   "orig".

The arch:all packages are needed to trigger this, because Britney
flags an arch:any package as "out of date" and stops the migration
there.  However, she is more lenient with arch:all packages.

What happens is that Britney realises that src:orig need to be updated
in testing (to remove liborig1).  This leaves src:orig with no
binaries left in testing (as the orig-doc from hijack is used) and it
is therefore removed as an obsolete source.  The obsolete removal then
exploded because Britney was also trying to remove the liborig1
package, which is no longer there.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:26 +00:00
Niels Thykier
6965dda5f7 Support versioned provides (without multi-arch)
Closes: #786803

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-04-11 16:27:26 +00:00