britney.py

#!/usr/bin/env python2.4 # -*- coding: utf-8 -*-

# Copyright (C) 2001-2004 Anthony Towns <ajt@debian.org>
#                         Andreas Barth <aba@debian.org>
#                         Fabio Tranchitella <kobold@debian.org>

# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

"""
= Introdution =

This is the Debian testing updater script, also known as "Britney".

Packages are usually installed into the `testing' distribution after
they have undergone some degree of testing in unstable. The goal of
this software is to do this task in a smart way, allowing testing
to be always fully installable and close to being a release candidate.

Britney source code is splitted in two different but related tasks:
the first one is the generation of the update excuses, while the
second tries to update testing with the valid candidates; first 
each package alone, then larger and even larger sets of packages
together. Each try is accepted if testing is not more uninstallable
after the update than before.

= Data Loading =

In order to analyze the entire Debian distribution, Britney needs to
load in memory the whole archive: this means more than 10.000 packages
for twelve architectures, as well as the dependency interconnection
between them. For this reason, the memory requirement for running this
software are quite high and at least 1 gigabyte of RAM should be available.

Britney loads the source packages from the `Sources' file and the binary
packages from the `Packages_${arch}' files, where ${arch} is substituted
with the supported architectures. While loading the data, the software
analyze the dependencies and build a directed weighted graph in memory
with all the interconnections between the packages (see Britney.read_sources
and Britney.read_binaries).

Other than source and binary packages, Britney loads the following data:

  * Bugs, which contains the count of release-critical bugs for a given
    version of a source package (see Britney.read_bugs).

  * Dates, which contains the date of the upload of a given version 
    of a source package (see Britney.read_dates).

  * Urgencies, which contains the urgency of the upload of a given
    version of a source package (see Britney.read_urgencies).

  * Approvals, which contains the list of approved testing-proposed-updates
    packages (see Britney.read_approvals).

  * Hints, which contains lists of commands which modify the standard behaviour
    of Britney (see Britney.read_hints).

For a more detailed explanation about the format of these files, please read
the documentation of the related methods. The exact meaning of them will be
instead explained in the chapter "Excuses Generation".

= Excuses =

An excuse is a detailed explanation of why a package can or cannot
be updated in the testing distribution from a newer package in 
another distribution (like for example unstable). The main purpose
of the excuses is to be written in an HTML file which will be 
published over HTTP. The maintainers will be able to parse it manually
or automatically to find the explanation of why their packages have
been updated or not.

== Excuses generation ==

These are the steps (with references to method names) that Britney
does for the generation of the update excuses.

 * If a source package is available in testing but it is not
   present in unstable and no binary packages in unstable are
   built from it, then it is marked for removal.

 * Every source package in unstable and testing-proposed-updates,
   if already present in testing, is checked for binary-NMUs, new
   or dropped binary packages in all the supported architectures
   (see Britney.should_upgrade_srcarch). The steps to detect if an
   upgrade is needed are:

    1. If there is a `remove' hint for the source package, the package
       is ignored: it will be removed and not updated.

    2. For every binary package build from the new source, it checks
       for unsatisfied dependencies, new binary package and updated
       binary package (binNMU) excluding the architecture-independent
       ones and the packages not built from the same source.

    3. For every binary package build from the old source, it checks
       if it is still built from the new source; if this is not true
       and the package is not architecture-independent, the script
       removes it from testing.

    4. Finally, if there is something worth doing (eg. a new or updated
       binary package) and nothing wrong it marks the source package
       as "Valid candidate", or "Not considered" if there is something
       wrong which prevented the update.

 * Every source package in unstable and testing-proposed-updates is
   checked for upgrade (see Britney.should_upgrade_src). The steps
   to detect if an upgrade is needed are:

    1. If the source package in testing is more recent the new one
       is ignored.

    2. If the source package doesn't exist (is fake), which means that
       a binary package refers to it but it is not present in the
       `Sources' file, the new one is ignored.

    3. If the package doesn't exist in testing, the urgency of the
       upload is ignored and set to the default (actually `low').

    4. If there is a `remove' hint for the source package, the package
       is ignored: it will be removed and not updated.

    5. If there is a `block' hint for the source package without an
       `unblock` hint or a `block-all source`, the package is ignored.

    7. If the suite is unstable, the update can go ahead only if the
       upload happend more then the minimum days specified by the
       urgency of the upload; if this is not true, the package is
       ignored as `too-young'. Note that the urgency is sticky, meaning
       that the highest urgency uploaded since the previous testing
       transition is taken into account.

    8. All the architecture-dependent binary packages and the
       architecture-independent ones for the `nobreakall' architectures
       have to be built from the source we are considering. If this is
       not true, then these are called `out-of-date' architectures and
       the package is ignored.

    9. The source package must have at least a binary package, otherwise
       it is ignored.

   10. If the suite is unstable, the count of release critical bugs for
       the new source package must be less then the count for the testing
       one. If this is not true, the package is ignored as `buggy'.

   11. If there is a `force' hint for the source package, then it is
       updated even if it is marked as ignored from the previous steps.

   12. If the suite is testing-proposed-updates, the source package can
       be updated only if there is an explicit approval for it.

   13. If the package will be ignored, mark it as "Valid candidate",
       otherwise mark it as "Not considered".

 * The list of `remove' hints is processed: if the requested source
   package is not already being updated or removed and the version
   actually in testing is the same specified with the `remove' hint,
   it is marked for removal.

 * The excuses are sorted by the number of days from the last upload
   (days-old) and by name.

 * A list of unconsidered excuses (for which the package is not upgraded)
   is built. Using this list, all the excuses depending on them is marked
   as invalid for "unpossible dependency".

 * The excuses are written in an HTML file.
"""

import os
import re
import sys
import string
import time
import copy
import optparse
import operator

import apt_pkg

from excuse import Excuse

__author__ = 'Fabio Tranchitella'
__version__ = '2.0.alpha1'

# source package
VERSION = 0
SECTION = 1
BINARIES = 2
MAINTAINER = 3
FAKESRC = 4

# binary package
SOURCE = 2
SOURCEVER = 3
ARCHITECTURE = 4
PREDEPENDS = 5
DEPENDS = 6
CONFLICTS = 7
PROVIDES = 8
RDEPENDS = 9
RCONFLICTS = 10


class Britney:
    """Britney, the debian testing updater script
    
    This is the script that updates the testing_ distribution. It is executed
    each day after the installation of the updated packages. It generates the 
    `Packages' files for the testing distribution, but it does so in an
    intelligent manner; it try to avoid any inconsistency and to use only
    non-buggy packages.

    For more documentation on this script, please read the Developers Reference.
    """

    HINTS_STANDARD = ("easy", "hint", "remove", "block", "unblock", "urgent", "approve")
    HINTS_ALL = ("force", "force-hint", "block-all") + HINTS_STANDARD

    def __init__(self):
        """Class constructor

        This method initializes and populates the data lists, which contain all
        the information needed by the other methods of the class.
        """
        self.date_now = int(((time.time() / (60*60)) - 15) / 24)

        # parse the command line arguments
        self.__parse_arguments()

        # initialize the apt_pkg back-end
        apt_pkg.init()

        # if requested, build the non-installable status and save it
        if not self.options.nuninst_cache:
            self.__log("Building the list of not installable packages for the full archive", type="I")
            self.sources = {'testing': self.read_sources(self.options.testing)}
            nuninst = {}
            for arch in self.options.architectures:
                self.binaries = {'testing': {arch: self.read_binaries(self.options.testing, "testing", arch)}}
                self.__log("> Checking for non-installable packages for architecture %s" % arch, type="I")
                result = self.get_nuninst(arch, build=True)
                nuninst.update(result)
                self.__log("> Found %d non-installable packages" % len(nuninst[arch]), type="I")
            self.write_nuninst(nuninst)
        else:
            self.__log("Not building the list of not installable packages, as requested", type="I")

        # read the source and binary packages for the involved distributions
        self.sources = {'testing': self.read_sources(self.options.testing),
                        'unstable': self.read_sources(self.options.unstable),
                        'tpu': self.read_sources(self.options.tpu),}
        self.binaries = {'testing': {}, 'unstable': {}, 'tpu': {}}
        for arch in self.options.architectures:
            self.binaries['testing'][arch] = self.read_binaries(self.options.testing, "testing", arch)
            self.binaries['unstable'][arch] = self.read_binaries(self.options.unstable, "unstable", arch)
            self.binaries['tpu'][arch] = self.read_binaries(self.options.tpu, "tpu", arch)

        # read the release-critical bug summaries for testing and unstable
        self.bugs = {'unstable': self.read_bugs(self.options.unstable),
                     'testing': self.read_bugs(self.options.testing),}
        self.normalize_bugs()

        # read additional data
        self.dates = self.read_dates(self.options.testing)
        self.urgencies = self.read_urgencies(self.options.testing)
        self.approvals = self.read_approvals(self.options.tpu)
        self.hints = self.read_hints(self.options.unstable)
        self.excuses = []
        self.dependencies = {}

    def __parse_arguments(self):
        """Parse the command line arguments

        This method parses and initializes the command line arguments.
        While doing so, it preprocesses some of the options to be converted
        in a suitable form for the other methods of the class.
        """
        # initialize the parser
        self.parser = optparse.OptionParser(version="%prog")
        self.parser.add_option("-v", "", action="count", dest="verbose", help="enable verbose output")
        self.parser.add_option("-c", "--config", action="store", dest="config", default="/etc/britney.conf",
                               help="path for the configuration file")
        self.parser.add_option("", "--architectures", action="store", dest="architectures", default=None,
                               help="override architectures from configuration file")
        self.parser.add_option("", "--actions", action="store", dest="actions", default=None,
                               help="override the list of actions to be performed")
        self.parser.add_option("", "--dry-run", action="store_true", dest="dry_run", default=False,
                               help="disable all outputs to the testing directory")
        self.parser.add_option("", "--compatible", action="store_true", dest="compatible", default=False,
                               help="enable full compatibility with old britney's output")
        self.parser.add_option("", "--control-files", action="store_true", dest="control_files", default=False,
                               help="enable control files generation")
        self.parser.add_option("", "--nuninst-cache", action="store_true", dest="nuninst_cache", default=False,
                               help="do not build the non-installability status, use the cache from file")
        (self.options, self.args) = self.parser.parse_args()

        # if the configuration file exists, than read it and set the additional options
        if not os.path.isfile(self.options.config):
            self.__log("Unable to read the configuration file (%s), exiting!" % self.options.config, type="E")
            sys.exit(1)

        # minimum days for unstable-testing transition and the list of hints
        # are handled as an ad-hoc case
        self.MINDAYS = {}
        self.HINTS = {}
        for k, v in [map(string.strip,r.split('=', 1)) for r in file(self.options.config) if '=' in r and not r.strip().startswith('#')]:
            if k.startswith("MINDAYS_"):
                self.MINDAYS[k.split("_")[1].lower()] = int(v)
            elif k.startswith("HINTS_"):
                self.HINTS[k.split("_")[1].lower()] = \
                    reduce(lambda x,y: x+y, [hasattr(self, "HINTS_" + i) and getattr(self, "HINTS_" + i) or (i,) for i in v.split()])
            elif not hasattr(self.options, k.lower()) or \
                 not getattr(self.options, k.lower()):
                setattr(self.options, k.lower(), v)

        # Sort the architecture list
        allarches = sorted(self.options.architectures.split())
        arches = [x for x in allarches if x in self.options.nobreakall_arches]
        arches += [x for x in allarches if x not in arches and x not in self.options.fucked_arches.split()]
        arches += [x for x in allarches if x not in arches and x not in self.options.break_arches.split()]
        arches += [x for x in allarches if x not in arches and x not in self.options.new_arches.split()]
        arches += [x for x in allarches if x not in arches]
        self.options.architectures = arches
        self.options.smooth_updates = self.options.smooth_updates.split()

    def __log(self, msg, type="I"):
        """Print info messages according to verbosity level
        
        An easy-and-simple log method which prints messages to the standard
        output. The type parameter controls the urgency of the message, and
        can be equal to `I' for `Information', `W' for `Warning' and `E' for
        `Error'. Warnings and errors are always printed, and information are
        printed only if the verbose logging is enabled.
        """
        if self.options.verbose or type in ("E", "W"):
            print "%s: [%s] - %s" % (type, time.asctime(), msg)

    # Data reading/writing methods
    # ----------------------------

    def read_sources(self, basedir):
        """Read the list of source packages from the specified directory
        
        The source packages are read from the `Sources' file within the
        directory specified as `basedir' parameter. Considering the
        large amount of memory needed, not all the fields are loaded
        in memory. The available fields are Version, Maintainer and Section.

        The method returns a list where every item represents a source
        package as a dictionary.
        """
        sources = {}
        package = None
        filename = os.path.join(basedir, "Sources")
        self.__log("Loading source packages from %s" % filename)
        Packages = apt_pkg.ParseTagFile(open(filename))
        get_field = Packages.Section.get
        while Packages.Step():
            pkg = get_field('Package')
            sources[pkg] = [get_field('Version'),
                            get_field('Section'),
                            [],
                            get_field('Maintainer'),
                            False,
                           ]
        return sources

    def read_binaries(self, basedir, distribution, arch):
        """Read the list of binary packages from the specified directory
        
        The binary packages are read from the `Packages_${arch}' files
        within the directory specified as `basedir' parameter, replacing
        ${arch} with the value of the arch parameter. Considering the
        large amount of memory needed, not all the fields are loaded
        in memory. The available fields are Version, Source, Pre-Depends,
        Depends, Conflicts, Provides and Architecture.
        
        After reading the packages, reverse dependencies are computed
        and saved in the `rdepends' keys, and the `Provides' field is
        used to populate the virtual packages list.

        The dependencies are parsed with the apt.pkg.ParseDepends method,
        and they are stored both as the format of its return value and
        text.

        The method returns a tuple. The first element is a list where
        every item represents a binary package as a dictionary; the second
        element is a dictionary which maps virtual packages to real
        packages that provide it.
        """

        packages = {}
        provides = {}
        sources = self.sources
        package = None

        filename = os.path.join(basedir, "Packages_%s" % arch)
        self.__log("Loading binary packages from %s" % filename)
        Packages = apt_pkg.ParseTagFile(open(filename))
        get_field = Packages.Section.get
        while Packages.Step():
            pkg = get_field('Package')
            version = get_field('Version')
            dpkg = [version,
                    get_field('Section'),
                    pkg, 
                    version,
                    get_field('Architecture'),
                    get_field('Pre-Depends'),
                    get_field('Depends'),
                    get_field('Conflicts'),
                    get_field('Provides'),
                    [],
                    [],
                   ]

            # retrieve the name and the version of the source package
            source = get_field('Source')
            if source:
                dpkg[SOURCE] = source.split(" ")[0]
                if "(" in source:
                    dpkg[SOURCEVER] = source[source.find("(")+1:source.find(")")]

            # if the source package is available in the distribution, then register this binary package
            if dpkg[SOURCE] in sources[distribution]:
                sources[distribution][dpkg[SOURCE]][BINARIES].append(pkg + "/" + arch)
            # if the source package doesn't exist, create a fake one
            else:
                sources[distribution][dpkg[SOURCE]] = [dpkg[SOURCEVER], None, [pkg + "/" + arch], None, True]

            # register virtual packages and real packages that provide them
            if dpkg[PROVIDES]:
                parts = map(string.strip, dpkg[PROVIDES].split(","))
                for p in parts:
                    if p not in provides:
                        provides[p] = []
                    provides[p].append(pkg)
                dpkg[PROVIDES] = parts
            else: dpkg[PROVIDES] = []

            # add the resulting dictionary to the package list
            packages[pkg] = dpkg

        # loop again on the list of packages to register reverse dependencies and conflicts
        register_reverses = self.register_reverses
        for pkg in packages:
            register_reverses(pkg, packages, provides, check_doubles=False)

        # return a tuple with the list of real and virtual packages
        return (packages, provides)

    def register_reverses(self, pkg, packages, provides, check_doubles=True, parse_depends=apt_pkg.ParseDepends):
        """Register reverse dependencies and conflicts for the specified package

        This method register the reverse dependencies and conflicts for
        a give package using `packages` as list of packages and `provides`
        as list of virtual packages.

        The method has an optional parameter parse_depends which is there
        just for performance reasons and is not meant to be overwritten.
        """
        # register the list of the dependencies for the depending packages
        dependencies = []
        if packages[pkg][DEPENDS]:
            dependencies.extend(parse_depends(packages[pkg][DEPENDS]))
        if packages[pkg][PREDEPENDS]:
            dependencies.extend(parse_depends(packages[pkg][PREDEPENDS]))
        # go through the list
        for p in dependencies:
            for a in p:
                # register real packages
                if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RDEPENDS]):
                    packages[a[0]][RDEPENDS].append(pkg)
                # register packages which provides a virtual package
                elif a[0] in provides:
                    for i in provides.get(a[0]):
                        if i not in packages: continue
                        if not check_doubles or pkg not in packages[i][RDEPENDS]:
                            packages[i][RDEPENDS].append(pkg)
        # register the list of the conflicts for the conflicting packages
        if packages[pkg][CONFLICTS]:
            for p in parse_depends(packages[pkg][CONFLICTS]):
                for a in p:
                    # register real packages
                    if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RCONFLICTS]):
                        packages[a[0]][RCONFLICTS].append(pkg)
                    # register packages which provides a virtual package
                    elif a[0] in provides:
                        for i in provides[a[0]]:
                            if i not in packages: continue
                            if not check_doubles or pkg not in packages[i][RCONFLICTS]:
                                packages[i][RCONFLICTS].append(pkg)
     
    def read_bugs(self, basedir):
        """Read the release critial bug summary from the specified directory
        
        The RC bug summaries are read from the `Bugs' file within the
        directory specified as `basedir' parameter. The file contains
        rows with the format:

        <package-name> <count-of-rc-bugs>

        The method returns a dictionary where the key is the binary package
        name and the value is the number of open RC bugs for it.
        """
        bugs = {}
        filename = os.path.join(basedir, "Bugs")
        self.__log("Loading RC bugs count from %s" % filename)
        for line in open(filename):
            l = line.split()
            if len(l) != 2: continue
            try:
                bugs[l[0]] = int(l[1])
            except ValueError:
                self.__log("Bugs, unable to parse \"%s\"" % line, type="E")
        return bugs

    def write_bugs(self, basedir, bugs):
        """Write the release critical bug summary to the specified directory

        For a more detailed explanation of the format, please check the method
        read_bugs.
        """
        filename = os.path.join(basedir, "Bugs")
        self.__log("Writing RC bugs count to %s" % filename)
        f = open(filename, 'w')
        for pkg in sorted(bugs.keys()):
            if bugs[pkg] == 0: continue
            f.write("%s %d\n" % (pkg, bugs[pkg]))
        f.close()

    def __maxver(self, pkg, dist):
        """Return the maximum version for a given package name
        
        This method returns None if the specified source package
        is not available in the `dist' distribution. If the package
        exists, then it returns the maximum version between the
        source package and its binary packages.
        """
        maxver = None
        if pkg in self.sources[dist]:
            maxver = self.sources[dist][pkg][VERSION]
        for arch in self.options.architectures:
            if pkg not in self.binaries[dist][arch][0]: continue
            pkgv = self.binaries[dist][arch][0][pkg][VERSION]
            if maxver == None or apt_pkg.VersionCompare(pkgv, maxver) > 0:
                maxver = pkgv
        return maxver

    def normalize_bugs(self):
        """Normalize the release critical bug summaries for testing and unstable
        
        The method doesn't return any value: it directly modifies the
        object attribute `bugs'.
        """
        # loop on all the package names from testing and unstable bug summaries
        for pkg in set(self.bugs['testing'].keys() + self.bugs['unstable'].keys()):

            # make sure that the key is present in both dictionaries
            if pkg not in self.bugs['testing']:
                self.bugs['testing'][pkg] = 0
            elif pkg not in self.bugs['unstable']:
                self.bugs['unstable'][pkg] = 0

            # retrieve the maximum version of the package in testing:
            maxvert = self.__maxver(pkg, 'testing')

            # if the package is not available in testing or it has the
            # same RC bug count, then do nothing
            if maxvert == None or \
               self.bugs['testing'][pkg] == self.bugs['unstable'][pkg]:
                continue

            # retrieve the maximum version of the package in testing:
            maxveru = self.__maxver(pkg, 'unstable')

            # if the package is not available in unstable, then do nothing
            if maxveru == None:
                continue
            # else if the testing package is more recent, then use the
            # unstable RC bug count for testing, too
            elif apt_pkg.VersionCompare(maxvert, maxveru) >= 0:
                self.bugs['testing'][pkg] = self.bugs['unstable'][pkg]

    def read_dates(self, basedir):
        """Read the upload date for the packages from the specified directory
        
        The upload dates are read from the `Date' file within the directory
        specified as `basedir' parameter. The file contains rows with the
        format:

        <package-name> <version> <date-of-upload>

        The dates are expressed as days starting from the 1970-01-01.

        The method returns a dictionary where the key is the binary package
        name and the value is tuple with two items, the version and the date.
        """
        dates = {}
        filename = os.path.join(basedir, "Dates")
        self.__log("Loading upload data from %s" % filename)
        for line in open(filename):
            l = line.split()
            if len(l) != 3: continue
            try:
                dates[l[0]] = (l[1], int(l[2]))
            except ValueError:
                self.__log("Dates, unable to parse \"%s\"" % line, type="E")
        return dates

    def write_dates(self, basedir, dates):
        """Write the upload date for the packages to the specified directory

        For a more detailed explanation of the format, please check the method
        read_dates.
        """
        filename = os.path.join(basedir, "Dates")
        self.__log("Writing upload data to %s" % filename)
        f = open(filename, 'w')
        for pkg in sorted(dates.keys()):
            f.write("%s %s %d\n" % ((pkg,) + dates[pkg]))
        f.close()


    def read_urgencies(self, basedir):
        """Read the upload urgency of the packages from the specified directory
        
        The upload urgencies are read from the `Urgency' file within the
        directory specified as `basedir' parameter. The file contains rows
        with the format:

        <package-name> <version> <urgency>

        The method returns a dictionary where the key is the binary package
        name and the value is the greatest urgency from the versions of the
        package that are higher then the testing one.
        """

        urgencies = {}
        filename = os.path.join(basedir, "Urgency")
        self.__log("Loading upload urgencies from %s" % filename)
        for line in open(filename):
            l = line.split()
            if len(l) != 3: continue

            # read the minimum days associated to the urgencies
            urgency_old = urgencies.get(l[0], self.options.default_urgency)
            mindays_old = self.MINDAYS.get(urgency_old, self.MINDAYS[self.options.default_urgency])
            mindays_new = self.MINDAYS.get(l[2], self.MINDAYS[self.options.default_urgency])

            # if the new urgency is lower (so the min days are higher), do nothing
            if mindays_old <= mindays_new:
                continue

            # if the package exists in testing and it is more recent, do nothing
            tsrcv = self.sources['testing'].get(l[0], None)
            if tsrcv and apt_pkg.VersionCompare(tsrcv[VERSION], l[1]) >= 0:
                continue

            # if the package doesn't exist in unstable or it is older, do nothing
            usrcv = self.sources['unstable'].get(l[0], None)
            if not usrcv or apt_pkg.VersionCompare(usrcv[VERSION], l[1]) < 0:
                continue

            # update the urgency for the package
            urgencies[l[0]] = l[2]

        return urgencies

    def read_approvals(self, basedir):
        """Read the approval commands from the specified directory
        
        The approval commands are read from the files contained by the 
        `Approved' directory within the directory specified as `basedir'
        parameter. The name of the files has to be the same of the
        authorized users for the approvals.
        
        The file contains rows with the format:

        <package-name> <version>

        The method returns a dictionary where the key is the binary package
        name followed by an underscore and the version number, and the value
        is the user who submitted the command.
        """
        approvals = {}
        for approver in self.options.approvers.split():
            filename = os.path.join(basedir, "Approved", approver)
            self.__log("Loading approvals list from %s" % filename)
            for line in open(filename):
                l = line.split()
                if len(l) != 2: continue
                approvals["%s_%s" % (l[0], l[1])] = approver
        return approvals

    def read_hints(self, basedir):
        """Read the hint commands from the specified directory
        
        The hint commands are read from the files contained by the `Hints'
        directory within the directory specified as `basedir' parameter. 
        The name of the files has to be the same of the authorized users
        for the hints.
        
        The file contains rows with the format:

        <command> <package-name>[/<version>]

        The method returns a dictionary where the key is the command, and
        the value is the list of affected packages.
        """
        hints = dict([(k,[]) for k in self.HINTS_ALL])

        for who in self.HINTS.keys():
            filename = os.path.join(basedir, "Hints", who)
            self.__log("Loading hints list from %s" % filename)
            for line in open(filename):
                line = line.strip()
                if line == "": continue
                l = line.split()
                if l[0] == 'finished':
                    break
                elif l[0] not in self.HINTS[who]:
                    continue
                elif l[0] in ["easy", "hint", "force-hint"]:
                    hints[l[0]].append((who, [k.split("/") for k in l if "/" in k]))
                elif l[0] in ["block-all"]:
                    hints[l[0]].extend([(y, who) for y in l[1:]])
                elif l[0] in ["block"]:
                    hints[l[0]].extend([(y, who) for y in l[1:]])
                elif l[0] in ["remove", "approve", "unblock", "force", "urgent"]:
                    hints[l[0]].extend([(k.split("/")[0], (k.split("/")[1],who) ) for k in l if "/" in k])

        for x in ["block", "block-all", "unblock", "force", "urgent", "remove"]:
            z = {}
            for a, b in hints[x]:
                if a in z:
                    self.__log("Overriding %s[%s] = %s with %s" % (x, a, z[a], b), type="W")
                z[a] = b
            hints[x] = z

        return hints

    def write_heidi(self, basedir, filename):
        """Write the output HeidiResult

        This method write the output for Heidi, which contains all the
        binary packages and the source packages in the form:
        
        <pkg-name> <pkg-version> <pkg-architecture> <pkg-section>
        <src-name> <src-version> <src-section>
        """
        filename = os.path.join(basedir, filename)
        self.__log("Writing Heidi results to %s" % filename)
        f = open(filename, 'w')

        # local copies
        sources = self.sources['testing']

        # write binary packages
        for arch in sorted(self.options.architectures):
            binaries = self.binaries['testing'][arch][0]
            for pkg_name in sorted(binaries):
                pkg = binaries[pkg_name]
                pkgv = pkg[VERSION]
                pkgarch = pkg[ARCHITECTURE]
                pkgsec = pkg[SECTION] or 'unknown'
                f.write('%s %s %s %s\n' % (pkg_name, pkgv, pkgarch, pkgsec))

        # write sources
        for src_name in sorted(sources):
            src = sources[src_name]
            srcv = src[VERSION]
            srcsec = src[FAKESRC] and 'faux' or src[SECTION] or 'unknown'
            f.write('%s %s source %s\n' % (src_name, srcv, srcsec))

        f.close()

    def write_controlfiles(self, basedir, suite):
        """Write the control files

        This method write the control files for the binary packages of all
        the architectures and for the source packages.
        """
        sources = self.sources[suite]

        self.__log("Writing new %s control files to %s" % (suite, basedir))
        for arch in self.options.architectures:
            filename = os.path.join(basedir, 'Packages_%s' % arch)
            f = open(filename, 'w')
            binaries = self.binaries[suite][arch][0]
            for pkg in binaries:
                output = "Package: %s\n" % pkg
                for key, k in ((SECTION, 'Section'), (ARCHITECTURE, 'Architecture'), (SOURCE, 'Source'), (VERSION, 'Version'), 
                          (PREDEPENDS, 'Pre-Depends'), (DEPENDS, 'Depends'), (PROVIDES, 'Provides'), (CONFLICTS, 'Conflicts')):
                    if not binaries[pkg][key]: continue
                    if key == SOURCE:
                        if binaries[pkg][SOURCE] == pkg:
                            if binaries[pkg][SOURCEVER] != binaries[pkg][VERSION]:
                                source = binaries[pkg][SOURCE] + " (" + binaries[pkg][SOURCEVER] + ")"
                            else: continue
                        else:
                            if binaries[pkg][SOURCEVER] != binaries[pkg][VERSION]:
                                source = binaries[pkg][SOURCE] + " (" + binaries[pkg][SOURCEVER] + ")"
                            else:
                                source = binaries[pkg][SOURCE]
                        output += (k + ": " + source + "\n")
                        if sources[binaries[pkg][SOURCE]][MAINTAINER]:
                            output += (k + ": " + sources[binaries[pkg][SOURCE]][MAINTAINER] + "\n")
                    elif key == PROVIDES:
                        if len(binaries[pkg][key]) > 0:
                            output += (k + ": " + ", ".join(binaries[pkg][key]) + "\n")
                    else:
                        output += (k + ": " + binaries[pkg][key] + "\n")
                f.write(output + "\n")
            f.close()

        filename = os.path.join(basedir, 'Sources')
        f = open(filename, 'w')
        for src in sources:
            output = "Package: %s\n" % src
            for k in ('Version', 'Section', 'Maintainer'):
                key = k.lower()
                if key not in sources[src] or not sources[src][key]: continue
                output += (k + ": " + sources[src][key] + "\n")
            f.write(output + "\n")
        f.close()

    def write_nuninst(self, nuninst):
        """Write the non-installable report"""
        f = open(self.options.noninst_status, 'w')
        f.write("Built on: " + time.strftime("%Y.%m.%d %H:%M:%S %z", time.gmtime(time.time())) + "\n")
        f.write("Last update: " + time.strftime("%Y.%m.%d %H:%M:%S %z", time.gmtime(time.time())) + "\n\n")
        f.write("".join([k + ": " + " ".join(nuninst[k]) + "\n" for k in nuninst]))
        f.close()

    def read_nuninst(self):
        """Read the non-installable report"""
        f = open(self.options.noninst_status)
        nuninst = {}
        for r in f:
            if ":" not in r: continue
            arch, packages = r.strip().split(":", 1)
            if arch.split("+", 1)[0] in self.options.architectures:
                nuninst[arch] = packages.split()
        return nuninst


    # Utility methods for package analisys
    # ------------------------------------

    def same_source(self, sv1, sv2):
        """Check if two version numbers are built from the same source

        This method returns a boolean value which is true if the two
        version numbers specified as parameters are built from the same
        source. The main use of this code is to detect binary-NMU.
        """
        if sv1 == sv2:
            return 1

        m = re.match(r'^(.*)\+b\d+$', sv1)
        if m: sv1 = m.group(1)
        m = re.match(r'^(.*)\+b\d+$', sv2)
        if m: sv2 = m.group(1)

        if sv1 == sv2:
            return 1

        if re.search("-", sv1) or re.search("-", sv2):
            m = re.match(r'^(.*-[^.]+)\.0\.\d+$', sv1)
            if m: sv1 = m.group(1)
            m = re.match(r'^(.*-[^.]+\.[^.]+)\.\d+$', sv1)
            if m: sv1 = m.group(1)

            m = re.match(r'^(.*-[^.]+)\.0\.\d+$', sv2)
            if m: sv2 = m.group(1)
            m = re.match(r'^(.*-[^.]+\.[^.]+)\.\d+$', sv2)
            if m: sv2 = m.group(1)

            return (sv1 == sv2)
        else:
            m = re.match(r'^([^-]+)\.0\.\d+$', sv1)
            if m and sv2 == m.group(1): return 1

            m = re.match(r'^([^-]+)\.0\.\d+$', sv2)
            if m and sv1 == m.group(1): return 1

            return 0

    def get_dependency_solvers(self, block, arch, distribution, excluded=[], strict=False):
        """Find the packages which satisfy a dependency block

        This method returns the list of packages which satisfy a dependency
        block (as returned by apt_pkg.ParseDepends) for the given architecture
        and distribution.

        It returns a tuple with two items: the first is a boolean which is
        True if the dependency is satisfied, the second is the list of the
        solving packages.
        """

        packages = []

        # local copies for better performances
        binaries = self.binaries[distribution][arch]

        # for every package, version and operation in the block
        for name, version, op in block:
            # look for the package in unstable
            if name not in excluded and name in binaries[0]:
                package = binaries[0][name]
                # check the versioned dependency (if present)
                if op == '' and version == '' or apt_pkg.CheckDep(package[VERSION], op, version):
                    packages.append(name)

            # look for the package in the virtual packages list and loop on them
            for prov in binaries[1].get(name, []):
                if prov in excluded or \
                   prov not in binaries[0]: continue
                package = binaries[0][prov]
                # check the versioned dependency (if present)
                # TODO: this is forbidden by the debian policy, which says that versioned
                #       dependencies on virtual packages are never satisfied. The old britney
                #       does it and we have to go with it, but at least a warning should be raised.
                if op == '' and version == '' or not strict and apt_pkg.CheckDep(package[VERSION], op, version):
                    packages.append(prov)
                    break

        return (len(packages) > 0, packages)

    def excuse_unsat_deps(self, pkg, src, arch, suite, excuse, excluded=[], conflicts=False):
        """Find unsatisfied dependencies for a binary package

        This method analyzes the dependencies of the binary package specified
        by the parameter `pkg', built from the source package `src', for the
        architecture `arch' within the suite `suite'. If the dependency can't
        be satisfied in testing and/or unstable, it updates the excuse passed
        as parameter.

        The dependency fields checked are Pre-Depends and Depends.
        """
        # retrieve the binary package from the specified suite and arch
        binary_u = self.binaries[suite][arch][0][pkg]

        # local copies for better performances
        parse_depends = apt_pkg.ParseDepends
        get_dependency_solvers = self.get_dependency_solvers
        strict = True # not self.options.compatible

        # analyze the dependency fields (if present)
        for type_key, type in ((PREDEPENDS, 'Pre-Depends'), (DEPENDS, 'Depends')):
            if not binary_u[type_key]:
                continue

            # for every block of dependency (which is formed as conjunction of disconjunction)
            for block, block_txt in zip(parse_depends(binary_u[type_key]), binary_u[type_key].split(',')):
                # if the block is satisfied in testing, then skip the block
                solved, packages = get_dependency_solvers(block, arch, 'testing', excluded, strict=strict)
                if solved:
                    for p in packages:
                        if p not in self.binaries[suite][arch][0]: continue
                        excuse.add_sane_dep(self.binaries[suite][arch][0][p][SOURCE])
                    continue

                # check if the block can be satisfied in unstable, and list the solving packages
                solved, packages = get_dependency_solvers(block, arch, suite, [], strict=strict)
                packages = [self.binaries[suite][arch][0][p][SOURCE] for p in packages]

                # if the dependency can be satisfied by the same source package, skip the block:
                # obviously both binary packages will enter testing togheter
                if src in packages: continue

                # if no package can satisfy the dependency, add this information to the excuse
                if len(packages) == 0:
                    excuse.addhtml("%s/%s unsatisfiable %s: %s" % (pkg, arch, type, block_txt.strip()))
                    if arch not in self.options.break_arches: excuse.add_unsat_dep(arch)
                    continue

                # for the solving packages, update the excuse to add the dependencies
                for p in packages:
                    if arch not in self.options.break_arches.split():
                        excuse.add_dep(p)
                    else:
                        excuse.add_break_dep(p, arch)

        return True

    # Package analisys methods
    # ------------------------

    def should_remove_source(self, pkg):
        """Check if a source package should be removed from testing
        
        This method checks if a source package should be removed from the
        testing distribution; this happen if the source package is not
        present in the unstable distribution anymore.

        It returns True if the package can be removed, False otherwise.
        In the former case, a new excuse is appended to the the object
        attribute excuses.
        """
        # if the soruce package is available in unstable, then do nothing
        if pkg in self.sources['unstable']:
            return False
        # otherwise, add a new excuse for its removal and return True
        src = self.sources['testing'][pkg]
        excuse = Excuse("-" + pkg)
        excuse.set_vers(src[VERSION], None)
        src[MAINTAINER] and excuse.set_maint(src[MAINTAINER].strip())
        src[SECTION] and excuse.set_section(src[SECTION].strip())

        # if the package is blocked, skip it
        if self.hints['block'].has_key('-' + pkg):
            exc.addhtml("Not touching package, as requested by %s (contact debian-release if update is needed)" % hints['block']['-' + pkg])
            return False

        excuse.addhtml("Valid candidate")
        self.excuses.append(excuse)
        return True

    def should_upgrade_srcarch(self, src, arch, suite):
        """Check if binary package should be upgraded

        This method checks if a binary package should be upgraded; this can
        happen also if the binary package is a binary-NMU for the given arch.
        The analisys is performed for the source package specified by the
        `src' parameter, checking the architecture `arch' for the distribution
        `suite'.
       
        It returns False if the given package doesn't need to be upgraded,
        True otherwise. In the former case, a new excuse is appended to
        the the object attribute excuses.
        """
        # retrieve the source packages for testing and suite
        source_t = self.sources['testing'][src]
        source_u = self.sources[suite][src]

        # build the common part of the excuse, which will be filled by the code below
        ref = "%s/%s%s" % (src, arch, suite != 'unstable' and "_" + suite or "")
        excuse = Excuse(ref)
        excuse.set_vers(source_t[VERSION], source_t[VERSION])
        source_u[MAINTAINER] and excuse.set_maint(source_u[MAINTAINER].strip())
        source_u[SECTION] and excuse.set_section(source_u[SECTION].strip())
        
        # if there is a `remove' hint and the requested version is the same of the
        # version in testing, then stop here and return False
        if src in self.hints["remove"] and \
           self.same_source(source_t[VERSION], self.hints["remove"][src][0]):
            excuse.addhtml("Removal request by %s" % (self.hints["remove"][src][1]))
            excuse.addhtml("Trying to remove package, not update it")
            excuse.addhtml("Not considered")
            self.excuses.append(excuse)
            return False

        # the starting point is that there is nothing wrong and nothing worth doing
        anywrongver = False
        anyworthdoing = False

        # for every binary package produced by this source in unstable for this architecture
        for pkg in sorted(filter(lambda x: x.endswith("/" + arch), source_u[BINARIES]), key=lambda x: x.split("/")[0]):
            pkg_name = pkg.split("/")[0]

            # retrieve the testing (if present) and unstable corresponding binary packages
            binary_t = pkg in source_t[BINARIES] and self.binaries['testing'][arch][0][pkg_name] or None
            binary_u = self.binaries[suite][arch][0][pkg_name]

            # this is the source version for the new binary package
            pkgsv = self.binaries[suite][arch][0][pkg_name][SOURCEVER]

            # if the new binary package is architecture-independent, then skip it
            if binary_u[ARCHITECTURE] == 'all':
                excuse.addhtml("Ignoring %s %s (from %s) as it is arch: all" % (pkg_name, binary_u[VERSION], pkgsv))
                continue

            # if the new binary package is not from the same source as the testing one, then skip it
            if not self.same_source(source_t[VERSION], pkgsv):
                anywrongver = True
                excuse.addhtml("From wrong source: %s %s (%s not %s)" % (pkg_name, binary_u[VERSION], pkgsv, source_t[VERSION]))
                break

            # find unsatisfied dependencies for the new binary package
            self.excuse_unsat_deps(pkg_name, src, arch, suite, excuse)

            # if the binary is not present in testing, then it is a new binary;
            # in this case, there is something worth doing
            if not binary_t:
                excuse.addhtml("New binary: %s (%s)" % (pkg_name, binary_u[VERSION]))
                anyworthdoing = True
                continue

            # at this point, the binary package is present in testing, so we can compare
            # the versions of the packages ...
            vcompare = apt_pkg.VersionCompare(binary_t[VERSION], binary_u[VERSION])

            # ... if updating would mean downgrading, then stop here: there is something wrong
            if vcompare > 0:
                anywrongver = True
                excuse.addhtml("Not downgrading: %s (%s to %s)" % (pkg_name, binary_t[VERSION], binary_u[VERSION]))
                break
            # ... if updating would mean upgrading, then there is something worth doing
            elif vcompare < 0:
                excuse.addhtml("Updated binary: %s (%s to %s)" % (pkg_name, binary_t[VERSION], binary_u[VERSION]))
                anyworthdoing = True

        # if there is nothing wrong and there is something worth doing or the source
        # package is not fake, then check what packages shuold be removed
        if not anywrongver and (anyworthdoing or self.sources[suite][src][FAKESRC]):
            srcv = self.sources[suite][src][VERSION]
            ssrc = self.same_source(source_t[VERSION], srcv)
            # for every binary package produced by this source in testing for this architecture
            for pkg in sorted([x.split("/")[0] for x in self.sources['testing'][src][BINARIES] if x.endswith("/"+arch)]):
                # if the package is architecture-independent, then ignore it
                if self.binaries['testing'][arch][0][pkg][ARCHITECTURE] == 'all':
                    excuse.addhtml("Ignoring removal of %s as it is arch: all" % (pkg))
                    continue
                # if the package is not produced by the new source package, then remove it from testing
                if pkg not in self.binaries[suite][arch][0]:
                    tpkgv = self.binaries['testing'][arch][0][pkg][VERSION]
                    excuse.addhtml("Removed binary: %s %s" % (pkg, tpkgv))
                    if ssrc: anyworthdoing = True

        # if there is nothing wrong and there is something worth doing, this is valid candidate
        if not anywrongver and anyworthdoing:
            excuse.addhtml("Valid candidate")
            self.excuses.append(excuse)
            return True
        # else if there is something worth doing (but something wrong, too) this package won't be considered
        elif anyworthdoing:
            excuse.addhtml("Not considered")
            self.excuses.append(excuse)

        # otherwise, return False
        return False

    def should_upgrade_src(self, src, suite):
        """Check if source package should be upgraded

        This method checks if a source package should be upgraded. The analisys
        is performed for the source package specified by the `src' parameter, 
        checking the architecture `arch' for the distribution `suite'.
       
        It returns False if the given package doesn't need to be upgraded,
        True otherwise. In the former case, a new excuse is appended to
        the the object attribute excuses.
        """

        # retrieve the source packages for testing (if available) and suite
        source_u = self.sources[suite][src]
        if src in self.sources['testing']:
            source_t = self.sources['testing'][src]
            # if testing and unstable have the same version, then this is a candidate for binary-NMUs only
            if apt_pkg.VersionCompare(source_t[VERSION], source_u[VERSION]) == 0:
                return False
        else:
            source_t = None

        # build the common part of the excuse, which will be filled by the code below
        ref = "%s%s" % (src, suite != 'unstable' and "_" + suite or "")
        excuse = Excuse(ref)
        excuse.set_vers(source_t and source_t[VERSION] or None, source_u[VERSION])
        source_u[MAINTAINER] and excuse.set_maint(source_u[MAINTAINER].strip())
        source_u[SECTION] and excuse.set_section(source_u[SECTION].strip())

        # the starting point is that we will update the candidate
        update_candidate = True
        
        # if the version in unstable is older, then stop here with a warning in the excuse and return False
        if source_t and apt_pkg.VersionCompare(source_u[VERSION], source_t[VERSION]) < 0:
            excuse.addhtml("ALERT: %s is newer in testing (%s %s)" % (src, source_t[VERSION], source_u[VERSION]))
            self.excuses.append(excuse)
            return False

        # check if the source package really exists or if it is a fake one
        if source_u[FAKESRC]:
            excuse.addhtml("%s source package doesn't exist" % (src))
            update_candidate = False

        # retrieve the urgency for the upload, ignoring it if this is a NEW package (not present in testing)
        urgency = self.urgencies.get(src, self.options.default_urgency)
        if not source_t and urgency != self.options.default_urgency:
            excuse.addhtml("Ignoring %s urgency setting for NEW package" % (urgency))
            urgency = self.options.default_urgency

        # if there is a `remove' hint and the requested version is the same of the
        # version in testing, then stop here and return False
        if src in self.hints["remove"]:
            if source_t and self.same_source(source_t[VERSION], self.hints['remove'][src][0]) or \
               self.same_source(source_u[VERSION], self.hints['remove'][src][0]):
                excuse.addhtml("Removal request by %s" % (self.hints["remove"][src][1]))
                excuse.addhtml("Trying to remove package, not update it")
                update_candidate = False

        # check if there is a `block' hint for this package or a `block-all source' hint
        blocked = None
        if src in self.hints["block"]:
            blocked = self.hints["block"][src]
        elif 'source' in self.hints["block-all"]:
            blocked = self.hints["block-all"]["source"]

        # if the source is blocked, then look for an `unblock' hint; the unblock request
        # is processed only if the specified version is correct
        if blocked:
            unblock = self.hints["unblock"].get(src,(None,None))
            if unblock[0] != None:
                if self.same_source(unblock[0], source_u[VERSION]):
                    excuse.addhtml("Ignoring request to block package by %s, due to unblock request by %s" % (blocked, unblock[1]))
                else:
                    excuse.addhtml("Unblock request by %s ignored due to version mismatch: %s" % (unblock[1], unblock[0]))
            else:
                excuse.addhtml("Not touching package, as requested by %s (contact debian-release if update is needed)" % (blocked))
                update_candidate = False

        # if the suite is unstable, then we have to check the urgency and the minimum days of
        # permanence in unstable before updating testing; if the source package is too young,
        # the check fails and we set update_candidate to False to block the update
        if suite == 'unstable':
            if src not in self.dates:
                self.dates[src] = (source_u[VERSION], self.date_now)
            elif not self.same_source(self.dates[src][0], source_u[VERSION]):
                self.dates[src] = (source_u[VERSION], self.date_now)

            days_old = self.date_now - self.dates[src][1]
            min_days = self.MINDAYS[urgency]
            excuse.setdaysold(days_old, min_days)
            if days_old < min_days:
                if src in self.hints["urgent"] and self.same_source(source_u[VERSION], self.hints["urgent"][src][0]):
                    excuse.addhtml("Too young, but urgency pushed by %s" % (self.hints["urgent"][src][1]))
                else:
                    update_candidate = False

        # at this point, we check what is the status of the builds on all the supported architectures
        # to catch the out-of-date ones
        pkgs = {src: ["source"]}
        for arch in self.options.architectures:
            oodbins = {}
            # for every binary package produced by this source in the suite for this architecture
            for pkg in sorted([x.split("/")[0] for x in self.sources[suite][src][BINARIES] if x.endswith("/"+arch)]):
                if pkg not in pkgs: pkgs[pkg] = []
                pkgs[pkg].append(arch)

                # retrieve the binary package and its source version
                binary_u = self.binaries[suite][arch][0][pkg]
                pkgsv = binary_u[SOURCEVER]

                # if it wasn't builded by the same source, it is out-of-date
                if not self.same_source(source_u[VERSION], pkgsv):
                    if pkgsv not in oodbins:
                        oodbins[pkgsv] = []
                    oodbins[pkgsv].append(pkg)
                    continue

                # if the package is architecture-dependent or the current arch is `nobreakall'
                # find unsatisfied dependencies for the binary package
                if binary_u[ARCHITECTURE] != 'all' or arch in self.options.nobreakall_arches:
                    self.excuse_unsat_deps(pkg, src, arch, suite, excuse)

            # if there are out-of-date packages, warn about them in the excuse and set update_candidate
            # to False to block the update; if the architecture where the package is out-of-date is
            # in the `fucked_arches' list, then do not block the update
            if oodbins:
                oodtxt = ""
                for v in oodbins.keys():
                    if oodtxt: oodtxt = oodtxt + "; "
                    oodtxt = oodtxt + "%s (from <a href=\"http://buildd.debian.org/build.php?" \
                        "arch=%s&pkg=%s&ver=%s\" target=\"_blank\">%s</a>)" % \
                        (", ".join(sorted(oodbins[v])), arch, src, v, v)
                text = "out of date on <a href=\"http://buildd.debian.org/build.php?" \
                    "arch=%s&pkg=%s&ver=%s\" target=\"_blank\">%s</a>: %s" % \
                    (arch, src, source_u[VERSION], arch, oodtxt)

                if arch in self.options.fucked_arches:
                    text = text + " (but %s isn't keeping up, so nevermind)" % (arch)
                else:
                    update_candidate = False

                if self.date_now != self.dates[src][1]:
                    excuse.addhtml(text)

        # if the source package has no binaries, set update_candidate to False to block the update
        if len(self.sources[suite][src][BINARIES]) == 0:
            excuse.addhtml("%s has no binaries on any arch" % src)
            update_candidate = False

        # if the suite is unstable, then we have to check the release-critical bug counts before
        # updating testing; if the unstable package have a RC bug count greater than the testing
        # one,  the check fails and we set update_candidate to False to block the update
        if suite == 'unstable':
            for pkg in pkgs.keys():
                if pkg not in self.bugs['testing']:
                    self.bugs['testing'][pkg] = 0
                if pkg not in self.bugs['unstable']:
                    self.bugs['unstable'][pkg] = 0

                if self.bugs['unstable'][pkg] > self.bugs['testing'][pkg]:
                    excuse.addhtml("%s (%s) is <a href=\"http://bugs.debian.org/cgi-bin/pkgreport.cgi?" \
                                   "which=pkg&data=%s&sev-inc=critical&sev-inc=grave&sev-inc=serious\" " \
                                   "target=\"_blank\">buggy</a>! (%d > %d)" % \
                                   (pkg, ", ".join(pkgs[pkg]), pkg, self.bugs['unstable'][pkg], self.bugs['testing'][pkg]))
                    update_candidate = False
                elif self.bugs['unstable'][pkg] > 0:
                    excuse.addhtml("%s (%s) is (less) <a href=\"http://bugs.debian.org/cgi-bin/pkgreport.cgi?" \
                                   "which=pkg&data=%s&sev-inc=critical&sev-inc=grave&sev-inc=serious\" " \
                                   "target=\"_blank\">buggy</a>! (%d <= %d)" % \
                                   (pkg, ", ".join(pkgs[pkg]), pkg, self.bugs['unstable'][pkg], self.bugs['testing'][pkg]))

        # check if there is a `force' hint for this package, which allows it to go in even if it is not updateable
        if not update_candidate and src in self.hints["force"] and \
           self.same_source(source_u[VERSION], self.hints["force"][src][0]):
            excuse.dontinvalidate = 1
            excuse.addhtml("Should ignore, but forced by %s" % (self.hints["force"][src][1]))
            update_candidate = True

        # if the suite is testing-proposed-updates, the package needs an explicit approval in order to go in
        if suite == "tpu":
            key = "%s_%s" % (src, source_u[VERSION])
            if key in self.approvals:
                excuse.addhtml("Approved by %s" % approvals[key])
            else:
                excuse.addhtml("NEEDS APPROVAL BY RM")
                update_candidate = False

        # if the package can be updated, it is a valid candidate
        if update_candidate:
            excuse.addhtml("Valid candidate")
        # else it won't be considered
        else:
            excuse.addhtml("Not considered")

        self.excuses.append(excuse)
        return update_candidate

    def reversed_exc_deps(self):
        """Reverse the excuses dependencies

        This method returns a dictionary where the keys are the package names
        and the values are the excuse names which depend on it.
        """
        res = {}
        for exc in self.excuses:
            for d in exc.deps:
                if d not in res: res[d] = []
                res[d].append(exc.name)
        return res

    def invalidate_excuses(self, valid, invalid):
        """Invalidate impossible excuses

        This method invalidates the impossible excuses, which depend
        on invalid excuses. The two parameters contains the list of
        `valid' and `invalid' excuses.
        """
        # build a lookup-by-name map
        exclookup = {}
        for e in self.excuses:
            exclookup[e.name] = e

        # build the reverse dependencies
        revdeps = self.reversed_exc_deps()

        # loop on the invalid excuses
        i = 0
        while i < len(invalid):
            # if there is no reverse dependency, skip the item
            if invalid[i] not in revdeps:
                i += 1
                continue
            # if there dependency can be satisfied by a testing-proposed-updates excuse, skip the item
            if (invalid[i] + "_tpu") in valid:
                i += 1
                continue
            # loop on the reverse dependencies
            for x in revdeps[invalid[i]]:
                # if the item is valid and it is marked as `dontinvalidate', skip the item
                if x in valid and exclookup[x].dontinvalidate:
                    continue

                # otherwise, invalidate the dependency and mark as invalidated and
                # remove the depending excuses
                exclookup[x].invalidate_dep(invalid[i])
                if x in valid:
                    p = valid.index(x)
                    invalid.append(valid.pop(p))
                    exclookup[x].addhtml("Invalidated by dependency")
                    exclookup[x].addhtml("Not considered")
            i = i + 1
 
    def write_excuses(self):
        """Produce and write the update excuses

        This method handles the update excuses generation: the packages are
        looked to determine whether they are valid candidates. For the details
        of this procedure, please refer to the module docstring.
        """

        self.__log("Update Excuses generation started", type="I")

        # list of local methods and variables (for better performance)
        sources = self.sources
        architectures = self.options.architectures
        should_remove_source = self.should_remove_source
        should_upgrade_srcarch = self.should_upgrade_srcarch
        should_upgrade_src = self.should_upgrade_src

        # this list will contain the packages which are valid candidates;
        # if a package is going to be removed, it will have a "-" prefix
        upgrade_me = []

        # for every source package in testing, check if it should be removed
        for pkg in sources['testing']:
            if should_remove_source(pkg):
                upgrade_me.append("-" + pkg)

        # for every source package in unstable check if it should be upgraded
        for pkg in sources['unstable']:
            if sources['unstable'][pkg][FAKESRC]: continue
            # if the source package is already present in testing,
            # check if it should be upgraded for every binary package
            if pkg in sources['testing'] and not sources['testing'][pkg][FAKESRC]:
                for arch in architectures:
                    if should_upgrade_srcarch(pkg, arch, 'unstable'):
                        upgrade_me.append("%s/%s" % (pkg, arch))

            # check if the source package should be upgraded
            if should_upgrade_src(pkg, 'unstable'):
                upgrade_me.append(pkg)

        # for every source package in testing-proposed-updates, check if it should be upgraded
        for pkg in sources['tpu']:
            if sources['tpu'][pkg][FAKESRC]: continue
            # if the source package is already present in testing,
            # check if it should be upgraded for every binary package
            if pkg in sources['testing']:
                for arch in architectures:
                    if should_upgrade_srcarch(pkg, arch, 'tpu'):
                        upgrade_me.append("%s/%s_tpu" % (pkg, arch))

            # check if the source package should be upgraded
            if should_upgrade_src(pkg, 'tpu'):
                upgrade_me.append("%s_tpu" % pkg)

        # process the `remove' hints, if the given package is not yet in upgrade_me
        for src in self.hints["remove"].keys():
            if src in upgrade_me: continue
            if ("-"+src) in upgrade_me: continue
            if src not in sources['testing']: continue

            # check if the version specified in the hint is the same of the considered package
            tsrcv = sources['testing'][src][VERSION]
            if not self.same_source(tsrcv, self.hints["remove"][src][0]): continue

            # add the removal of the package to upgrade_me and build a new excuse
            upgrade_me.append("-%s" % (src))
            excuse = Excuse("-%s" % (src))
            excuse.set_vers(tsrcv, None)
            excuse.addhtml("Removal request by %s" % (self.hints["remove"][src][1]))
            excuse.addhtml("Package is broken, will try to remove")
            self.excuses.append(excuse)

        # sort the excuses by daysold and name
        self.excuses.sort(lambda x, y: cmp(x.daysold, y.daysold) or cmp(x.name, y.name))

        # extract the not considered packages, which are in the excuses but not in upgrade_me
        unconsidered = [e.name for e in self.excuses if e.name not in upgrade_me]

        # invalidate impossible excuses
        for e in self.excuses:
            for d in e.deps:
                if d not in upgrade_me and d not in unconsidered:
                    e.addhtml("Unpossible dep: %s -> %s" % (e.name, d))
        self.invalidate_excuses(upgrade_me, unconsidered)

        # sort the list of candidates
        self.upgrade_me = sorted(upgrade_me)

        # write excuses to the output file
        self.__log("> Writing Excuses to %s" % self.options.excuses_output, type="I")

        f = open(self.options.excuses_output, 'w')
        f.write("<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/REC-html40/strict.dtd\">\n")
        f.write("<html><head><title>excuses...</title>")
        f.write("<meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\"></head><body>\n")
        f.write("<p>Generated: " + time.strftime("%Y.%m.%d %H:%M:%S %z", time.gmtime(time.time())) + "</p>\n")
        f.write("<ul>\n")
        for e in self.excuses:
            f.write("<li>%s" % e.html())
        f.write("</ul></body></html>\n")
        f.close()

        self.__log("Update Excuses generation completed", type="I")

    # Upgrade run
    # -----------

    def newlyuninst(self, nuold, nunew):
        """Return a nuninst statstic with only new uninstallable packages

        This method subtract the uninstallabla packages of the statistic
        `nunew` from the statistic `nuold`.

        It returns a dictionary with the architectures as keys and the list
        of uninstallable packages as values.
        """
        res = {}
        for arch in nuold:
            if arch not in nunew: continue
            res[arch] = [x for x in nunew[arch] if x not in nuold[arch]]
        return res

    def get_nuninst(self, requested_arch=None, build=False):
        """Return the uninstallability statistic for all the architectures

        To calculate the uninstallability counters, the method checks the
        installability of all the packages for all the architectures, and
        tracking dependencies in a recursive way. The architecture
        indipendent packages are checked only for the `nobreakall`
        architectures.

        It returns a dictionary with the architectures as keys and the list
        of uninstallable packages as values.
        """
        # if we are not asked to build the nuninst, read it from the cache
        if not build:
            return self.read_nuninst()

        nuninst = {}

        # local copies for better performances
        binaries = self.binaries['testing']
        check_installable = self.check_installable

        # when a new uninstallable package is discovered, check again all the
        # reverse dependencies and if they are uninstallable, too, call itself
        # recursively
        def add_nuninst(pkg, arch):
            if pkg not in nuninst[arch]:
                nuninst[arch].append(pkg)
                for p in binaries[arch][0][pkg][RDEPENDS]:
                    r = check_installable(p, arch, 'testing', excluded=nuninst[arch], conflicts=True)
                    if not r:
                        add_nuninst(p, arch)

        # for all the architectures
        for arch in self.options.architectures:
            if requested_arch and arch != requested_arch: continue
            # if it is in the nobreakall ones, check arch-indipendent packages too
            if arch not in self.options.nobreakall_arches:
                skip_archall = True
            else: skip_archall = False

            # check all the packages for this architecture, calling add_nuninst if a new
            # uninstallable package is found
            nuninst[arch] = []
            for pkg_name in binaries[arch][0]:
                r = check_installable(pkg_name, arch, 'testing', excluded=nuninst[arch], conflicts=True)
                if not r:
                    add_nuninst(pkg_name, arch)

            # if they are not required, removed architecture-indipendent packages
            nuninst[arch + "+all"] = nuninst[arch][:]
            if skip_archall:
                for pkg in nuninst[arch + "+all"]:
                    bpkg = binaries[arch][0][pkg]
                    if bpkg[ARCHITECTURE] == 'all':
                        nuninst[arch].remove(pkg)

        # return the dictionary with the results
        return nuninst

    def eval_nuninst(self, nuninst, original=None):
        """Return a string which represents the uninstallability counters

        This method returns a string which represents the uninstallability
        counters reading the uninstallability statistics `nuninst` and, if
        present, merging the results with the `original` one.

        An example of the output string is:
        1+2: i-0:a-0:a-0:h-0:i-1:m-0:m-0:p-0:a-0:m-0:s-2:s-0

        where the first part is the number of broken packages in non-break
        architectures + the total number of broken packages for all the
        architectures.
        """
        res = []
        total = 0
        totalbreak = 0
        for arch in self.options.architectures:
            if arch in nuninst:
                n = len(nuninst[arch])
            elif original and arch in original:
                n = len(original[arch])
            else: continue
            if arch in self.options.break_arches:
                totalbreak = totalbreak + n
            else:
                total = total + n
            res.append("%s-%d" % (arch[0], n))
        return "%d+%d: %s" % (total, totalbreak, ":".join(res))

    def eval_uninst(self, nuninst):
        """Return a string which represents the uninstallable packages

        This method returns a string which represents the uninstallable
        packages reading the uninstallability statistics `nuninst`.

        An example of the output string is:
            * i386: broken-pkg1, broken-pkg2
        """
        parts = []
        for arch in self.options.architectures:
            if arch in nuninst and len(nuninst[arch]) > 0:
                parts.append("    * %s: %s\n" % (arch,", ".join(sorted(nuninst[arch]))))
        return "".join(parts)

    def is_nuninst_asgood_generous(self, old, new):
        diff = 0
        for arch in self.options.architectures:
            if arch in self.options.break_arches: continue
            diff = diff + (len(new[arch]) - len(old[arch]))
        return diff <= 0

    def check_installable(self, pkg, arch, suite, excluded=[], conflicts=False):
        """Check if a package is installable

        This method analyzes the dependencies of the binary package specified
        by the parameter `pkg' for the architecture `arch' within the suite
        `suite'. If the dependency can be satisfied in the given `suite` and
        `conflicts` parameter is True, then the co-installability with 
        conflicts handling is checked.

        The dependency fields checked are Pre-Depends and Depends.

        The method returns a boolean which is True if the given package is
        installable.
        """
        # retrieve the binary package from the specified suite and arch
        binary_u = self.binaries[suite][arch][0][pkg]

        # local copies for better performances
        parse_depends = apt_pkg.ParseDepends
        get_dependency_solvers = self.get_dependency_solvers

        # analyze the dependency fields (if present)
        for type in (PREDEPENDS, DEPENDS):
            if not binary_u[type]:
                continue

            # for every block of dependency (which is formed as conjunction of disconjunction)
            for block in parse_depends(binary_u[type]):
                # if the block is not satisfied, return False
                solved, packages = get_dependency_solvers(block, arch, 'testing', excluded, strict=True)
                if not solved:
                    return False

        # otherwise, the package is installable (not considering conflicts)
        # if the conflicts handling is enabled, then check conflicts before
        # saying that the package is really installable
        if conflicts:
            return self.check_conflicts(pkg, arch, excluded, {}, {})

        return True

    def check_conflicts(self, pkg, arch, broken, system, conflicts):
        """Check if a package can be installed satisfying the conflicts

        This method checks if the `pkg` package from the `arch` architecture
        can be installed (excluding `broken` packages) within the system
        `system` along with all its dependencies. This means that all the
        conflicts relationships are checked in order to achieve the test
        co-installability of the package.

        The method returns a boolean which is True if the given package is
        co-installable in the given system.
        """

        # local copies for better performances
        binaries = self.binaries['testing'][arch]
        parse_depends = apt_pkg.ParseDepends
        check_depends = apt_pkg.CheckDep

        # unregister conflicts, local method to remove conflicts
        # registered from a given package.
        def unregister_conflicts(pkg, conflicts):
            for c in conflicts.keys():
                i = 0
                while i < len(conflicts[c]):
                    if conflicts[c][i][3] == pkg:
                        del conflicts[c][i]
                    else: i = i + 1
                if len(conflicts[c]) == 0:
                    del conflicts[c]

        def remove_package(pkg, system, conflicts):
            for k in system:
                if pkg in system[k][1]:
                    system[k][1].remove(pkg)
            unregister_conflicts(pkg, conflicts)

        # handle a conflict, local method to solve a conflict which happened
        # in the system; the behaviour of the conflict-solver is:
        #   1. If there are alternatives for the package which must be removed,
        #      try them, and if one of them resolves the system return True;
        #   2. If none of the alternatives can solve the conflict, then call
        #      itself for the package which depends on the conflicting package.
        #   3. If the top of the dependency tree is reached, then the conflict
        #      can't be solved, so return False.
        def handle_conflict(pkg, source, system, conflicts):
            # skip packages which don't have reverse dependencies
            if source not in system or system[source][1] == []:
                remove_package(source, system, conflicts)
                return (system, conflicts)
            # reached the top of the tree
            if not system[source][1][0]:
                return False
            # remove its conflicts
            unregister_conflicts(source, conflicts)
            # if there are alternatives, try them
            alternatives = system[source][0]
            for alt in alternatives:
                if satisfy(alt, [x for x in alternatives if x != alt], pkg_from=system[source][1],
                        system=system, conflicts=conflicts, excluded=[source]):
                    remove_package(source, system, conflicts)
                    return (system, conflicts)
            # there are no good alternatives, so remove the package which depends on it
            for p in system[source][1]:
                # the package does not exist, we reached the top of the tree
                if not p: return False
                # we are providing the package we conflict on (eg. exim4 and mail-transfer-agent), skip it
                if p == pkg: continue
                output = handle_conflict(pkg, p, system, conflicts)
                if output:
                    system, conflicts = output
                else: return False
            remove_package(source, system, conflicts)
            return (system, conflicts)

        # dependency tree satisfier, local method which tries to satisfy the dependency
        # tree for a given package. It calls itself recursively in order to check the
        # co-installability of the full tree of dependency of the starting package.
        # If a conflict is detected, it tries to handle it calling the handle_conflict
        # method; if it can't be resolved, then it returns False.
        def satisfy(pkg, pkg_alt=None, pkg_from=None, system=system, conflicts=conflicts, excluded=[]):
            # if it is real package and it is already installed, skip it and return True
            if pkg in binaries[0]:
                if pkg in system:
                    if type(pkg_from) == list:
                        system[pkg][1].extend(pkg_from)
                    else:
                        system[pkg][1].append(pkg_from)
                    system[pkg] = (system[pkg][1], filter(lambda x: x in pkg_alt, system[pkg][0]))
                    return True
                binary_u = binaries[0][pkg]
            else: binary_u = None

            # if it is a virtual package
            providers = []
            if pkg_from and pkg in binaries[1]:
                providers = binaries[1][pkg]
                # it is both real and virtual, so the providers are alternatives
                if binary_u:
                    providers = filter(lambda x: (not pkg_alt or x not in pkg_alt) and x != pkg, providers)
                    if not pkg_alt:
                        pkg_alt = []
                    pkg_alt.extend(providers)
                # try all the alternatives and if none of them suits, give up and return False
                else:
                    # if we already have a provider in the system, everything is ok and return True
                    if len(filter(lambda x: x in providers and x not in excluded, system)) > 0:
                        return True
                    for p in providers:
                        # try to install the providers skipping excluded packages,
                        # which we already tried but do not work
                        if p in excluded: continue
                        elif satisfy(p, [a for a in providers if a != p], pkg_from):
                            return True
                    # if none of them suits, return False
                    return False

            # if the package doesn't exist, return False
            if not binary_u: return False

            # it is broken, but we have providers
            if pkg in broken and pkg_from:
                for p in providers:
                    # try to install the providers skipping excluded packages,
                    # which we already tried but do not work
                    if p in excluded: continue
                    elif satisfy(p, [a for a in providers if a != p], pkg_from):
                        return True
                return False

            # install the package into the system, recording which package required it
            if type(pkg_from) != list:
                pkg_from = [pkg_from]
            system[pkg] = (pkg_alt or [], pkg_from)

            # register provided packages
            if binary_u[PROVIDES]:
                for p in binary_u[PROVIDES]:
                    if p in system:
                        # do not consider packages providing the one which we are checking
                        if len(system[p][1]) == 1 and system[p][1][0] == None: continue
                        system[p][1].append(pkg)
                    else:
                        system[p] = ([], [pkg])

            # check the conflicts
            if pkg in conflicts:
                for name, version, op, conflicting in conflicts[pkg]:
                    if conflicting in binary_u[PROVIDES] and system[conflicting][1] == [pkg]: continue
                    if op == '' and version == '' or check_depends(binary_u[VERSION], op, version):
                        # if conflict is found, check if it can be solved removing
                        # already-installed packages without broking the system; if
                        # this is not possible, give up and return False
                        output = handle_conflict(pkg, conflicting, system.copy(), conflicts.copy())
                        if output:
                            system, conflicts = output
                        else:
                            del system[pkg]
                            return False

            # register conflicts from the just-installed package
            if binary_u[CONFLICTS]:
                for block in map(operator.itemgetter(0), parse_depends(binary_u[CONFLICTS] or [])):
                    name, version, op = block
                    # skip conflicts for packages provided by itself
                    # if the conflicting package is in the system (and it is not a self-conflict)
                    if not (name in binary_u[PROVIDES] and system[name][1] == [pkg]) and \
                       block[0] != pkg and block[0] in system:
                        if block[0] in binaries[0]:
                            binary_c = binaries[0][block[0]]
                        else: binary_c = None
                        if op == '' and version == '' or binary_c and check_depends(binary_c[VERSION], op, version):
                            # if conflict is found, check if it can be solved removing
                            # already-installed packages without broking the system; if
                            # this is not possible, give up and return False
                            output = handle_conflict(pkg, name, system.copy(), conflicts.copy())
                            if output:
                                system, conflicts = output
                            else:
                                del system[pkg]
                                unregister_conflicts(pkg, conflicts)
                                return False
                    # register the conflict)
                    if block[0] not in conflicts:
                        conflicts[block[0]] = []
                    conflicts[block[0]].append((name, version, op, pkg))

            # list all its dependencies ...
            dependencies = []
            for key in (PREDEPENDS, DEPENDS):
                if not binary_u[key]: continue
                dependencies.extend(parse_depends(binary_u[key]))

            # ... and go through them
            for block in dependencies:
                # list the possible alternatives, in case of a conflict
                alternatives = map(operator.itemgetter(0), block)
                valid = False
                for name, version, op in block:
                    # otherwise, if it is already installed or it is installable, the block is satisfied
                    if name in system or satisfy(name, [a for a in alternatives if a != name], pkg):
                        valid = True
                        break
                # if the block can't be satisfied, the package is not installable so
                # we need to remove it, its conflicts and its provided packages and
                # return False
                if not valid:
                    del system[pkg]
                    unregister_conflicts(pkg, conflicts)
                    for p in providers:
                        if satisfy(p, [a for a in providers if a != p], pkg_from):
                            return True
                    return False

            # if all the blocks have been satisfied, the package is installable
            return True

        # check the package at the top of the tree
        return satisfy(pkg)

    def doop_source(self, pkg):
        """Apply a change to the testing distribution as requested by `pkg`

        This method apply the changes required by the action `pkg` tracking
        them so it will be possible to revert them.

        The method returns a list of the package name, the suite where the
        package comes from, the list of packages affected by the change and
        the dictionary undo which can be used to rollback the changes.
        """
        undo = {'binaries': {}, 'sources': {}, 'virtual': {}, 'nvirtual': []}

        affected = []
        arch = None

        # local copies for better performances
        sources = self.sources
        binaries = self.binaries['testing']

        # removal of single-arch binary package = "-<package>/<arch>"
        if pkg[0] == "-" and "/" in pkg:
            pkg_name, arch = pkg.split("/")
            pkg_name = pkg_name[1:]
            if arch.endswith("_tpu"):
                arch, suite = arch.split("_")
            else: suite = "testing"
        # arch = "<source>/<arch>",
        elif "/" in pkg:
            pkg_name, arch = pkg.split("/")
            suite = "unstable"
        # removal of source packages = "-<source>",
        elif pkg[0] == "-":
            pkg_name = pkg[1:]
            suite = "testing"
        # testing-proposed-updates = "<source>_tpu"
        elif pkg[0].endswith("_tpu"):
            pkg_name = pkg[:-4]
            suite = "tpu"
        # normal update of source packages = "<source>"
        else:
            pkg_name = pkg
            suite = "unstable"

        # remove all binary packages (if the source already exists)
        if not (arch and pkg[0] == '-'):
            if pkg_name in sources['testing']:
                source = sources['testing'][pkg_name]
                # remove all the binaries
                for p in source[BINARIES]:
                    binary, parch = p.split("/")
                    if arch and parch != arch: continue
                    # if a smooth update is possible for the package, skip it
                    if not self.options.compatible and suite == 'unstable' and \
                       binary not in self.binaries[suite][parch][0] and \
                       ('ALL' in self.options.smooth_updates or \
                        binaries[parch][0][binary][SECTION] in self.options.smooth_updates):
                        continue
                    # save the old binary for undo
                    undo['binaries'][p] = binaries[parch][0][binary]
                    # all the reverse dependencies are affected by the change
                    for j in binaries[parch][0][binary][RDEPENDS]:
                        key = (j, parch)
                        if key not in affected: affected.append(key)
                    # remove the provided virtual packages
                    for j in binaries[parch][0][binary][PROVIDES]:
                        key = j + "/" + parch
                        if key not in undo['virtual']:
                            undo['virtual'][key] = binaries[parch][1][j][:]
                        binaries[parch][1][j].remove(binary)
                        if len(binaries[parch][1][j]) == 0:
                            del binaries[parch][1][j]
                    # finally, remove the binary package
                    del binaries[parch][0][binary]
                # remove the source package
                if not arch:
                    undo['sources'][pkg_name] = source
                    del sources['testing'][pkg_name]
            else:
                # the package didn't exist, so we mark it as to-be-removed in case of undo
                undo['sources']['-' + pkg_name] = True

        # single binary removal
        elif pkg_name in binaries[arch][0]:
            undo['binaries'][pkg_name + "/" + arch] = binaries[arch][0][pkg_name]
            for j in binaries[arch][0][pkg_name][RDEPENDS]:
                key = (j, arch)
                if key not in affected: affected.append(key)
            del binaries[arch][0][pkg_name]

        # add the new binary packages (if we are not removing)
        if pkg[0] != "-":
            source = sources[suite][pkg_name]
            for p in source[BINARIES]:
                binary, parch = p.split("/")
                if arch and parch != arch: continue
                key = (binary, parch)
                # obviously, added/modified packages are affected
                if key not in affected: affected.append(key)
                # if the binary already exists (built from another ource)
                if binary in binaries[parch][0]:
                    # save the old binary package
                    undo['binaries'][p] = binaries[parch][0][binary]
                    # all the reverse dependencies are affected by the change
                    for j in binaries[parch][0][binary][RDEPENDS]:
                        key = (j, parch)
                        if key not in affected: affected.append(key)
                    # all the reverse conflicts and their dependency tree are affected by the change
                    for j in binaries[parch][0][binary][RCONFLICTS]:
                        key = (j, parch)
                        if key not in affected: affected.append(key)
                        for p in self.get_full_tree(j, parch, 'testing'):
                            key = (p, parch)
                            if key not in affected: affected.append(key)
                # add/update the binary package
                binaries[parch][0][binary] = self.binaries[suite][parch][0][binary]
                # register new provided packages
                for j in binaries[parch][0][binary][PROVIDES]:
                    key = j + "/" + parch
                    if j not in binaries[parch][1]:
                        undo['nvirtual'].append(key)
                        binaries[parch][1][j] = []
                    elif key not in undo['virtual']:
                        undo['virtual'][key] = binaries[parch][1][j][:]
                    binaries[parch][1][j].append(binary)
                # all the reverse dependencies are affected by the change
                for j in binaries[parch][0][binary][RDEPENDS]:
                    key = (j, parch)
                    if key not in affected: affected.append(key)

            # register reverse dependencies and conflicts for the new binary packages
            for p in source[BINARIES]:
                binary, parch = p.split("/")
                if arch and parch != arch: continue
                self.register_reverses(binary, binaries[parch][0] , binaries[parch][1])

            # add/update the source package
            if not arch:
                sources['testing'][pkg_name] = sources[suite][pkg_name]

        # return the package name, the suite, the list of affected packages and the undo dictionary
        return (pkg_name, suite, affected, undo)

    def get_full_tree(self, pkg, arch, suite):
        """Calculate the full dependency tree for the given package

        This method returns the full dependency tree for the package `pkg`,
        inside the `arch` architecture for the suite `suite`.
        """
        packages = [pkg]
        binaries = self.binaries[suite][arch][0]
        l = n = 0
        while len(packages) > l:
            l = len(packages)
            for p in packages[n:]:
                packages.extend([x for x in binaries[p][RDEPENDS] if x not in packages and x in binaries])
            n = l
        return packages

    def iter_packages(self, packages, selected, hint=False, nuninst=None):
        """Iter on the list of actions and apply them one-by-one

        This method apply the changes from `packages` to testing, checking the uninstallability
        counters for every action performed. If the action do not improve the it, it is reverted.
        The method returns the new uninstallability counters and the remaining actions if the
        final result is successful, otherwise (None, None).
        """
        extra = []
        deferred = []
        skipped = []
        mark_passed = False
        position = len(packages)

        if nuninst:
            nuninst_comp = nuninst.copy()
        else:
            nuninst_comp = self.nuninst_orig.copy()

        # local copies for better performances
        check_installable = self.check_installable
        binaries = self.binaries['testing']
        sources = self.sources
        architectures = self.options.architectures
        nobreakall_arches = self.options.nobreakall_arches
        new_arches = self.options.new_arches
        break_arches = self.options.break_arches
        dependencies = self.dependencies
        compatible = self.options.compatible

        # pre-process a hint batch
        pre_process = {}
        if selected and hint:
            for pkg in selected:
                pkg_name, suite, affected, undo = self.doop_source(pkg)
                pre_process[pkg] = (pkg_name, suite, affected, undo)

        lundo = []
        if not hint:
            self.output_write("recur: [%s] %s %d/%d\n" % ("", ",".join(selected), len(packages), len(extra)))

        # loop on the packages (or better, actions)
        while packages:
            pkg = packages.pop(0)

            # this is the marker for the first loop
            if not compatible and not mark_passed and position < 0:
                mark_passed = True
                packages.extend(deferred)
                del deferred
            else: position -= 1

            # defer packages if their dependency has been already skipped
            if not compatible and not mark_passed:
                defer = False
                for p in dependencies.get(pkg, []):
                    if p in skipped:
                        deferred.append(pkg)
                        skipped.append(pkg)
                        defer = True
                        break
                if defer: continue

            if not hint:
                self.output_write("trying: %s\n" % (pkg))

            better = True
            nuninst = {}

            # apply the changes
            if pkg in pre_process:
                pkg_name, suite, affected, undo = pre_process[pkg]
            else:
                pkg_name, suite, affected, undo = self.doop_source(pkg)
            if hint:
                lundo.append((undo, pkg, pkg_name, suite))

            # check the affected packages on all the architectures
            for arch in ("/" in pkg and (pkg.split("/")[1].split("_")[0],) or architectures):
                if arch not in nobreakall_arches:
                    skip_archall = True
                else: skip_archall = False

                nuninst[arch] = [x for x in nuninst_comp[arch] if x in binaries[arch][0]]
                nuninst[arch + "+all"] = [x for x in nuninst_comp[arch + "+all"] if x in binaries[arch][0]]
                broken = nuninst[arch + "+all"]
                to_check = [x[0] for x in affected if x[1] == arch]
                # broken packages (first round)
                repaired = []
                broken_changed = True
                last_broken = None
                while broken_changed:
                    broken_changed = False
                    for p in to_check:
                        if p == last_broken: break
                        if p not in binaries[arch][0]: continue
                        r = check_installable(p, arch, 'testing', excluded=broken, conflicts=True)
                        if not r and p not in broken:
                            last_broken = p
                            broken.append(p)
                            broken_changed = True
                            if not (skip_archall and binaries[arch][0][p][ARCHITECTURE] == 'all'):
                                nuninst[arch].append(p)
                        elif r and p in broken:
                            last_broken = p
                            repaired.append(p)
                            broken.remove(p)
                            broken_changed = True
                            if not (skip_archall and binaries[arch][0][p][ARCHITECTURE] == 'all'):
                                nuninst[arch].remove(p)

                # broken packages (second round, reverse dependencies of the first round)
                l = 0
                broken_changed = True
                last_broken = None
                while broken_changed:
                    broken_changed = False
                    for j in broken + repaired:
                        if j not in binaries[arch][0]: continue
                        for p in binaries[arch][0][j][RDEPENDS]:
                            if p in broken or p not in binaries[arch][0]: continue
                            r = check_installable(p, arch, 'testing', excluded=broken, conflicts=True)
                            if not r and p not in broken:
                                l = -1
                                last_broken = j
                                broken.append(p)
                                broken_changed = True
                                if not (skip_archall and binaries[arch][0][p][ARCHITECTURE] == 'all'):
                                    nuninst[arch].append(p)
                            elif r and p in nuninst[arch + "+all"]:
                                last_broken = p
                                repaired.append(p)
                                broken.remove(p)
                                broken_changed = True
                                if not (skip_archall and binaries[arch][0][p][ARCHITECTURE] == 'all'):
                                    nuninst[arch].remove(p)
                    if l != -1 and last_broken == j: break

                # if we are processing hints, go ahead
                if hint:
                    nuninst_comp[arch] = nuninst[arch]
                    nuninst_comp[arch + "+all"] = nuninst[arch + "+all"]
                    continue

                # if the uninstallability counter is worse than before, break the loop
                if (("/" in pkg and arch not in new_arches) or \
                    (arch not in break_arches)) and len(nuninst[arch]) > len(nuninst_comp[arch]):
                    better = False
                    break

            # if we are processing hints or the package is already accepted, go ahead
            if hint or pkg in selected: continue

            # check if the action improved the uninstallability counters
            if better:
                lundo.append((undo, pkg, pkg_name, suite))
                selected.append(pkg)
                packages.extend(extra)
                extra = []
                self.output_write("accepted: %s\n" % (pkg))
                self.output_write("   ori: %s\n" % (self.eval_nuninst(self.nuninst_orig)))
                self.output_write("   pre: %s\n" % (self.eval_nuninst(nuninst_comp)))
                self.output_write("   now: %s\n" % (self.eval_nuninst(nuninst, nuninst_comp)))
                if len(selected) <= 20:
                    self.output_write("   all: %s\n" % (" ".join(selected)))
                else:
                    self.output_write("  most: (%d) .. %s\n" % (len(selected), " ".join(selected[-20:])))
                for k in nuninst:
                    nuninst_comp[k] = nuninst[k]
            else:
                self.output_write("skipped: %s (%d <- %d)\n" % (pkg, len(extra), len(packages)))
                self.output_write("    got: %s\n" % (self.eval_nuninst(nuninst, "/" in pkg and nuninst_comp or None)))
                self.output_write("    * %s: %s\n" % (arch, ", ".join(sorted([b for b in nuninst[arch] if b not in nuninst_comp[arch]]))))

                extra.append(pkg)
                if not mark_passed:
                    skipped.append(pkg)

                # undo the changes (source)
                for k in undo['sources'].keys():
                    if k[0] == '-':
                        del sources['testing'][k[1:]]
                    else: sources['testing'][k] = undo['sources'][k]

                # undo the changes (new binaries)
                if pkg[0] != '-' and pkg_name in sources[suite]:
                    for p in sources[suite][pkg_name][BINARIES]:
                        binary, arch = p.split("/")
                        if "/" in pkg and arch != pkg[pkg.find("/")+1:]: continue
                        del binaries[arch][0][binary]

                # undo the changes (binaries)
                for p in undo['binaries'].keys():
                    binary, arch = p.split("/")
                    if binary[0] == "-":
                        del binaries[arch][0][binary[1:]]
                    else: binaries[arch][0][binary] = undo['binaries'][p]

                # undo the changes (virtual packages)
                for p in undo['nvirtual']:
                    j, arch = p.split("/")
                    del binaries[arch][1][j]
                for p in undo['virtual']:
                    j, arch = p.split("/")
                    if j[0] == '-':
                        del binaries[arch][1][j[1:]]
                    else: binaries[arch][1][j] = undo['virtual'][p]

        # if we are processing hints, return now
        if hint:
            return (nuninst_comp, [], lundo)

        self.output_write(" finish: [%s]\n" % ",".join(selected))
        self.output_write("endloop: %s\n" % (self.eval_nuninst(self.nuninst_orig)))
        self.output_write("    now: %s\n" % (self.eval_nuninst(nuninst_comp)))
        self.output_write(self.eval_uninst(self.newlyuninst(self.nuninst_orig, nuninst_comp)))
        self.output_write("\n")

        return (nuninst_comp, extra, lundo)

    def do_all(self, maxdepth=0, init=None, actions=None):
        """Testing update runner

        This method tries to update testing checking the uninstallability
        counters before and after the actions to decide if the update was
        successful or not.
        """
        selected = []
        if actions:
            upgrade_me = actions[:]
        else:
            upgrade_me = self.upgrade_me[:]
        nuninst_start = self.nuninst_orig

        # these are special parameters for hints processing
        undo = False
        force = False
        earlyabort = False
        if maxdepth == "easy" or maxdepth < 0:
            force = maxdepth < 0
            earlyabort = True
            maxdepth = 0

        # if we have a list of initial packages, check them
        if init:
            self.output_write("leading: %s\n" % (",".join(init)))
            for x in init:
                if x not in upgrade_me:
                    self.output_write("failed: %s\n" % (x))
                    return None
                selected.append(x)
                upgrade_me.remove(x)
        
        self.output_write("start: %s\n" % self.eval_nuninst(nuninst_start))
        self.output_write("orig: %s\n" % self.eval_nuninst(nuninst_start))

        if earlyabort:
            extra = upgrade_me[:]
            (nuninst_end, extra, lundo) = self.iter_packages(init, selected, hint=True)
            undo = True
            self.output_write("easy: %s\n" % (self.eval_nuninst(nuninst_end)))
            self.output_write(self.eval_uninst(self.newlyuninst(nuninst_start, nuninst_end)) + "\n")
            if not force and not self.is_nuninst_asgood_generous(self.nuninst_orig, nuninst_end):
                nuninst_end, extra = None, None
        else:
            lundo = []
            if init:
                (nuninst_end, extra, tundo) = self.iter_packages(init, selected, hint=True)
                lundo.extend(tundo)
                undo = True
            else: nuninst_end = None
            (nuninst_end, extra, tundo) = self.iter_packages(upgrade_me, selected, nuninst=nuninst_end)
            lundo.extend(tundo)
            if not self.is_nuninst_asgood_generous(self.nuninst_orig, nuninst_end):
                nuninst_end, extra = None, None

        if nuninst_end:
            self.output_write("Apparently successful\n")
            self.output_write("final: %s\n" % ",".join(sorted(selected)))
            self.output_write("start: %s\n" % self.eval_nuninst(nuninst_start))
            self.output_write(" orig: %s\n" % self.eval_nuninst(self.nuninst_orig))
            self.output_write("  end: %s\n" % self.eval_nuninst(nuninst_end))
            if force:
                self.output_write("force breaks:\n")
                self.output_write(self.eval_uninst(self.newlyuninst(nuninst_start, nuninst_end)) + "\n")
            self.output_write("SUCCESS (%d/%d)\n" % (len(actions or self.upgrade_me), len(extra)))
            self.nuninst_orig = nuninst_end
            if not actions:
                self.upgrade_me = sorted(extra)
                if not self.options.compatible:
                    self.sort_actions()
        else:
            self.output_write("FAILED\n")
            if not undo: return

            # undo all the changes
            for (undo, pkg, pkg_name, suite) in lundo:
                # undo the changes (source)
                for k in undo['sources'].keys():
                    if k[0] == '-':
                        del self.sources['testing'][k[1:]]
                    else: self.sources['testing'][k] = undo['sources'][k]

                # undo the changes (new binaries)
                if pkg[0] != '-' and pkg_name in self.sources[suite]:
                    for p in self.sources[suite][pkg_name][BINARIES]:
                        binary, arch = p.split("/")
                        if "/" in pkg and arch != pkg[pkg.find("/")+1:]: continue
                        del self.binaries['testing'][arch][0][binary]

                # undo the changes (binaries)
                for p in undo['binaries'].keys():
                    binary, arch = p.split("/")
                    if binary[0] == "-":
                        del self.binaries['testing'][arch][0][binary[1:]]
                    else: self.binaries['testing'][arch][0][binary] = undo['binaries'][p]

                # undo the changes (virtual packages)
                for p in undo['nvirtual']:
                    j, arch = p.split("/")
                    del self.binaries['testing'][arch][1][j]
                for p in undo['virtual']:
                    j, arch = p.split("/")
                    if j[0] == '-':
                        del self.binaries['testing'][arch][1][j[1:]]
                    else: self.binaries['testing'][arch][1][j] = undo['virtual'][p]

    def upgrade_testing(self):
        """Upgrade testing using the unstable packages

        This method tries to upgrade testing using the packages from unstable.
        Before running the do_all method, it tries the easy and force-hint
        commands.
        """

        self.__log("Starting the upgrade test", type="I")
        self.__output = open(self.options.upgrade_output, 'w')
        self.output_write("Generated on: %s\n" % (time.strftime("%Y.%m.%d %H:%M:%S %z", time.gmtime(time.time()))))
        self.output_write("Arch order is: %s\n" % ", ".join(self.options.architectures))

        self.__log("> Calculating current uninstallability counters", type="I")
        self.nuninst_orig = self.get_nuninst()

        if not self.options.actions:
            # process `easy' hints
            for x in self.hints['easy']:
                self.do_hint("easy", x[0], x[1])

            # process `force-hint' hints
            for x in self.hints["force-hint"]:
                self.do_hint("force-hint", x[0], x[1])

        # run the first round of the upgrade
        self.__log("> First loop on the packages with depth = 0", type="I")

        # separate runs for break arches
        allpackages = []
        normpackages = self.upgrade_me[:]
        archpackages = {}
        for a in self.options.break_arches.split():
            archpackages[a] = [p for p in normpackages if p.endswith("/" + a)]
            normpackages = [p for p in normpackages if not p.endswith("/" + a)]
        self.upgrade_me = normpackages
        self.output_write("info: main run\n")
        self.do_all()
        allpackages += self.upgrade_me
        for a in self.options.break_arches.split():
            backup = self.options.break_arches
            self.options.break_arches = " ".join([x for x in self.options.break_arches.split() if x != a])
            self.upgrade_me = archpackages[a]
            self.output_write("info: broken arch run for %s\n" % (a))
            self.do_all()
            allpackages += self.upgrade_me
            self.options.break_arches = backup
        self.upgrade_me = allpackages

        if self.options.actions:
            return

        # process `hint' hints
        hintcnt = 0
        for x in self.hints["hint"][:50]:
            if hintcnt > 50:
                self.output_write("Skipping remaining hints...")
                break
            if self.do_hint("hint", x[0], x[1]):
                hintcnt += 1

        # run the auto hinter
        if not self.options.compatible:
            self.auto_hinter()

        # smooth updates
        if not self.options.compatible and len(self.options.smooth_updates) > 0:
            self.__log("> Removing old packages left in testing from smooth updates", type="I")
            removals = self.old_libraries()
            if len(removals) > 0:
                self.output_write("Removing packages left in testing for smooth updates (%d):\n%s" % \
                    (len(removals), self.old_libraries_format(removals)))
                self.do_all(actions=removals)
                removals = self.old_libraries()

        if not self.options.compatible:
            self.output_write("List of old libraries in testing (%d):\n%s" % \
                (len(removals), self.old_libraries_format(removals)))

        # output files
        if not self.options.dry_run:
            # re-write control files
            if self.options.control_files:
                self.write_controlfiles(self.options.testing, 'testing')

            # write bugs and dates
            self.write_bugs(self.options.testing, self.bugs['testing'])
            self.write_dates(self.options.testing, self.dates)

            # write HeidiResult
            self.write_heidi(self.options.testing, 'HeidiResult')

        self.__output.close()
        self.__log("Test completed!", type="I")

    def do_hint(self, type, who, pkgvers):
        """Process hints

        This method process `easy`, `hint` and `force-hint` hints. If the
        requested version is not in unstable, than the hint is skipped.
        """
        hintinfo = {"easy": "easy",
                    "hint": 0,
                    "force-hint": -1,}

        self.__log("> Processing '%s' hint from %s" % (type, who), type="I")
        self.output_write("Trying %s from %s: %s\n" % (type, who, " ".join( ["%s/%s" % (p,v) for (p,v) in pkgvers])))

        ok = True
        # loop on the requested packages and versions
        for pkg, v in pkgvers:
            # remove architecture
            if "/" in pkg:
                pkg = pkg[:pkg.find("/")]

            # skip removal requests
            if pkg[0] == "-":
                continue
            # handle testing-proposed-updates
            elif pkg.endswith("_tpu"):
                pkg = pkg[:-4]
                if pkg not in self.sources['tpu']: continue
                if apt_pkg.VersionCompare(self.sources['tpu'][pkg][VERSION], v) != 0:
                    self.output_write(" Version mismatch, %s %s != %s\n" % (pkg, v, self.sources['tpu'][pkg][VERSION]))
                    ok = False
            # does the package exist in unstable?
            elif pkg not in self.sources['unstable']:
                self.output_write(" Source %s has no version in unstable\n" % pkg)
                ok = False
            elif apt_pkg.VersionCompare(self.sources['unstable'][pkg][VERSION], v) != 0:
                self.output_write(" Version mismatch, %s %s != %s\n" % (pkg, v, self.sources['unstable'][pkg][VERSION]))
                ok = False
        if not ok:
            self.output_write("Not using hint\n")
            return False

        self.do_all(hintinfo[type], map(operator.itemgetter(0), pkgvers))
        return True

    def sort_actions(self):
        """Sort actions in a smart way

        This method sorts the list of actions in a smart way. In details, it uses
        as base sort the number of days the excuse is old, then reordering packages
        so the ones with most reverse dependencies are at the end of the loop.
        If an action depends on another one, it is put after it.
        """
        upgrade_me = [x.name for x in self.excuses if x.name in self.upgrade_me]
        for e in self.excuses:
            if e.name not in upgrade_me: continue
            # try removes at the end of the loop
            elif e.name[0] == '-':
                upgrade_me.remove(e.name)
                upgrade_me.append(e.name)
            # otherwise, put it in a good position checking its dependencies
            else:
                pos = []
                udeps = [upgrade_me.index(x) for x in e.deps if x in upgrade_me and x != e.name]
                if len(udeps) > 0:
                    pos.append(max(udeps))
                sdeps = [upgrade_me.index(x) for x in e.sane_deps if x in upgrade_me and x != e.name]
                if len(sdeps) > 0:
                    pos.append(min(sdeps))
                if len(pos) == 0: continue
                upgrade_me.remove(e.name)
                upgrade_me.insert(max(pos)+1, e.name)
                self.dependencies[e.name] = e.deps

        # replace the list of actions with the new one
        self.upgrade_me = upgrade_me

    def auto_hinter(self):
        """Auto hint circular dependencies

        This method tries to auto hint circular dependencies analyzing the update
        excuses relationships. If they build a circular dependency, which we already
        know as not-working with the standard do_all algorithm, try to `easy` them.
        """
        self.__log("> Processing hints from the auto hinter", type="I")

        # consider only excuses which are valid candidates
        excuses = dict([(x.name, x) for x in self.excuses if x.name in self.upgrade_me])

        def find_related(e, hint, first=False):
            if e not in excuses:
                return False
            excuse = excuses[e]
            if e in self.sources['testing'] and self.sources['testing'][e][VERSION] == excuse.ver[1]:
                return True
            if not first:
                hint[e] = excuse.ver[1]
            if len(excuse.deps) == 0:
                return hint
            for p in excuse.deps:
                if p in hint: continue
                if not find_related(p, hint):
                    return False
            return hint

        # loop on them
        cache = []
        for e in excuses:
            excuse = excuses[e]
            if e in self.sources['testing'] and self.sources['testing'][e][VERSION] == excuse.ver[1] or \
               len(excuse.deps) == 0:
                continue
            hint = find_related(e, {}, True)
            if hint and e in hint and hint not in cache:
                self.do_hint("easy", "autohinter", hint.items())
                cache.append(hint)

    def old_libraries(self):
        """Detect old libraries left in testing for smooth transitions

        This method detect old libraries which are in testing but no longer
        built from the source package: they are still there because other
        packages still depend on them, but they should be removed as soon
        as possible.
        """
        sources = self.sources['testing']
        testing = self.binaries['testing']
        unstable = self.binaries['unstable']
        removals = []
        for arch in self.options.architectures:
            for pkg_name in testing[arch][0]:
                pkg = testing[arch][0][pkg_name]
                if pkg_name not in unstable[arch][0] and \
                   not self.same_source(sources[pkg[SOURCE]][VERSION], pkg[SOURCEVER]):
                    removals.append("-" + pkg_name + "/" + arch)
        return removals

    def old_libraries_format(self, libs):
        """Format old libraries in a smart table"""
        libraries = {}
        for i in libs:
            pkg, arch = i.split("/")
            pkg = pkg[1:]
            if pkg in libraries:
                    libraries[pkg].append(arch)
            else:
                    libraries[pkg] = [arch]
        return "\n".join(["  " + k + ": " + " ".join(libraries[k]) for k in libraries]) + "\n"

    def output_write(self, msg):
        """Simple wrapper for output writing"""
        self.__output.write(msg)

    def main(self):
        """Main method
        
        This is the entry point for the class: it includes the list of calls
        for the member methods which will produce the output files.
        """
        # if no actions are provided, build the excuses and sort them
        if not self.options.actions:
            self.write_excuses()
            if not self.options.compatible:
                self.sort_actions()
        # otherwise, use the actions provided by the command line
        else: self.upgrade_me = self.options.actions.split()

        # run the upgrade test
        self.upgrade_testing()

if __name__ == '__main__':
    Britney().main()

---