758 Commits

Author SHA1 Message Date
Martin Pitt
eac1bf30a9 Re-fix merging of source packages
We need complete copies of the SourcePackage objects, as we might later find
some binaries which are only in unstable.

Traceback (most recent call last):
  File "/home/ubuntu-archive/proposed-migration/code/b2/britney.py", line 3290, in <module>
    Britney().main()
  File "/home/ubuntu-archive/proposed-migration/code/b2/britney.py", line 3261, in main
    self.write_excuses()
  File "/home/ubuntu-archive/proposed-migration/code/b2/britney.py", line 1963, in write_excuses
    if should_upgrade_srcarch(pkg, arch, 'unstable'):
  File "/home/ubuntu-archive/proposed-migration/code/b2/britney.py", line 1534, in should_upgrade_srcarch
    False)
  File "/home/ubuntu-archive/proposed-migration/code/b2/britney.py", line 2294, in _compute_groups
    if binaries_t[parch][0][binary].source != source_name:
KeyError: 'kicad'
2016-10-12 16:33:47 +02:00
Martin Pitt
2e3a6c2cdb Python 3.2 workaround: Drop usage of clear()
The clear() collection method is not yet available in Python 3.2, but as long
as snakefruit still runs Ubuntu 12.04 we need to get along without it.
2016-10-11 11:00:53 +02:00
Martin Pitt
9f3b7a2864 Python 3.2 workaround: Drop usage of enum module
enum is not yet available in Python 3.2, but as long as snakefruit still runs
Ubuntu 12.04 we need to get along without it.
2016-10-11 11:00:53 +02:00
Martin Pitt
6ba940c7f7 Add "run-autopkgtest" tool to re-run tests 2016-10-11 11:00:53 +02:00
Martin Pitt
943621deb2 Add autopkgtest integration
Add new module autopkgtest.py with the logic for determining the tests for a
source package, requesting tests via AMQP, fetching results from swift, and
keeping track of pending tests between run. This also caches the downloaded
results from swift, as re-dowloading them all is very expensive.

Integrate this into britney.py:

 * In should_upgrade_src(), check whether a package is built everywhere and
   installable (run_autopkgtest), and good enough to run autopkgtests for it
and its reverse dependencies.

 * In write_excuses(), generate test requests for all excuses and create blocks
   for those that cause test regresssions.

This introduces two new hints:

 * force-badtest pkg/ver: Failing results for that package will be ignored.
   This is useful to deal with broken tests that get imported from Debian or
   are from under-maintained packages, or broke due to some infrastructure
   changes. These are long-lived usually.

 * force-skiptest pkg/ver: Test results *triggered by* that package (i. e.
   reverse dependencies) will be ignored. This is mostly useful for landing
   packages that trigger a huge amount of tests (glibc, perl) where some tests
   are just too flaky to get them all passing, and one just wants to land it
   after the remaining failures have been checked. This should be used rarely
   and the hints should be removed immediately again.

Add integration tests that call britney in various scenarios on constructed
fake archives, with mocked AMQP and Swift results.
2016-10-11 11:00:53 +02:00
Martin Pitt
d99e769988 Make linux* wait on corresponding -meta package
We want to treat linux-$flavor and linux-meta-$flavor as one set in britney
which goes in together or not at all. We never want to promote linux-$flavor
without the accompanying linux-meta-$flavor.

Introduce a synthetic linux* → linux-meta* dependency to enforce this grouping.
2016-10-11 11:00:53 +02:00
Dimitri John Ledkov
420e83b2ca Teach excuse_unsat_deps() about components ogre model 2016-10-11 10:56:51 +02:00
Colin Watson
73650fc12c Change date format to seconds-since-epoch 2016-10-11 10:56:51 +02:00
Colin Watson
78a4959597 Allow blocking uploads by way of bugs with the block-proposed tag set 2016-10-11 10:56:51 +02:00
Colin Watson
372c34783a Link to wiki documentation 2016-10-11 10:56:51 +02:00
Colin Watson
1f50273a25 Update contact information for Ubuntu 2016-10-11 10:56:51 +02:00
Colin Watson
afc2a1d7f8 Show links to Launchpad package pages in excuses 2016-10-11 10:56:51 +02:00
Colin Watson
20945927de buildd.debian.org -> launchpad.net 2016-10-11 10:56:51 +02:00
Colin Watson
c9b69afc1c britney.conf: add Ubuntu hinters
Delegate "block" and "unblock" hints to members of ~ubuntu-touch-release.

Add active members of the SRU team to hint permissions.
2016-10-11 10:56:51 +02:00
Colin Watson
ae28048c2c britney.conf: Ubuntu configuration
- use relative paths
 - set Ubuntu architectures
 - make all output files series specific
 - mark -proposed as a partial suite
 - set mindays to 0 for all urgencies
 - drop tpu and pu
 - disable smooth updates
 - disable removals of obsolete source packages
 - disable components, using old merged package lists for now
2016-10-11 10:56:51 +02:00
Martin Pitt
07e1f9459b Store Testsuite: list in sources
Extend read_sources to store the components of Testsuite: as a list.
2016-10-11 10:56:51 +02:00
Colin Watson
3ee64804ed Add a --distribution option 2016-10-11 10:56:51 +02:00
Colin Watson
7380e2eb92 Add new --series option
Replace %(SERIES) macro in config files with given value.
2016-10-11 10:56:50 +02:00
Colin Watson
b1134d9712 Only issue "From wrong source" for binaries not at the same version in testing
Same-version could happen for merged binaries in partial-unstable mode.
2016-10-11 10:56:50 +02:00
Colin Watson
200377bef1 Optionally merge packages from testing to unstable
To cope with a partial unstable suite like Ubuntu's -proposed pocket.
2016-10-11 10:56:50 +02:00
Colin Watson
75dcfddccd Make sure that containing directories exist before writing output files
We don't use os.makedirs(dir, exist_ok=True) as that is too strict: it fails if
the directory already exists with different permissions (e. g. with 775). Thus
introduce a helper function ensuredir().
2016-10-11 09:25:50 +02:00
Robert Bruce Park
7be991e720 Make Heidi optional
Setting HEIDI_OUTPUT to an empty value will skip the heidi generation, to speed
up britney in scenarios where heidi is not needed.
2016-10-11 09:24:00 +02:00
Martin Pitt
d1ea68dd51 Skip second-stage upgrade testing if UPGRADE_OUTPUT option is absent or empty
This is useful to speed up britney in dry run mode, if we are only interested
in builds and package tests.
2016-10-11 09:24:00 +02:00
Colin Watson
b549788210 Skip bug-based processing if BugsV is missing 2016-10-11 09:24:00 +02:00
Colin Watson
475f8366b1 Use default urgency for all packages if Urgencies is missing 2016-10-11 09:24:00 +02:00
Martin Pitt
c40edb6cff Always reject packages with unsatisfiable dependencies
Partially revert commit ac66e311 which caused packages with unsatisfiable
dependencies to only get rejected if they were not in testing. In Ubuntu we
always want to block those.
2016-10-11 09:24:00 +02:00
Martin Pitt
4b5c4dc266 Consider packages with M-A qualifiers for reverse dependencies
Strip of Multi-Arch qualifiers ":any" and ":native" when building the
dependency fields, as they are not part of the package name.

This will fix cases like

  Package: ipython3
  Depends: python3:any (>= 3)

and include ipython3 in python3's reverse dependencies.

Closes: #794194
2016-10-11 09:23:57 +02:00
Emilio Pozuelo Monfort
250c9314a6 Also do it from britney_nobreakall.conf
Signed-off-by: Emilio Pozuelo Monfort <pochu@debian.org>
2016-10-10 01:32:25 +02:00
Emilio Pozuelo Monfort
e7eb6603fe Remove mips64el from break/outofsync arches
Signed-off-by: Emilio Pozuelo Monfort <pochu@debian.org>
2016-10-10 00:59:58 +02:00
Colin Watson
e5e6d7e5b7 Make removal of obsolete source packages configurable
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:29:09 +00:00
Colin Watson
fe7cc466e1 Rename fucked_arches to outofsync_arches
To avoid getting in trouble when bringing up new commercially-supported
architectures.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:29:09 +00:00
Niels Thykier
0664c580b0 Prefer s.x to s[X] for SourcePackage objects
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:29:09 +00:00
Niels Thykier
5a19b4f74d Create a SourcePackage class
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:29:09 +00:00
Niels Thykier
204359e3bc inst-tester: Fix typo in a variable name
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:29:09 +00:00
Niels Thykier
cd7cdda61e britney: Remove out a tuple lookup in get_dependency_solvers
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:29:09 +00:00
Niels Thykier
f414b12130 britney: Make read_binaries read all binaries for a suite
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:29:09 +00:00
Niels Thykier
ea166f5bd3 britney -> util: Extract a "create_provides_map"
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:05:54 +00:00
Niels Thykier
6de0c3408c AgePolicy: Create the (new) date file automatically
Closes: Debian/britney2#13
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:05:54 +00:00
Robert Bruce Park
190be31014 Python loop performance enhancements.
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:05:54 +00:00
Niels Thykier
34d27bbd15 britney.py: Make hinted removal excuses valid
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:05:53 +00:00
Mathieu Trudel-Lapierre
31669ac5bd Update whitespace; make the output easier to read
FAILED/SUCCESS lines would be separated by a whitespace from the list
of architectures, but not itself followed by whitespace. This is slightly
confusing, as one could interpret it as being a heading for the following
block of tested packages, rather that the final result of the previous
block.
2016-10-03 18:05:53 +00:00
Colin Watson
f527835076 Clarify excuse HTML/text for the mindays == 0 case 2016-10-03 18:05:53 +00:00
Niels Thykier
ec21cf1342 Add an ignore-rc-bugs hint
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:05:53 +00:00
Niels Thykier
29ecbd80d0 Let the age policy register its own hints
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:05:53 +00:00
Niels Thykier
d43311be03 Support policies registering their own hints
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:05:53 +00:00
Niels Thykier
9efe92de39 Extract an AgeDayHint class from Hint
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:05:53 +00:00
Niels Thykier
e5846fdc8b hints: Move Hint creation out of add_hints
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:05:53 +00:00
Niels Thykier
948d15d536 HintParser: Support adding new hints to the parser
This includes refining "HINTS_ALL" to cover all hints added at
runtime.

Currently, it is not very useful.  However, a later commit will allow
policies to use this feature.

Signed-off-by: Niels Thykier <niels@thykier.net>
2016-10-03 18:05:53 +00:00
Niels Thykier
13417c18e4 hint-tester: Use hint-parser for parsing excuse hints
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-09-23 05:27:44 +00:00
Niels Thykier
6328b15a9c Change argument order for HintCollection.add_hint
Signed-off-by: Niels Thykier <niels@thykier.net>
2016-09-23 05:27:44 +00:00