1463 Commits

Author SHA1 Message Date
Michael Hudson-Doyle
dfe2205bc7 add a message when list of valid/invalid excuses fails to validate 2020-11-20 17:37:44 +00:00
Iain Lane
f0e46d7bfa britney: Disable Built-Using policy
Launchpad currently doesn't enforce this, so it doesn't make too much
sense to do so in proposed-migration.

Once https://bugs.launchpad.net/launchpad/+bug/1868558 is fixed, we
should think about probably re-enabling this.
2020-11-20 17:37:44 +00:00
Iain Lane
8fb4d5d841 BlockPolicy, ExcuseFinder: refer to DISTRO-release
On Ubuntu we want this to say ubuntu-release, on Debian debian-release.
Substitute the first part with the name of the distro.
2020-11-20 17:37:44 +00:00
Iain Lane
0dbeb64fde Add Ubuntu configuration 2020-11-20 17:37:44 +00:00
Iain Lane
b453096702 excuse: When writing HTML, say that packages are not considered when they need further action
We currently have excuses.html saying:

  x264 (2:0.155.2917+git0a84d98-2 to 2:0.159.2999+git296494a-2)

      Migration status for x264 (2:0.155.2917+git0a84d98-2 to 2:0.159.2999+git296494a-2): BLOCKED: Cannot migrate due to another item, which is blocked (please check which dependencies are stuck)
      Issues preventing migration:
      Implicit dependency: x264 nageru (not considered)
      Invalidated by implicit-dependency
      Implicit dependency: x264 ffmpeg
      Implicit dependency: x264 gst-plugins-ugly1.0
      Implicit dependency: x264 handbrake
      Implicit dependency: x264 libquicktime
      Implicit dependency: x264 mplayer
      Implicit dependency: x264 mythtv
      Implicit dependency: x264 obs-studio
      Implicit dependency: x264 vlc
      Implicit dependency: x264 xpra
      Additional info:
      8 days old

Apart from `nageru` it's not clear here that some further packages need
working on to move this migration forward. In fact, the package
`gst-plugins-ugly1.0` is FTBFS

  Migration status for gst-plugins-ugly1.0 (1.16.2-2build2 to
  1.16.2-2build3): BLOCKED: Maybe temporary, maybe blocked but Britney
  is missing information (check below)

as is xpra. The remaining packages are simply those which are part of
this SONAME transition, as such they have

  Migration status for mplayer (2:1.3.0-8build6 to 2:1.3.0-8build7):
  BLOCKED: Cannot migrate due to another item, which is blocked (please
  check which dependencies are stuck)

All of these excuses are (or should be) "invalid", but only the ones
which are not BLOCKED_BY_ANOTHER_ITEM require direct action. We should
show this, so that maintainers can see what they need to concentrate on.

When outputting depdencency problems, consider the policy verdict of the
excuse. If it's one that indicates further work is required for that
excuse, indicate this by outputting "(not considered)".
2020-11-20 17:37:44 +00:00
Iain Lane
2ed6da1e0d autopkgtest: When skipping linux-* packages for Ubuntu, consider the source suite
We often introduce new kernels post-release, and we still want to not
trigger tests for the kernel image packages. Check for -meta in the
*source* suite as well as the target.

Fix the -meta name calculation for linux-signed-foo.
2020-11-20 17:37:44 +00:00
Iain Lane
e871a3993d autopkgtest: Add back hack to only trigger selected tests for gcc-N
We might want to consider doing this in a more generic way.
2020-11-20 17:37:44 +00:00
Iain Lane
96715dcd2b autopkgtest: Import amqplib when submitting test requests
Commit b82f4fc63238e48a8b7c76f32576dee2cd57d403 moved the importing of
amqplib to be done when required, but it missed a spot.
2020-11-20 17:37:44 +00:00
Iain Lane
7d6d4debdf autopkgtest, DependsPolicy: Run tests where we didn't care about the depends problem
We currently skip running autopkgtests where there is an installability
problem, but in a few cases the depends policy notes these only, but
otherwise doesn't block migration on them.

In these cases, let's try to run the autopkgtests anyway. There will be
a few instances of uninstallability here, but since we migrate the
packages we should give them a chance to be tested.
2020-11-20 17:37:44 +00:00
Iain Lane
94b6ac273a autopkgtest: Submit multiple triggers to amqp as a list
We currently concatenate all triggers together into a string, but the
AMQP consumer expects this to be a list.

When using AMQP, keep the triggers as a list. Ensure that the "real"
trigger (the package being tested) is kept first, as before.
2020-11-20 17:37:44 +00:00
Iain Lane
4b74228867 britney: Allow breaking of packages on one arch when removing cruft
If all smooth updating/cruft is on a BREAK_ARCH then this can
legitimately happen.
2020-11-20 17:37:44 +00:00
Iain Lane
fab4a6eed9 InstallabilityTester: Update the pseudo-essential cache when we remove a "choice" too
When we're removing choice, we need to look into all the choice groups
and invalidate the cache if it's in there.
2020-11-20 17:37:44 +00:00
Iain Lane
e5c3f86393 autopkgtest: Don't output an info line for ignored failures
We already skip displaying most other types of results where no action
is required including ALWAYSFAIL, should do it for hints too
2020-11-20 17:37:44 +00:00
Iain Lane
fe748849e0 suiteloader: Handle NBS
If we have this situation

  - Binary in target taken over by new source
  - Not cleaned up yet
  - New source updated in source suite

We can have *three* copies of a source package visible to britney at
once. Handle this case by recording the source package a pkgid was seen
in before, and trying to remove from that and the current source.
2020-11-20 17:37:44 +00:00
Iain Lane
67b9f56741 suiteloader: Don't need to do an 'in' check when adding to a set 2020-11-20 17:37:44 +00:00
Iain Lane
429df37b66 autopkgtest: Retry downloads
If we have a slightly flaky network / endpoint, retrying can help us to
keep going rather than crash & restart.
2020-11-20 17:37:44 +00:00
Iain Lane
a5828a269e excusefinder: Support not having arch-all buildds
On some distros (Ubuntu), arch:all packages are built along with one of
the architectures. We shouldn't be listing 'all' as its own arch in this
case. Instead we filter out the binaries except for on the
'all_buildarch'.
2020-11-20 17:37:44 +00:00
Iain Lane
c9d45ecb2b Support writing excuses YAML to xz and gz files 2020-11-20 17:37:44 +00:00
Colin Watson
65351ba511 buildd.debian.org -> launchpad.net 2020-11-20 17:37:44 +00:00
Iain Lane
eb8f1c9fc7 Allow HEIDI_OUTPUT to be empty 2020-11-20 17:37:44 +00:00
Iain Lane
95b71e0ddb autopkgtest: Save pending.json after every test request
We just had the autopkgtest queues DoSed because britney was crashing
after requesting each reverse dependency for a perl upload, but before
it had written pending.json out so it knew what not to request again.
This was 25,000 requests per arch...

Let's write pending.json straight after sending each request, so that
the next run - even after a crash - won't re-request the same things
again.
2020-11-20 17:37:44 +00:00
Iain Lane
07b9c517c8 BuildDependsPolicy: handle old python3-apt
Very old versions, such as that run on the Ubuntu archive machine, don't
have the arch parameter.

Handle that by setting APT::Architecture instead
2020-11-20 17:37:44 +00:00
Iain Lane
9523425cb2 autopkgtest: Support Ubuntu's run_id format
These have a hash appended.

We don't actually use the baseline retrying, which is where the ID
parsing is used, but we might as well handle this, not least so we don't
crash.
2020-11-20 17:37:44 +00:00
Iain Lane
c9dfcf9b06 autopkgtest: Allow adt_reference_max_age to be empty 2020-11-20 17:37:44 +00:00
Iain Lane
15f431d37c autopkgtest: Add new force-reset-test hint type
This allows hinters to reset the baseline for a package/arch, so that
previous failures are ignore and not considered regressions
2020-11-20 17:37:44 +00:00
Iain Lane
27c2476e8f autopkgtest: Only request reference tests when we want them 2020-11-20 17:37:44 +00:00
Iain Lane
e5b7f5f539 autopkgtest: Support autopkgtest.u.c in-progress URL
This is at /running. We use 'are we running on swift?' as a proxy for
deciding which URL format to use
2020-11-20 17:37:44 +00:00
Iain Lane
af085da46e autopkgtest: Handle 'blacklisted' version
The apt version comparison sorts 'blacklisted' greater than most version
numbers, which means that we accidentally apply force hints for version
'blacklisted' to all uploads. Since this is the only case of a hacked
version number, let's special case it so that 'blacklisted' hints only
match packages with 'blacklisted' version.
2020-11-20 17:37:44 +00:00
Iain Lane
34c8b05010 autopkgtest: store the submission time when making requests
This is so that we can see more easily see if requests are stuck
2020-11-20 17:37:44 +00:00
Iain Lane
b2054e6cb2 Repair requests without testpkg-version
We're supposed to synthesise an "unknown" version for these, but a bug
in the worker meant we didn't do that in some cases and these leaked
into swift. Let's repair it client-side.
2020-11-20 17:37:44 +00:00
Iain Lane
b2e863fd6f autopkgtest: Publish AMQP messages persistently
We want to ask the broker to write them out to disk, hoepfully
mitigating against it crashing.
2020-11-20 17:37:44 +00:00
Iain Lane
2623053f82 test_autopkgtest: Re-enable test for package not built on arch
This is a test carried over from Ubuntu which ensures that a package
which builds no binaries on an arch doesn't have tests requested.

It was disabled. Enable it.
2020-11-20 17:37:44 +00:00
Iain Lane
af091a504c autopkgtest: Trigger dkms when there's a dep on dkms
Most DKMS packages do not declare Testsuite: autopkgtest-pkg-dkms, but
we can detect this anyway, and this way we can enforce that the module
is buildable.
2020-11-20 17:37:44 +00:00
Łukasz 'sil2100' Zemczak
d960c15706 Add a new policy to message bugs on SRU regressions
It works like this. We wait until all tests have finished running. and
then grab their results. If there are any regressions, we mail each bug
with a link to pending-sru.html. There's a state file which records the
mails we've sent out, so that we don't mail the same bug multiple times.
2020-11-20 17:37:44 +00:00
Tiago Daitx
256a48a4ad Link to update-excuse bugs on update_excuses
LP: #1683749
2020-11-20 17:37:44 +00:00
Iain Lane
bae4fdbd45 LinuxPolicy: Make linux* wait on corresponding -meta package
We want to treat linux-$flavor and linux-meta-$flavor as one set in
britney which goes in together or not at all. We never want to promote
linux-$flavor without the accompanying linux-meta-$flavor.

Add a new LinuxPolicy which runs after most of the other policies, which
invalidates linux-foo if linux-meta-foo is invalid.
2020-11-20 17:37:44 +00:00
Robert Bruce Park
36e8b65c8b Email Policy, send emails when packages are rejected. 2020-11-20 17:37:44 +00:00
Robert Bruce Park
9c777eac1b Reject packages if entire source ppa won't migrate 2020-11-11 09:41:40 +00:00
Iain Lane
5b5c99e401 excuse, excusefinder: Allow excuses to invalidate each other
Sometimes (for the LinuxPolicy), excuses don't have enough information
when they are processed. They need to defer some of their work until
another excuse has been evaluated and then perhaps get invalidated.

Introduce the concept of "external invalidation" to deal with this. A
policy can keep references to other excuses around, and then invalidate
them later (when processing another one). The excuse finder needs to
know this happened to remove the excuse from the list of "actionable
excuses".

For example. The Linux policy invalidates linux-foo if linux-meta-foo is
invalid. If linux-foo is encountered first, we will not yet know if
linux-meta-foo is going to be invalid. We effectively defer the
evaluation until after linux-meta-foo has been dealt with, and then we
know whether we need to invalidate linux-foo or not.
2020-11-11 09:41:40 +00:00
Iain Lane
d2dbdadf1f autopkgtest: Hardcode some linux-meta tests
We should really fix this to use Testsuite-Triggers / hint-testsuite-triggers
2020-11-11 09:41:40 +00:00
Iain Lane
56c066b753 Allow piuparts policy to be disabled by config 2020-11-11 09:41:40 +00:00
Iain Lane
7ed335893e Implement Ubuntu component relationship constraints (ogre model) 2020-11-11 09:41:40 +00:00
Iain Lane
3fa9eceb89 Store the component in BinaryPackage and SourcePackage 2020-11-11 09:41:40 +00:00
Colin Watson
4f9a33ddab Allow blocking uploads by way of bugs with the block-proposed tag set 2020-11-11 09:41:40 +00:00
Iain Lane
1487096a15 test_autopkgtest: Split setup commands into a separate class
This lets other policies share this code for their integration tests
2020-10-20 14:57:51 +01:00
Colin Watson
4e59e3fb6a Change date format to seconds-since-epoch 2020-10-20 14:57:51 +01:00
Colin Watson
964e414dbe Add a --distribution option 2020-10-20 14:57:51 +01:00
Colin Watson
f4dc6b2277 Replace %(SERIES) macro in config files with value passed to --series 2020-10-20 14:57:51 +01:00
Colin Watson
5f95cf771e Only issue "From wrong source" for binaries not at the same version in testing
Same-version could happen for merged binaries in partial-unstable mode.
2020-10-20 14:57:51 +01:00
Iain Lane
77a2a0df19 utils: Fix pycodestyle E302 (missing blank line) 2020-10-20 14:57:51 +01:00