79 Commits

Author SHA1 Message Date
Dan Streetman
b11b83f0e2 pull-pkg: change UCA source pkg class to subclass of PPA source pkg class
The UCA source pkg class was its own specific class, but with the
addition of the generic PPA source pkg class, the UCA class is just
a specific case of PPA.  Changing it to a subclass of the PPA class
simplifies code.
2019-11-25 10:10:06 -05:00
Dan Streetman
7084bfc8bc pull-pkg: add pull-ppa-* functionality
Add functionality, and frontend pull-ppa-* scripts, to be able to pull
from PPA archives.
2019-11-25 10:10:06 -05:00
Dan Streetman
3491b0cff9 pull-pkg: change pullpkg into class PullPkg
instead of pullpkg.py containing a simple method to call, change it
into a normal class PullPkg that callers can create and use.
2019-11-25 10:10:06 -05:00
Dan Streetman
fb750e38bb lpapicache: find binaries from lp
Also update getBinaries() to allow retreival by binary name

This significantly speeds up binary file lookup for non-published
package versions, since we can get the list of binary urls, but
then have to look up the BPPH for each of those urls, which is slow.
If the user only wants a specific binary package and/or arch, this
speeds up getting that, especially for packages with a lot of binary
files (like the kernel).
2019-11-25 10:10:06 -05:00
Dan Streetman
3a413760f3 ubuntutools: add pull-* --no-verify-signature option, don't fail if no pub key
Change dsc verification to fail only if the public key was available, but
signature verification failed.  If no public key is available for the dsc,
print warning only. (LP: #1700846)

Also add pull-* parameter --no-verify-signature to manually prevent failure
when signature verification fails.
2019-11-25 10:10:06 -05:00
Dan Streetman
f4f16f95f7 pull-uca: update UbuntuCloudArchiveSourcePackage
look in its PPA for its SPPHs
look up the development (latest) UCA release if not specified
use the 'series' param as the UCA release, instead of 'uca_release' param
2019-11-25 09:57:54 -05:00
Dan Streetman
b96885f05e ubuntutools/archive: convert external rmadison call to native implementation
instead of a function that calls the system program rmadison, use
a fully-functional class to interface with the madison api, as well
as the debian snapshot api
2019-11-25 09:57:54 -05:00
Dan Streetman
f944d3146a pull-pkg: find src pkg name from binary pkg name 2019-11-25 09:57:54 -05:00
Dan Streetman
9f58f5cb80 pull-pkg: find latest pkg in series in SourcePackage
Allow specifying series instead of version, to find the latest
version in that series; or only specify package name, to get the
latest version in the devel series
2019-11-25 09:57:54 -05:00
Dan Streetman
b0c22e1d57 pull-pkg: update archive.py SourcePackage
verify it can parse/load DSC when created
throw PackageNotFoundException from lp_spph if can't find package
update test case expected order of url processing
2019-11-25 09:57:54 -05:00
Dan Streetman
506e3db601 pull-pkg: add SourcePackage.pull_binaries() function 2019-11-25 09:57:54 -05:00
Mattia Rizzolo
982ebe6a46
Trust the installed debian-keyring when checking validity of dsc signatures. 2019-10-28 14:35:01 +01:00
Mattia Rizzolo
68b41fc82f
Drop workaround for an old pylint bug
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2019-09-10 14:25:35 +02:00
Mattia Rizzolo
434ca8952e
Merge branch 'python3' of git+ssh://git.launchpad.net/ubuntu-dev-tools
MR: https://code.launchpad.net/~ubuntu-dev/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/372305
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2019-09-10 14:19:07 +02:00
Dan Streetman
a7d166b52f pull-pkg: use ubuntutools/version/Version
instead of debian.debian_support.Version, use ubuntutools version,
which extends it and adds the strip_epoch() function which is
needed to convert full version string to version used in filenames
2019-09-09 12:25:21 -04:00
Stefano Rivera
06642ef9f7 Remove some more u'' strings 2019-09-05 16:10:26 -03:00
Stefano Rivera
301569e809 Pull out Python 2 support hacks 2019-09-04 19:24:44 -03:00
Stefano Rivera
bb765237db Port Python scripts to Python 3, remove python 2 modules. (Closes: #938740, LP: #1099537) 2019-09-04 19:24:44 -03:00
Stefano Rivera
2d3765522e Use the Popen() encoding flag to decode to unicode
This should make behavior identical on Python 2 & 3.
2019-09-04 19:24:44 -03:00
Mattia Rizzolo
e0f6b6f97b
archive: Disable dsc signature verification for non-Debian.
Ubuntu doesn't have a unified keyring of developers like Debian has, so
it is not feasible to check for the dsc signatures.

Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2019-08-12 13:41:16 +02:00
Mattia Rizzolo
3fb7516d72
Default to checking signatures while pulling a .dsc.
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2019-08-05 13:50:33 +02:00
Robie Basak
a761ccfc72 Fix rmadison parsing for Python 3
In Python 3, it matters that the output of rmadison must be decoded
before it can be parsed.
2017-06-30 15:47:29 +01:00
Robie Basak
4c66fba4d9 pull-debian-source: --no-verify-signature option
Using pull-debian-source fails on some very old packages such as on
texinfo 4.8.dfsg.1-4. I have hand-verified that the signature is good
(though with no trust path), so presumably this is because the signature
has rotated out of debian-keyring.

Add a --no-verify-signature option so that developers can still make use
of the find-and-download functionality of this tool, albeit without
signture verification.
2017-06-30 15:47:29 +01:00
Benjamin Drung
cc7170eccb Fix all flake8 issues 2017-05-01 00:20:03 +02:00
Dan Streetman
03fda64eae pull-uca-source: update to allow using binary pkg names, -pockets, and specific versions 2017-02-03 10:21:37 -05:00
Scott Moser
12c86a8fda ubuntutools/archive.py/rmadison: return results for sid if asked about unstable
Some versions of rmadison return results with 'unstable' when asked about sid.
Others return 'sid'.  This makes a query of 'unstable' return results
for sid.

End result is fixing 'pull-debian-source hello' on wily.
2015-10-22 10:00:21 -04:00
Dimitri John Ledkov
866adfd768 Actually, now that more things expect streaming bytes over the wire
the mirror tests don't stall anymore. Also use a codec.reader to slurp
up unicode bytes for json.load.
2014-12-18 23:18:00 +00:00
Dimitri John Ledkov
5da114b070 More consistent binary handling throughout.
Don't mock open(..., "b") with str / StringIO.
Silence source package pull, buffered/mocked output does not flush.
Disable mirror tests on python3, stall/hang.
2014-12-18 23:03:23 +00:00
Dimitri John Ledkov
ddabeed530 Fix ups 2014-12-18 21:34:13 +00:00
Dimitri John Ledkov
a7dedd9296 archive.py 2014-12-16 02:38:52 +00:00
Benjamin Drung
c2aa97dba2 Move devscripts.logger to ubuntutools.logger. 2013-03-19 00:18:02 +01:00
Stefano Rivera
4056e98a43 ubuntutools.archive: Improve error handling around rmadison calls
(LP: #1010951)
2012-06-20 22:32:20 +02:00
Stefano Rivera
52fc42ca8f Make ubuntutools.archive a little more testable, by making its urlopener an
instance property.

Replace these with mocks, as needed, rather than mocking out urllib2.urlopen.

Makes the test suite pass again.
2012-03-26 10:00:46 +02:00
Stefano Rivera
872957da4e Whitespace 2012-03-26 09:41:05 +02:00
Daniel Hahler
ee569590a6 ubuntutools/archive.py: use ProxyHandler in _download_file.
This makes use of the system proxy (e.g. http_proxy).
2012-03-16 16:22:14 +01:00
Stefano Rivera
70a035f13d Use httplib2 everywhere that we do https. The python stdlib doesn't do
certificate verification.
2011-11-22 15:57:02 +02:00
Stefano Rivera
09e468211a Add changelog retrieval to lpapicache, and use this in syncpackage and
requestsync. The changelogs should be available in Launchpad sooner than
Debian PTS.
2011-11-13 22:50:34 +02:00
Stefano Rivera
bf0709edf6 Cleanup progress bar if there's an exception, too 2011-09-10 01:25:23 +02:00
Stefano Rivera
ca2abaa613 * ubuntutools.archive:
- Add quiet option to silence downloading.
  - Use wget-style progress bar (fixed width) (LP: #845787)
2011-09-09 22:44:16 +02:00
Stefano Rivera
17b287e7bf ubuntutools.archive: Don't write .dsc files until we pull the entire
source package, just hold it in memory. Avoids littering the current
directory (LP: #838361)
2011-09-04 19:01:01 +02:00
Colin Watson
a4000c606c drop quiet download stuff, not worth an API debate 2011-08-18 15:10:19 +01:00
Colin Watson
d624e9d18f factor out dsc comparison into a method on ubuntutools.archive.Dsc 2011-08-18 15:06:05 +01:00
Colin Watson
d26a7521b0 make it possible to quieten download progress information, particularly useful if downloading only the small .dsc files 2011-08-17 02:54:37 +01:00
Stefano Rivera
aa99618dd6 backportpackage: Backport from local source packages again (LP: #801945) 2011-08-16 01:17:25 +02:00
Evan Broder
19c33e5e32 * ubuntutools.subprocess:
- New drop-in replacement wrapper module around subprocess that
    backports the restore_signals kwarg and defaults close_fds=True
  - Switch everything previously using subprocess to use
    ubuntutools.subprocess instead (LP: #785854)
2011-06-24 16:32:07 +02:00
Stefano Rivera
6aee8410eb Work around pylint bug: http://www.logilab.org/ticket/46273 2011-05-30 15:55:15 +02:00
Stefano Rivera
1c248875b4 ubuntutools.archive: Display any errors rmadison emits, rather than
guessing at the cause. (LP: #788447)
2011-05-30 10:21:12 +02:00
Evan Broder
d82e65e0e2 * ubuntutools.subprocess:
- New drop-in replacement wrapper module around subprocess that
    backports the restore_signals kwarg and defaults close_fds=True
  - Switch everything previously using subprocess to use
    ubuntutools.subprocess instead (LP: #785854)
2011-05-24 20:22:37 +02:00
Benjamin Drung
e8b3a18c45 Move suspicious-source and wrap-and-sort from ubuntu-dev-tools into devscripts. 2011-05-23 23:41:00 +02:00
Scott Kitterman
f5ce76d52e In ubuntutools/archive.py, instead of an AssertionError liburi-perl is not installed provide a similar message to the one rmadison provides. 2011-04-10 00:41:30 -04:00