736 Commits

Author SHA1 Message Date
Dan Streetman
6ca12331d6 archive: use proper component
source packages sometimes have different component than their
bpphs, so use the correct component when downloading binaries

LP: #1943819
2021-09-16 21:01:19 -04:00
Dan Streetman
5fcc4b5b46 misc: handle ConnectionError as NotFoundError 2021-09-16 20:29:14 -04:00
Dan Streetman
a3ff68be5a misc: download to tmp file, to avoid leftover 0-size file on error 2021-09-16 19:24:00 -04:00
Mattia Rizzolo
ceb020d0fa
lpapicache: fix sorting in Archive.getUploadersForPackage().
LP: #1862372
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2021-08-17 15:55:14 +02:00
Mattia Rizzolo
fbbcee9cc1
reflow code for the next commit
Gbp-Dch: Ignore
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2021-08-17 15:51:23 +02:00
Alex Murray
296e498fe9
archive: Fix PersonalPackageArchiveSourcePackage to yield URLs
Yielding the result of super()_source_urls() / _binary_urls() yields the
generator object itself which generates the list of URLs - not the URLs
which would be returned from this generator. Instead use yield from which
forwards the yield onto the generator object itself.

Fixes LP: #1938659

Signed-off-by: Alex Murray <alex.murray@canonical.com>
2021-08-03 10:39:55 +09:30
Dan Streetman
b4ca04efaa test: fix archive tests
Assumptions were made about the implementation by mocking
that are no longer true, and the tests generally need to be
fixed to be more robust about testing
2021-07-14 17:41:10 -04:00
Dan Streetman
0eaf71737d test: fix example package class
this class can be much simpler and more robust and doesn't require
checking in the example package files to git
2021-07-14 17:41:10 -04:00
Dan Streetman
1b12d8b4e3 archive: convert to use pathlib instead of os.path
Mostly convert to using pathlib
2021-07-14 17:41:10 -04:00
Dan Streetman
be6e09a02b archive: don't use existing file if no verification methods provided
we shouldn't use an existing file if we aren't checking its checksums
2021-07-14 17:41:10 -04:00
Dan Streetman
86276665cd archive: Update PersonalPackageArchiveSourcePackage to handle private PPA
Unfortunately private PPAs require downloading files from the special server
private-ppa.launchpad.net, and all the usual urls provided by the LP api fail.
So add code to handle using those custom private URLs, and use authentication
when downloading.
2021-07-14 17:41:10 -04:00
Dan Streetman
3f2983c157 misc: Change download() method to use python requests and optional authentication
Since private PPAs require authentication, use python requests library instead
of urlopen(), since requests handles authentication easily
2021-07-14 17:41:10 -04:00
Dan Streetman
4f6d6bf2d8 misc: add extract_authentication method
This pulls the username:password out of a URL
2021-07-14 17:41:10 -04:00
Dan Streetman
df93a225a8 misc: replace os.path with Pathlib
also change some strings to use f-strings
2021-07-14 17:41:10 -04:00
Dan Streetman
d8df8cc869 misc: add download_bytes() and deprecate mode param for download_text()
Passing 'mode' assumes use of open(), but callers don't care about
implementation, just if the returned object is text or bytes
2021-07-14 17:41:04 -04:00
Dan Streetman
243a728d6c Move DownloadError into ubuntutools/misc 2021-07-13 14:08:59 -04:00
Dan Streetman
6e18d60de4 lpapicache: add Archive.getMySubscriptionURL()
Private PPA require using username/password to access their files,
so make this information available. This gets the currently logged in
user's "subscription URL", which includes the authentication data.
2021-07-13 07:55:02 -04:00
Dan Streetman
952b331c22 archive: fix flake8 test failure 2021-06-04 12:14:10 -04:00
Dan Streetman
a0315dac8c archive: only download dsc file to the workdir from pull() method
LP: #1928946
2021-06-02 11:24:53 -04:00
Dan Streetman
9be49e7b93 archive: deprecate poorly-named pull_dsc() method 2021-05-28 16:24:15 -04:00
Dan Streetman
90824e056c archive: move check for verify_signature into check_dsc_signature 2021-05-28 16:18:50 -04:00
Dan Streetman
1093c372eb archive: allow passing absolute path to _download_file 2021-05-28 16:17:29 -04:00
Dan Streetman
ff66707a4c Add mode param to download_text() to allow using custom modes like 'rb' 2021-05-28 16:14:51 -04:00
Krytarik Raido
728849964e
Logging: Fix oversight from the last logging refactor
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2021-02-24 14:26:18 +01:00
Mattia Rizzolo
09537bd44d
Merge branch 'fix-sponsor-patch' of git+ssh://git.launchpad.net/~logan/ubuntu-dev-tools
MR: https://code.launchpad.net/~logan/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/398509
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2021-02-24 14:24:50 +01:00
Logan Rosen
bc24ef23de sponsor-patch: fix bugs from py3 migration 2021-02-22 22:24:52 -05:00
Dan Streetman
0eeb93ee0c logging: update ubuntutools.getLogger() to output to stdout/stderr correctly
Python logging by default sends all output to stderr, but that's not what
normal programs usually do and is not expected for these scripts.

Change the ubuntutools.getLogger() method to return a logger with handlers
correctly set up to send INFO level (or lower) logs to stdout and WARNING
level (or higher; technically INFO+1 level or higher) logs to stderr.

This results in normally logged output going to stdout and warnings/errors
going to stderr, as expected.
2021-02-02 06:25:12 -05:00
Dan Streetman
b35712fa40 archive: wrap filtered masters in list
LP: #1913330
2021-02-02 06:25:12 -05:00
Dan Streetman
ee9b8756d9 archive: make SourcePackage ABC abstract class to enforce subclasses provide distribution 2021-02-02 06:25:12 -05:00
Dan Streetman
9792f5b95c archive: don't download from archive if using UCA staging ppa
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
d8d4fd9285 archive: log warnings around 'staging' pocket
The 'staging' pocket contains binaries that don't match published binaries,
so log a warning if we fallback to that pocket, and log a warning if pulling
binaries using that pocket.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
9a38ad1b7b archive: add UCA parsing/finding code for UCA releases/pockets
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
db181f4aa6 archive: use UCA Project to find valid UCA release names
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
04ae04e17b archive: simplify PPA source package class
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
ae3837be1d lp: add Project and ProjectSeries classes
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
20036e6c36 ubuntutools/pullpkg: add --login parameter
Just in case we want to pull from a private ppa...

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
e13a4eb45e ubuntutools/archive: deprecate passing 'lp' object to SourcePackage()
There is no point to doing this, as the Launchpad object is a singleton;
the caller should just call Launchpad.login_with() directly and leave
the SourcePackage class out of it.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
cb865e3b86 ubuntutools/lp: change default login to anonymous
If not specified, default to logging in anonymously.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
61f0023c37 archive: use verify_file_checksums()
This reduces duplicate verification steps, and results in logging
error from the verification function if there is a size mismatch,
instead of the silent verification failure that was present in case
neither sha checksum was provided.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
e5a42b1ba1 misc: add verify_file_checksums() function
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
359cb18d8d lpapicache: replace httplib2 use with call to download_text()
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
b89ad962f4 misc: add download_text() method
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
58ca95e719 pullpkg: provide dst to download()
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
a26a154bc9 misc: move logic into download() to handle plain file paths, and src == dst
Update download() function to handle src of plain path, by prepending
'file://' to it.  Also handle the case of src and dst pointing to the same
file.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
fed562405d archive: simplify handling of dsc file
There are only 2 options for getting the dsc file:
1) it's provided as 'dscfile' param
2) we look it up via lp_spph

For case #1, update the constructor to immediately read the provided file
to create the self._dsc object, and set package and version from that.

For case #2, simplify various functions to simply use the normal
_download_file_from_urls() function to get the dsc file from the
normal locations (mirrors, masters, directly from lp), using the same
process as any other file, including checking checksum(s).

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
248cf38d79 misc: update download() function
-require 'dst' parameter
-allow 'dst' parameter to be dir or dest file
-use contextlib.suppress instead of try/except/pass

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
3d0921ee54 archive: simplify _source_urls() and _binary_urls()
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
c29b660dbf lpapicache: make sure source/binary file metadata includes all keys
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
b2f4ceee8e archive: use _download_file_from_urls()
Both pull() and pull_binaries() are changed to use the common function.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
b6e0b5b388 archive: add _download_file_from_urls()
This will consolidate the download functionality from pull() and
pull_binaries()

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00