147 Commits

Author SHA1 Message Date
ferbraher
4a4c4e0a27 Parsing arch parameter to getBinaryPackage() 2025-03-04 13:08:59 -06:00
Benjamin Drung
d7362d9ed8 Use Python f-strings
```
flynt -ll 99 -tc -tj -a pbuilder-dist pm-helper running-autopkgtests ubuntu-build ubuntutools
```
2024-11-02 17:49:20 +01:00
Simon Quigley
bed2dc470d Add support for the non-free-firmware components in all tools already referencing non-free. 2023-07-26 13:04:12 -05:00
Steve Langasek
89e788bf48 Remove references to deprecated http://people.canonical.com/~ubuntu-archive. 2023-05-30 19:37:11 -07:00
Benjamin Drung
aa556af89d Use f-strings
pylint complains about C0209: Formatting a regular string which could be
a f-string (consider-using-f-string)

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 19:32:58 +01:00
Benjamin Drung
444b319c12 Implement refactorings found by pylint
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 16:58:24 +01:00
Benjamin Drung
4449cf2437 Fix warnings found by pylint
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 15:51:29 +01:00
Benjamin Drung
f6fde2e217 fix: Use lazy % formatting in logging functions
pylint complains about W1201: Use lazy % formatting in logging functions
(logging-not-lazy) and W1203: Use lazy % formatting in logging functions
(logging-fstring-interpolation).

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 11:13:07 +01:00
Benjamin Drung
72add78e9d Fix errors found by pylint
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 10:19:24 +01:00
Benjamin Drung
b1bc7e1cdc Address pylint complaints
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-30 23:10:31 +01:00
Benjamin Drung
4e27045f49 style: Sort Python imports with isort
```
isort -l 99 --profile=black .
```

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-30 21:28:47 +01:00
Benjamin Drung
3354b526b5 style: Format Python code with black
```
PYTHON_SCRIPTS=$(grep -l -r '^#! */usr/bin/python3$' .)
black -C -l 99 . $PYTHON_SCRIPTS
```

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-30 19:45:36 +01:00
Mattia Rizzolo
dabe475067
ubuntutools/archive.py: fix crash in SourcePackage()._source_urls()
Fix operation of SourcePackage._source_urls() (as used, for example, in
SourcePackage.pull() called by backportpackage) to also work when the
class is instantiated with a URL as .dsc.

This is a regression caused by 1b12d8b4e3315de3bf417b40a3c66279f309d72c
(first in v0.184) that moved from os.path.join() to Pathlib, but
os.path.join() was also used to join URLs.

Thanks: Unit 193 for the initial patch.
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2022-09-29 10:34:51 +02:00
Mattia Rizzolo
3a903ca628
archive.py: support python 3.6
this is needed for the backports to bionic

Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2021-12-09 17:56:50 +01:00
Dan Streetman
d05e023dfe ubuntutools: don't sys.exit if unpack fails, just log it 2021-11-19 08:11:31 -05:00
Dan Streetman
6ca12331d6 archive: use proper component
source packages sometimes have different component than their
bpphs, so use the correct component when downloading binaries

LP: #1943819
2021-09-16 21:01:19 -04:00
Alex Murray
296e498fe9
archive: Fix PersonalPackageArchiveSourcePackage to yield URLs
Yielding the result of super()_source_urls() / _binary_urls() yields the
generator object itself which generates the list of URLs - not the URLs
which would be returned from this generator. Instead use yield from which
forwards the yield onto the generator object itself.

Fixes LP: #1938659

Signed-off-by: Alex Murray <alex.murray@canonical.com>
2021-08-03 10:39:55 +09:30
Dan Streetman
1b12d8b4e3 archive: convert to use pathlib instead of os.path
Mostly convert to using pathlib
2021-07-14 17:41:10 -04:00
Dan Streetman
be6e09a02b archive: don't use existing file if no verification methods provided
we shouldn't use an existing file if we aren't checking its checksums
2021-07-14 17:41:10 -04:00
Dan Streetman
86276665cd archive: Update PersonalPackageArchiveSourcePackage to handle private PPA
Unfortunately private PPAs require downloading files from the special server
private-ppa.launchpad.net, and all the usual urls provided by the LP api fail.
So add code to handle using those custom private URLs, and use authentication
when downloading.
2021-07-14 17:41:10 -04:00
Dan Streetman
3f2983c157 misc: Change download() method to use python requests and optional authentication
Since private PPAs require authentication, use python requests library instead
of urlopen(), since requests handles authentication easily
2021-07-14 17:41:10 -04:00
Dan Streetman
d8df8cc869 misc: add download_bytes() and deprecate mode param for download_text()
Passing 'mode' assumes use of open(), but callers don't care about
implementation, just if the returned object is text or bytes
2021-07-14 17:41:04 -04:00
Dan Streetman
243a728d6c Move DownloadError into ubuntutools/misc 2021-07-13 14:08:59 -04:00
Dan Streetman
952b331c22 archive: fix flake8 test failure 2021-06-04 12:14:10 -04:00
Dan Streetman
a0315dac8c archive: only download dsc file to the workdir from pull() method
LP: #1928946
2021-06-02 11:24:53 -04:00
Dan Streetman
9be49e7b93 archive: deprecate poorly-named pull_dsc() method 2021-05-28 16:24:15 -04:00
Dan Streetman
90824e056c archive: move check for verify_signature into check_dsc_signature 2021-05-28 16:18:50 -04:00
Dan Streetman
1093c372eb archive: allow passing absolute path to _download_file 2021-05-28 16:17:29 -04:00
Dan Streetman
b35712fa40 archive: wrap filtered masters in list
LP: #1913330
2021-02-02 06:25:12 -05:00
Dan Streetman
ee9b8756d9 archive: make SourcePackage ABC abstract class to enforce subclasses provide distribution 2021-02-02 06:25:12 -05:00
Dan Streetman
9792f5b95c archive: don't download from archive if using UCA staging ppa
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
d8d4fd9285 archive: log warnings around 'staging' pocket
The 'staging' pocket contains binaries that don't match published binaries,
so log a warning if we fallback to that pocket, and log a warning if pulling
binaries using that pocket.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
9a38ad1b7b archive: add UCA parsing/finding code for UCA releases/pockets
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
db181f4aa6 archive: use UCA Project to find valid UCA release names
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
04ae04e17b archive: simplify PPA source package class
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
e13a4eb45e ubuntutools/archive: deprecate passing 'lp' object to SourcePackage()
There is no point to doing this, as the Launchpad object is a singleton;
the caller should just call Launchpad.login_with() directly and leave
the SourcePackage class out of it.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
61f0023c37 archive: use verify_file_checksums()
This reduces duplicate verification steps, and results in logging
error from the verification function if there is a size mismatch,
instead of the silent verification failure that was present in case
neither sha checksum was provided.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
a26a154bc9 misc: move logic into download() to handle plain file paths, and src == dst
Update download() function to handle src of plain path, by prepending
'file://' to it.  Also handle the case of src and dst pointing to the same
file.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
fed562405d archive: simplify handling of dsc file
There are only 2 options for getting the dsc file:
1) it's provided as 'dscfile' param
2) we look it up via lp_spph

For case #1, update the constructor to immediately read the provided file
to create the self._dsc object, and set package and version from that.

For case #2, simplify various functions to simply use the normal
_download_file_from_urls() function to get the dsc file from the
normal locations (mirrors, masters, directly from lp), using the same
process as any other file, including checking checksum(s).

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
3d0921ee54 archive: simplify _source_urls() and _binary_urls()
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
b2f4ceee8e archive: use _download_file_from_urls()
Both pull() and pull_binaries() are changed to use the common function.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
b6e0b5b388 archive: add _download_file_from_urls()
This will consolidate the download functionality from pull() and
pull_binaries()

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
494e0d6ddd archive: change log level of some messages
Change some messages to the correct log level for their importanance.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Stefano Rivera
90cb51becb archive.py: Evaluate the filter() fixing syncpackage. 2021-01-29 22:18:22 -07:00
Dan Streetman
390be3f0b3 archive: remove DebianSourcePackage.pull_dsc()
No longer necessary to override this function as the superclass
now does everything the subclass function does.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-01-22 16:19:49 -05:00
Dimitri John Ledkov
02c531dfb5
archive.py: use Regular, Ports, and Internal mirrors by default. Thus enabling pull-lp-debs to work with ports architectures, and inside launchpad builds too. 2021-01-19 10:43:48 +00:00
Dan Streetman
ea30b9f5bc ubuntutools/misc: add download() function
Move the download functionality from the archive.py into the common
misc.py file.
2020-03-13 10:02:05 -04:00
Dan Streetman
11429b213c archive: remove _lp_url() 2020-03-06 13:01:37 +01:00
Dan Streetman
eaa07014e0 archive: verify and use existing already-downloaded files 2020-03-06 13:01:37 +01:00
Dan Streetman
8682c44957 archive: update pull_dsc to use _source_urls() and verify_file_checksum() 2020-03-06 13:01:37 +01:00