708 Commits

Author SHA1 Message Date
Benjamin Drung
17bed46ffb feat: Add some type hints
Add some type hints to satisfy mypy.

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 10:35:22 +01:00
Benjamin Drung
72add78e9d Fix errors found by pylint
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 10:19:24 +01:00
Benjamin Drung
ab64467f33 Run pylint during package build again
Commit ae74f71a1e9d4be043162b19d23f2d44c964c771 removed the pylint unit
test saying that unit tests are not needed to just run flake8 or pylint.

Since pylint is useful, add it back, but this time call it directly and
not embed it into a unit test.

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 00:05:15 +01:00
Benjamin Drung
b1bc7e1cdc Address pylint complaints
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-30 23:10:31 +01:00
Benjamin Drung
4e27045f49 style: Sort Python imports with isort
```
isort -l 99 --profile=black .
```

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-30 21:28:47 +01:00
Benjamin Drung
3354b526b5 style: Format Python code with black
```
PYTHON_SCRIPTS=$(grep -l -r '^#! */usr/bin/python3$' .)
black -C -l 99 . $PYTHON_SCRIPTS
```

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-30 19:45:36 +01:00
Benjamin Drung
096d5612e7 sponsor-patch: Use --skip-patches when extract source package
Use `--skip-patches` when extract source packages with `dpkg-source`.
`--no-preparation` is a source package build option and `--skip-patches`
is the correct extract option.

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2022-11-16 18:37:56 +01:00
Benjamin Drung
b510dbd91e sponsor-patch: Ignore exit code 1 of debdiff call
sponsor-patch calls `debdiff` which exits with 1 if there are
differences. So accept exit codes 0 and 1 as expected.

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2022-11-15 16:43:27 +01:00
Mattia Rizzolo
dabe475067
ubuntutools/archive.py: fix crash in SourcePackage()._source_urls()
Fix operation of SourcePackage._source_urls() (as used, for example, in
SourcePackage.pull() called by backportpackage) to also work when the
class is instantiated with a URL as .dsc.

This is a regression caused by 1b12d8b4e3315de3bf417b40a3c66279f309d72c
(first in v0.184) that moved from os.path.join() to Pathlib, but
os.path.join() was also used to join URLs.

Thanks: Unit 193 for the initial patch.
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2022-09-29 10:34:51 +02:00
Dan Streetman
562e6b13cd lpapicache: force lp access on login to workaround possibly invalid cached creds 2022-08-09 12:08:50 -04:00
Dan Streetman
9c1561ff26 lpapicache: remove try-except around login that only logs the error and then re-raises 2022-08-09 12:07:31 -04:00
Graham Inggs
0dde3262d1
ubuntutools/lp: Python 3.10 compatibility 2022-01-20 15:38:03 +01:00
Mattia Rizzolo
3a903ca628
archive.py: support python 3.6
this is needed for the backports to bionic

Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2021-12-09 17:56:50 +01:00
Dan Streetman
8fe22fbbb6 Remove hugdaylist script
The last email announcing a 'hug day' appears to have been over
5 years ago, and the hugdaylist script doesn't seem to work
right anymore anyway.

This also removes ubuntutools/lp/libsupport, which has no functions
used by anything else.
2021-11-22 13:00:52 -05:00
Dan Streetman
4d72d184db pullpkg: change pull_upload_queue params into specific keyword-only params 2021-11-19 08:11:31 -05:00
Dan Streetman
20261960f6 pullpkg: unpack downloaded src from upload queue 2021-11-19 08:11:31 -05:00
Dan Streetman
d05e023dfe ubuntutools: don't sys.exit if unpack fails, just log it 2021-11-19 08:11:31 -05:00
Dan Streetman
de295fe524 ubuntutools/misc: return dst Path object from download() 2021-11-19 08:11:31 -05:00
Dan Streetman
1e2036399e ubuntutools/misc: use iter_content instead of raw stream access
Reading the raw stream doesn't decode files that are content-encoded,
which is true for the 'changes' file from launchpad, so it's saved
to disk gzipped which isn't what is expected.

Using the python requests iter_content method instead uses the
built-in stream decoding that the requests library provides, and
saves the file uncompressed/unencoded.

Note that since the encoded Content-Length won't match the resulting
unencoded actual length of data we save to file, this also turns off
the progress bar for any files that have Content-Encoding.
2021-11-19 08:11:31 -05:00
Dan Streetman
85125e3c90 ubuntutools/misc: allow specifying blocksize to download methods 2021-11-19 08:11:15 -05:00
Dan Streetman
cfa45994d0 ubuntutools/misc: create helper class to display download progress bar 2021-11-19 08:09:08 -05:00
Dan Streetman
266085d587 misc: fix flake8 complaints 2021-09-17 07:25:47 -04:00
Dan Streetman
6ca12331d6 archive: use proper component
source packages sometimes have different component than their
bpphs, so use the correct component when downloading binaries

LP: #1943819
2021-09-16 21:01:19 -04:00
Dan Streetman
5fcc4b5b46 misc: handle ConnectionError as NotFoundError 2021-09-16 20:29:14 -04:00
Dan Streetman
a3ff68be5a misc: download to tmp file, to avoid leftover 0-size file on error 2021-09-16 19:24:00 -04:00
Mattia Rizzolo
ceb020d0fa
lpapicache: fix sorting in Archive.getUploadersForPackage().
LP: #1862372
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2021-08-17 15:55:14 +02:00
Mattia Rizzolo
fbbcee9cc1
reflow code for the next commit
Gbp-Dch: Ignore
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2021-08-17 15:51:23 +02:00
Alex Murray
296e498fe9
archive: Fix PersonalPackageArchiveSourcePackage to yield URLs
Yielding the result of super()_source_urls() / _binary_urls() yields the
generator object itself which generates the list of URLs - not the URLs
which would be returned from this generator. Instead use yield from which
forwards the yield onto the generator object itself.

Fixes LP: #1938659

Signed-off-by: Alex Murray <alex.murray@canonical.com>
2021-08-03 10:39:55 +09:30
Dan Streetman
b4ca04efaa test: fix archive tests
Assumptions were made about the implementation by mocking
that are no longer true, and the tests generally need to be
fixed to be more robust about testing
2021-07-14 17:41:10 -04:00
Dan Streetman
0eaf71737d test: fix example package class
this class can be much simpler and more robust and doesn't require
checking in the example package files to git
2021-07-14 17:41:10 -04:00
Dan Streetman
1b12d8b4e3 archive: convert to use pathlib instead of os.path
Mostly convert to using pathlib
2021-07-14 17:41:10 -04:00
Dan Streetman
be6e09a02b archive: don't use existing file if no verification methods provided
we shouldn't use an existing file if we aren't checking its checksums
2021-07-14 17:41:10 -04:00
Dan Streetman
86276665cd archive: Update PersonalPackageArchiveSourcePackage to handle private PPA
Unfortunately private PPAs require downloading files from the special server
private-ppa.launchpad.net, and all the usual urls provided by the LP api fail.
So add code to handle using those custom private URLs, and use authentication
when downloading.
2021-07-14 17:41:10 -04:00
Dan Streetman
3f2983c157 misc: Change download() method to use python requests and optional authentication
Since private PPAs require authentication, use python requests library instead
of urlopen(), since requests handles authentication easily
2021-07-14 17:41:10 -04:00
Dan Streetman
4f6d6bf2d8 misc: add extract_authentication method
This pulls the username:password out of a URL
2021-07-14 17:41:10 -04:00
Dan Streetman
df93a225a8 misc: replace os.path with Pathlib
also change some strings to use f-strings
2021-07-14 17:41:10 -04:00
Dan Streetman
d8df8cc869 misc: add download_bytes() and deprecate mode param for download_text()
Passing 'mode' assumes use of open(), but callers don't care about
implementation, just if the returned object is text or bytes
2021-07-14 17:41:04 -04:00
Dan Streetman
243a728d6c Move DownloadError into ubuntutools/misc 2021-07-13 14:08:59 -04:00
Dan Streetman
6e18d60de4 lpapicache: add Archive.getMySubscriptionURL()
Private PPA require using username/password to access their files,
so make this information available. This gets the currently logged in
user's "subscription URL", which includes the authentication data.
2021-07-13 07:55:02 -04:00
Dan Streetman
952b331c22 archive: fix flake8 test failure 2021-06-04 12:14:10 -04:00
Dan Streetman
a0315dac8c archive: only download dsc file to the workdir from pull() method
LP: #1928946
2021-06-02 11:24:53 -04:00
Dan Streetman
9be49e7b93 archive: deprecate poorly-named pull_dsc() method 2021-05-28 16:24:15 -04:00
Dan Streetman
90824e056c archive: move check for verify_signature into check_dsc_signature 2021-05-28 16:18:50 -04:00
Dan Streetman
1093c372eb archive: allow passing absolute path to _download_file 2021-05-28 16:17:29 -04:00
Dan Streetman
ff66707a4c Add mode param to download_text() to allow using custom modes like 'rb' 2021-05-28 16:14:51 -04:00
Krytarik Raido
728849964e
Logging: Fix oversight from the last logging refactor
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2021-02-24 14:26:18 +01:00
Mattia Rizzolo
09537bd44d
Merge branch 'fix-sponsor-patch' of git+ssh://git.launchpad.net/~logan/ubuntu-dev-tools
MR: https://code.launchpad.net/~logan/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/398509
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2021-02-24 14:24:50 +01:00
Logan Rosen
bc24ef23de sponsor-patch: fix bugs from py3 migration 2021-02-22 22:24:52 -05:00
Dan Streetman
0eeb93ee0c logging: update ubuntutools.getLogger() to output to stdout/stderr correctly
Python logging by default sends all output to stderr, but that's not what
normal programs usually do and is not expected for these scripts.

Change the ubuntutools.getLogger() method to return a logger with handlers
correctly set up to send INFO level (or lower) logs to stdout and WARNING
level (or higher; technically INFO+1 level or higher) logs to stderr.

This results in normally logged output going to stdout and warnings/errors
going to stderr, as expected.
2021-02-02 06:25:12 -05:00
Dan Streetman
b35712fa40 archive: wrap filtered masters in list
LP: #1913330
2021-02-02 06:25:12 -05:00