63 Commits

Author SHA1 Message Date
Robie Basak
232a73de31 ubuntutools/misc: swap iter_content for raw stream
This is a partial revert of 1e20363.

When downloading a .diff.gz source package file, we do expect it to be
written to disk still compressed. If we were to uncompress it, then we
would get a size mismatch and even if we were to ignore that, we'd get a
hash mismatch.

On the other hand when downloading a changes file we need to make sure
that is written to disk uncompressed.

To make this work in both cases we can ask the HTTP server for no
special content encoding using "Accept-Encoding: identity". This is what
wget requests, for example. Then we can write the output to the file
without performing any decoding at our end by using the raw response
object again.

This fixes both cases.

LP: #2025748
2023-07-06 11:28:21 +01:00
Benjamin Drung
c8a757eb07 Format Python code with black 23.1
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-04-04 12:11:36 +02:00
Benjamin Drung
aa556af89d Use f-strings
pylint complains about C0209: Formatting a regular string which could be
a f-string (consider-using-f-string)

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 19:32:58 +01:00
Benjamin Drung
444b319c12 Implement refactorings found by pylint
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 16:58:24 +01:00
Benjamin Drung
4449cf2437 Fix warnings found by pylint
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 15:51:29 +01:00
Benjamin Drung
f6fde2e217 fix: Use lazy % formatting in logging functions
pylint complains about W1201: Use lazy % formatting in logging functions
(logging-not-lazy) and W1203: Use lazy % formatting in logging functions
(logging-fstring-interpolation).

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 11:13:07 +01:00
Benjamin Drung
17bed46ffb feat: Add some type hints
Add some type hints to satisfy mypy.

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-31 10:35:22 +01:00
Benjamin Drung
b1bc7e1cdc Address pylint complaints
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-30 23:10:31 +01:00
Benjamin Drung
4e27045f49 style: Sort Python imports with isort
```
isort -l 99 --profile=black .
```

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-30 21:28:47 +01:00
Benjamin Drung
3354b526b5 style: Format Python code with black
```
PYTHON_SCRIPTS=$(grep -l -r '^#! */usr/bin/python3$' .)
black -C -l 99 . $PYTHON_SCRIPTS
```

Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
2023-01-30 19:45:36 +01:00
Dan Streetman
de295fe524 ubuntutools/misc: return dst Path object from download() 2021-11-19 08:11:31 -05:00
Dan Streetman
1e2036399e ubuntutools/misc: use iter_content instead of raw stream access
Reading the raw stream doesn't decode files that are content-encoded,
which is true for the 'changes' file from launchpad, so it's saved
to disk gzipped which isn't what is expected.

Using the python requests iter_content method instead uses the
built-in stream decoding that the requests library provides, and
saves the file uncompressed/unencoded.

Note that since the encoded Content-Length won't match the resulting
unencoded actual length of data we save to file, this also turns off
the progress bar for any files that have Content-Encoding.
2021-11-19 08:11:31 -05:00
Dan Streetman
85125e3c90 ubuntutools/misc: allow specifying blocksize to download methods 2021-11-19 08:11:15 -05:00
Dan Streetman
cfa45994d0 ubuntutools/misc: create helper class to display download progress bar 2021-11-19 08:09:08 -05:00
Dan Streetman
266085d587 misc: fix flake8 complaints 2021-09-17 07:25:47 -04:00
Dan Streetman
5fcc4b5b46 misc: handle ConnectionError as NotFoundError 2021-09-16 20:29:14 -04:00
Dan Streetman
a3ff68be5a misc: download to tmp file, to avoid leftover 0-size file on error 2021-09-16 19:24:00 -04:00
Dan Streetman
3f2983c157 misc: Change download() method to use python requests and optional authentication
Since private PPAs require authentication, use python requests library instead
of urlopen(), since requests handles authentication easily
2021-07-14 17:41:10 -04:00
Dan Streetman
4f6d6bf2d8 misc: add extract_authentication method
This pulls the username:password out of a URL
2021-07-14 17:41:10 -04:00
Dan Streetman
df93a225a8 misc: replace os.path with Pathlib
also change some strings to use f-strings
2021-07-14 17:41:10 -04:00
Dan Streetman
d8df8cc869 misc: add download_bytes() and deprecate mode param for download_text()
Passing 'mode' assumes use of open(), but callers don't care about
implementation, just if the returned object is text or bytes
2021-07-14 17:41:04 -04:00
Dan Streetman
243a728d6c Move DownloadError into ubuntutools/misc 2021-07-13 14:08:59 -04:00
Dan Streetman
ff66707a4c Add mode param to download_text() to allow using custom modes like 'rb' 2021-05-28 16:14:51 -04:00
Dan Streetman
e5a42b1ba1 misc: add verify_file_checksums() function
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
b89ad962f4 misc: add download_text() method
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
a26a154bc9 misc: move logic into download() to handle plain file paths, and src == dst
Update download() function to handle src of plain path, by prepending
'file://' to it.  Also handle the case of src and dst pointing to the same
file.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
248cf38d79 misc: update download() function
-require 'dst' parameter
-allow 'dst' parameter to be dir or dest file
-use contextlib.suppress instead of try/except/pass

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
8aa2d602c5 pullpkg: add support for pulling from upload queue 2020-03-13 10:02:05 -04:00
Dan Streetman
ea30b9f5bc ubuntutools/misc: add download() function
Move the download functionality from the archive.py into the common
misc.py file.
2020-03-13 10:02:05 -04:00
Dan Streetman
8ebd086aac ubuntutools/misc: add verify_file_checksum function 2020-03-06 13:01:37 +01:00
Dan Streetman
9f428e471b ubuntutools/misc: convert print() to std logging 2020-03-06 13:01:37 +01:00
Dan Streetman
7c097b19ba ubuntutools: add --status param to pull-pkg
Allow specifying what package statuses should be searched for.

By default search only for Pending and Published, unless a specific
version number is being searched for.
2019-11-25 10:10:06 -05:00
Dan Streetman
d0aa64a51b ubuntutools/misc: define POCKETS and DEFAULT_POCKETS
POCKETS is all valid pockets (capitalized), DEFAULT_POCKETS is all in
POCKETS except 'Backports'.
2019-11-25 09:57:54 -05:00
Mattia Rizzolo
21adb8f61f
fix flake8 warning
Signed-off-by: Mattia Rizzolo <mattia@debian.org>
2019-09-26 11:05:48 +02:00
Stefano Rivera
317b68bc08 Use a context manager for opening files 2019-09-11 13:38:42 -03:00
Stefano Rivera
7a6b779e77 ubuntutools.misc: Replace Popen() calls with check_output() 2019-09-11 13:38:42 -03:00
Stefano Rivera
301569e809 Pull out Python 2 support hacks 2019-09-04 19:24:44 -03:00
Stefano Rivera
43ad610a66 Add encoding support to our Popen wrapper 2019-09-04 19:24:44 -03:00
Benjamin Drung
cc7170eccb Fix all flake8 issues 2017-05-01 00:20:03 +02:00
Dimitri John Ledkov
1558b91dde misc.py 2014-12-18 20:34:04 +00:00
Stefano Rivera
9ba1790863 pull-lp-source, requestbackport: Take the latest version from any
non-backports pocket. Implemented by making lpapicache's getSourcePackage
smarter.
2011-11-23 01:45:49 +02:00
Benjamin Drung
e591ab20cf backportpackage: Use Ubuntu and Debian as fall back check in codename_to_distribution to allow backporting to Ubuntu from a Debian system (LP: #823832). 2011-09-08 18:21:57 +02:00
Benjamin Drung
e9efe8d0f0 pull-lp-source: Support source packages with a bad version string
(LP: #844682).
2011-09-08 16:40:29 +02:00
Benjamin Drung
ce2f96bc0d Make pylint happier and code more PEP-8 compliant. 2011-08-20 11:31:02 +02:00
Benjamin Drung
a03fa654f3 Move debian-distro-info, distro-info, and ubuntu-distro-info from
ubuntu-dev-tools into distro-info.
2011-06-25 17:53:44 +02:00
Evan Broder
19c33e5e32 * ubuntutools.subprocess:
- New drop-in replacement wrapper module around subprocess that
    backports the restore_signals kwarg and defaults close_fds=True
  - Switch everything previously using subprocess to use
    ubuntutools.subprocess instead (LP: #785854)
2011-06-24 16:32:07 +02:00
Benjamin Drung
de32133e75 distro_info.py: Add validity check method. 2011-06-15 00:01:49 +02:00
Evan Broder
16300e471f Move vendor_to_distroinfo and codename_to_distribution into ubuntutools.misc 2011-06-11 06:04:09 -07:00
Evan Broder
2398d09ba4 ubuntutools.misc: Add a new "system_distribution_chain", which returns
a list starting with the current distribution and working its way up
each distribution's parent.
2011-06-11 05:05:35 -07:00
Evan Broder
d82e65e0e2 * ubuntutools.subprocess:
- New drop-in replacement wrapper module around subprocess that
    backports the restore_signals kwarg and defaults close_fds=True
  - Switch everything previously using subprocess to use
    ubuntutools.subprocess instead (LP: #785854)
2011-05-24 20:22:37 +02:00