122 Commits

Author SHA1 Message Date
Dan Streetman
9be49e7b93 archive: deprecate poorly-named pull_dsc() method 2021-05-28 16:24:15 -04:00
Dan Streetman
90824e056c archive: move check for verify_signature into check_dsc_signature 2021-05-28 16:18:50 -04:00
Dan Streetman
1093c372eb archive: allow passing absolute path to _download_file 2021-05-28 16:17:29 -04:00
Dan Streetman
b35712fa40 archive: wrap filtered masters in list
LP: #1913330
2021-02-02 06:25:12 -05:00
Dan Streetman
ee9b8756d9 archive: make SourcePackage ABC abstract class to enforce subclasses provide distribution 2021-02-02 06:25:12 -05:00
Dan Streetman
9792f5b95c archive: don't download from archive if using UCA staging ppa
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
d8d4fd9285 archive: log warnings around 'staging' pocket
The 'staging' pocket contains binaries that don't match published binaries,
so log a warning if we fallback to that pocket, and log a warning if pulling
binaries using that pocket.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
9a38ad1b7b archive: add UCA parsing/finding code for UCA releases/pockets
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
db181f4aa6 archive: use UCA Project to find valid UCA release names
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
04ae04e17b archive: simplify PPA source package class
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
e13a4eb45e ubuntutools/archive: deprecate passing 'lp' object to SourcePackage()
There is no point to doing this, as the Launchpad object is a singleton;
the caller should just call Launchpad.login_with() directly and leave
the SourcePackage class out of it.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
61f0023c37 archive: use verify_file_checksums()
This reduces duplicate verification steps, and results in logging
error from the verification function if there is a size mismatch,
instead of the silent verification failure that was present in case
neither sha checksum was provided.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
a26a154bc9 misc: move logic into download() to handle plain file paths, and src == dst
Update download() function to handle src of plain path, by prepending
'file://' to it.  Also handle the case of src and dst pointing to the same
file.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
fed562405d archive: simplify handling of dsc file
There are only 2 options for getting the dsc file:
1) it's provided as 'dscfile' param
2) we look it up via lp_spph

For case #1, update the constructor to immediately read the provided file
to create the self._dsc object, and set package and version from that.

For case #2, simplify various functions to simply use the normal
_download_file_from_urls() function to get the dsc file from the
normal locations (mirrors, masters, directly from lp), using the same
process as any other file, including checking checksum(s).

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
3d0921ee54 archive: simplify _source_urls() and _binary_urls()
Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
b2f4ceee8e archive: use _download_file_from_urls()
Both pull() and pull_binaries() are changed to use the common function.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
b6e0b5b388 archive: add _download_file_from_urls()
This will consolidate the download functionality from pull() and
pull_binaries()

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Dan Streetman
494e0d6ddd archive: change log level of some messages
Change some messages to the correct log level for their importanance.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-02-02 06:25:12 -05:00
Stefano Rivera
90cb51becb archive.py: Evaluate the filter() fixing syncpackage. 2021-01-29 22:18:22 -07:00
Dan Streetman
390be3f0b3 archive: remove DebianSourcePackage.pull_dsc()
No longer necessary to override this function as the superclass
now does everything the subclass function does.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
2021-01-22 16:19:49 -05:00
Dimitri John Ledkov
02c531dfb5
archive.py: use Regular, Ports, and Internal mirrors by default. Thus enabling pull-lp-debs to work with ports architectures, and inside launchpad builds too. 2021-01-19 10:43:48 +00:00
Dan Streetman
ea30b9f5bc ubuntutools/misc: add download() function
Move the download functionality from the archive.py into the common
misc.py file.
2020-03-13 10:02:05 -04:00
Dan Streetman
11429b213c archive: remove _lp_url() 2020-03-06 13:01:37 +01:00
Dan Streetman
eaa07014e0 archive: verify and use existing already-downloaded files 2020-03-06 13:01:37 +01:00
Dan Streetman
8682c44957 archive: update pull_dsc to use _source_urls() and verify_file_checksum() 2020-03-06 13:01:37 +01:00
Dan Streetman
20dd65b281 archive: verify binary file checksums 2020-03-06 13:01:37 +01:00
Dan Streetman
45253b398c archive: remove unneeded 'found' flag
This var isn't needed - just using for/else is better
2020-03-06 13:01:37 +01:00
Dan Streetman
e1f2cbf831 archive: remove _binary_files_info()
also use spph/bpph provided urls
2020-03-06 13:01:37 +01:00
Dan Streetman
907061c15e lpapicache: support 'include_meta' param in binaryFileUrls()
support the param for both native BPPH as well as Debian Snapshot
emulated BPPH

add sourceFileUrls() function

add helper functions to get file urls, checksums, and size
2020-03-06 13:01:37 +01:00
Dan Streetman
d98e16226b archive: change _download_file verify param to dscverify
The param is specific to dsc-verifiable files, so name it accordingly.
2020-03-06 12:18:10 +01:00
Dan Streetman
53fa2336b8 archive: cache Debian srcpkg when pulling binaries 2020-03-06 12:18:10 +01:00
Dan Streetman
f5694d7cf6 archive: pass src pkg file size to _download_file 2020-03-06 12:18:10 +01:00
Dan Streetman
7900a09763 archive: fix DebianSourcePackage implementation of _source_urls() 2020-03-06 12:18:10 +01:00
Dan Streetman
3d7f542e49 archive: allow pull_binaries() to accept arch=None 2020-02-13 17:43:24 -05:00
Dan Streetman
ebe460aad1 lpapicache: allow getBinaries(arch=None) to get all archs
this was the previous behavior and some scripts still call with no args.

LP: #1862286
2020-02-13 17:43:22 -05:00
Dan Streetman
63f614ebe3 archive: if using local file, avoid error trying to copy file to itself 2020-02-13 17:39:01 -05:00
Colin Watson
1e1702e676 Use +sourcefiles URLs where possible
In
https://code.launchpad.net/~cjwatson/launchpad/archive-unambiguous-files-traversals/+merge/345118,
I added support for downloading source package files from Launchpad that
can cope with the situation where different versions of a source package
have the same file name with different contents (normally impossible,
but it can happen with imported archives and due to some old bugs).  Use
this where possible.

LP: #1860456
2020-01-24 16:34:02 +00:00
Dan Streetman
82c8c438f7 ubuntutools: use file extension when possible to determine deb/ddeb/udeb 2019-11-25 16:40:18 -05:00
Dan Streetman
e37e9db7eb pullpkg: add --security param to query ubuntu sec team proposed ppa 2019-11-25 10:10:06 -05:00
Dan Streetman
90e8fe81e1 replace ubuntutools.logger with standard python logging 2019-11-25 10:10:06 -05:00
Dan Streetman
c9c7fed1f6 pull-uca-*: search in reverse order of UCA releases, if none specified
if only the version of a UCA package is specified, search each UCA
archive from latest backwards.  This avoids having to specify both
the package version *and* UCA release.
2019-11-25 10:10:06 -05:00
Dan Streetman
7c097b19ba ubuntutools: add --status param to pull-pkg
Allow specifying what package statuses should be searched for.

By default search only for Pending and Published, unless a specific
version number is being searched for.
2019-11-25 10:10:06 -05:00
Dan Streetman
0f61836b10 ubuntutools: update archive/lpapicache to optionally search all series
For PPA and UCA repos, the latest build is not necessarily in the
'development' release; so for those SourcePackage classes, search
all the 'active' series (starting with latest devel) for any
matches to the provided package name.  This allows not having to
specify the series name when looking in PPA/UCA repos for the 'latest'
version of a specific package.
2019-11-25 10:10:06 -05:00
Dan Streetman
b11b83f0e2 pull-pkg: change UCA source pkg class to subclass of PPA source pkg class
The UCA source pkg class was its own specific class, but with the
addition of the generic PPA source pkg class, the UCA class is just
a specific case of PPA.  Changing it to a subclass of the PPA class
simplifies code.
2019-11-25 10:10:06 -05:00
Dan Streetman
7084bfc8bc pull-pkg: add pull-ppa-* functionality
Add functionality, and frontend pull-ppa-* scripts, to be able to pull
from PPA archives.
2019-11-25 10:10:06 -05:00
Dan Streetman
3491b0cff9 pull-pkg: change pullpkg into class PullPkg
instead of pullpkg.py containing a simple method to call, change it
into a normal class PullPkg that callers can create and use.
2019-11-25 10:10:06 -05:00
Dan Streetman
fb750e38bb lpapicache: find binaries from lp
Also update getBinaries() to allow retreival by binary name

This significantly speeds up binary file lookup for non-published
package versions, since we can get the list of binary urls, but
then have to look up the BPPH for each of those urls, which is slow.
If the user only wants a specific binary package and/or arch, this
speeds up getting that, especially for packages with a lot of binary
files (like the kernel).
2019-11-25 10:10:06 -05:00
Dan Streetman
3a413760f3 ubuntutools: add pull-* --no-verify-signature option, don't fail if no pub key
Change dsc verification to fail only if the public key was available, but
signature verification failed.  If no public key is available for the dsc,
print warning only. (LP: #1700846)

Also add pull-* parameter --no-verify-signature to manually prevent failure
when signature verification fails.
2019-11-25 10:10:06 -05:00
Dan Streetman
f4f16f95f7 pull-uca: update UbuntuCloudArchiveSourcePackage
look in its PPA for its SPPHs
look up the development (latest) UCA release if not specified
use the 'series' param as the UCA release, instead of 'uca_release' param
2019-11-25 09:57:54 -05:00
Dan Streetman
b96885f05e ubuntutools/archive: convert external rmadison call to native implementation
instead of a function that calls the system program rmadison, use
a fully-functional class to interface with the madison api, as well
as the debian snapshot api
2019-11-25 09:57:54 -05:00