feat: modular providers + commands architecture, direct downloads for MassIVE/JPOST/iProX (0.0.17)

README.md

@@ -8,7 +8,7 @@
 
 You can:
 - download public and private PRIDE files
-- download public MassIVE datasets directly from `MSV...` accessions
+- download public MassIVE (`MSV...`), JPOST (`JPST...`), and iProX (`IPX...`) datasets directly. MassIVE goes through FTPS at `massive-ftp.ucsd.edu`; JPOST uses the JSON PROXI endpoint at `repository.jpostdb.org` for listings and `ftp.jpostdb.org` for transfers; iProX fetches the dataset's ProteomeXchange XML from `download.iprox.org` and downloads files over anonymous HTTPS
 - download by category (`RAW`, `SEARCH`, `RESULT`, etc.)
 - stream project and file metadata
 - search projects by keyword and filters
@@ -80,15 +80,31 @@ pridepy download-all-public-raw-files \
   --checksum-check
 ```
 
-### 3) Download a public MassIVE dataset directly
+### 3) Download a public MassIVE, JPOST, or iProX dataset directly
 
 ```bash
+# MassIVE
 pridepy download-all-public-raw-files \
   -a MSV000082297 \
   -o ./downloads/MSV000082297
+
+# JPOST
+pridepy download-all-public-raw-files \
+  -a JPST002311 \
+  -o ./downloads/JPST002311
+
+# iProX
+pridepy download-all-public-raw-files \
+  -a IPX0017413000 \
+  -o ./downloads/IPX0017413000
 ```
 
-For direct `MSV...` downloads, `pridepy` enumerates the dataset from MassIVE's public FTP tree. Raw downloads follow MassIVE's own collection layout, so `download-all-public-raw-files` downloads the files stored under the dataset's `raw/` collection.
+For these direct downloads, `pridepy` enumerates the dataset from the repository:
+- **MassIVE** lists files by walking the FTPS tree at `massive-ftp.ucsd.edu` (TLS is required by the server).
+- **JPOST** lists files through the JSON PROXI endpoint at `https://repository.jpostdb.org/proxi/datasets/<JPSTxxxxxx>` and downloads them from `ftp.jpostdb.org` over plain FTP. The PROXI listing avoids the source-IP connection limit JPOST enforces on FTP.
+- **iProX** fetches the dataset's ProteomeXchange XML from `http://download.iprox.org/<accession>/PX_<accession>.xml`, then downloads each referenced file from the same host over anonymous HTTPS. iProX exposes Aspera (`faspe://`) with username/password for very large bulk transfers; `pridepy` uses the public HTTPS endpoint instead so no iProX credentials are required.
+
+Raw downloads follow each repository's own collection layout, so `download-all-public-raw-files` downloads the files stored under the dataset's `raw/` collection. Direct downloads support resume (REST for FTP, byte-Range for HTTPS), per-file retries, parallel workers (`-w N` up to 3), and post-transfer size verification against the server-reported size.
 
 ### 4) Download only selected categories
 
@@ -99,7 +115,7 @@ pridepy download-all-public-category-files \
   -c RAW,SEARCH
 ```
 
-You can also request a specific MassIVE collection through the same category interface:
+You can also request a specific MassIVE / JPOST / iProX collection through the same category interface:
 
 ```bash
 pridepy download-all-public-category-files \
@@ -244,14 +260,15 @@ print(f"RAW files: {len(raw_files)}")
 print(raw_files[0]["fileName"])
 ```
 
-For MassIVE accessions, the same method returns the files found under the dataset's `raw/` collection:
+For MassIVE / JPOST / iProX accessions, the same method returns the files found under the dataset's `raw/` collection:
 
 ```python
 from pridepy.files.files import Files
 
 files = Files()
-raw_files = files.get_all_raw_file_list("MSV000082297")
-print(f"MassIVE raw files: {len(raw_files)}")
+for accession in ("MSV000082297", "JPST002311", "IPX0017413000"):
+    raw_files = files.get_all_raw_file_list(accession)
+    print(f"{accession} raw files: {len(raw_files)}")
 ```
 
 ### Example: search projects

pridepy/commands/__init__.py

@@ -0,0 +1,20 @@
+"""Cross-cutting download commands.
+
+Each module under this package owns one user-facing command that doesn't
+fit any single provider:
+
+- ``by_url``: download a list of explicit URLs (ftp/http/https)
+- ``by_list``: download a subset of a project's files by filename
+
+ProteomeXchange used to live here too but moved to
+:class:`pridepy.providers.proteomexchange.ProteomeXchangeProvider` because
+it conforms to the ``Provider`` interface (takes an accession or URL and
+returns file records). It is deliberately not auto-registered with the
+provider registry — PXD/PRD accessions continue to route through
+:class:`pridepy.providers.pride.PrideProvider`; ProteomeXchangeProvider is
+the explicit gateway for the cross-repository XML view, invoked via the
+``download-px-raw-files`` CLI command and ``Files.download_px_raw_files``.
+
+The ``pridepy.files.files.Files`` facade keeps shim methods that
+delegate here, so existing test patches on ``Files.X`` keep working.
+"""

pridepy/commands/by_list.py

@@ -0,0 +1,59 @@
+"""Download a subset of project files identified by a filename list."""
+import logging
+from typing import List, Optional
+
+from pridepy.providers import registry
+
+
+def download_files_by_list(
+    accession: str,
+    file_names: List[str],
+    output_folder: str,
+    skip_if_downloaded_already: bool,
+    protocol: str = "ftp",
+    aspera_maximum_bandwidth: str = "100M",
+    checksum_check: bool = False,
+    parallel_files: int = 1,
+) -> None:
+    """Download a subset of project files identified by a filename list.
+
+    Resolves each requested filename via the project metadata API and
+    delegates to the provider's ``download_files`` so the existing batch +
+    protocol fallback engine is reused.
+
+    :param accession: PRIDE or MassIVE project accession (public)
+    :param file_names: filenames to download
+    :param output_folder: directory to write downloaded files into
+    :param skip_if_downloaded_already: skip files already present locally
+    :param protocol: preferred protocol; falls back across others on failure
+    :param aspera_maximum_bandwidth: aspera ascp bandwidth cap
+    :param checksum_check: download project checksums and validate
+    :param parallel_files: number of files to download simultaneously for globus
+    :raises ValueError: if ``file_names`` is empty or none match the project
+    """
+    if not file_names:
+        raise ValueError("file_names must contain at least one filename")
+
+    provider = registry.resolve(accession)
+    all_files = provider.list_files(accession)
+
+    requested = set(file_names)
+    matched = [f for f in all_files if f.get("fileName") in requested]
+    missing = sorted(requested - {f.get("fileName") for f in matched})
+    if missing:
+        logging.warning("Files not found in project %s: %s", accession, missing)
+    if not matched:
+        raise ValueError(
+            f"No matching files in project {accession} for: {sorted(requested)}"
+        )
+
+    provider.download_files(
+        accession=accession,
+        records=matched,
+        output_folder=output_folder,
+        skip_if_downloaded_already=skip_if_downloaded_already,
+        protocol=protocol,
+        parallel_files=parallel_files,
+        checksum_check=checksum_check,
+        aspera_maximum_bandwidth=aspera_maximum_bandwidth,
+    )