Skip to content

Feat/multiple download performance testing#308

Merged
jacob50231 merged 26 commits into
uc-cdis:chore/merge-gsocfrom
Dhiren-Mhatre:feat/multiple-download-performance-testing
Jul 17, 2026
Merged

Feat/multiple download performance testing#308
jacob50231 merged 26 commits into
uc-cdis:chore/merge-gsocfrom
Dhiren-Mhatre:feat/multiple-download-performance-testing

Conversation

@jacob50231

Copy link
Copy Markdown
Contributor

New Features

  • Enhanced Asynchronous Multiple File Download: Introduces a high-performance async_download_multiple method utilizing hybrid concurrency (multiprocessing + asyncio) with memory-efficient queue-based management
  • New CLI Command: Adds gen3 download-multiple-async command with advanced options like resuming downloads (--skip-completed), flexible filename formatting, and progress bars
  • Performance Testing Suite: Added comprehensive benchmarking framework comparing the new implementation against CDIS Data Client for download speed, memory usage, and concurrency efficiency

Dependency updates
Updated fastavro from 1.8.4 to 1.11.1
Updated pypfb to include extras: pypfb = {extras = ["gen3"], version = "^0.5.33"}
Updated importlib-metadata from 8.5.0 to 4.13.0

Dhiren-Mhatre and others added 26 commits August 5, 2025 15:23
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
@jacob50231
jacob50231 merged commit c81d0e5 into uc-cdis:chore/merge-gsoc Jul 17, 2026
35 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants