Skip to content

Chore/merge gsoc#309

Open
jacob50231 wants to merge 28 commits into
masterfrom
chore/merge-gsoc
Open

Chore/merge gsoc#309
jacob50231 wants to merge 28 commits into
masterfrom
chore/merge-gsoc

Conversation

@jacob50231

Copy link
Copy Markdown
Contributor

New Features

  • Enhanced Asynchronous Multiple File Download: Introduces a high-performance async_download_multiple method utilizing hybrid concurrency (multiprocessing + asyncio) with memory-efficient queue-based management
  • New CLI Command: Adds gen3 download-multiple-async command with advanced options like resuming downloads (--skip-completed), flexible filename formatting, and progress bars
  • Performance Testing Suite: Added comprehensive benchmarking framework comparing the new implementation against CDIS Data Client for download speed, memory usage, and concurrency efficiency

Dependency updates
Updated fastavro from 1.8.4 to 1.11.1
Updated pypfb to include extras: pypfb = {extras = ["gen3"], version = "^0.5.33"}
Updated importlib-metadata from 8.5.0 to 4.13.0

Dhiren-Mhatre and others added 28 commits August 5, 2025 15:23
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
Signed-off-by: Dhiren-Mhatre <kp064669@gmail.com>
…formance-testing

Feat/multiple download performance testing
@github-actions

Copy link
Copy Markdown

The style in this PR agrees with black. ✔️

This formatting comment was generated automatically by a script in uc-cdis/wool.

@github-actions

Copy link
Copy Markdown

Integration Tests

Test summary after running integration tests

filepath passed failed skipped SUBTOTAL
tests/test_dbgap.py 4 0 1 5
tests/test_ras_passport.py 0 0 2 2
tests/test_data_upload.py 8 0 1 9
tests/test_graph_submit_and_query.py 11 2 1 14
tests/test_presigned_url.py 7 0 0 7
tests/test_centralized_auth.py 5 0 0 5
tests/test_google_data_access.py 1 0 0 1
tests/test_audit_service.py 1 0 0 1
tests/test_drs_endpoint.py 2 0 0 2
tests/test_gen3_sdk.py 1 0 0 1
TOTAL 40 2 5 47

Test summary after rerunning failed integration tests

filepath passed failed SUBTOTAL
tests/test_graph_submit_and_query.py 1 1 2
TOTAL 1 1 2

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

@github-actions

Copy link
Copy Markdown

Integration Tests

Test summary after running integration tests

filepath passed failed skipped SUBTOTAL
tests/test_dbgap.py 4 0 1 5
tests/test_ras_passport.py 0 0 2 2
tests/test_data_upload.py 8 0 1 9
tests/test_graph_submit_and_query.py 12 1 1 14
tests/test_presigned_url.py 7 0 0 7
tests/test_centralized_auth.py 5 0 0 5
tests/test_google_data_access.py 1 0 0 1
tests/test_audit_service.py 1 0 0 1
tests/test_drs_endpoint.py 2 0 0 2
tests/test_gen3_sdk.py 1 0 0 1
TOTAL 41 1 5 47

Test summary after rerunning failed integration tests

filepath passed SUBTOTAL
tests/test_graph_submit_and_query.py 1 1
TOTAL 1 1

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants