Skip to content

[SPARK-56120][PYTHON][TEST] Add ASV micro-benchmarks for SQL_WINDOW_AGG_ARROW_UDF#55056

Closed
Yicong-Huang wants to merge 1 commit intoapache:masterfrom
Yicong-Huang:SPARK-56120/bench/window-agg-arrow-udf
Closed

[SPARK-56120][PYTHON][TEST] Add ASV micro-benchmarks for SQL_WINDOW_AGG_ARROW_UDF#55056
Yicong-Huang wants to merge 1 commit intoapache:masterfrom
Yicong-Huang:SPARK-56120/bench/window-agg-arrow-udf

Conversation

@Yicong-Huang
Copy link
Copy Markdown
Contributor

@Yicong-Huang Yicong-Huang commented Mar 27, 2026

What changes were proposed in this pull request?

Add ASV micro-benchmarks for SQL_WINDOW_AGG_ARROW_UDF.

Also adds optional runner_conf parameter to MockProtocolWriter.write_worker_input for passing configuration like window_bound_types.

Why are the changes needed?

Part of SPARK-55724. Establishes performance baselines before refactoring this eval type.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

./python/asv run --python=same -b "WindowAggArrow" -a repeat=3 --show-stderr
··· ================ ============ ================
    --                            udf             
    ---------------- -----------------------------
        scenario       sum_udf     mean_multi_udf 
    ================ ============ ================
     few_groups_sm    11.3±0.3ms     9.85±0.1ms   
     few_groups_lg    45.7±0.1ms     44.4±0.1ms   
     many_groups_sm    317±4ms        273±5ms     
     many_groups_lg    177±2ms        158±1ms     
       wide_cols       85.1±1ms      83.2±0.7ms   
    ================ ============ ================
··· ================ ========= ================
    --                          udf            
    ---------------- --------------------------
        scenario      sum_udf   mean_multi_udf 
    ================ ========= ================
     few_groups_sm      468M         467M      
     few_groups_lg      492M         490M      
     many_groups_sm     474M         473M      
     many_groups_lg     505M         503M      
       wide_cols        478M         477M      
    ================ ========= ================

Was this patch authored or co-authored using generative AI tooling?

No.

@Yicong-Huang Yicong-Huang marked this pull request as draft March 27, 2026 09:55
@Yicong-Huang
Copy link
Copy Markdown
Contributor Author

wait for fix in #55059

@Yicong-Huang Yicong-Huang force-pushed the SPARK-56120/bench/window-agg-arrow-udf branch from 09348d5 to 4513cfa Compare March 30, 2026 23:49
@Yicong-Huang Yicong-Huang marked this pull request as ready for review March 30, 2026 23:57
@zhengruifeng
Copy link
Copy Markdown
Contributor

merged to master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants