test(workflow-core): extend small-class coverage and add LargeBinary/OnIceberg specs - #7025
Merged
Merged
Conversation
…OnIceberg specs Add 42 tests across five classes (no source changes; extends are insertions only): - PartitionInfo: hash-attribute content/order sensitivity in satisfies/merge, the RangePartition always-Unknown merge and companion collapse to UnknownPartition, and the polymorphic JSON contract (type-id tagging + base-type round-trip). - JGitVersionControl: initRepo branch extraction, empty-tree and nested-directory node building, symbolic refs, per-commit content history, binary-safe copies, and staged deletions reported as uncommitted. - Tuple: constructor null guards, positional access, inMemSize growth, and the Builder's missing/duplicate/size-mismatch error paths. - LargeBinary (new spec): URI validation with exact messages, bucket/object-key parsing incl. the empty-path branch, value semantics, the thread-local base-URI constructor, and Jackson @JsonValue/@JsonCreator behavior. - OnIceberg (new spec): expireSnapshots collapsing snapshots while retaining live rows, idempotence, the snapshot-less no-op, and the missing-table error — driven against a local Hadoop catalog on a temp warehouse (no REST/S3 needed).
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7025 +/- ##
============================================
+ Coverage 79.20% 80.40% +1.19%
- Complexity 3786 3900 +114
============================================
Files 1160 1161 +1
Lines 46118 46145 +27
Branches 5114 5134 +20
============================================
+ Hits 36529 37102 +573
+ Misses 7969 7387 -582
- Partials 1620 1656 +36
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 363 | 0.222 | 26,000/35,560/35,560 us | 🔴 +97.5% / 🔴 +119.5% |
| 🔴 | bs=100 sw=10 sl=64 | 765 | 0.467 | 127,566/150,400/150,400 us | 🔴 +32.3% / 🔴 +35.8% |
| 🔴 | bs=1000 sw=10 sl=64 | 906 | 0.553 | 1,101,038/1,226,526/1,226,526 us | 🔴 +10.9% / 🔴 +15.4% |
Baseline details
Latest main 45b7b20 from 2026-07-31T13:32:11.856Z
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 363 tuples/sec | 727.01 tuples/sec | 765.25 tuples/sec | -50.1% | -52.6% |
| bs=10 sw=10 sl=64 | MB/s | 0.222 MB/s | 0.444 MB/s | 0.467 MB/s | -50.0% | -52.5% |
| bs=10 sw=10 sl=64 | p50 | 26,000 us | 13,167 us | 12,703 us | +97.5% | +104.7% |
| bs=10 sw=10 sl=64 | p95 | 35,560 us | 18,674 us | 16,199 us | +90.4% | +119.5% |
| bs=10 sw=10 sl=64 | p99 | 35,560 us | 20,833 us | 19,557 us | +70.7% | +81.8% |
| bs=100 sw=10 sl=64 | throughput | 765 tuples/sec | 927.24 tuples/sec | 965.22 tuples/sec | -17.5% | -20.7% |
| bs=100 sw=10 sl=64 | MB/s | 0.467 MB/s | 0.566 MB/s | 0.589 MB/s | -17.5% | -20.7% |
| bs=100 sw=10 sl=64 | p50 | 127,566 us | 106,033 us | 103,830 us | +20.3% | +22.9% |
| bs=100 sw=10 sl=64 | p95 | 150,400 us | 113,668 us | 110,778 us | +32.3% | +35.8% |
| bs=100 sw=10 sl=64 | p99 | 150,400 us | 118,853 us | 118,713 us | +26.5% | +26.7% |
| bs=1000 sw=10 sl=64 | throughput | 906 tuples/sec | 938.19 tuples/sec | 992.33 tuples/sec | -3.4% | -8.7% |
| bs=1000 sw=10 sl=64 | MB/s | 0.553 MB/s | 0.573 MB/s | 0.606 MB/s | -3.4% | -8.7% |
| bs=1000 sw=10 sl=64 | p50 | 1,101,038 us | 1,064,784 us | 1,016,959 us | +3.4% | +8.3% |
| bs=1000 sw=10 sl=64 | p95 | 1,226,526 us | 1,105,948 us | 1,062,501 us | +10.9% | +15.4% |
| bs=1000 sw=10 sl=64 | p99 | 1,226,526 us | 1,130,141 us | 1,082,611 us | +8.5% | +13.3% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,550.48,200,128000,363,0.222,26000.49,35560.31,35560.31
1,100,10,64,20,2614.98,2000,1280000,765,0.467,127566.04,150399.55,150399.55
2,1000,10,64,20,22082.78,20000,12800000,906,0.553,1101037.84,1226525.51,1226525.51
Contributor
There was a problem hiding this comment.
Pull request overview
This PR increases unit test coverage in common/workflow-core by extending existing specs and adding new specs for previously under-tested small classes, aligning with issue #7023’s goal of closing coverage gaps without changing production code.
Changes:
- Expanded
PartitionInfoSpec,TupleSpec, andJGitVersionControlSpecwith additional behavioral/edge-case assertions. - Added a new
LargeBinarySpeccovering URI validation, parsing, value semantics, thread-local base-URI behavior, and Jackson (de)serialization. - Added a new
OnIcebergSpecexercisingexpireSnapshotsbehavior against a local Hadoop-backed Iceberg catalog (file-based warehouse).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/workflow/PartitionInfoSpec.scala | Adds tests for Hash/Range semantics and polymorphic JSON type tagging/round-trips. |
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/tuple/TupleSpec.scala | Adds tests for constructor null-guards, positional access, sizing, and builder error paths. |
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/tuple/LargeBinarySpec.scala | New spec validating LargeBinary URI rules, parsing, thread-local base URI generation, and JSON behavior. |
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/util/dataset/JGitVersionControlSpec.scala | Adds tests for init branch name, empty/deep trees, symref hash resolution, binary-safe reads, and staged deletions. |
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/result/iceberg/OnIcebergSpec.scala | New spec validating snapshot expiration behavior and missing-table error behavior using a local Hadoop catalog. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Xinyuan Lin <xinyual3@uci.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Adds 42 tests across five
common/workflow-coreclasses (49 -> 91 in these suites). No source changes; the three extended specs are pure insertions.PartitionInfo:satisfies/mergesensitivity to hash-attribute contents and order,RangePartition's always-Unknown merge override and the companion's collapse toUnknownPartitionon an empty attribute list, plus the polymorphic-JSON contract (each subtype tagged with its registered type id, base-type round-trip preserving range bounds).JGitVersionControl:initRepo's branch extraction matching what JGit reports,getRootFileNodeOfCommiton an empty tree and on depth-two nested directories, symbolic refs (HEAD) on both read paths, per-commit content after re-staging a modified file, binary-safe copies through both stream APIs, and a staged deletion reported as uncommitted until committed.Tuple: constructor null guards, positionalgetFieldand its out-of-range failure,inMemSizegrowth, and the Builder's missing-attribute / duplicate-add /addSequentiallysize-mismatch paths.LargeBinary(new spec): URI validation with exact messages (including the case-sensitives3://guard), bucket/object-key parsing with the empty-path branch, value semantics, the thread-local base-URI constructor and its unset failure, and Jackson@JsonValue/@JsonCreatorbehavior.OnIceberg(new spec):expireSnapshotscollapsing three snapshots to one while keeping every live row, idempotence, a no-op on a snapshot-less table, and theNoSuchTableExceptionmessage — driven against a local Hadoop catalog on a tempfile:/warehouse, so no REST/S3 backend is needed.A few defensive branches were left uncovered rather than contrived (unreachable
return nullguards ininitRepo, and a parent-node creation branch the tree walk can't reach); they're listed in the review notes.Any related issues, documentation, discussions?
Closes #7023.
How was this PR tested?
sbt -java-home <jbr-17> "WorkflowCore/testOnly *PartitionInfoSpec *JGitVersionControlSpec *TupleSpec *LargeBinarySpec *OnIcebergSpec"-> 91 succeeded, 0 failed.Test/scalafmtCheck+Test/scalafix --checkclean.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8 [1M context])