Skip to content

Run tests on the JUnit Platform (TestNG + Jupiter engines)#18781

Open
gortiz wants to merge 1 commit into
apache:masterfrom
gortiz:junit5-testng-engine
Open

Run tests on the JUnit Platform (TestNG + Jupiter engines)#18781
gortiz wants to merge 1 commit into
apache:masterfrom
gortiz:junit5-testng-engine

Conversation

@gortiz

@gortiz gortiz commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Run Pinot's tests on the JUnit Platform (TestNG + Jupiter side by side)

TL;DR

This moves the test build onto the JUnit Platform so we can write new tests with the
modern JUnit Jupiter API (we pin JUnit 6) — without rewriting any of the ~10,000
existing TestNG tests
. They keep running, unchanged, through the TestNG engine for the
JUnit Platform
. New and old tests run together
in a single Surefire execution.

This is intentionally a foundation-only PR: build wiring + the two unavoidable
suite-file translations + one example test + a short guide. No bulk migration.

JUnit 5 or 6? We pin the JUnit 6 BOM (6.0.x) — the current major. JUnit 6 unified
the Platform / Jupiter / Vintage version numbers (Platform was 1.x, Jupiter 5.x; now all
6.0.x) and raised the runtime baseline to Java 17 — a non-issue on our JDK 21 test
runs. The Jupiter authoring API is the same programming model many people still call
"JUnit 5"; choosing 6 avoids a near-term 5→6 bump for newly written tests.

Why

TestNG has served us well, but Jupiter is where the ecosystem and tooling have moved. The
JUnit Platform is engine-pluggable — JUnit Vintage runs JUnit 4, Jupiter runs JUnit 5/6, and
the TestNG engine runs TestNG — so adopting it is a build change, not a test rewrite.
Once we're on the platform, a lot of capability we don't have today becomes available
incrementally, test-by-test, at each author's discretion:

  • Extensions instead of inheritance. Jupiter's @ExtendWith / RegisterExtension model
    replaces today's deep ControllerTest → ClusterTest → BaseClusterIntegrationTest base-class
    chains with composable behavior. Cross-cutting setup (clusters, ZK, leak detection) becomes
    a reusable extension rather than something every test must inherit.
  • @Nested inner classes with their own @BeforeEach/@AfterEach. Group related cases
    with scoped, layered setup instead of one giant flat test class.
  • Declarative resource cleanup with @AutoClose (and @TempDir) — fields are closed
    automatically after the test, removing hand-written teardown and a class of leaks.
  • Parameterized tests as a first-class feature@ParameterizedTest with @ValueSource,
    @CsvSource, @MethodSource, @EnumSource. A cleaner, less boilerplate-heavy replacement
    for @DataProvider (and great for exhaustive type/null-handling coverage, which we care
    about a lot).
  • Dynamic tests (@TestFactory) — generate cases at runtime, e.g. one test per
    query file, per segment format, or per discovered fixture.
  • Built-in parallel execution — Jupiter's junit-platform.properties parallelism (per
    method/class, with @Execution/@ResourceLock for safety) gives a standard knob for
    speeding up suites, instead of TestNG's bespoke config.
  • Better tooling reach. Being on the JUnit Platform unlocks tools that target it — for
    example Fray, a concurrency-testing /
    deterministic-scheduling framework that hunts for races and deadlocks. Pinot has a lot of
    concurrency-heavy code (upsert metadata, consumer coordination, Helix state); a JUnit
    Platform integration for that kind of tooling would be valuable and is only practical once
    we're on the platform.

Crucially, none of this is forced. Existing tests stay as TestNG and keep their
org.testng.Assert semantics. We adopt the new capabilities where they pay off.

What's in this PR

  1. Root pom.xml — import the JUnit 6 BOM (unified versioning), add testng-engine,
    junit-jupiter, and junit-platform-launcher as inherited test dependencies, and drop the
    surefire-testng provider so Surefire auto-selects its junit-platform provider. The
    launcher is on the test classpath so it stays version-aligned with the engines in the
    forked JVM.
  2. pinot-controller / pinot-integration-tests — the TestNG engine does not support
    testng.xml suite files (the only two in the build). TestNG @Test(groups=...) are exposed
    to the platform as tags, so the controller stateful/stateless split becomes two
    tag-filtered Surefire executions, and the custom-cluster integration suite becomes an
    include glob. The controller executions use reuseForks=true to preserve the shared-cluster
    setup the single-JVM suites relied on (verified: the second stateful class reuses the
    cluster, starting zero new controllers). The now-dead ControllerTestSetup
    (@BeforeGroups/@AfterGroups, no @Test) is removed.
  3. BooleanUtilsTest — a small, real Jupiter test (adds genuine coverage) demonstrating
    @ParameterizedTest and assertThrows, running in the same module as 685 TestNG tests to
    prove coexistence.
  4. docs/junit-migration.md — conventions for new Jupiter tests and a TestNG→Jupiter
    mapping table.

Verification

  • pinot-spi: 685 existing TestNG tests pass under the new provider; 698 with the new
    Jupiter test — both engines run together in one execution.
  • pinot-controller: both tag-filtered executions fire; group→tag mapping confirmed; the
    shared-cluster optimization is preserved.
  • @Listeners (e.g. NettyTestNGListener for Netty leak detection) is honored by the engine.
  • Coverage is unaffected: JaCoCo attaches via @{argLine} (provider-agnostic), and
    surefire-report reads the standard TEST-*.xml the platform also emits.
  • Enforcer dependencyConvergence, spotless, checkstyle, and license all pass.

Compatibility / risk

  • No behavior change for existing tests — they run as TestNG via the engine.
  • Engine compatibility is confirmed: testng-engine 1.1.0 is CI-tested against JUnit 6,
    supports TestNG ≥ 6.14.3 (we use 7.12.0), and needs Surefire ≥ 3.5.2 (we use 3.5.4).
  • The one thing to watch on merge is a full CI run confirming per-module test counts and that
    coverage artifacts upload exactly as before.

@gortiz gortiz requested review from Jackie-Jiang and xiangfu0 June 16, 2026 14:14
Move the test build onto the JUnit Platform so new JUnit 5 (Jupiter) tests can
be written while the existing ~10k TestNG tests keep running unchanged via the
TestNG engine for the JUnit Platform (org.junit.support:testng-engine).

- Import the JUnit BOM (unified 6.x), add testng-engine, junit-jupiter and
  junit-platform-launcher as inherited test dependencies, and drop the
  surefire-testng provider so Surefire auto-selects the junit-platform provider.
  The launcher is on the test classpath so it stays version-aligned with the
  engines in the forked JVM.
- The engine does not support testng.xml suite files. Replace the controller
  statefull/stateless split and the custom-cluster integration suite with
  JUnit Platform tag / include filters (TestNG groups map to platform tags).
  Use reuseForks=true on the controller executions to keep the shared-cluster
  setup that the single-JVM suites relied on; remove the now-dead
  ControllerTestSetup (@BeforeGroups/@AfterGroups, no @test).
- Add a Jupiter example test (BooleanUtilsTest) and docs/junit5-migration.md.
@gortiz gortiz force-pushed the junit5-testng-engine branch from 68b6492 to 38c0a24 Compare June 16, 2026 14:16
@codecov-commenter

codecov-commenter commented Jun 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (7b1d115) to head (38c0a24).
⚠️ Report is 21 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff               @@
##             master    #18781       +/-   ##
==============================================
+ Coverage     64.67%   100.00%   +35.32%     
+ Complexity     1309         6     -1303     
==============================================
  Files          3381         3     -3378     
  Lines        209821         6   -209815     
  Branches      32805         0    -32805     
==============================================
- Hits         135697         6   -135691     
+ Misses        63230         0    -63230     
+ Partials      10894         0    -10894     
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-21 100.00% <ø> (+35.32%) ⬆️
temurin 100.00% <ø> (+35.32%) ⬆️
unittests ?
unittests1 ?
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants