Skip to content

[Feature][Flink] Define FlinkTableJob and FlinkStreamTableJob Java base classes#4441

Open
ocb3916 wants to merge 4 commits into
apache:devfrom
ocb3916:phase0.2
Open

[Feature][Flink] Define FlinkTableJob and FlinkStreamTableJob Java base classes#4441
ocb3916 wants to merge 4 commits into
apache:devfrom
ocb3916:phase0.2

Conversation

@ocb3916

@ocb3916 ocb3916 commented Jul 21, 2026

Copy link
Copy Markdown

What is the purpose of the change

Implements Phase 0.2 of the Scala removal plan (#4408).

Defines FlinkTableJob and FlinkStreamTableJob as the Java abstract base classes that will
eventually replace the Scala FlinkTableTrait and FlinkStreamTableTrait traits — for Table API
(batch) jobs and jobs that mix the DataStream API with the Table API, respectively. Both classes
live in the existing streampark-flink-shims-base module and follow the same
ready → handle → destroy lifecycle established by FlinkStreamingJob in sub-issue 0.1, so all
three "Job" base classes share a consistent shape.

Brief change log

  • Add FlinkTableJob abstract class with lifecycle: ready → handle → destroy, exposing the
    underlying TableEnvironment via getTableEnv() (composition) instead of extending / mimicking
    the interface like the legacy trait did
  • Add FlinkStreamTableJob abstract class with the same lifecycle, exposing both
    StreamExecutionEnvironment (getEnv()) and StreamTableEnvironment (getTableEnv()) via
    composition
  • Add sql(String) / sql(String, Consumer<String>) on both classes, bridging to the existing
    Scala FlinkSqlExecutor.executeSql so the public API surface stays Scala-free
  • Add markConvertedToDataStream() on FlinkStreamTableJob, replacing the legacy trait's
    automatic Table → DataStream conversion detection (no longer possible under composition) with
    an explicit call
  • Place both classes under org.apache.streampark.flink.core.javaapi rather than the
    issue-specified org.apache.streampark.flink.core.java — see note below
  • Add JUnit 5 tests for both classes verifying lifecycle order, getTableEnv()/getEnv()
    composition, SQL execution, and the markConvertedToDataStream() flag
  • Extract shared lifecycle/logic to fix SonarCloud duplication warnings:
    • AbstractFlinkJob (package-private) — common ready → handle → execute → destroy lifecycle, sql(...), and app.name lookup shared by both job classes
    • FlinkJobSupport — SQL callback bridging (Java Consumer <-> Scala Function1)
    • JobTestParams — shared ParameterTool builder used by both test classes
  • Replace System.out.println with SLF4J logging in both execute(...) overrides

A note on the package name

The issue body specifies org.apache.streampark.flink.core.java. That name breaks Scala
compilation of every sibling file in streampark-flink-shims-base that references
java.util/java.io/java.lang — a nested package literally named java shadows the real
java.* standard library for any Scala file sharing the enclosing package. Reproduced both ways
in a clean build:

  • org.apache.streampark.flink.core.javaBUILD FAILURE, 33 errors, all java.util/java.io/
    java.lang unresolved in unrelated sibling .scala files
  • org.apache.streampark.flink.core.javaapiBUILD SUCCESS, no other files affected

Went with javaapi for this PR. Open to other naming suggestions if javaapi isn't preferred.

Design note: composition instead of full delegation

The legacy traits extend/mimic Flink's TableEnvironment / StreamTableEnvironment directly,
re-implementing ~40-50 delegate methods each (FlinkStreamTableTrait additionally needed a
$-prefix naming hack to avoid clashes between the two APIs). The new classes hold the
environment(s) as fields instead, exposed via getTableEnv() / getEnv(). This removes the
delegate boilerplate and the naming hack, and means the classes won't go stale as the Flink Table
API changes — but it is a breaking change for code that currently calls Table API methods
directly on a trait instance (e.g. job.sqlQuery(...)job.getTableEnv().sqlQuery(...)).
Flagging for discussion since the Phase 11 migration guide will need to cover it.

Verifying this change

mvn test -pl streampark-flink/streampark-flink-shims/streampark-flink-shims-base

Tests run: 9, Failures: 0, Errors: 0

Does this pull request potentially affect one of the following parts?

  • The public API
  • The web application
  • The build system

Documentation

- Add Java lifecycle base classes for Table API and Streaming+Table API
  jobs, following the same ready/handle/destroy lifecycle established by
  FlinkStreamingJob (sub-issue 0.1).
- Use composition (getTableEnv()/getEnv()) instead of extending/mimicking
  TableEnvironment / StreamTableEnvironment, avoiding ~40-50 delegate
  methods per class that the legacy Scala traits needed.
- Add JUnit 5 smoke tests covering lifecycle order, SQL bridging, and
  the markConvertedToDataStream() flag.

Part of apache#4408 (Flink Scala-to-Java migration), Phase 0.2.
@github-actions github-actions Bot added the FLINK label Jul 21, 2026
@ocb3916 ocb3916 changed the title [Flink] Define FlinkTableJob and FlinkStreamTableJob Java base classes [Feature][Flink] Define FlinkTableJob and FlinkStreamTableJob Java base classes Jul 21, 2026
ocb3916 added 3 commits July 21, 2026 15:34
- Add Java lifecycle base classes for Table API and Streaming+Table API
  jobs, following the same ready/handle/destroy lifecycle established by
  FlinkStreamingJob (sub-issue 0.1).
- Use composition (getTableEnv()/getEnv()) instead of extending/mimicking
  TableEnvironment / StreamTableEnvironment, avoiding ~40-50 delegate
  methods per class that the legacy Scala traits needed.
- Add JUnit 5 smoke tests covering lifecycle order, SQL bridging, and
  the markConvertedToDataStream() flag.

Part of apache#4408 (Flink Scala-to-Java migration), Phase 0.2.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant