[Feature][Flink] Define FlinkTableJob and FlinkStreamTableJob Java base classes#4441
Open
ocb3916 wants to merge 4 commits into
Open
[Feature][Flink] Define FlinkTableJob and FlinkStreamTableJob Java base classes#4441ocb3916 wants to merge 4 commits into
ocb3916 wants to merge 4 commits into
Conversation
- Add Java lifecycle base classes for Table API and Streaming+Table API jobs, following the same ready/handle/destroy lifecycle established by FlinkStreamingJob (sub-issue 0.1). - Use composition (getTableEnv()/getEnv()) instead of extending/mimicking TableEnvironment / StreamTableEnvironment, avoiding ~40-50 delegate methods per class that the legacy Scala traits needed. - Add JUnit 5 smoke tests covering lifecycle order, SQL bridging, and the markConvertedToDataStream() flag. Part of apache#4408 (Flink Scala-to-Java migration), Phase 0.2.
- Add Java lifecycle base classes for Table API and Streaming+Table API jobs, following the same ready/handle/destroy lifecycle established by FlinkStreamingJob (sub-issue 0.1). - Use composition (getTableEnv()/getEnv()) instead of extending/mimicking TableEnvironment / StreamTableEnvironment, avoiding ~40-50 delegate methods per class that the legacy Scala traits needed. - Add JUnit 5 smoke tests covering lifecycle order, SQL bridging, and the markConvertedToDataStream() flag. Part of apache#4408 (Flink Scala-to-Java migration), Phase 0.2.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What is the purpose of the change
Implements Phase 0.2 of the Scala removal plan (#4408).
Defines
FlinkTableJobandFlinkStreamTableJobas the Java abstract base classes that willeventually replace the Scala
FlinkTableTraitandFlinkStreamTableTraittraits — for Table API(batch) jobs and jobs that mix the DataStream API with the Table API, respectively. Both classes
live in the existing
streampark-flink-shims-basemodule and follow the sameready → handle → destroylifecycle established byFlinkStreamingJobin sub-issue 0.1, so allthree "Job" base classes share a consistent shape.
Brief change log
FlinkTableJobabstract class with lifecycle:ready → handle → destroy, exposing theunderlying
TableEnvironmentviagetTableEnv()(composition) instead of extending / mimickingthe interface like the legacy trait did
FlinkStreamTableJobabstract class with the same lifecycle, exposing bothStreamExecutionEnvironment(getEnv()) andStreamTableEnvironment(getTableEnv()) viacomposition
sql(String)/sql(String, Consumer<String>)on both classes, bridging to the existingScala
FlinkSqlExecutor.executeSqlso the public API surface stays Scala-freemarkConvertedToDataStream()onFlinkStreamTableJob, replacing the legacy trait'sautomatic
Table → DataStreamconversion detection (no longer possible under composition) withan explicit call
org.apache.streampark.flink.core.javaapirather than theissue-specified
org.apache.streampark.flink.core.java— see note belowgetTableEnv()/getEnv()composition, SQL execution, and the
markConvertedToDataStream()flagAbstractFlinkJob(package-private) — commonready → handle → execute → destroylifecycle,sql(...), andapp.namelookup shared by both job classesFlinkJobSupport— SQL callback bridging (JavaConsumer<-> ScalaFunction1)JobTestParams— sharedParameterToolbuilder used by both test classesSystem.out.printlnwith SLF4J logging in bothexecute(...)overridesA note on the package name
The issue body specifies
org.apache.streampark.flink.core.java. That name breaks Scalacompilation of every sibling file in
streampark-flink-shims-basethat referencesjava.util/java.io/java.lang— a nested package literally namedjavashadows the realjava.*standard library for any Scala file sharing the enclosing package. Reproduced both waysin a clean build:
org.apache.streampark.flink.core.java→BUILD FAILURE, 33 errors, alljava.util/java.io/java.langunresolved in unrelated sibling.scalafilesorg.apache.streampark.flink.core.javaapi→BUILD SUCCESS, no other files affectedWent with
javaapifor this PR. Open to other naming suggestions ifjavaapiisn't preferred.Design note: composition instead of full delegation
The legacy traits extend/mimic Flink's
TableEnvironment/StreamTableEnvironmentdirectly,re-implementing ~40-50 delegate methods each (
FlinkStreamTableTraitadditionally needed a$-prefix naming hack to avoid clashes between the two APIs). The new classes hold theenvironment(s) as fields instead, exposed via
getTableEnv()/getEnv(). This removes thedelegate boilerplate and the naming hack, and means the classes won't go stale as the Flink Table
API changes — but it is a breaking change for code that currently calls Table API methods
directly on a trait instance (e.g.
job.sqlQuery(...)→job.getTableEnv().sqlQuery(...)).Flagging for discussion since the Phase 11 migration guide will need to cover it.
Verifying this change
mvn test -pl streampark-flink/streampark-flink-shims/streampark-flink-shims-baseTests run: 9, Failures: 0, Errors: 0
Does this pull request potentially affect one of the following parts?
Documentation