Skip to content

[Feature][Flink] Define FlinkStreamingJob Java lifecycle base class#4436

Draft
och5351 wants to merge 1 commit into
apache:devfrom
och5351:feature/convert-java-flinkstreamingjob
Draft

[Feature][Flink] Define FlinkStreamingJob Java lifecycle base class#4436
och5351 wants to merge 1 commit into
apache:devfrom
och5351:feature/convert-java-flinkstreamingjob

Conversation

@och5351

@och5351 och5351 commented Jul 19, 2026

Copy link
Copy Markdown

What is the purpose of the change

Implements #4435 — Phase 0.1 of the Scala removal plan (#4408).

Introduces a pure-Java streampark-flink-core module and defines FlinkStreamingJob as the Java abstract base class that replaces the deleted Scala FlinkStreaming trait.

Brief change log

previous code
package org.apache.streampark.flink.core.scala

class StreamingContext(val parameter: ParameterTool, private val environment: StreamExecutionEnvironment)
  extends StreamExecutionEnvironment(environment.getJavaEnv) {
  def start(): JobExecutionResult = execute()
  override def execute(): JobExecutionResult = {
    val appName = parameter.getAppName(required = true)
    execute(appName)
  }
}

trait FlinkStreaming extends Serializable with Logger {
  implicit final lazy val parameter: ParameterTool = context.parameter
  implicit var context: StreamingContext = _
  var jobExecutionResult: JobExecutionResult = _

  final def main(args: Array[String]): Unit = {
    init(args)
    ready()
    handle()
    jobExecutionResult = context.start()
    destroy()
  }

  private[this] def init(args: Array[String]): Unit = {
    SystemPropertyUtils.setAppHome(KEY_APP_HOME, classOf[FlinkStreaming])
    context = new StreamingContext(FlinkStreamingInitializer.initialize(args, config))
  }

  def ready(): Unit = {}
  def config(env: StreamExecutionEnvironment, parameter: ParameterTool): Unit = {}
  def handle(): Unit
  def destroy(): Unit = {}
}
  • Add new Maven module streampark-flink/streampark-flink-core (no Scala version suffix, no Scala dependencies)
  • Define FlinkStreamingJob abstract class with lifecycle: configure → ready → handle → execute → destroy
  • Register the new module in streampark-flink/pom.xml
  • Add JUnit 5 tests verifying lifecycle order, class name default, and ParameterTool propagation

Verifying this change

mvn test -pl streampark-flink/streampark-flink-core

Tests run: 3, Failures: 0, Errors: 0

Does this pull request potentially affect one of the following parts?

  • The public API
  • The web application
  • The build system

Documentation

  • Relates to issue #4435
  • Part of umbrella issue #4408

@och5351
och5351 force-pushed the feature/convert-java-flinkstreamingjob branch 3 times, most recently from 4a33862 to 15c1919 Compare July 19, 2026 07:50
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public abstract class FlinkStreamingJob {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We greatly appreciate your contribution. I have observed that the current Java implementation differs considerably from the Scala version, and that several core classes (FlinkStreamTable, FlinkTable) are not yet updated. The recommendation is to keep the original Scala module's logic unchanged and only reimplement it in Java, which would reduce the risk to a minimum. Subsequent iterations and modifications can be made thereafter.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review @wolfboys !

  1. Regarding FlinkStreamTable and FlinkTable — this PR intentionally covers only Phase 0.1 of the [Proposal] Remove Scala from StreamPark — design and migration plan #4408 roadmap, limiting the scope to FlinkStreamingJob. The remaining classes are planned for subsequent phases. I apologize if that context wasn't clear enough in the PR description. If my approach is off track, I would really appreciate your feedback.

  2. Regarding the difference from the Scala version — the lifecycle structure (init → ready → handle → execute → destroy) follows the original. The differences are intentional: since FlinkStreamingInitializer and StreamingContext are themselves Scala components targeted for removal in later phases, the Java version uses Flink's standard APIs directly instead of depending on them. This was a deliberate decision to avoid StreamPark's own wrapper classes, but I'd love your thoughts on whether this direction is the right one.

Comment thread streampark-flink/streampark-flink-core/.flattened-pom.xml Outdated
@och5351
och5351 force-pushed the feature/convert-java-flinkstreamingjob branch from d730d14 to 2135e1e Compare July 19, 2026 12:37
@sonarqubecloud

Copy link
Copy Markdown

@shangeyao

shangeyao commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Thanks for picking up #4408 Phase 0.1 — the scoped approach and lifecycle tests look good.

Before merge, I think we need closer parity with the old FlinkStreaming platform contract:

  1. Init path — Wire init through FlinkStreamingInitializer (or an equivalent Java port): --conf, setAppHome, merged ParameterTool / FlinkConfiguration. The current ParameterTool.fromArgs() + bare StreamExecutionEnvironment.getExecutionEnvironment() won't work for Console-submitted jobs.
  2. App name — Use StreamPark app-name resolution (getAppName semantics: app.name, pipeline.name, deflate), not only raw parameter.get("app.name", ...). The existing test doesn't verify the name passed to execute().
  3. main() entry — Console launches user mainClass; the old trait provided main(). Please document or provide a standard entry pattern for FlinkStreamingJob.
  4. Context vs [Flink] Remove dev framework and connector wrappers #4432 — Clarify in the PR description that this reintroduces streampark-flink-core as the Java replacement after [Flink] Remove dev framework and connector wrappers #4432, per [Proposal] Remove Scala from StreamPark — design and migration plan #4408 — not a rollback.

Non-blocking: consider finally for destroy(), align package name with #4408 (core.java?), add streampark-common (provided).

Happy to re-review once the blocking items are addressed. +1 on phased follow-ups for FlinkTable / FlinkStreamTable.

@wolfboys
wolfboys self-requested a review July 21, 2026 01:46
@wolfboys

wolfboys commented Jul 23, 2026

Copy link
Copy Markdown
Member

Thank you for your valuable contribution.
After careful evaluation, we found that the current PR introduces significant changes compared with the existing implementation and may impact the original design and migration approach. To ensure a minimal, stable, and manageable migration process, the PMC has discussed and reviewed the proposal, and we have redefined the roadmap for removing Scala.

Currently, we plan to proceed with Stage 1 — Flink Core: 4445 , 4446 , 4447 , 4448 , migrating modules incrementally according to the proposed execution plan(Principle: no logic changes; migrate modules to Java with readable, idiomatic code). We kindly suggest following the steps outlined in the issue: flink core

Thanks again for your effort and contribution. We look forward to continuing the collaboration on this migration.

@och5351
och5351 marked this pull request as draft July 25, 2026 00:26
@och5351

och5351 commented Jul 25, 2026

Copy link
Copy Markdown
Author

@wolfboys

Thanks for providing the roadmap. I'll begin with issue #4445 and work on making meaningful contributions to the project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants