[Feature][Task] Support Amazon EMR Serverless task plugin

[norrishuang]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

DolphinScheduler currently supports Amazon EMR on EC2 task type for managing EC2-based clusters and running computing tasks. However, there is no support for Amazon EMR Serverless, which is a serverless deployment option that allows users to run Spark and Hive workloads without managing cluster infrastructure.
EMR Serverless automatically provisions and scales compute resources on demand, offering a simpler operational model and cost optimization through pay-per-use pricing. It has become the recommended way to run EMR workloads for many use cases.
This feature request proposes adding a new EMR_SERVERLESS task type that enables users to:

1. Submit jobs — Submit Spark or Hive jobs to a pre-created EMR Serverless application via the StartJobRun API
2. Monitor job status — Automatically poll job state (SUBMITTED → PENDING → SCHEDULED → RUNNING → SUCCESS/FAILED/CANCELLED) via the GetJobRun API
3. Cancel jobs — Automatically cancel running jobs when a DolphinScheduler task is killed, via the CancelJobRun API
4. Failover recovery — Recover tracking of running jobs when Worker nodes restart

Task Parameters

Parameter	Description
Application Id	EMR Serverless application ID (e.g. 00fkht2eodujab09)
Execution Role Arn	IAM role ARN for job execution
Job Name	Optional job name for identification
StartJobRunRequest JSON	JSON containing JobDriver and ConfigurationOverrides for the job

Use case

• Data engineering teams running scheduled Spark ETL pipelines without managing EMR clusters
• Ad-hoc Hive query workloads that benefit from serverless auto-scaling
• Cost-sensitive environments where pay-per-use is preferred over always-on clusters
• Organizations migrating from EMR on EC2 to EMR Serverless for operational simplicity

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct