Skip to content

executing-ecs-task-with-lambda.md

Latest commit

 

History

History
520 lines (382 loc) · 17.5 KB

executing-ecs-task-with-lambda.md

File metadata and controls

520 lines (382 loc) · 17.5 KB
Raw

Below is a structured “deep note” on executing an ECS job using Lambda, tuned to the architecture you’re building (video transcoding on Fargate).

1. Hook – Why this is interesting

Imagine this requirement:

  • “Every time a large video lands in S3, start an isolated FFmpeg container that processes just that video, with its own CPU/memory, network, and IAM permissions, then disappears.”

You don’t want:

  • a forever-running EC2 worker,
  • a monolithic app polling S3,
  • or a Lambda trying to do heavy FFmpeg work itself.

You want a tiny brain (Lambda) that reacts to events and spins up short-lived workers (ECS tasks) on demand.

This is exactly what “executing an ECS job using Lambda” gives you.

2. What is this?

Concept:
Use a Lambda function as an event-driven controller that calls RunTask on ECS to start a one-shot ECS task (e.g., Fargate) for each unit of work (one video). The ECS task runs with its own IAM roles, resources, and container image, and exits when it’s done.

Why it exists:

  • Lambda is great at reacting (S3 events, API calls, queues), but bad at long-running heavy CPU work.
  • ECS tasks are great at running heavy, long, containerized jobs, but not event-aware by themselves.
  • Glue them together and you get: “event → dedicated container job → done.”

3. One-Sentence Summary

Use Lambda as the lightweight scheduler that reacts to events and starts properly-permissioned, one-shot ECS tasks (Fargate) to do the real heavy work.

4. Intuition – A mental model

Think of the system as:

Doorbell (Lambda)        Movers (ECS tasks)
───────────────          ────────────────
 hears a knock             carry the furniture
 checks what it is         do heavy work
 calls movers              leave when finished
  • Lambda = brains: quick, event-driven, low resource.
  • ECS task = muscle: slower to start, but powerful, isolated, and configurable.

Lambda does not “become” the worker. Instead, it orders a worker from ECS, specifying:

  • which task definition to run (container image + CPU/memory),
  • what environment variables to give (job inputs),
  • which IAM roles the worker may use (what it can touch).

5. Motivation – Why not just Lambda or just ECS?

Problem before this pattern:

  1. Only Lambda:
    • Limited runtime, ephemeral filesystem, poor fit for long FFmpeg jobs on large videos.
    • Hard to control CPU/memory and container-level tooling compared to ECS.
  2. Only ECS:
    • You still need something to decide when to start tasks.
    • If you keep workers running, you pay even when there’s no work.
    • Polling S3 from ECS is wasteful and adds latency.

This pattern is better because:

  • Lambda gives you event-driven orchestration and scale-to-zero brain.
  • ECS tasks give you containerized, resource-tuned workers for the heavy job.
  • IAM separation lets the dispatcher (Lambda) and the worker (ECS task) have different, minimal permissions.

6. Where is it used?

This pattern appears a lot in real systems:

  • Media platforms: per-video transcoding, thumbnail generation, audio waveform extraction.
  • Data engineering: per-file ETL, CSV → Parquet conversion, batch ingest.
  • ML pipelines: feature extraction, batch inference, data preprocessing jobs.
  • Security & compliance: per-object scanning, log normalization, report generation.
  • CI-style workflows: “job runners” that spin up containers for tests or builds.

Anywhere you want “one job = one container”, a short-lived, isolated ECS task triggered by an event is a good fit.

7. High-Level Workflow

Let’s outline the flow you are building:

1. S3 receives a new input video
     ↓ (S3 event)
2. Lambda is triggered
     ↓
3. Lambda:
   - parses event (bucket/key)
   - constructs RunTask request
   - calls ECS RunTask with overrides (JOB_ID, INPUT_BUCKET, INPUT_KEY, OUTPUT_BUCKET)
     ↓
4. ECS:
   - uses task definition (image, CPU, memory, roles)
   - runs Fargate task with those env vars
     ↓
5. Worker container:
   - reads env vars
   - downloads input from S3
   - runs FFmpeg
   - uploads outputs to S3
   - updates DB (optional)
   - exits

Lambda never opens the video file itself; it only passes coordinates and asks ECS to run the actual work.

8. Internals – The three IAM identities

The most important “internal” logic is IAM. There are three distinct identities:

Lambda execution role
    ↓ calls
ECS service (RunTask)
    ↓ assumes
Task execution role
    ↓ sets up container, pulls image, logs, etc.

Task role
    ↓ used by your app code for S3, DB, etc.

  • Lambda execution role

    • What your Lambda runs as.
    • Needs permission to start tasks and pass roles.

  • Task execution role

    • Used by ECS infrastructure (before your app starts).
    • Pulls image from ECR, writes logs, fetches secrets.

  • Task role

    • Used by your FFmpeg container.
    • Grants access to S3 buckets, DB, queues.

Understanding this separation is key to understanding why iam:PassRole exists.

9. Deep Breakdown – Step by step

9.1 Lambda side – the dispatcher

Responsibility:

  • React to an event (S3).
  • Translate it into a job request.
  • Ask ECS to run a task for that job.

Inputs:

  • S3Event (bucket, key).
  • Config: task definition ARN, cluster ARN, subnets, security groups, output bucket.

Outputs:

  • An ECS task started (or an error).

Core logic (simplified):

const command = new RunTaskCommand({
  taskDefinition: "arn:...:task-definition/video-transcoder:1",
  cluster: "arn:...:cluster/video-transcoding-cluster",
  launchType: "FARGATE",
  networkConfiguration: { ... },
  overrides: {
    containerOverrides: [
      {
        name: "video-transcoder",
        environment: [
          { name: "JOB_ID", value: crypto.randomUUID() },
          { name: "INPUT_BUCKET", value: tempBucketName },
          { name: "INPUT_KEY", value: objectKey },
          { name: "OUTPUT_BUCKET", value: process.env.OUTPUT_BUCKET },
        ],
      },
    ],
  },
});

const response = await ecsClient.send(command);

if (response.failures?.length) {
  throw new Error(JSON.stringify(response.failures));
}

IAM logic you must satisfy:

  • Lambda execution role:
    • ecs:RunTask on the task definition.
    • iam:PassRole on:
      • the ECS task execution role,
      • the task role (if present).

This is the “role delegation” point.

9.2 ECS task definition – the blueprint

Responsibility:

  • Describe how to run the worker container.

Contains:

  • Container image (from ECR).
  • CPU/memory.
  • Environment variables (base).
  • Task execution role ARN.
  • Task role ARN.
  • Network mode (awsvpc).
  • Logging configuration.

The task definition is a blueprint, not a running thing. Lambda’s RunTask is: “Instantiate this blueprint once, with some overrides.”

9.3 Fargate task – the actual worker

Responsibility:

  • Execute one job (one video).

Inputs:

  • Job context via env vars (JOB_ID, INPUT_BUCKET, INPUT_KEY, OUTPUT_BUCKET).
  • Credentials from task role (via ECS + STS).
  • Network connectivity (via subnets, security groups).

Outputs:

  • Transcoded files in output bucket.
  • Optional status updates in DB or queues.

Internal logic (conceptually):

// inside container
const jobId = process.env.JOB_ID;
const inputBucket = process.env.INPUT_BUCKET;
const inputKey = process.env.INPUT_KEY;
const outputBucket = process.env.OUTPUT_BUCKET;

// 1. download from S3
// 2. run ffmpeg
// 3. upload outputs
// 4. update DB / status
// 5. exit

When it exits, ECS marks task as STOPPED; CloudWatch logs record its lifecycle.

10. Visual Explanation

Architecture view

                 +---------------------+
                 |       S3 Bucket     |
                 |  (input videos)     |
                 +----------+----------+
                            |
                            | S3 Event
                            v
                   +--------+--------+
                   |     Lambda      |
                   | (dispatcher)    |
                   +--------+--------+
                            |
                            | ecs:RunTask + iam:PassRole
                            v
                  +---------+---------+
                  |        ECS        |
                  |   Task on Fargate |
                  +---------+---------+
                            |
                            | uses Task Role
                            v
                 +----------+----------+
                 |     S3 Bucket       |
                 |   (output videos)   |
                 +---------------------+

IAM view

Lambda execution role
  ├─ ecs:RunTask on video-transcoder:*
  └─ iam:PassRole on:
       - ecsTaskExecutionRole
       - FfmpegTaskRole

Task execution role
  └─ permissions: ECR pull, CloudWatch logs, secrets

Task role
  └─ permissions: S3 read/write, DB access, etc.

11. Important Terms and Concepts

  • RunTask
    API call that tells ECS: “Start N tasks using this task definition and configuration.” It’s scheduling authority.

  • Task definition
    A JSON blueprint describing how containers should run: image, CPU/mem, env vars, IAM roles, etc.

  • Task execution role
    Role assumed by the ECS agent / infrastructure to pull container images and send logs.

  • Task role (task IAM role)
    Role assumed by your container code so it can call S3, DynamoDB, etc. No credentials in code needed.

  • Lambda execution role
    Role under which the Lambda handler runs. Governs what AWS APIs the Lambda itself can call (like RunTask).

  • iam:PassRole
    Permission that lets a caller say: “Service X should use role Y.” This is delegation, not self-assumption.

12. Step-by-Step Example (Slow Motion)

Let’s walk one S3 upload all the way through:

  1. User uploads videos/source/film.mp4 to S3.
  2. S3 fires an event to your Lambda’s trigger.
  3. Lambda handler runs:
    • Extracts bucket = temp-bucket, key = videos/source/film.mp4.
    • Generates JOB_ID = 550e8400-e29b-41d4-a716-446655440000.
    • Prepares RunTask request with:
      • taskDefinition: video-transcoder:1.
      • cluster: video-transcoding-cluster.
      • network configuration (subnets, SGs).
      • containerOverrides:
        • JOB_ID, INPUT_BUCKET, INPUT_KEY, OUTPUT_BUCKET.
  4. Lambda calls RunTask.
  5. IAM evaluates:
    • Lambda role has ecs:RunTask?
    • Lambda role has iam:PassRole for:
      • ecsTaskExecutionRole?
      • FfmpegTaskRole?
  6. If allowed:
    • ECS pulls the image from ECR using task execution role.
    • ECS starts a Fargate task in your subnets.
  7. The task container starts:
    • It receives env vars you passed.
    • It receives temporary credentials for task role.
  8. Your worker code:
    • Downloads film.mp4 from temp-bucket.
    • Runs FFmpeg to produce e.g. film_720p.mp4, film_1080p.mp4.
    • Uploads outputs to final-videos-bucket.
    • Updates DB (if you implemented it).
    • Exits.
  9. ECS marks the task as STOPPED.
  10. Logs are visible in CloudWatch for debugging.

Lambda never handled the file data directly – it just orchestrated.

13. Edge Cases & Failure Modes

  • Lambda’s RunTask succeeds but ECS fails to start the task

    • The API call returns HTTP 200 with a failures array.
    • You must check response.failures in Lambda.

  • Network misconfiguration

    • Fargate tasks in private subnets without NAT or VPC endpoints can’t reach:
      • S3,
      • ECR,
      • CloudWatch.
    • You see image pull errors, timeouts, or no logs.

  • Missing iam:PassRole

    • Lambda gets AccessDeniedException when calling RunTask.
    • The task definition itself might be correct, but the caller is not allowed to delegate the roles.

  • Task role too permissive

    • Worker can read/write more buckets or resources than necessary.
    • In a multi-tenant environment, this is a security risk.

  • Poison job

    • Bad input file causes FFmpeg to crash repeatedly.
    • Without a DLQ or retry strategy, you might keep retrying the same failing job.

14. Gotchas (Common mistakes)

  1. Confusing task execution role and task role

    • Execution role = infra side; task role = app side.
    • Giving S3 permissions to the execution role won’t help your app code.

  2. Forgetting iam:PassRole

    • Having only ecs:RunTask is not enough.
    • You must allow Lambda to pass the exact roles used by the task.

  3. Hardcoding infra details in code

    • Task definition ARN, cluster ARN, subnet IDs, security group IDs embedded in code.
    • Better: configure via env vars so you can change environments without code changes.

  4. Not checking for response.failures

    • The call can succeed but still not start the task.
    • Always inspect the response.

  5. Running Lambda logic in the task

    • Trying to do worker-like work in Lambda (e.g., running FFmpeg in Lambda directly) instead of ECS defeats the purpose.

15. Trade-offs

Benefits:

  • Event-driven, scale-to-zero control plane.
  • Clear separation between orchestration (Lambda) and execution (ECS task).
  • Containerized, resource-tuned workers with per-job isolation.
  • Strong IAM isolation between dispatcher and worker.

Costs:

  • More moving parts (Lambda, ECS, ECR, VPC, IAM roles).
  • Need to understand iam:PassRole and trust policies.
  • Need to design for failures across multiple services (S3, Lambda, ECS, S3 again).
  • Slight latency overhead for task startup (cold start of Fargate tasks).

As a system designer, you’re trading simplicity (single service) for flexibility and robustness (two services with clear responsibilities).

16. Comparison – Alternatives

Approach Pros Cons When to use
Lambda only (FFmpeg in Lambda) Simple architecture, no ECS Runtime limits, bad for large videos Tiny workload, short jobs
ECS service (always-on workers) Less orchestration logic Pay for idling workers, custom scheduling Constant, steady load
ECS job via Lambda (your pattern) Event-driven, scale-to-zero, per-job isolation More IAM & infra complexity Spiky, heavy, per-file jobs

For large, spiky video processing, Lambda → ECS task is usually the sweet spot.

17. Questions to Think About (do not answer yet)

  • What new failure modes do you introduce by splitting orchestration and execution into different services?
  • How would you add retries and a DLQ to this pipeline so bad inputs don’t cause infinite loops?
  • How would you track job status across S3, Lambda, ECS, and your DB in a reliable way?
  • What happens if two S3 events for the same file arrive? How would you design idempotency?
  • How could you extend this to run multiple different kinds of jobs (e.g., thumbnails vs full transcode) off the same S3 events?
  • When would you prefer Step Functions over plain Lambda for orchestrating ECS jobs?

18. Key Takeaways

  • Lambda is your event-driven scheduler, not your video processor.
  • ECS tasks (Fargate) are your one-job workers, tuned for heavy, long-running tasks.
  • ECS task definitions encode:
    • container image,
    • resources,
    • IAM roles (execution & task).
  • ecs:RunTask gives Lambda permission to start tasks of a particular type.
  • iam:PassRole gives Lambda permission to delegate specific IAM roles to ECS tasks.
  • The mental model is delegation: Lambda asks ECS to run a worker with certain powers, and PassRole decides if that delegation is allowed.

19. Minimal Self-Test (no answers here)

Try to answer these from memory:

  1. Explain in one sentence what Lambda is doing when it executes an ECS job.
  2. Draw an ASCII diagram showing S3 → Lambda → ECS task → S3, including the three IAM roles involved.
  3. What is the difference between the ECS task execution role and the task role?
  4. In your own words, why does Lambda need iam:PassRole when calling RunTask?
  5. What happens if Lambda has ecs:RunTask but not iam:PassRole for the roles referenced in the task definition?
  6. Imagine Fargate tasks run in a private subnet with no NAT and no VPC endpoints. What breaks and why?
  7. How would you add job IDs so you can match ECS tasks back to S3 events or DB records?
  8. Compare: running FFmpeg inside Lambda vs running FFmpeg inside an ECS Fargate task. What is the main trade-off?
  9. What checks should you add in your Lambda to be sure the ECS task really started?
  10. How would you prevent a compromised Lambda from abusing iam:PassRole to escalate privileges?

20. What to Learn Next

If you want to go deeper, natural next topics:

  • ECS task role vs task execution role and how credentials are vended to containers.
  • Designing robust retry and DLQ patterns for S3 → Lambda → ECS.
  • ECS networking for Fargate (NAT, VPC endpoints, security groups).
  • Observability: mapping logs and metrics across S3, Lambda, ECS, and your application.
  • Step Functions vs Lambda as orchestrators for multi-step workflows.

Once you see this pattern clearly, you’ll notice it everywhere: small brain, big muscles — a lightweight control plane launching heavyweight workers with carefully scoped permissions.