StreamForge

StreamForge is a video transcoding platform that uploads raw videos, stores them in MinIO, and processes them asynchronously into streaming-friendly HLS formats via a worker queue. It includes a web dashboard for managing videos and monitoring transcoding progress.

Features

• Video Upload: Upload videos via API or web dashboard with drag-and-drop support
• Multi-Resolution Transcoding: Automatically transcodes videos to 360p, 720p, and 1080p HLS formats
• Real-Time Progress Tracking: Monitor transcoding progress with live updates
• Job Cancellation: Cancel queued or processing transcoding jobs
• Video Metadata Extraction: Automatically extracts and stores video metadata (duration, resolution, codec, bitrate, etc.)
• Web Dashboard: User-friendly interface for uploading, viewing, and managing videos
• HLS Streaming: Adaptive bitrate streaming with master playlist support
• Health Checks: Monitor system health and dependencies

Architecture Overview

flowchart TB
    Client[Client/Browser] -->|Upload Video| API[API Server<br/>NestJS]
    API -->|Store| MinIO[MinIO<br/>Object Storage]
    API -->|Create Record| DB[(PostgreSQL<br/>Database)]
    API -->|Publish Job| RabbitMQ[RabbitMQ<br/>Message Queue]
    API -->|Track Progress| Redis[Redis<br/>Cache]
    
    RabbitMQ -->|Consume Job| Worker[Worker<br/>NestJS + FFmpeg]
    Worker -->|Download| MinIO
    Worker -->|Transcode| FFmpeg[FFmpeg<br/>Transcoding]
    Worker -->|Upload HLS| MinIO
    Worker -->|Update Progress| Redis
    Worker -->|Update Status| DB
    
    Client -->|Stream HLS| API
    API -->|Serve Playlist| MinIO
    
    style API fill:#667eea,color:#fff
    style Worker fill:#764ba2,color:#fff
    style MinIO fill:#10b981,color:#fff
    style DB fill:#3b82f6,color:#fff
    style RabbitMQ fill:#f59e0b,color:#fff
    style Redis fill:#ef4444,color:#fff

Loading

Component Responsibilities

• API Server: Handles HTTP requests, file uploads, video management, and serves HLS playlists
• Worker: Processes transcoding jobs from the queue, downloads videos, transcodes with FFmpeg, and uploads results
• PostgreSQL: Stores video metadata and status
• MinIO: Stores raw videos and processed HLS files (segments and playlists)
• RabbitMQ: Queues transcoding jobs for asynchronous processing
• Redis: Tracks real-time transcoding progress with TTL

Stack

• Backend: NestJS 11, TypeORM, PostgreSQL
• Storage: MinIO (S3-compatible)
• Queue: RabbitMQ
• Cache: Redis
• Transcoding: FFmpeg
• Frontend: Vanilla HTML/CSS/JavaScript (served by API)

Apps

• api — REST API, web dashboard, upload, metadata, orchestration
• worker — Async transcoding jobs, queue consumers
• api-e2e, worker-e2e — Integration tests

Web Dashboard

StreamForge includes a built-in web dashboard for managing videos without using the API directly.

Accessing the Dashboard

Once the API server is running, access the dashboard at:

• Main Dashboard: http://localhost:3000/
• Video Viewer: http://localhost:3000/video?id={videoId}
• HLS Test Player: http://localhost:3000/test-player

Dashboard Features

Main Dashboard (GET /)

• Video Upload

  • Drag and drop video files
  • Click to browse and select files
  • Real-time upload progress bar
  • Supports common video formats (mp4, mov, avi, mkv, webm, etc.)

• Video List

  • Grid view of all uploaded videos
  • Status badges showing current processing state:
    • 🟡 UPLOADING / QUEUED - Waiting to be processed
    • 🔵 PROCESSING - Currently transcoding
    • 🟢 COMPLETED - Ready for streaming
    • 🔴 FAILED - Processing error occurred
    • ⚪ CANCELLED - Job was cancelled
  • Real-time progress bars for active transcoding jobs
  • Click any video card to view details

• Job Cancellation

  • Cancel button appears for QUEUED or PROCESSING videos
  • Click "Cancel" to stop transcoding
  • Status updates to CANCELLED immediately

• Real-Time Updates

  • Progress bars update automatically every 2 seconds
  • Status badges update as jobs complete
  • No page refresh needed

Video Viewer (GET /video)

• HLS video player for completed videos
• Adaptive bitrate streaming (automatically selects best quality)
• Video metadata display
• Playback controls

Quick Start

Prerequisites

• Node.js 20+ and npm
• Docker and Docker Compose
• FFmpeg (for worker)

Step-by-Step Setup

1. Clone and install dependencies

   npm install

2. Start infrastructure services

   docker-compose up -d

   This starts:

   • PostgreSQL (port 5432)
   • Redis (port 6379)
   • RabbitMQ (port 5672, Management UI: http://localhost:15672)
   • MinIO (port 9000, Console: http://localhost:9001)

3. Configure environment variables (optional) Create a .env file or use defaults. See Environment Variables section.

4. Start the API server

   npx nx serve api

   API runs on http://localhost:3000

   • Dashboard: http://localhost:3000/
   • Swagger Docs: http://localhost:3000/api/docs

5. Start the worker (in a separate terminal)

   npx nx serve worker

6. Access the dashboard Open http://localhost:3000/ in your browser and start uploading videos!

Accessing Services

• API & Dashboard: http://localhost:3000
• Swagger Documentation: http://localhost:3000/api/docs
• MinIO Console: http://localhost:9001 (admin/admin)
• RabbitMQ Management: http://localhost:15672 (admin/admin)

Development

Running Tests

# API unit/integration tests
npx nx test api

# Worker tests
npx nx test worker

# E2E tests
npx nx e2e api-e2e
npx nx e2e worker-e2e

Building

# Build API
npx nx build api

# Build Worker
npx nx build worker

Linting

npx nx lint api
npx nx lint worker

Project Graph

npx nx graph