⚡️ FileFlow: Distributed Multimedia Pipeline Edge Router

Live Demo: https://fileflow-frontend.vercel.app/
Architecture Deep Dive: View ARCHITECTURE.md

A highly-concurrent, distributed file processing platform built from the ground up for massive throughput. FileFlow allows users to upload thousands of images, videos, and documents simultaneously and monitors their exact state natively in real-time as background worker nodes process the queue.

🏗 System Architecture

FileFlow relies on a distributed microservice architecture to allow the Gateway (Frontend/API) to remain entirely decoupled from the Heavy Compute (Workers).

1. Next.js Edge Dashboard: A reactive, polished Next.js interface utilizing Server-Sent Events (SSE) to natively track pipeline states (Queued, Uploading, Processing, Done) with sub-second latency.
2. Express Edge Gateway: Manages user authentication, generates S3 Presigned URLs for zero-bottleneck direct-to-storage uploads, and registers jobs securely into the task broker.
3. BullMQ & Redis: The lightning-fast asynchronous message broker linking the UI Gateway to the Compute Cluster.
4. MinIO (S3 Compatible): High-availability block storage containing the "Raw Ingest" and "Processed Outputs".
5. Background Workers: A fleet of parallel, deployable Node instances polling the Redis queue. They securely stream from S3, perform necessary processing operations, push the mutated delta back to S3, and mark job completion.

🚀 Getting Started

Follow these steps to run the complete environment locally:

1. Boot up Infrastructure (Postgres, Redis, MinIO)

docker-compose up -d

2. Start the Backend API (Terminal 1)

cd Backend
npm install
npm run dev

3. Start the Background Worker (Terminal 2)

cd worker
npm install
npm run dev

4. Start the Frontend Application (Terminal 3)

cd frontend
npm install
npm run dev

🎬 How to test: Open http://localhost:3000 and register an account. On the Upload page, click the Simulate Heavy Load button. This will automatically inject multiple files into the ecosystem, demonstrating the distributed S3 upload process, Redis background queue parsing, concurrent Node worker computation, and real-time SSE frontend updates simultaneously!

🛠 Tech Stack

• Frontend: Next.js 15, Tailwind CSS, Built-in Real-time Server Sent Events (SSE).
• Backend Edge: Node.js, Express.js, Postgres (pg), AWS SDK.
• Task Queue: Redis, BullMQ.
• Background Worker: Standalone Node.js processes pulling directly from distributed streams.
• Storage: MinIO object storage.