Modern Data Lakehouse Platform

A fully containerized, production-ready data lakehouse for e-commerce analytics. Built on Apache Iceberg with a medallion architecture (Bronze → Silver → Gold), orchestrated by Airflow, stored in MinIO via Spark, queried via Dremio and Jupyter, and monitored with Prometheus and Grafana — all running locally with Docker Compose.

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                     MODERN DATA PLATFORM                        │
│                                                                 │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐  │
│  │ INGEST   │───▶│ PROCESS  │───▶│  STORE   │───▶│  QUERY   │  │
│  │ Airflow  │    │  Spark   │    │  MinIO   │    │  Dremio  │  │
│  │ DAG ETL  │    │ 1M+2W    │    │ Iceberg  │    │ Jupyter  │  │
│  └──────────┘    └──────────┘    └──────────┘    └──────────┘  │
│                                        │                        │
│                                   ┌────▼────┐                   │
│                                   │ Nessie  │                   │
│                                   │Catalog  │                   │
│                                   └─────────┘                   │
│                                                                 │
│  ┌──────────────────────────────────────────┐                   │
│  │  MONITOR: Prometheus + Grafana           │                   │
│  └──────────────────────────────────────────┘                   │
└─────────────────────────────────────────────────────────────────┘

Medallion Architecture:
  RAW CSV ──▶ BRONZE (Iceberg) ──▶ SILVER (Iceberg) ──▶ GOLD (Iceberg)
              orders                orders_enriched      daily_sales
              customers             (joined + clean)     product_performance
              products                                   customer_metrics

Tech Stack

Layer	Technology	Version
Orchestration	Apache Airflow	2.8.0
Processing	Apache Spark	3.5 (1 master + 2 workers)
Table Format	Apache Iceberg	1.5.0
Catalog	Project Nessie	0.90.4
Object Storage	MinIO	latest
Query Engine	Dremio OSS	latest
Notebooks	JupyterLab (PySpark)	spark-3.5.0
Metadata DB	PostgreSQL	15
Monitoring	Prometheus + Grafana	latest

Prerequisites

Docker Desktop with at least 8 GB RAM (Settings → Resources → Memory)
20 GB+ free disk space
The following ports must be free:

3000   Grafana
4040-4050  Spark App UIs
5432   PostgreSQL
7077   Spark Master RPC
8080   Spark Master UI
8081   Airflow Webserver
8888   JupyterLab
9000   MinIO API
9001   MinIO Console
9047   Dremio
9090   Prometheus
19120  Nessie API
31010  Dremio ODBC/JDBC
45678  Dremio Arrow Flight

Quick Start

# 1. Clone and enter the project
cd modern-data-platform

# 2. Build images and start all services
docker compose up --build -d

# 3. Check all containers are healthy
docker compose ps

# 4. Health check all services
echo "=== Platform Health ===" && \
echo -n "Spark:       " && curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080 && \
echo -n "Airflow:     " && curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8081/health && \
echo -n "Dremio:      " && curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9047 && \
echo -n "MinIO:       " && curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9001 && \
echo -n "Prometheus:  " && curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9090 && \
echo -n "Grafana:     " && curl -s -o /dev/null -w "%{http_code}\n" http://localhost:3000 && \
echo -n "Nessie:      " && curl -s -o /dev/null -w "%{http_code}\n" http://localhost:19120/api/v2/config && \
echo "=== Done ==="

All services must return 200.

Service URLs

Service	URL	Credentials
Airflow	http://localhost:8081	admin / admin
Spark Master UI	http://localhost:8080	—
Dremio	http://localhost:9047	wizard on first login
JupyterLab	http://localhost:8888	see token below
MinIO Console	http://localhost:9001	minioadmin / minioadmin123
Prometheus	http://localhost:9090	—
Grafana	http://localhost:3000	admin / admin
Nessie API	http://localhost:19120	—
PostgreSQL	localhost:5432	admin / admin123

# Get the Jupyter token
docker logs jupyter 2>&1 | grep "token="

Step-by-Step Pipeline

Step 1 — Generate Raw Data

# Option A: trigger the Airflow DAG (recommended)
# Open http://localhost:8081 → ecommerce_etl_pipeline → ▶ Trigger

# Option B: generate directly
docker exec airflow-webserver python /opt/scripts/generate_data.py

Creates in data/raw/:

customers.csv   1,000 records
products.csv      200 records
orders.csv      5,000 records

Step 2 — Setup Bronze Iceberg Tables (Namespaces)

docker exec spark-master spark-submit \
  --master spark://spark-master:7077 \
  /opt/spark-jobs/01_setup_bronze.py

Expected output:

✓ Namespace iceberg.bronze created
✓ Namespace iceberg.silver created
✓ Namespace iceberg.gold created
✓ Table bronze.orders (partitioned by days(order_date))
✓ Table bronze.customers
✓ Table bronze.products

Step 3 — Load CSV → Bronze Iceberg (MinIO)

docker exec spark-master spark-submit \
  --master spark://spark-master:7077 \
  /opt/spark-jobs/02_load_bronze.py

Expected output:

✓ Loaded 5,000 records → iceberg.bronze.orders
✓ Loaded 1,000 records → iceberg.bronze.customers
✓ Loaded   200 records → iceberg.bronze.products

Verify in MinIO Console (http://localhost:9001) → bucket warehouse:

warehouse/
├── bronze/
│   ├── orders_.../
│   │   ├── data/*.parquet
│   │   └── metadata/*.json
│   ├── customers_.../
│   └── products_.../

Step 4 — Iceberg Features Demo (Time Travel + Schema Evolution)

docker exec spark-master spark-submit \
  --master spark://spark-master:7077 \
  /opt/spark-jobs/03_iceberg_features.py

This demonstrates:

Table history and snapshot listing
Time travel queries (AS OF snapshot)
Schema evolution (adding a column without rewrite)
Partition statistics

Step 5 — Silver Layer (Enrichment + Joins)

docker exec spark-master spark-submit \
  --master spark://spark-master:7077 \
  /opt/spark-jobs/04_silver_layer.py

Joins orders + customers + products, excludes cancelled orders, derives:

calculated_revenue = quantity × unit_price
order_value_tier (High / Normal)
order_year, order_month (partition columns)

Expected output:

→ First run: CREATE TABLE AS SELECT
✓ iceberg.silver.orders_enriched — ~3,500 records
  (incremental runs use MERGE INTO)

Step 6 — Gold Layer (Aggregations)

docker exec spark-master spark-submit \
  --master spark://spark-master:7077 \
  /opt/spark-jobs/05_gold_layer.py

Creates three analytical aggregates (DROP + CREATE on each run):

Table	Description
`gold.daily_sales`	Revenue, orders, unique customers per day
`gold.product_performance`	Units sold, revenue per product/category
`gold.customer_metrics`	Lifetime value, order count, first/last order per customer

Step 7 — Analytics Queries

docker exec spark-master spark-submit \
  --master spark://spark-master:7077 \
  /opt/spark-jobs/06_analytics.py

Runs KPI reports and prints summary statistics to stdout.

Step 8 — Verify in Nessie

# All tables registered in Nessie catalog
curl -s http://localhost:19120/api/v2/trees/main/entries | python3 -m json.tool

# Commit history
curl -s http://localhost:19120/api/v2/trees/main/history | python3 -m json.tool | head -60

Expected entries: bronze.orders, bronze.customers, bronze.products, silver.orders_enriched, gold.daily_sales, gold.product_performance, gold.customer_metrics

Dremio Configuration

Step 1 — First Login

Open http://localhost:9047 and complete the setup wizard.

Step 2 — Add Nessie Source

Sources → + → Nessie

General tab:

Name:            nessie
Endpoint URL:    http://nessie:19120/api/v2
Authentication:  None

Storage tab:

AWS Access Key:    minioadmin
AWS Secret Key:    minioadmin123
AWS Root Path:     warehouse
Encrypt connection: ☐ unchecked

Add the following properties (click Add Property for each):

Property	Value
`fs.s3a.endpoint`	`minio:9000` ⚠️ NO http:// prefix
`fs.s3a.path.style.access`	`true`
`dremio.s3.compat`	`true`
`fs.s3a.aws.credentials.provider`	`org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider`

⚠️ Critical: use minio:9000 NOT http://minio:9000. Dremio adds the scheme automatically. Using the full URL causes UnknownHostException: http.

Metadata tab:

Dataset Discovery:  Every 1 Minute(s)
Dataset Details:    All Datasets / Every 1 Minute(s)
☑ Automatically format files into physical datasets
☑ Remove dataset definitions if underlying data is unavailable

→ Save

Step 3 — Add MinIO as S3 Source

Sources → + → Amazon S3

General tab:

Name:              minio
Authentication:    AWS Access Key
AWS Access Key:    minioadmin
AWS Secret Key:    minioadmin123
Encrypt connection: ☐ unchecked
Root Path:         /

Advanced Options — Add Properties:

Property	Value
`fs.s3a.endpoint`	`minio:9000`
`fs.s3a.path.style.access`	`true`
`dremio.s3.compat`	`true`

→ Save

Step 4 — Refresh Metadata

Run each statement individually in the SQL Runner:

ALTER TABLE nessie.bronze.orders            REFRESH METADATA FORCE UPDATE;
ALTER TABLE nessie.bronze.customers         REFRESH METADATA FORCE UPDATE;
ALTER TABLE nessie.bronze.products          REFRESH METADATA FORCE UPDATE;
ALTER TABLE nessie.silver.orders_enriched   REFRESH METADATA FORCE UPDATE;
ALTER TABLE nessie.gold.daily_sales         REFRESH METADATA FORCE UPDATE;
ALTER TABLE nessie.gold.product_performance REFRESH METADATA FORCE UPDATE;
ALTER TABLE nessie.gold.customer_metrics    REFRESH METADATA FORCE UPDATE;

Step 5 — Run Queries

-- Verify tables
SHOW SCHEMAS IN nessie;
SHOW TABLES IN nessie.bronze;

-- Row counts
SELECT COUNT(*) FROM nessie.bronze.orders;
SELECT COUNT(*) FROM nessie.silver.orders_enriched;
SELECT COUNT(*) FROM nessie.gold.daily_sales;

-- Top 5 categories by revenue
SELECT category, SUM(calculated_revenue) AS revenue
FROM nessie.silver.orders_enriched
GROUP BY category
ORDER BY revenue DESC
LIMIT 5;

-- Daily sales last 30 days
SELECT order_date, total_revenue, num_orders
FROM nessie.gold.daily_sales
ORDER BY order_date DESC
LIMIT 30;

-- Top customers by lifetime value
SELECT customer_id, customer_segment, state, lifetime_value
FROM nessie.gold.customer_metrics
ORDER BY lifetime_value DESC
LIMIT 10;

-- Iceberg time travel
SELECT * FROM nessie.bronze.orders
AT BRANCH main
LIMIT 5;

JupyterLab — PySpark Exploration

# Get the access token
docker logs jupyter 2>&1 | grep "token="

Open http://localhost:8888, create a new notebook:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("Exploration") \
    .master("spark://spark-master:7077") \
    .getOrCreate()

# Read Iceberg tables
orders = spark.table("iceberg.bronze.orders")
orders.printSchema()
orders.show(5)

# Gold layer analytics
spark.table("iceberg.gold.daily_sales") \
     .orderBy("order_date", ascending=False) \
     .show(10)

# Time travel
spark.read.format("iceberg") \
     .option("as-of-timestamp", "2024-01-01 00:00:00") \
     .load("iceberg.bronze.orders") \
     .count()

Monitoring

Prometheus

Open http://localhost:9090 → check /targets — all should be green.

Useful PromQL queries:

# CPU usage
rate(process_cpu_seconds_total[5m])

# Memory (MB)
process_resident_memory_bytes / 1024 / 1024

# Active Spark workers
spark_master_aliveWorkers

# JVM heap (Dremio)
jvm_memory_bytes_used{area="heap"}

Grafana

Open http://localhost:3000 (admin / admin).

Add Prometheus datasource: Connections → Data Sources → Prometheus → URL: http://prometheus:9090 → Save & Test

Import ready-made JVM dashboard: Dashboards → Import → ID 4701 → select Prometheus → Import

Recommended panels:

Panel	Query	Visualization
CPU Usage	`rate(process_cpu_seconds_total[5m])`	Time series
Memory (MB)	`process_resident_memory_bytes / 1024 / 1024`	Gauge
Spark Workers	`spark_master_aliveWorkers`	Stat

Data Model

customers (1,000 records)
├── customer_id (PK), email, first_name, last_name
├── city, state, registration_date
└── customer_segment  (Premium | Standard | Basic)

products (200 records)
├── product_id (PK), product_name
├── category  (Electronics | Clothing | Home | Sports | Books)
└── price, stock_quantity

orders (5,000 records)
├── order_id (PK), customer_id (FK), product_id (FK)
├── order_date, quantity, total_amount
├── status  (completed | pending | cancelled)
└── payment_method  (Credit Card | PayPal | Debit Card)

Silver layer — orders_enriched:

Column	Source
`customer_segment`, `state`	joined from customers
`category`, `unit_price`	joined from products
`calculated_revenue`	`quantity × unit_price`
`order_value_tier`	High if `total_amount > 500`
`order_year`, `order_month`	partition columns from `order_date`

Gold layer tables:

Table	Key Columns
`daily_sales`	`order_date · num_orders · total_revenue · unique_customers`
`product_performance`	`product_id · category · units_sold · total_revenue`
`customer_metrics`	`customer_id · total_orders · lifetime_value · first/last_order_date`

Project Structure

modern-data-platform/
├── docker-compose.yml                    ← All services definition
├── start.sh                              ← Quick start script
│
├── airflow/
│   └── dags/
│       └── ecommerce_etl_pipeline.py    ← Full medallion DAG (8 tasks)
│
├── spark/
│   ├── Dockerfile                        ← Custom image with Iceberg/Nessie JARs
│   ├── spark-defaults.conf               ← Iceberg + S3 defaults
│   └── jobs/
│       ├── 01_setup_bronze.py            ← Create namespaces + tables
│       ├── 02_load_bronze.py             ← CSV → Iceberg Bronze
│       ├── 03_iceberg_features.py        ← Time travel + schema evolution demo
│       ├── 04_silver_layer.py            ← Bronze → Silver (CTAS + MERGE)
│       ├── 05_gold_layer.py              ← Silver → Gold (DROP + CREATE)
│       ├── 06_analytics.py               ← KPI reports
│       └── run_pipeline.sh               ← Run all 6 steps in sequence
│
├── scripts/
│   └── generate_data.py                  ← Standalone data generator
│
├── sql/
│   └── init.sql                          ← PostgreSQL init (schema + namespace props)
│
├── monitoring/
│   ├── prometheus/
│   │   └── prometheus.yml
│   └── grafana/
│       ├── dashboards/
│       └── datasources/
│
└── data/                                 ← Created at runtime
    ├── raw/                              ← Source CSV files
    ├── bronze/                           ← Local Parquet (Airflow output)
    ├── silver/
    └── gold/

Full Reset

# Stop everything and remove all data
docker compose down --volumes --remove-orphans

# Remove custom Spark image (will be rebuilt)
docker rmi modern-data-platform-spark:latest 2>/dev/null || true

# Clean local data
rm -rf data/raw/* data/bronze/* data/silver/* data/gold/*
rm -rf airflow/logs/*

# Rebuild and restart
docker compose up --build -d

Useful Commands

# View running services
docker compose ps

# Follow logs of a specific service
docker compose logs -f airflow-webserver
docker compose logs -f spark-master
docker compose logs -f dremio

# Run full Spark pipeline in one command
docker exec spark-master bash /opt/spark-jobs/run_pipeline.sh

# Run individual Spark job
docker exec spark-master spark-submit \
  --master spark://spark-master:7077 \
  /opt/spark-jobs/04_silver_layer.py

# Open PostgreSQL shell
docker exec -it postgres psql -U admin -d airflow

# Inspect Nessie catalog
curl -s http://localhost:19120/api/v2/trees/main/entries | python3 -m json.tool

# Check MinIO buckets
docker exec minio-setup mc ls myminio/warehouse --recursive

# Restart a single service
docker compose restart dremio

# Stop (keep volumes)
docker compose down

# Stop and wipe everything
docker compose down -v --remove-orphans

Troubleshooting

Problem	Cause	Fix
Services won't start	Not enough RAM	Set Docker Desktop to 8 GB+
Port conflict	Port already in use	Change host ports in `docker-compose.yml`
Airflow DAG fails	Missing CSV files	Run `generate_data.py` first
Spark can't reach MinIO	Wrong endpoint or bucket	Check `minio:9000` reachable from spark container; verify bucket `warehouse` exists
Dremio `UnknownHostException: http`	Endpoint set to `http://minio:9000`	Use `minio:9000` without `http://` in Nessie source properties
Dremio `metadata is unchanged`	Stale cache	Run `FORGET TABLE` then `REFRESH METADATA FORCE UPDATE` per table
Tables not visible in Dremio	Spark pipeline not finished	Confirm steps 1–6 completed; check Nessie entries via API
Grafana shows no data	Time range too narrow	Set time range to "Last 1 hour"
Jupyter can't connect to Spark	spark-master not healthy	Run `docker compose ps`; restart spark-master if needed

Production Upgrade Path

Component	Local Setup	Production Option
Object Storage	MinIO	AWS S3 / Google GCS / Azure ADLS
Spark	3-node Docker cluster	EMR / Dataproc / Databricks
Query Engine	Dremio OSS	Dremio Cloud / Trino
Orchestration	Airflow LocalExecutor	MWAA / Airflow CeleryExecutor / Prefect
Catalog	Nessie OSS	Nessie Cloud / AWS Glue / Polaris
Streaming	—	Kafka + Spark Structured Streaming
Transformations	—	dbt on Silver/Gold layers
Monitoring	Prometheus + Grafana	Datadog / CloudWatch / Grafana Cloud

Name		Name	Last commit message	Last commit date
Latest commit History 22 Commits
airflow/dags		airflow/dags
data		data
monitoring/prometheus		monitoring/prometheus
screen		screen
scripts		scripts
spark		spark
sql		sql
README.md		README.md
docker-compose.yml		docker-compose.yml
dremio_nessie_etl_V2.sql		dremio_nessie_etl_V2.sql
start.sh		start.sh

Folders and files

Latest commit

History

Repository files navigation

Modern Data Lakehouse Platform

Architecture Overview

Tech Stack

Prerequisites

Quick Start

Service URLs

Step-by-Step Pipeline

Step 1 — Generate Raw Data

Step 2 — Setup Bronze Iceberg Tables (Namespaces)

Step 3 — Load CSV → Bronze Iceberg (MinIO)

Step 4 — Iceberg Features Demo (Time Travel + Schema Evolution)

Step 5 — Silver Layer (Enrichment + Joins)

Step 6 — Gold Layer (Aggregations)

Step 7 — Analytics Queries

Step 8 — Verify in Nessie

Dremio Configuration

Step 1 — First Login

Step 2 — Add Nessie Source

Step 3 — Add MinIO as S3 Source

Step 4 — Refresh Metadata

Step 5 — Run Queries

JupyterLab — PySpark Exploration

Monitoring

Prometheus

Grafana

Data Model

Project Structure

Full Reset

Useful Commands

Troubleshooting

Production Upgrade Path

References

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages