Skip to content

brianbirir/virtual-iot-simulator

BranchesTags

Repository files navigation

Virtual IoT Simulator

A large-scale IoT device simulator capable of running thousands of concurrent virtual devices that generate realistic telemetry and publish it over configurable protocols.

The system is composed of three services:

  • Device Runtime (Go) — the data plane. Runs one goroutine per device, generates telemetry, and streams it over the configured protocol adapter.
  • Simulation Orchestrator (Python) — the control plane. Loads device profiles, manages the fleet lifecycle, and exposes both a CLI and a REST API.
  • Frontend (React + TypeScript) — a web dashboard for controlling the fleet and observing live telemetry via the orchestrator REST API.

Status: All phases complete. See docs/SYSTEM.md for the full design document.

System Overview

 ┌─────────────────────────────────────────────────────────────────────────┐
 │                        Operator / CI                                    │
 │                                                                         │
 │   iot-sim spawn / stop / status / stream      REST API :8000            │
 │   iot-sim scenario run ramp_up.py             POST /api/v1/devices/...  │
 └──────────────┬──────────────────────────────────────┬───────────────────┘
                │ gRPC (SpawnDevices, StopDevices,      │ HTTP / SSE
                │ InjectFault, StreamTelemetry …)       │
 ┌──────────────▼───────────────────────────────────────▼───────────────────┐
 │                  Simulation Orchestrator  (Python)                        │
 │                                                                           │
 │  ┌─────────────┐  ┌──────────────┐  ┌───────────────┐  ┌─────────────┐  │
 │  │  Typer CLI  │  │  FastAPI app │  │  RuntimePool  │  │  Scenario   │  │
 │  │  (cli.py)   │  │  (api.py)    │  │  (pool.py)    │  │  Engine     │  │
 │  └──────┬──────┘  └──────┬───────┘  └───────┬───────┘  │ (scenario  │  │
 │         └────────────────┴──────────────────►│          │  .py)      │  │
 │                                              │          └─────────────┘  │
 │                             consistent-hash ring                          │
 └──────────────────────────────────┬───────────────────────────────────────┘
                                    │ gRPC :50051
 ┌──────────────────────────────────▼───────────────────────────────────────┐
 │                    Device Runtime  (Go)                                   │
 │                                                                           │
 │  ┌──────────────────────────────────────────────────────┐                 │
 │  │  Manager                                             │                 │
 │  │   spawnOne() ──► VirtualDevice goroutine × N         │                 │
 │  │                  ┌─────────────────────────────┐     │                 │
 │  │                  │  ticker → Generator.Next()  │     │                 │
 │  │                  │  applyFaults()              │     │                 │
 │  │                  │  Publisher.Publish()        │     │                 │
 │  │                  └──────────────┬──────────────┘     │                 │
 │  │                                 │ fan-in chan         │                 │
 │  │  Broadcaster ◄──────────────────┘                    │                 │
 │  └──────────────────────────────────────────────────────┘                 │
 │                                                                           │
 │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐                    │
 │  │  MQTT pool   │  │  HTTP POST   │  │  AMQP chan   │  ← protocol        │
 │  │  (paho)      │  │  (net/http)  │  │  pool        │    adapters        │
 │  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘                    │
 │         │                 │                  │                            │
 │  :8080 /metrics (Prometheus)    /healthz  /readyz                         │
 └─────────┼─────────────────┼─────────────────┼──────────────────────────-─┘
           │                 │                  │
 ┌─────────▼─────┐   ┌───────▼──────┐   ┌──────▼───────┐   ┌─────────────┐
 │   Mosquitto   │   │  HTTP target │   │   RabbitMQ   │   │  Prometheus │
 │   :1883       │   │  (any)       │   │   :5672      │   │  :9090      │
 └───────────────┘   └──────────────┘   └──────────────┘   └──────┬──────┘
                                                                   │
                                                            ┌──────▼──────┐
                                                            │   Grafana   │
                                                            │   :3000     │
                                                            └─────────────┘

Data generators (per telemetry field, seeded deterministically): gaussian · static · brownian (Ornstein-Uhlenbeck) · diurnal (sinusoidal) · markov (state machine)

Fault injection (applied per device tick): DISCONNECT · LATENCY_SPIKE · DATA_CORRUPTION · BATTERY_DRAIN · CLOCK_DRIFT

flowchart TB
    subgraph Operator["Operator / CI"]
        CLI["iot-sim CLI\nspawn · stop · status · stream\nscenario run"]
        RESTC["REST client\ncurl / SDK"]
    end

    subgraph Orchestrator["Simulation Orchestrator (Python)"]
        direction TB
        API["FastAPI\n:8000"]
        CLIAPP["Typer CLI"]
        POOL["RuntimePool\nconsistent-hash ring"]
        SCENARIO["ScenarioEngine\nSimClock · ScenarioContext"]
        CLI --> CLIAPP
        RESTC --> API
        API --> POOL
        CLIAPP --> POOL
        SCENARIO --> POOL
    end

    subgraph Runtime["Device Runtime (Go)  :50051"]
        direction TB
        GRPC["gRPC Server\nDeviceRuntimeService"]
        MGR["Manager\nspawnOne · Stop · InjectFault"]
        BC["Broadcaster\nfan-in channel"]

        subgraph Devices["VirtualDevice goroutine × N"]
            GEN["Generator.Next()\ngaussian · brownian\ndiurnal · markov · static"]
            FAULT["applyFaults()\nDISCONNECT · LATENCY_SPIKE\nDATA_CORRUPTION · CLOCK_DRIFT"]
            PUB["Publisher.Publish()"]
            GEN --> FAULT --> PUB
        end

        GRPC --> MGR --> Devices
        Devices --> BC
    end

    subgraph Frontend["Frontend (React)  :3001"]
        UI["Dashboard · Devices · Telemetry\nnginx → proxy /api"]
    end

    subgraph Infra["Infrastructure"]
        MQTT["Mosquitto\n:1883"]
        PROM["Prometheus\n:9090"]
        GRAF["Grafana\n:3000"]
        RABBIT["RabbitMQ\n:5672"]
        HTTP["HTTP target\n(any)"]
    end

    UI -->|"REST / SSE"| API
    POOL -->|gRPC| GRPC
    PUB -->|MQTT| MQTT
    PUB -->|AMQP| RABBIT
    PUB -->|HTTP POST| HTTP
    Runtime -->|"/metrics :8080"| PROM
    PROM --> GRAF
Loading

Repository Layout

.
├── device-runtime/          # Go gRPC server + virtual device engine
│   ├── cmd/runtime/         # Binary entry point
│   └── internal/
│       ├── device/          # VirtualDevice, Manager, RuntimeClock
│       ├── generator/       # Gaussian, Static generators + factory
│       ├── protocol/        # Publisher interface + Console adapter
│       └── server/          # gRPC handlers, Broadcaster, interceptors
├── orchestrator/            # Python orchestrator
│   ├── orchestrator/
│   │   ├── api.py           # FastAPI REST app
│   │   ├── cli.py           # Typer CLI (iot-sim)
│   │   ├── config.py        # Profile loader + Pydantic validation
│   │   └── grpc_client.py   # Typed async gRPC client
│   ├── tests/
│   ├── Pipfile              # Pipenv dependency manifest
│   └── pyproject.toml       # Package metadata + entry points
├── frontend/                # React + TypeScript web dashboard
│   ├── src/
│   │   ├── api/             # TanStack Query hooks + fetch client
│   │   ├── components/      # Shared MUI layout components
│   │   └── pages/           # Dashboard, Devices, Telemetry pages
│   ├── package.json
│   └── vite.config.ts       # Dev server with /api proxy to :8000
├── proto/simulator/v1/      # Protobuf definitions (source of truth)
├── profiles/                # Device profile YAML files
├── deployments/             # Docker Compose, Dockerfiles, nginx config
├── docs/SYSTEM.md           # Full technical design document
├── IMPLEMENTATION_PLAN.md   # Phase-by-phase implementation plan
├── buf.yaml                 # Buf lint/breaking-change config
└── Makefile                 # Build, test, and code-gen targets

Prerequisites

Tool Version Purpose
Docker + Compose ≥ 24 Run the full stack
Go ≥ 1.21 Device runtime (local dev)
Python ≥ 3.12 Orchestrator (local dev)
Node.js ≥ 20 Frontend (local dev)
pipenv latest Python dependency management
buf latest Protobuf linting and code generation

Quick Start

Docker (recommended)

The entire stack — runtime, orchestrator, frontend, MQTT broker, Prometheus, and Grafana — starts with a single command:

docker compose -f deployments/docker-compose.yaml up --build
Service URL Description
Frontend http://localhost:3001 React dashboard
Orchestrator API http://localhost:8000 FastAPI + Swagger UI at /docs
Runtime admin http://localhost:8080 /healthz · /readyz · /metrics
Grafana http://localhost:3000 Dashboards (admin / admin)
Prometheus http://localhost:9090 Metrics explorer

Startup order is enforced via healthcheck dependencies: mosquitto → runtime → orchestrator → frontend

Local Development

1. Generate protobuf code

make proto-gen

This runs buf generate for Go and grpc_tools.protoc for Python, writing generated files to device-runtime/gen/go/ and orchestrator/gen/python/.

2. Start the Device Runtime

make go-build
./device-runtime/runtime --port 50051 --admin-port 8080 --log-level info

The runtime exposes:

  • :50051 — gRPC (DeviceRuntimeService)
  • :8080 — Admin HTTP (/healthz, /readyz)

3. Install Python dependencies

cd orchestrator
pipenv install

4. Install Frontend dependencies

If you have Node.js ≥ 20 installed locally:

cd frontend
npm install

If Node.js is not installed locally, use Docker to install packages or generate the lock file:

# Generate / update package-lock.json
docker run --rm -v "$(pwd)/frontend:/app" -w /app node:20-alpine npm install

# Install packages only (no node_modules on host, lock file only)
docker run --rm -v "$(pwd)/frontend:/app" -w /app node:20-alpine npm install --package-lock-only

This is also required when adding new packages so that package-lock.json stays in sync before running make docker-build.

5. Control devices via CLI

# Spawn 5 temperature sensors
pipenv run iot-sim spawn --profile ../profiles/temperature_sensor.yaml --count 5

# Check fleet status
pipenv run iot-sim status

# Stream live telemetry to the terminal
pipenv run iot-sim stream --type temperature_sensor

# Stop all devices
pipenv run iot-sim stop --all

6. Control devices via the REST API

# Start the API server (default: http://localhost:8000)
pipenv run iot-sim serve

# Spawn devices
curl -X POST http://localhost:8000/api/v1/devices/spawn \
  -H "Content-Type: application/json" \
  -d '{"profile": "../profiles/temperature_sensor.yaml", "count": 5}'

# Fleet status
curl http://localhost:8000/api/v1/devices/status

# Stream telemetry (SSE)
curl -N "http://localhost:8000/api/v1/devices/stream?device_type=temperature_sensor"

# Stop all
curl -X POST http://localhost:8000/api/v1/devices/stop \
  -H "Content-Type: application/json" \
  -d '{"all": true}'

Interactive API docs (Swagger UI) are available at http://localhost:8000/docs.

Device Profiles

Profiles are YAML files in profiles/ that define a device type's telemetry schema, protocol, and generator configuration. See docs/DEVICE_PROFILES.md for the full reference.

Development

Run tests

# Go
make go-test

# Python
cd orchestrator && pipenv run pytest tests/ -v

Makefile targets

Target Description
make proto-gen Generate Go + Python code from .proto files
make proto-lint Lint proto files with Buf
make proto-breaking Check for breaking changes against main
make go-build Build the runtime binary
make go-test Run Go tests
make py-test Run Python tests
make all Lint → generate → build → test (all)

Documentation

  • docs/SYSTEM.md — full technical design: architecture, data models, concurrency model, API reference, and implementation status per phase.
  • docs/DEVICE_PROFILES.md — device profile YAML reference with all fields and generator types.

About

Simulate IoT devices to test IoT ingestion platforms hence removing the need of actual devices. Good for quickening development in cases where devices are scarce to access

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages