Skip to content

Commit e0cd822

Browse files
committed
FEATURE: Add SQL plugin support
Signed-off-by: Pascal Zimmermann <pascal.zimmermann@theiotstudio.com>
1 parent 5662acd commit e0cd822

44 files changed

Lines changed: 2715 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

sql/Makefile

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# Makefile for SQL Plugin Testing
2+
3+
.PHONY: help test test-watch test-coverage db-up db-down db-restart db-logs db-clean unit-tests integration-tests
4+
5+
help: ## Show this help message
6+
@echo 'Usage: make [target]'
7+
@echo ''
8+
@echo 'Available targets:'
9+
@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf " %-20s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
10+
11+
# Database Management
12+
db-up: ## Start all test databases
13+
docker-compose -f docker-compose.test.yaml up -d
14+
@echo "Waiting for databases to be ready..."
15+
@sleep 10
16+
@echo "Databases are ready!"
17+
@echo ""
18+
@echo "PostgreSQL: localhost:5432"
19+
@echo "MySQL: localhost:3306"
20+
@echo "MariaDB: localhost:3307"
21+
@echo "Adminer UI: http://localhost:8080"
22+
23+
db-down: ## Stop all test databases
24+
docker-compose -f docker-compose.test.yaml down
25+
26+
db-restart: ## Restart all test databases
27+
docker-compose -f docker-compose.test.yaml restart
28+
29+
db-logs: ## Show database logs
30+
docker-compose -f docker-compose.test.yaml logs -f
31+
32+
db-clean: ## Stop and remove all databases and volumes
33+
docker-compose -f docker-compose.test.yaml down -v
34+
@echo "All database containers and volumes removed"
35+
36+
db-status: ## Show status of database containers
37+
docker-compose -f docker-compose.test.yaml ps
38+
39+
# Testing
40+
unit-tests: ## Run unit tests
41+
npm test -- replace-sql-builtin-variables.test.ts
42+
43+
test: ## Run all tests
44+
npm test
45+
46+
test-watch: ## Run tests in watch mode
47+
npm test -- --watch
48+
49+
test-coverage: ## Run tests with coverage report
50+
npm test -- --coverage
51+
52+
test-verbose: ## Run tests with verbose output
53+
npm test -- --verbose
54+
55+
# Database Verification
56+
verify-postgres: ## Verify PostgreSQL test data
57+
docker exec -it perses-sql-test-postgres psql -U perses -d perses_test -c "\
58+
SELECT 'system_metrics' as table_name, COUNT(*) as row_count FROM system_metrics \
59+
UNION ALL SELECT 'sensor_readings', COUNT(*) FROM sensor_readings \
60+
UNION ALL SELECT 'http_requests', COUNT(*) FROM http_requests \
61+
UNION ALL SELECT 'transactions', COUNT(*) FROM transactions;"
62+
63+
verify-mysql: ## Verify MySQL test data
64+
docker exec -it perses-sql-test-mysql mysql -u perses -pperses_test_password perses_test -e "\
65+
SELECT 'system_metrics' as table_name, COUNT(*) as row_count FROM system_metrics \
66+
UNION ALL SELECT 'sensor_readings', COUNT(*) FROM sensor_readings \
67+
UNION ALL SELECT 'http_requests', COUNT(*) FROM http_requests \
68+
UNION ALL SELECT 'transactions', COUNT(*) FROM transactions;"
69+
70+
verify-mariadb: ## Verify MariaDB test data
71+
docker exec -it perses-sql-test-mariadb mysql -u perses -pperses_test_password perses_test -e "\
72+
SELECT 'system_metrics' as table_name, COUNT(*) as row_count FROM system_metrics \
73+
UNION ALL SELECT 'sensor_readings', COUNT(*) FROM sensor_readings \
74+
UNION ALL SELECT 'http_requests', COUNT(*) FROM http_requests \
75+
UNION ALL SELECT 'transactions', COUNT(*) FROM transactions;"
76+
77+
verify-all: verify-postgres verify-mysql verify-mariadb ## Verify all databases
78+
79+
# Development
80+
dev-setup: db-up ## Complete development setup
81+
npm install
82+
@echo ""
83+
@echo "✅ Development environment ready!"
84+
@echo "Run 'make test' to run tests"
85+
@echo "Run 'make test-watch' for watch mode"
86+
87+
clean-all: db-clean ## Clean everything (databases, node_modules, dist)
88+
rm -rf node_modules dist
89+
90+
# Quick commands
91+
quick-test: db-up test ## Start databases and run tests
92+
93+
integration: db-up verify-all ## Start databases and verify data
94+
@echo ""
95+
@echo "✅ Integration test environment ready!"
96+
97+
# Install dependencies
98+
install: ## Install npm dependencies
99+
npm install
100+
101+
build: ## Build the plugin
102+
npm run build
103+
104+
lint: ## Run linter
105+
npm run lint
106+
107+
type-check: ## Run TypeScript type checking
108+
npm run type-check
109+
110+
# Complete test workflow
111+
full-test: clean-all install db-up verify-all test test-coverage ## Complete test workflow
112+
@echo ""
113+
@echo "✅ Full test workflow completed!"

sql/PR_READINESS.md

Whitespace-only changes.

sql/README.md

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# SQL Plugin
2+
3+
A SQL datasource plugin for Perses that supports multiple SQL databases with interactive querying capabilities.
4+
5+
## Supported Databases
6+
7+
- **PostgreSQL** (with TimescaleDB support)
8+
- **MySQL**
9+
- **MariaDB**
10+
11+
## Features
12+
13+
- **SQL Datasource**: Configure connections to SQL databases via Perses proxy
14+
- **Time Series Queries**: Query time-series data with automatic visualization
15+
- **Explore Mode**: Interactive SQL query builder with Table and Graph views (Prometheus-style)
16+
- **SQL Macros**: Use `$__timeFilter`, `$__timeFrom`, `$__timeTo`, `$__interval` in queries
17+
- **Multi-Database Support**: Connect to PostgreSQL, MySQL, and MariaDB
18+
- **TLS/SSL Support**: Secure connections with customizable SSL modes and certificate management
19+
- **Secret Management**: Store credentials securely using Perses secrets
20+
- **Connection Pooling**: Efficient connection management via backend proxy
21+
22+
## Installation
23+
24+
This plugin is part of the Perses plugins monorepo. Install dependencies:
25+
26+
```bash
27+
cd perses/plugins/sql
28+
npm install
29+
```
30+
31+
## Usage
32+
33+
### Creating a SQL Datasource
34+
35+
1. Navigate to **Configuration → Datasources** in Perses UI
36+
2. Click **Add Datasource**
37+
3. Select **SQL Datasource**
38+
4. Configure:
39+
- **Driver**: Choose database type (PostgreSQL, MySQL, or MariaDB)
40+
- **Host**: Database host and port (e.g., `localhost:5432`)
41+
- **Database**: Database name
42+
- **Secret**: Select a secret containing credentials (BasicAuth)
43+
- **SSL Mode** (PostgreSQL): `disable`, `require`, `verify-ca`, or `verify-full`
44+
- **TLS Certificates** (optional): CA cert, client cert, client key
45+
46+
### Using the Explore Mode
47+
48+
1. Navigate to **Explore** in Perses UI
49+
2. Select **SQL Explorer**
50+
3. Choose your SQL datasource
51+
4. Write your SQL query in the editor
52+
5. Switch between **Table** and **Graph** views
53+
6. Click **Run Query** to execute
54+
55+
Example query:
56+
```sql
57+
SELECT
58+
timestamp AS time,
59+
cpu_percent AS value,
60+
host
61+
FROM system_metrics
62+
WHERE $__timeFilter(timestamp)
63+
ORDER BY timestamp DESC
64+
LIMIT 100
65+
```
66+
67+
### Time Series Queries in Dashboards
68+
69+
Add a panel to your dashboard:
70+
71+
1. **Add Panel**
72+
2. Select **Query Type**: Time Series Query
73+
3. Select **Plugin**: SQL Time Series Query
74+
4. Select **Datasource**: Your SQL datasource
75+
5. Write your query:
76+
77+
```sql
78+
SELECT
79+
time_bucket('$__interval seconds', timestamp) AS time,
80+
host AS label,
81+
AVG(cpu_percent) AS value
82+
FROM system_metrics
83+
WHERE $__timeFilter(timestamp)
84+
GROUP BY time, host
85+
ORDER BY time
86+
```
87+
88+
### Available SQL Macros
89+
90+
The plugin automatically replaces these macros in your queries:
91+
92+
- **`$__timeFilter(column)`**: Generates `column >= 'start' AND column <= 'end'`
93+
- **`$__timeFrom`**: Start timestamp of the time range
94+
- **`$__timeTo`**: End timestamp of the time range
95+
- **`$__interval`**: Calculated interval in seconds (for time bucketing)
96+
- **`$__interval_ms`**: Calculated interval in milliseconds
97+
98+
Example with macros:
99+
```sql
100+
SELECT
101+
time_bucket('$__interval seconds', created_at) AS time,
102+
category,
103+
COUNT(*) AS count
104+
FROM transactions
105+
WHERE $__timeFilter(created_at)
106+
AND status = 'completed'
107+
GROUP BY time, category
108+
ORDER BY time
109+
```
110+
111+
Gets transformed to:
112+
```sql
113+
SELECT
114+
time_bucket('300 seconds', created_at) AS time,
115+
category,
116+
COUNT(*) AS count
117+
FROM transactions
118+
WHERE created_at >= '2026-01-25T10:00:00Z' AND created_at <= '2026-01-25T11:00:00Z'
119+
AND status = 'completed'
120+
GROUP BY time, category
121+
ORDER BY time
122+
```
123+
124+
## Query Requirements
125+
126+
### For Time Series Visualization
127+
128+
Your query must return:
129+
- **Time column**: Named `time` or aliased as `time` (timestamp format)
130+
- **Value column**: Named `value` or aliased as `value` (numeric)
131+
- **Label columns** (optional): For series differentiation (e.g., `host`, `sensor_id`)
132+
133+
### Supported Time Formats
134+
135+
- ISO 8601: `2026-01-25T10:00:00Z`
136+
- Unix timestamp (seconds): `1706176800`
137+
- Unix timestamp (milliseconds): `1706176800000`
138+
139+
## Database-Specific Features
140+
141+
### PostgreSQL / TimescaleDB
142+
143+
- Supports `time_bucket()` for time aggregation
144+
- SSL modes: `disable`, `require`, `verify-ca`, `verify-full`
145+
- Advanced connection options via PostgreSQL config
146+
147+
### MySQL / MariaDB
148+
149+
- Standard SQL queries
150+
- TLS/SSL support
151+
- Connection timeout and max connections configuration
152+
153+
## Development
154+
155+
### Prerequisites
156+
157+
- Node.js >= 22
158+
- npm >= 10
159+
- Docker (for test databases)
160+
161+
### Setup
162+
163+
```bash
164+
# Install dependencies
165+
npm install
166+
167+
# Start test databases
168+
make db-up
169+
170+
# Run tests
171+
npm test
172+
173+
# Type checking
174+
npm run type-check
175+
176+
# Start development server
177+
percli plugin start .
178+
```
179+
180+
### Testing
181+
182+
```bash
183+
# Run all tests
184+
npm test
185+
186+
# Run with coverage
187+
npm test -- --coverage
188+
189+
# Start test databases
190+
make db-up
191+
192+
# Verify test data
193+
make verify-all
194+
195+
# Stop databases
196+
make db-down
197+
```
198+
199+
## Architecture
200+
201+
- **Frontend** (TypeScript/React): Datasource editor, query editor, explore mode
202+
- **Backend** (Go): SQL proxy in Perses core (`internal/api/impl/proxy/`)
203+
- **Schemas** (CUE): Configuration validation
204+
205+
All database connections and query execution happen server-side via the Perses backend proxy for security.
206+
207+
## Examples
208+
209+
See `test-data/` directory for sample database schemas and queries:
210+
- `test-data/postgres/` - PostgreSQL with TimescaleDB examples
211+
- `test-data/mysql/` - MySQL examples
212+
- `test-data/mariadb/` - MariaDB examples
213+
214+
## Contributing
215+
216+
See [CONTRIBUTING.md](CONTRIBUTING.md) for development guidelines.
217+
218+
## License
219+
220+
Apache License 2.0

sql/cue.mod/module.cue

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
module: "github.com/perses/plugin/sql@v0"
2+
language: {
3+
version: "v0.9.2"
4+
}
5+
source: {
6+
kind: "git"
7+
}
8+
deps: {
9+
"github.com/perses/perses/cue@v0": {
10+
v: "v0.53.0-rc.0"
11+
default: true
12+
}
13+
"github.com/perses/shared/cue@v0": {
14+
v: "v0.53.0-rc.1"
15+
default: true
16+
}
17+
}

0 commit comments

Comments
 (0)