Skip to content

Commit d6ef4fe

Browse files
docs: update README with multi-store APIs, scalable clusters, scenario tests
- Cluster: make cluster NODES=N, add-node.sh for live addition - Docker: dynamic node addition via docker run - K8s: kubectl scale example - Multi-store HTTP APIs: /api/stores CRUD, /api/stores/{name}/cache/{key} - RESP: SELECT, STORES commands - Scenario tests section with SCENARIO_SEED reproducibility - Updated Makefile reference, access points table, key features
1 parent 16811bb commit d6ef4fe

1 file changed

Lines changed: 103 additions & 27 deletions

File tree

README.md

Lines changed: 103 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -74,14 +74,24 @@ open http://localhost:3000 # admin / admin123
7474
- Go 1.23.2+
7575
- `redis-cli` (optional, for RESP testing)
7676

77-
### Local Cluster (3 nodes)
77+
### Local Cluster (N nodes)
7878
```bash
79-
# Build and start a fresh 3-node cluster
79+
# Start a 3-node cluster (default)
8080
make cluster
8181

82+
# Start a 5-node cluster
83+
make cluster NODES=5
84+
85+
# Start a 10-node cluster
86+
make cluster NODES=10
87+
8288
# Check cluster health
8389
curl -s http://localhost:9080/health | python3 -m json.tool
8490

91+
# Add a node to a running cluster (auto-discovers ports and seeds)
92+
./scripts/add-node.sh
93+
./scripts/add-node.sh --node-name=node-6
94+
8595
# Stop the cluster
8696
make cluster-stop
8797

@@ -102,6 +112,13 @@ docker pull rishabhverma17/hypercache:latest
102112
# Start full stack (3-node cluster + Elasticsearch + Grafana + Filebeat)
103113
docker compose -f docker-compose.cluster.yml up -d
104114

115+
# Add a 4th node to a running Docker cluster (auto-joins via gossip)
116+
docker run -d --name hypercache-node4 \
117+
--network cache_hypercache-cluster \
118+
-p 8083:8080 -p 9083:9080 \
119+
rishabhverma17/hypercache:latest \
120+
--protocol resp --node-id node-4
121+
105122
# Or build locally and start
106123
make docker-build && make docker-up
107124

@@ -111,22 +128,28 @@ docker compose -f docker-compose.cluster.yml down
111128

112129
### Kubernetes
113130
```bash
131+
# Deploy a 3-node cluster
114132
kubectl apply -f k8s/hypercache-cluster.yaml
133+
134+
# Scale to 5 nodes
135+
kubectl scale statefulset hypercache -n hypercache --replicas=5
136+
137+
# Check cluster
138+
kubectl get pods -n hypercache
139+
kubectl exec -n hypercache hypercache-0 -- wget -qO- http://localhost:9080/health
115140
```
116141

117142
### 📊 Access Points
118143
| Service | URL | Notes |
119144
|---------|-----|-------|
120-
| Node 1 HTTP API | http://localhost:9080 | Health, cache, filter, metrics |
121-
| Node 2 HTTP API | http://localhost:9081 | |
122-
| Node 3 HTTP API | http://localhost:9082 | |
123-
| Node 1 RESP | `redis-cli -p 8080` | Redis-compatible |
124-
| Node 2 RESP | `redis-cli -p 8081` | |
125-
| Node 3 RESP | `redis-cli -p 8082` | |
145+
| Node N HTTP API | http://localhost:9079+N | Health, cache, stores, filter, metrics |
146+
| Node N RESP | `redis-cli -p 8079+N` | Redis-compatible |
126147
| Prometheus Metrics | http://localhost:9080/metrics | Per-node metrics |
127148
| Grafana | http://localhost:3000 | admin / admin123 |
128149
| Elasticsearch | http://localhost:9200 | |
129150

151+
Default 3-node cluster: HTTP on 9080/9081/9082, RESP on 8080/8081/8082.
152+
130153
## 🧪 **Testing**
131154

132155
### Unit Tests
@@ -151,18 +174,39 @@ health, metrics, CRUD, cross-node replication, delete replication, value types,
151174

152175
### HTTP API Examples
153176
```bash
154-
# Store a key
177+
# --- Default store (backward-compatible) ---
155178
curl -X PUT http://localhost:9080/api/cache/mykey \
156179
-H "Content-Type: application/json" \
157180
-d '{"value": "hello world"}'
158181

159-
# Retrieve it
160182
curl http://localhost:9080/api/cache/mykey
161-
162-
# Delete it
163183
curl -X DELETE http://localhost:9080/api/cache/mykey
164184

165-
# Check Cuckoo filter stats
185+
# --- Multi-store APIs ---
186+
187+
# List all stores
188+
curl http://localhost:9080/api/stores
189+
190+
# Create a new store (config is immutable after creation)
191+
curl -X POST http://localhost:9080/api/stores \
192+
-H "Content-Type: application/json" \
193+
-d '{"name":"sessions","eviction_policy":"ttl","max_memory":"1GB","default_ttl":"30m","cuckoo_filter":true,"persistence":"aof"}'
194+
195+
# Get store info and stats
196+
curl http://localhost:9080/api/stores/sessions
197+
198+
# Write to a specific store
199+
curl -X PUT http://localhost:9080/api/stores/sessions/cache/user:123 \
200+
-H "Content-Type: application/json" \
201+
-d '{"value": {"token":"abc","role":"admin"}}'
202+
203+
# Read from a specific store
204+
curl http://localhost:9080/api/stores/sessions/cache/user:123
205+
206+
# Drop a store (cannot drop "default")
207+
curl -X DELETE http://localhost:9080/api/stores/sessions
208+
209+
# Cuckoo filter stats
166210
curl http://localhost:9080/api/filter/stats
167211

168212
# Prometheus metrics
@@ -177,24 +221,45 @@ redis-cli -p 8081 GET foo # verify replication
177221
redis-cli -p 8080 DEL foo
178222
redis-cli -p 8080 INFO
179223
redis-cli -p 8080 DBSIZE
224+
225+
# Multi-store commands
226+
redis-cli -p 8080 STORES # list all stores
227+
redis-cli -p 8080 SELECT sessions # switch to "sessions" store
228+
redis-cli -p 8080 SET user:123 token # writes to "sessions" store
229+
redis-cli -p 8080 SELECT default # switch back
180230
```
181231

232+
### Scenario Tests
233+
Real-world pattern tests that run on every push and daily:
234+
```bash
235+
# Run all scenario tests
236+
go test -v -race ./tests/scenarios/...
237+
238+
# Reproduce a randomized test that failed in CI
239+
SCENARIO_SEED=1712438400 go test -run TestRandomizedMixedWorkload ./tests/scenarios/...
240+
```
241+
242+
**Deterministic** (same every run): session overflow, concurrent read/write, TTL expiry, persistence recovery, store lifecycle, hot key thundering herd.
243+
244+
**Randomized** (seed-based, different each run): mixed workload, burst writes, concurrent multi-store.
245+
182246
### Makefile Reference
183247
```
184-
make build Build the binary
185-
make run Run single node (RESP)
186-
make cluster Start 3-node local cluster
187-
make cluster-stop Stop all HyperCache processes
188-
make clean Remove binaries, logs, data
189-
make test-unit Run unit tests with coverage
190-
make test-integration Run integration tests
191-
make bench Run benchmarks
192-
make lint Run golangci-lint
193-
make fmt Format code
194-
make docker-build Build Docker image
195-
make docker-up Start Docker stack
196-
make docker-down Stop Docker stack
197-
make deps Download and tidy dependencies
248+
make build Build the binary
249+
make run Run single node (RESP)
250+
make cluster Start N-node cluster (default 3)
251+
make cluster NODES=5 Start 5-node cluster
252+
make cluster-stop Stop all HyperCache processes
253+
make clean Remove binaries, logs, data
254+
make test-unit Run unit tests with coverage
255+
make test-integration Run integration tests
256+
make bench Run benchmarks
257+
make lint Run golangci-lint
258+
make fmt Format code
259+
make docker-build Build Docker image
260+
make docker-up Start Docker stack
261+
make docker-down Stop Docker stack
262+
make deps Download and tidy dependencies
198263
```
199264

200265
## 🏆 **Key Features**
@@ -204,6 +269,7 @@ make deps Download and tidy dependencies
204269
- Works with any Redis client library
205270
- Drop-in replacement for many Redis use cases
206271
- Standard commands: GET, SET, DEL, EXISTS, PING, INFO, FLUSHALL, DBSIZE
272+
- Multi-store commands: SELECT, STORES
207273

208274
### **Distributed Resilience**
209275
- **Full Replication**: Every node stores every key — maximum availability, any node serves any request
@@ -224,9 +290,19 @@ make deps Download and tidy dependencies
224290
### **Containerized Deployment**
225291
- **Docker Hub Integration**: Pre-built multi-arch images (amd64, arm64)
226292
- **Docker Compose Support**: One-command cluster deployment with monitoring
293+
- **Scalable Clusters**: `make cluster NODES=N` for local, `kubectl scale` for K8s
294+
- **Dynamic Node Addition**: Add nodes to a running cluster — gossip handles join
227295
- **Kubernetes Ready**: StatefulSet manifests with service discovery
228296
- **CI/CD Pipeline**: GitHub Actions for lint, test, build, and publish
229297

298+
### **Multi-Store Architecture**
299+
- **Named Stores**: Create independent stores with different configs (LRU, LFU, TTL, FIFO)
300+
- **Per-Store Isolation**: Independent memory limits, eviction policies, TTL, persistence, cuckoo filters
301+
- **Runtime Management**: Create/drop stores via HTTP API or RESP `SELECT` command
302+
- **Immutable Config**: Store settings are set once at creation — drop and recreate to change
303+
- **Store Registry**: Runtime-created stores survive restarts (`stores.json` persistence)
304+
- **Environment Variable Config**: 8 env vars for Docker/K8s — no YAML needed
305+
230306
### **Advanced Memory Management**
231307
- **Per-Store Eviction Policies**: Independent LRU, LFU, or session-based eviction per store
232308
- **Smart Memory Pool**: Pressure monitoring (warning/critical/panic) with automatic cleanup

0 commit comments

Comments
 (0)