@@ -74,14 +74,24 @@ open http://localhost:3000 # admin / admin123
7474- Go 1.23.2+
7575- ` redis-cli ` (optional, for RESP testing)
7676
77- ### Local Cluster (3 nodes)
77+ ### Local Cluster (N nodes)
7878``` bash
79- # Build and start a fresh 3-node cluster
79+ # Start a 3-node cluster (default)
8080make cluster
8181
82+ # Start a 5-node cluster
83+ make cluster NODES=5
84+
85+ # Start a 10-node cluster
86+ make cluster NODES=10
87+
8288# Check cluster health
8389curl -s http://localhost:9080/health | python3 -m json.tool
8490
91+ # Add a node to a running cluster (auto-discovers ports and seeds)
92+ ./scripts/add-node.sh
93+ ./scripts/add-node.sh --node-name=node-6
94+
8595# Stop the cluster
8696make cluster-stop
8797
@@ -102,6 +112,13 @@ docker pull rishabhverma17/hypercache:latest
102112# Start full stack (3-node cluster + Elasticsearch + Grafana + Filebeat)
103113docker compose -f docker-compose.cluster.yml up -d
104114
115+ # Add a 4th node to a running Docker cluster (auto-joins via gossip)
116+ docker run -d --name hypercache-node4 \
117+ --network cache_hypercache-cluster \
118+ -p 8083:8080 -p 9083:9080 \
119+ rishabhverma17/hypercache:latest \
120+ --protocol resp --node-id node-4
121+
105122# Or build locally and start
106123make docker-build && make docker-up
107124
@@ -111,22 +128,28 @@ docker compose -f docker-compose.cluster.yml down
111128
112129### Kubernetes
113130``` bash
131+ # Deploy a 3-node cluster
114132kubectl apply -f k8s/hypercache-cluster.yaml
133+
134+ # Scale to 5 nodes
135+ kubectl scale statefulset hypercache -n hypercache --replicas=5
136+
137+ # Check cluster
138+ kubectl get pods -n hypercache
139+ kubectl exec -n hypercache hypercache-0 -- wget -qO- http://localhost:9080/health
115140```
116141
117142### 📊 Access Points
118143| Service | URL | Notes |
119144| ---------| -----| -------|
120- | Node 1 HTTP API | http://localhost:9080 | Health, cache, filter, metrics |
121- | Node 2 HTTP API | http://localhost:9081 | |
122- | Node 3 HTTP API | http://localhost:9082 | |
123- | Node 1 RESP | ` redis-cli -p 8080 ` | Redis-compatible |
124- | Node 2 RESP | ` redis-cli -p 8081 ` | |
125- | Node 3 RESP | ` redis-cli -p 8082 ` | |
145+ | Node N HTTP API | http://localhost:9079+N | Health, cache, stores, filter, metrics |
146+ | Node N RESP | ` redis-cli -p 8079+N ` | Redis-compatible |
126147| Prometheus Metrics | http://localhost:9080/metrics | Per-node metrics |
127148| Grafana | http://localhost:3000 | admin / admin123 |
128149| Elasticsearch | http://localhost:9200 | |
129150
151+ Default 3-node cluster: HTTP on 9080/9081/9082, RESP on 8080/8081/8082.
152+
130153## 🧪 ** Testing**
131154
132155### Unit Tests
@@ -151,18 +174,39 @@ health, metrics, CRUD, cross-node replication, delete replication, value types,
151174
152175### HTTP API Examples
153176``` bash
154- # Store a key
177+ # --- Default store (backward-compatible) ---
155178curl -X PUT http://localhost:9080/api/cache/mykey \
156179 -H " Content-Type: application/json" \
157180 -d ' {"value": "hello world"}'
158181
159- # Retrieve it
160182curl http://localhost:9080/api/cache/mykey
161-
162- # Delete it
163183curl -X DELETE http://localhost:9080/api/cache/mykey
164184
165- # Check Cuckoo filter stats
185+ # --- Multi-store APIs ---
186+
187+ # List all stores
188+ curl http://localhost:9080/api/stores
189+
190+ # Create a new store (config is immutable after creation)
191+ curl -X POST http://localhost:9080/api/stores \
192+ -H " Content-Type: application/json" \
193+ -d ' {"name":"sessions","eviction_policy":"ttl","max_memory":"1GB","default_ttl":"30m","cuckoo_filter":true,"persistence":"aof"}'
194+
195+ # Get store info and stats
196+ curl http://localhost:9080/api/stores/sessions
197+
198+ # Write to a specific store
199+ curl -X PUT http://localhost:9080/api/stores/sessions/cache/user:123 \
200+ -H " Content-Type: application/json" \
201+ -d ' {"value": {"token":"abc","role":"admin"}}'
202+
203+ # Read from a specific store
204+ curl http://localhost:9080/api/stores/sessions/cache/user:123
205+
206+ # Drop a store (cannot drop "default")
207+ curl -X DELETE http://localhost:9080/api/stores/sessions
208+
209+ # Cuckoo filter stats
166210curl http://localhost:9080/api/filter/stats
167211
168212# Prometheus metrics
@@ -177,24 +221,45 @@ redis-cli -p 8081 GET foo # verify replication
177221redis-cli -p 8080 DEL foo
178222redis-cli -p 8080 INFO
179223redis-cli -p 8080 DBSIZE
224+
225+ # Multi-store commands
226+ redis-cli -p 8080 STORES # list all stores
227+ redis-cli -p 8080 SELECT sessions # switch to "sessions" store
228+ redis-cli -p 8080 SET user:123 token # writes to "sessions" store
229+ redis-cli -p 8080 SELECT default # switch back
180230```
181231
232+ ### Scenario Tests
233+ Real-world pattern tests that run on every push and daily:
234+ ``` bash
235+ # Run all scenario tests
236+ go test -v -race ./tests/scenarios/...
237+
238+ # Reproduce a randomized test that failed in CI
239+ SCENARIO_SEED=1712438400 go test -run TestRandomizedMixedWorkload ./tests/scenarios/...
240+ ```
241+
242+ ** Deterministic** (same every run): session overflow, concurrent read/write, TTL expiry, persistence recovery, store lifecycle, hot key thundering herd.
243+
244+ ** Randomized** (seed-based, different each run): mixed workload, burst writes, concurrent multi-store.
245+
182246### Makefile Reference
183247```
184- make build Build the binary
185- make run Run single node (RESP)
186- make cluster Start 3-node local cluster
187- make cluster-stop Stop all HyperCache processes
188- make clean Remove binaries, logs, data
189- make test-unit Run unit tests with coverage
190- make test-integration Run integration tests
191- make bench Run benchmarks
192- make lint Run golangci-lint
193- make fmt Format code
194- make docker-build Build Docker image
195- make docker-up Start Docker stack
196- make docker-down Stop Docker stack
197- make deps Download and tidy dependencies
248+ make build Build the binary
249+ make run Run single node (RESP)
250+ make cluster Start N-node cluster (default 3)
251+ make cluster NODES=5 Start 5-node cluster
252+ make cluster-stop Stop all HyperCache processes
253+ make clean Remove binaries, logs, data
254+ make test-unit Run unit tests with coverage
255+ make test-integration Run integration tests
256+ make bench Run benchmarks
257+ make lint Run golangci-lint
258+ make fmt Format code
259+ make docker-build Build Docker image
260+ make docker-up Start Docker stack
261+ make docker-down Stop Docker stack
262+ make deps Download and tidy dependencies
198263```
199264
200265## 🏆 ** Key Features**
@@ -204,6 +269,7 @@ make deps Download and tidy dependencies
204269- Works with any Redis client library
205270- Drop-in replacement for many Redis use cases
206271- Standard commands: GET, SET, DEL, EXISTS, PING, INFO, FLUSHALL, DBSIZE
272+ - Multi-store commands: SELECT, STORES
207273
208274### ** Distributed Resilience**
209275- ** Full Replication** : Every node stores every key — maximum availability, any node serves any request
@@ -224,9 +290,19 @@ make deps Download and tidy dependencies
224290### ** Containerized Deployment**
225291- ** Docker Hub Integration** : Pre-built multi-arch images (amd64, arm64)
226292- ** Docker Compose Support** : One-command cluster deployment with monitoring
293+ - ** Scalable Clusters** : ` make cluster NODES=N ` for local, ` kubectl scale ` for K8s
294+ - ** Dynamic Node Addition** : Add nodes to a running cluster — gossip handles join
227295- ** Kubernetes Ready** : StatefulSet manifests with service discovery
228296- ** CI/CD Pipeline** : GitHub Actions for lint, test, build, and publish
229297
298+ ### ** Multi-Store Architecture**
299+ - ** Named Stores** : Create independent stores with different configs (LRU, LFU, TTL, FIFO)
300+ - ** Per-Store Isolation** : Independent memory limits, eviction policies, TTL, persistence, cuckoo filters
301+ - ** Runtime Management** : Create/drop stores via HTTP API or RESP ` SELECT ` command
302+ - ** Immutable Config** : Store settings are set once at creation — drop and recreate to change
303+ - ** Store Registry** : Runtime-created stores survive restarts (` stores.json ` persistence)
304+ - ** Environment Variable Config** : 8 env vars for Docker/K8s — no YAML needed
305+
230306### ** Advanced Memory Management**
231307- ** Per-Store Eviction Policies** : Independent LRU, LFU, or session-based eviction per store
232308- ** Smart Memory Pool** : Pressure monitoring (warning/critical/panic) with automatic cleanup
0 commit comments