Skip to content

Commit 375f765

Browse files
kamirclaude
kamir
and
claude
committed
feat: add LFS client SDKs (Java, Python, JS, browser) and demo applications
Multi-language LFS client SDKs and end-to-end demo applications for producing and consuming large files through the KafScale LFS proxy. SDKs: - Java SDK with retry/backoff, configurable timeouts, Maven build - Python SDK with retry/backoff, envelope codec, pip-installable - JavaScript SDK (Node.js) with streaming upload support - Browser SDK (JS) for SPA-based LFS file management Demos: - E60: Medical imaging LFS demo (README/design) - E61: Video streaming LFS demo (README/design) - E62: Industrial IoT LFS demo (README/design) - E70: Java SDK producer demo with video upload - E71: Python SDK demo with video LFS upload - E72: Browser-native LFS SDK single-page app Infrastructure: - Docker Compose stack for local LFS development - Synology S3-compatible storage deployment - Browser demo UI (HTML/CSS/JS) - LFS task tracking documents > This is PR 2 of 2. Depends on PR KafScale#128 (core LFS proxy). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 3200deb commit 375f765

97 files changed

Lines changed: 18784 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎deploy/docker-compose/Makefile‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Copyright 2026 Alexander Alten (novatechflow), NovaTechflow (novatechflow.com).
2+
# This project is supported and financed by Scalytics, Inc. (www.scalytics.io).
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
.PHONY: up down logs ps restart health test-upload test-download clean help
17+
18+
REGISTRY ?= 192.168.0.131:5100
19+
TAG ?= dev
20+
21+
up: ## Start all services
22+
REGISTRY=$(REGISTRY) TAG=$(TAG) docker-compose up -d
23+
24+
down: ## Stop all services
25+
docker-compose down
26+
27+
logs: ## View logs (follow mode)
28+
docker-compose logs -f
29+
30+
ps: ## Show service status
31+
docker-compose ps
32+
33+
restart: ## Restart all services
34+
docker-compose restart
35+
36+
health: ## Check service health
37+
@echo "=== Health Checks ==="
38+
@echo -n "etcd: "; curl -s http://localhost:2379/health | head -c 50 || echo "FAIL"
39+
@echo -n "minio: "; curl -s http://localhost:9000/minio/health/live || echo "FAIL"
40+
@echo -n "lfs-proxy: "; curl -s http://localhost:9094/readyz || echo "FAIL"
41+
@echo -n "broker: "; nc -z localhost 9092 && echo "OK" || echo "FAIL"
42+
43+
test-upload: ## Test LFS upload (creates 1KB test file)
44+
@echo "Uploading test file..."
45+
@dd if=/dev/urandom bs=1024 count=1 2>/dev/null | \
46+
curl -s -X POST http://localhost:8080/lfs/produce \
47+
-H "X-Kafka-Topic: test-uploads" \
48+
-H "Content-Type: application/octet-stream" \
49+
--data-binary @- | jq .
50+
51+
test-download: ## Test LFS download (requires KEY variable)
52+
@if [ -z "$(KEY)" ]; then echo "Usage: make test-download KEY=default/topic/lfs/..."; exit 1; fi
53+
@curl -s -X POST http://localhost:8080/lfs/download \
54+
-H "Content-Type: application/json" \
55+
-d '{"bucket":"kafscale","key":"$(KEY)","mode":"presign"}' | jq .
56+
57+
clean: ## Stop services and remove volumes
58+
docker-compose down -v
59+
60+
registry-check: ## Check local registry
61+
@echo "=== Registry Catalog ==="
62+
@curl -s http://$(REGISTRY)/v2/_catalog | jq .
63+
64+
help: ## Show this help
65+
@grep -E '^[a-zA-Z_-]+:.*?##' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "%-15s %s\n", $$1, $$2}'

‎deploy/docker-compose/README.md‎

Lines changed: 313 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
1+
<!--
2+
Copyright 2026 Alexander Alten (novatechflow), NovaTechflow (novatechflow.com).
3+
This project is supported and financed by Scalytics, Inc. (www.scalytics.io).
4+
5+
Licensed under the Apache License, Version 2.0 (the "License");
6+
you may not use this file except in compliance with the License.
7+
You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
-->
17+
18+
# KafScale Docker Compose
19+
20+
Local development platform using Docker Compose with images from a local registry.
21+
22+
## Prerequisites
23+
24+
1. Docker and Docker Compose installed
25+
2. Images pushed to local registry (`192.168.0.131:5100`)
26+
3. Docker configured for insecure registry (see below)
27+
28+
### Configure Insecure Registry
29+
30+
Docker Desktop → Settings → Docker Engine:
31+
32+
```json
33+
{
34+
"insecure-registries": ["192.168.0.131:5100"]
35+
}
36+
```
37+
38+
### Push Images to Local Registry
39+
40+
```bash
41+
# From repository root
42+
make stage-release STAGE_REGISTRY=192.168.0.131:5100 STAGE_TAG=dev
43+
```
44+
45+
### Verify Images
46+
47+
```bash
48+
curl http://192.168.0.131:5100/v2/_catalog
49+
```
50+
51+
## Quick Start
52+
53+
```bash
54+
cd deploy/docker-compose
55+
56+
# Start all services
57+
docker-compose up -d
58+
59+
# View logs
60+
docker-compose logs -f
61+
62+
# Stop all services
63+
docker-compose down
64+
```
65+
66+
## Services
67+
68+
| Service | Port | Description |
69+
|---------|------|-------------|
70+
| **etcd** | 2379 | Coordination store |
71+
| **minio** | 9000, 9001 | S3 storage (API, Console) |
72+
| **broker** | 9092 | KafScale Kafka broker |
73+
| **lfs-proxy** | 8080, 9093 | LFS HTTP API, Kafka protocol |
74+
| **console** | 3080 | Web management console |
75+
| **e72-browser-demo** | 3072 | Browser LFS demo (optional) |
76+
77+
## Access Points
78+
79+
| Service | URL |
80+
|---------|-----|
81+
| LFS HTTP API | http://localhost:8080 |
82+
| MinIO Console | http://localhost:9001 |
83+
| KafScale Console | http://localhost:3080 |
84+
| E72 Browser Demo | http://localhost:3072 |
85+
| Prometheus Metrics | http://localhost:9095/metrics |
86+
| Health Check | http://localhost:9094/readyz |
87+
88+
## Testing
89+
90+
### Broker Advertised Address
91+
92+
The broker must advertise its container hostname so other services can connect.
93+
Docker Compose sets:
94+
95+
- `KAFSCALE_BROKER_HOST=broker`
96+
- `KAFSCALE_BROKER_PORT=9092`
97+
98+
### Health Check
99+
100+
```bash
101+
curl http://localhost:9094/readyz
102+
```
103+
104+
### LFS Upload
105+
106+
```bash
107+
# Upload a file
108+
curl -X POST http://localhost:8080/lfs/produce \
109+
-H "X-Kafka-Topic: test-uploads" \
110+
-H "Content-Type: application/octet-stream" \
111+
--data-binary @myfile.bin
112+
113+
# Upload with key
114+
curl -X POST http://localhost:8080/lfs/produce \
115+
-H "X-Kafka-Topic: test-uploads" \
116+
-H "X-Kafka-Key: $(echo -n 'my-key' | base64)" \
117+
-H "Content-Type: video/mp4" \
118+
--data-binary @video.mp4
119+
```
120+
121+
### Large Uploads (Beast Mode)
122+
123+
Docker Compose ships with a large-upload profile:
124+
125+
- `KAFSCALE_LFS_PROXY_MAX_BLOB_SIZE=7516192768` (7 GB)
126+
- `KAFSCALE_LFS_PROXY_CHUNK_SIZE=16777216` (16 MB parts)
127+
- `KAFSCALE_LFS_PROXY_HTTP_READ_TIMEOUT_SEC=1800`
128+
- `KAFSCALE_LFS_PROXY_HTTP_WRITE_TIMEOUT_SEC=1800`
129+
- `KAFSCALE_LFS_PROXY_HTTP_IDLE_TIMEOUT_SEC=120`
130+
131+
These settings allow 6+ GB streaming uploads without hitting default limits.
132+
133+
### LFS Download
134+
135+
```bash
136+
# Get presigned URL
137+
curl -X POST http://localhost:8080/lfs/download \
138+
-H "Content-Type: application/json" \
139+
-d '{"bucket":"kafscale","key":"default/test-uploads/lfs/...","mode":"presign"}'
140+
```
141+
142+
## Traceability
143+
144+
Traceability is enabled in the compose file by default. It consists of:
145+
- **LFS Ops Tracker** events emitted by the LFS proxy to `__lfs_ops_state`
146+
- **Console LFS Dashboard** consuming those events and exposing APIs/UI
147+
148+
### Where to see it
149+
150+
1) **Console UI**
151+
- Open http://localhost:3080
152+
- Navigate to the **LFS** tab for objects, topics, live events, and S3 browser.
153+
154+
2) **Raw events from Kafka**
155+
```bash
156+
kcat -b localhost:9092 -C -t __lfs_ops_state -o beginning
157+
```
158+
159+
### Key settings (compose)
160+
161+
LFS proxy tracker:
162+
- `KAFSCALE_LFS_TRACKER_ENABLED=true`
163+
- `KAFSCALE_LFS_TRACKER_TOPIC=__lfs_ops_state`
164+
- `KAFSCALE_LFS_TRACKER_BATCH_SIZE=100`
165+
- `KAFSCALE_LFS_TRACKER_FLUSH_MS=100`
166+
- `KAFSCALE_LFS_TRACKER_ENSURE_TOPIC=true`
167+
- `KAFSCALE_LFS_TRACKER_PARTITIONS=3`
168+
- `KAFSCALE_LFS_TRACKER_REPLICATION_FACTOR=1`
169+
170+
Console LFS dashboard:
171+
- `KAFSCALE_CONSOLE_LFS_ENABLED=true`
172+
- `KAFSCALE_CONSOLE_KAFKA_BROKERS=broker:9092`
173+
- `KAFSCALE_CONSOLE_LFS_S3_*` set to MinIO credentials
174+
175+
### Kafka (via kcat)
176+
177+
```bash
178+
# List topics
179+
kcat -b localhost:9092 -L
180+
181+
# Produce message (goes through regular broker, not LFS)
182+
echo "hello" | kcat -b localhost:9092 -P -t test-topic
183+
184+
# Consume messages
185+
kcat -b localhost:9092 -C -t test-topic -o beginning
186+
```
187+
188+
## Configuration
189+
190+
### Environment Variables
191+
192+
Edit `.env` to customize:
193+
194+
```bash
195+
# Registry settings
196+
REGISTRY=192.168.0.131:5100
197+
TAG=dev
198+
199+
# MinIO credentials
200+
MINIO_ROOT_USER=minioadmin
201+
MINIO_ROOT_PASSWORD=minioadmin
202+
```
203+
204+
### Console Port Configuration
205+
206+
The console listens on `KAFSCALE_CONSOLE_HTTP_ADDR` (default `:8080`). In the compose file
207+
we set it to `:3080` and map `3080:3080`.
208+
209+
### Console Login
210+
211+
The console UI requires credentials. Compose sets:
212+
- `KAFSCALE_UI_USERNAME=kafscaleadmin`
213+
- `KAFSCALE_UI_PASSWORD=kafscale`
214+
215+
Override these in `docker-compose.yaml` or via your own `.env.local` if needed.
216+
217+
### Override Registry/Tag
218+
219+
```bash
220+
REGISTRY=my-registry.local:5000 TAG=v1.5.0 docker-compose up -d
221+
```
222+
223+
## Architecture
224+
225+
```
226+
┌─────────────────────────────────────────────────────────────────┐
227+
│ Docker Compose Network │
228+
│ │
229+
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
230+
│ │ etcd │ │ minio │ │ broker │ │ console │ │
231+
│ │ :2379 │ │ :9000/01 │ │ :9092 │ │ :3080 │ │
232+
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └──────────┘ │
233+
│ │ │ │ │
234+
│ └───────────────┼───────────────┘ │
235+
│ │ │
236+
│ ┌───────┴───────┐ │
237+
│ │ lfs-proxy │ │
238+
│ │ :8080 (HTTP) │ │
239+
│ │ :9093 (Kafka) │ │
240+
│ └───────────────┘ │
241+
└─────────────────────────────────────────────────────────────────┘
242+
│
243+
┌─────────┴─────────┐
244+
│ Host Machine │
245+
│ │
246+
│ localhost:8080 │ ← LFS HTTP API
247+
│ localhost:9092 │ ← Kafka Broker
248+
│ localhost:9001 │ ← MinIO Console
249+
│ localhost:3080 │ ← KafScale Console
250+
└───────────────────┘
251+
```
252+
253+
## Troubleshooting
254+
255+
### Services not starting
256+
257+
```bash
258+
# Check service status
259+
docker-compose ps
260+
261+
# View logs for specific service
262+
docker-compose logs lfs-proxy
263+
264+
# Restart a service
265+
docker-compose restart lfs-proxy
266+
```
267+
268+
### Image pull fails
269+
270+
```bash
271+
# Verify registry is accessible
272+
curl http://192.168.0.131:5100/v2/_catalog
273+
274+
# Check Docker daemon config
275+
docker info | grep -A5 "Insecure Registries"
276+
```
277+
278+
### LFS upload fails
279+
280+
```bash
281+
# Check LFS proxy logs
282+
docker-compose logs lfs-proxy
283+
284+
# Verify MinIO is healthy
285+
curl http://localhost:9000/minio/health/live
286+
287+
# Check bucket exists
288+
docker-compose exec minio mc ls local/
289+
```
290+
291+
### Reset everything
292+
293+
```bash
294+
# Stop and remove all containers, volumes
295+
docker-compose down -v
296+
297+
# Start fresh
298+
docker-compose up -d
299+
```
300+
301+
## Volumes
302+
303+
| Volume | Purpose |
304+
|--------|---------|
305+
| `etcd-data` | etcd persistent storage |
306+
| `minio-data` | MinIO object storage |
307+
| `broker-data` | Kafka broker data |
308+
309+
To persist data across restarts, volumes are used. To reset:
310+
311+
```bash
312+
docker-compose down -v
313+
```

0 commit comments

Comments
 (0)