Skip to content

Commit 9a88258

Browse files
andreivcodesandreivcodes
committed
Document Docker workflows and cleanup
Add separate README instructions for manual Docker and Docker Compose flows, and update teardown commands to match the observed behavior of optional profile services. Add the compose and migration artifacts required by the documented workflow, tighten Docker build ignores, and fix a near-indexer Clippy issue hit during pre-commit validation.
1 parent 6f40e33 commit 9a88258

5 files changed

Lines changed: 394 additions & 5 deletions

File tree

‎.dockerignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,9 @@ Thumbs.db
2424
.env
2525
.env.local
2626
.env.*.local
27+
**/.env
28+
**/.env.local
29+
**/.env.*.local
2730

2831
# Documentation that's not needed in container
2932
docs/
@@ -52,4 +55,5 @@ web/.next/
5255
# Docker files themselves
5356
Dockerfile*
5457
docker-compose*.yml
58+
compose.yml
5559
.dockerignore

‎README.md‎

Lines changed: 201 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -74,33 +74,231 @@ This monorepo ships as independently deployable services plus a shared PostgreSQ
7474
| `telegram-indexer` | Telegram ingestion worker | No | `DATABASE_URL`, Telegram credentials, durable session state |
7575
| `migration` | One-off schema migration job | No | Run before schema-dependent rollouts |
7676

77+
### Docker
78+
79+
Use plain Docker when you want to run each container explicitly instead of relying on [`compose.yml`](compose.yml).
80+
This workflow publishes the same host ports as Docker Compose, so stop one stack before starting the other.
81+
82+
Build the base images:
83+
84+
```bash
85+
docker build -f migration/Dockerfile -t hos-api/migration .
86+
docker build -f api/Dockerfile -t hos-api/api .
87+
docker build -f web/Dockerfile -t hos-api/web .
88+
```
89+
90+
Create a network and start PostgreSQL:
91+
92+
```bash
93+
docker network create hos-api-local
94+
docker volume create hos-api-postgres
95+
96+
docker run -d \
97+
--name hos-api-db \
98+
--network hos-api-local \
99+
-p 55432:5432 \
100+
-e POSTGRES_DB=hos_api \
101+
-e POSTGRES_USER=hos \
102+
-e POSTGRES_PASSWORD=hos \
103+
-v hos-api-postgres:/var/lib/postgresql/data \
104+
postgres:16-alpine
105+
```
106+
107+
Wait for PostgreSQL and run migrations:
108+
109+
```bash
110+
until docker exec hos-api-db pg_isready -U hos -d hos_api; do sleep 1; done
111+
112+
docker run --rm \
113+
--name hos-api-migrate \
114+
--network hos-api-local \
115+
-e DATABASE_URL=postgresql://hos:hos@hos-api-db:5432/hos_api \
116+
hos-api/migration up
117+
```
118+
119+
Start the API and web containers:
120+
121+
```bash
122+
docker run -d \
123+
--name hos-api-api \
124+
--network hos-api-local \
125+
-p 3000:3000 \
126+
-e DATABASE_URL=postgresql://hos:hos@hos-api-db:5432/hos_api \
127+
-e APP_LOGGING__LEVEL=info \
128+
-e PORT=3000 \
129+
hos-api/api
130+
131+
docker run -d \
132+
--name hos-api-web \
133+
--network hos-api-local \
134+
-p 3001:3000 \
135+
-e VENEAR_API_BASE_URL=http://hos-api-api:3000/api/v1/venear \
136+
-e NEXT_PUBLIC_VENEAR_API_BASE_URL=http://hos-api-api:3000/api/v1/venear \
137+
hos-api/web
138+
```
139+
140+
Optional indexers:
141+
142+
```bash
143+
docker build -f indexers/discourse-indexer/Dockerfile -t hos-api/discourse-indexer .
144+
docker run -d \
145+
--name hos-api-discourse-indexer \
146+
--network hos-api-local \
147+
-e DATABASE_URL=postgresql://hos:hos@hos-api-db:5432/hos_api \
148+
-e APP_LOGGING__LEVEL=info \
149+
hos-api/discourse-indexer
150+
151+
docker build -f indexers/near-indexer/Dockerfile -t hos-api/near-indexer .
152+
docker run -d \
153+
--name hos-api-near-indexer \
154+
--network hos-api-local \
155+
-e DATABASE_URL=postgresql://hos:hos@hos-api-db:5432/hos_api \
156+
-e APP_LOGGING__LEVEL=info \
157+
-e APP_NEAR__FASTNEAR_NUM_THREADS=2 \
158+
-e APP_NEAR__PROVIDER=fastnear \
159+
-e APP_NEAR__FASTNEAR_API_KEY="${FASTNEAR_API_KEY:-}" \
160+
hos-api/near-indexer
161+
```
162+
163+
Telegram requires credentials and persistent session storage:
164+
165+
```bash
166+
docker build -f indexers/telegram-indexer/Dockerfile -t hos-api/telegram-indexer .
167+
docker volume create hos-api-telegram-session
168+
169+
docker run -d \
170+
--name hos-api-telegram-indexer \
171+
--network hos-api-local \
172+
-e DATABASE_URL=postgresql://hos:hos@hos-api-db:5432/hos_api \
173+
-e TELEGRAM_API_ID=... \
174+
-e TELEGRAM_API_HASH=... \
175+
-e TELEGRAM_CHANNELS=... \
176+
-e TELEGRAM_SESSION_FILE=/var/lib/telegram/telegram_session.bin \
177+
-v hos-api-telegram-session:/var/lib/telegram \
178+
hos-api/telegram-indexer
179+
```
180+
181+
Useful checks:
182+
183+
```bash
184+
docker ps --filter name=hos-api
185+
docker logs -f hos-api-api
186+
curl http://127.0.0.1:3000/health
187+
curl http://127.0.0.1:3001/health
188+
```
189+
190+
Stop and clean up:
191+
192+
```bash
193+
docker rm -f \
194+
hos-api-web \
195+
hos-api-api \
196+
hos-api-discourse-indexer \
197+
hos-api-near-indexer \
198+
hos-api-telegram-indexer \
199+
hos-api-db \
200+
2>/dev/null || true
201+
202+
docker network rm hos-api-local 2>/dev/null || true
203+
204+
# Optional: also remove local persistent state.
205+
docker volume rm hos-api-postgres hos-api-telegram-session 2>/dev/null || true
206+
```
207+
208+
### Docker Compose
209+
210+
The repo now includes [`compose.yml`](compose.yml) for a full local container stack:
211+
212+
- `db` starts PostgreSQL 16 with a named data volume
213+
- `migrate` runs `migration up` before any schema-dependent service starts
214+
- `api` and `web` are enabled by default
215+
- `discourse-indexer`, `near-indexer`, and `telegram-indexer` are opt-in profiles
216+
217+
This workflow publishes the same host ports as the plain Docker commands above, so stop one stack before starting the other.
218+
219+
Build the images:
220+
221+
```bash
222+
docker compose build
223+
```
224+
225+
Start the default stack:
226+
227+
```bash
228+
docker compose up -d
229+
```
230+
231+
Access the services:
232+
233+
- API: `http://127.0.0.1:3000`
234+
- Swagger UI: `http://127.0.0.1:3000/swagger-ui/`
235+
- Web: `http://127.0.0.1:3001`
236+
- PostgreSQL from the host: `postgresql://hos:hos@127.0.0.1:55432/hos_api`
237+
238+
Enable the indexers when you want them:
239+
240+
```bash
241+
docker compose --profile discourse up -d discourse-indexer
242+
docker compose --profile near up -d near-indexer
243+
TELEGRAM_API_ID=... TELEGRAM_API_HASH=... TELEGRAM_CHANNELS=... \
244+
docker compose --profile telegram up -d telegram-indexer
245+
```
246+
247+
Useful checks:
248+
249+
```bash
250+
docker compose ps
251+
docker compose logs -f migrate api web
252+
curl http://127.0.0.1:3000/health
253+
curl http://127.0.0.1:3001/health
254+
```
255+
256+
Stop the stack:
257+
258+
```bash
259+
# Include the optional profiles so any profile containers you started are removed too.
260+
docker compose --profile discourse --profile near --profile telegram down
261+
docker compose --profile discourse --profile near --profile telegram down -v
262+
```
263+
264+
Notes for this compose stack:
265+
266+
- The compose file intentionally uses `COMPOSE_DATABASE_URL` instead of inheriting the root `.env` `DATABASE_URL`. That avoids accidentally pointing containers at a host-local database.
267+
- `web/.env.local` is excluded from Docker build context, so local frontend env files no longer leak into image builds.
268+
- The `near-indexer` profile lowers FastNEAR fetch threads to `2` by default. It still benefits from `FASTNEAR_API_KEY`, or you can switch to `NEAR_PROVIDER=lake`.
269+
- The `telegram-indexer` profile stores session state in the `telegram-session` named volume via `TELEGRAM_SESSION_FILE=/var/lib/telegram/telegram_session.bin`.
270+
77271
### Recommended deployment sequence
78272

79273
1. Provision PostgreSQL and set `DATABASE_URL` for every Rust service.
80-
2. Run migrations before starting the API or any indexer.
274+
2. Run the `migration` container before starting the API or any indexer.
81275
3. Deploy `api` and verify `/health` before attaching downstream services.
82276
4. Deploy `web` with `VENEAR_API_BASE_URL` pointed at the API's reachable URL.
83277
5. Deploy only the indexers you need for your data coverage.
84278
6. Validate logs, health checks, and database connectivity after each rollout.
85279

86280
### Minimum runtime configuration
87281

88-
- Shared Rust services: `DATABASE_URL`, optional `RUST_LOG`, `SERVICE_VERSION`, and `DEPLOYMENT_ENVIRONMENT`
282+
- Shared Rust services: `DATABASE_URL`
283+
- Common optional overrides: `APP_LOGGING__LEVEL`, `APP_LOGGING__FORMAT`, `APP_TELEMETRY__ENVIRONMENT`, and `APP_TELEMETRY__SERVICE_VERSION`
89284
- API: optional `PORT` or `APP_SERVER__PORT`
90285
- Web: `VENEAR_API_BASE_URL` or `NEXT_PUBLIC_VENEAR_API_BASE_URL`
91286
- Telegram indexer: `TELEGRAM_API_ID`, `TELEGRAM_API_HASH`, `TELEGRAM_CHANNELS`, and either `TELEGRAM_SESSION_DATA` or persistent storage for `TELEGRAM_SESSION_FILE`
92-
- NEAR indexer: optional FastNEAR credentials when you use the FastNEAR provider
287+
- NEAR indexer: optional `FASTNEAR_API_KEY`, or set `APP_NEAR__PROVIDER=lake` if you do not want FastNEAR
93288

94289
### Operational notes
95290

96291
- The API `/health` endpoint is useful for liveness, but it still returns HTTP 200 with a `degraded` payload when the database is unavailable. Treat it as a basic health signal, not your only database readiness gate.
97292
- The Telegram indexer defaults to a local `telegram_session.bin` file. In ephemeral container platforms, provide `TELEGRAM_SESSION_DATA` or attach durable storage so redeploys do not force re-authentication.
98293
- The worker services are background jobs, not HTTP apps. They should rely on restart-on-failure policies, logs, and telemetry rather than path-based health checks.
294+
- The NEAR and Discourse workers can hit upstream rate limits during initial backfills. For production rollouts, expect to tune concurrency and provider credentials rather than treating the defaults as infinite-throughput settings.
99295

100296
### Deployment artifacts in this repo
101297

298+
- `compose`: `/compose.yml`
102299
- `api`: `/api/Dockerfile` and `/api/railway.toml`
103300
- `web`: `/web/Dockerfile` and `/web/railway.toml`
301+
- `migration`: `/migration/Dockerfile`
104302
- `discourse-indexer`: `/indexers/discourse-indexer/Dockerfile` and `/indexers/discourse-indexer/railway.toml`
105303
- `near-indexer`: `/indexers/near-indexer/Dockerfile` and `/indexers/near-indexer/railway.toml`
106304
- `telegram-indexer`: `/indexers/telegram-indexer/Dockerfile` and `/indexers/telegram-indexer/railway.toml`

‎compose.yml‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
name: hos-api
2+
3+
x-rust-environment: &rust-environment
4+
DATABASE_URL: ${COMPOSE_DATABASE_URL:-postgresql://hos:hos@db:5432/hos_api}
5+
APP_LOGGING__FORMAT: json
6+
APP_TELEMETRY__ENVIRONMENT: ${APP_TELEMETRY__ENVIRONMENT:-docker-compose}
7+
APP_TELEMETRY__SERVICE_VERSION: ${APP_TELEMETRY__SERVICE_VERSION:-local}
8+
9+
x-rust-service: &rust-service
10+
restart: unless-stopped
11+
environment: *rust-environment
12+
13+
services:
14+
db:
15+
image: postgres:16-alpine
16+
restart: unless-stopped
17+
environment:
18+
POSTGRES_DB: ${POSTGRES_DB:-hos_api}
19+
POSTGRES_USER: ${POSTGRES_USER:-hos}
20+
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-hos}
21+
ports:
22+
- "${POSTGRES_PORT:-55432}:5432"
23+
volumes:
24+
- postgres-data:/var/lib/postgresql/data
25+
healthcheck:
26+
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-hos} -d ${POSTGRES_DB:-hos_api}"]
27+
interval: 5s
28+
timeout: 5s
29+
retries: 12
30+
start_period: 5s
31+
32+
migrate:
33+
build:
34+
context: .
35+
dockerfile: migration/Dockerfile
36+
depends_on:
37+
db:
38+
condition: service_healthy
39+
environment: *rust-environment
40+
restart: "no"
41+
command: ["up"]
42+
43+
api:
44+
<<: *rust-service
45+
build:
46+
context: .
47+
dockerfile: api/Dockerfile
48+
depends_on:
49+
db:
50+
condition: service_healthy
51+
migrate:
52+
condition: service_completed_successfully
53+
environment:
54+
<<: *rust-environment
55+
APP_LOGGING__LEVEL: ${API_LOG_LEVEL:-info}
56+
PORT: 3000
57+
ports:
58+
- "${API_PORT:-3000}:3000"
59+
60+
web:
61+
build:
62+
context: .
63+
dockerfile: web/Dockerfile
64+
depends_on:
65+
api:
66+
condition: service_started
67+
restart: unless-stopped
68+
environment:
69+
NODE_ENV: production
70+
PORT: 3000
71+
VENEAR_API_BASE_URL: ${COMPOSE_VENEAR_API_BASE_URL:-http://api:3000/api/v1/venear}
72+
NEXT_PUBLIC_VENEAR_API_BASE_URL: ${COMPOSE_VENEAR_API_BASE_URL:-http://api:3000/api/v1/venear}
73+
ports:
74+
- "${WEB_PORT:-3001}:3000"
75+
76+
discourse-indexer:
77+
<<: *rust-service
78+
profiles: ["discourse"]
79+
build:
80+
context: .
81+
dockerfile: indexers/discourse-indexer/Dockerfile
82+
depends_on:
83+
db:
84+
condition: service_healthy
85+
migrate:
86+
condition: service_completed_successfully
87+
environment:
88+
<<: *rust-environment
89+
APP_LOGGING__LEVEL: ${DISCOURSE_LOG_LEVEL:-info}
90+
91+
near-indexer:
92+
<<: *rust-service
93+
profiles: ["near"]
94+
build:
95+
context: .
96+
dockerfile: indexers/near-indexer/Dockerfile
97+
depends_on:
98+
db:
99+
condition: service_healthy
100+
migrate:
101+
condition: service_completed_successfully
102+
environment:
103+
<<: *rust-environment
104+
APP_LOGGING__LEVEL: ${NEAR_LOG_LEVEL:-info}
105+
APP_NEAR__CHAIN: ${NEAR_CHAIN:-mainnet}
106+
APP_NEAR__FASTNEAR_API_KEY: ${FASTNEAR_API_KEY:-}
107+
APP_NEAR__FASTNEAR_NUM_THREADS: ${FASTNEAR_NUM_THREADS:-2}
108+
APP_NEAR__PROVIDER: ${NEAR_PROVIDER:-fastnear}
109+
APP_NEAR__START_HEIGHT: ${NEAR_START_HEIGHT:-158320040}
110+
111+
telegram-indexer:
112+
<<: *rust-service
113+
profiles: ["telegram"]
114+
build:
115+
context: .
116+
dockerfile: indexers/telegram-indexer/Dockerfile
117+
depends_on:
118+
db:
119+
condition: service_healthy
120+
migrate:
121+
condition: service_completed_successfully
122+
environment:
123+
<<: *rust-environment
124+
APP_LOGGING__LEVEL: ${TELEGRAM_LOG_LEVEL:-info}
125+
TELEGRAM_API_HASH: ${TELEGRAM_API_HASH:-}
126+
TELEGRAM_API_ID: ${TELEGRAM_API_ID:-}
127+
TELEGRAM_CHANNELS: ${TELEGRAM_CHANNELS:-}
128+
TELEGRAM_SESSION_DATA: ${TELEGRAM_SESSION_DATA:-}
129+
TELEGRAM_SESSION_FILE: /var/lib/telegram/telegram_session.bin
130+
volumes:
131+
- telegram-session:/var/lib/telegram
132+
133+
volumes:
134+
postgres-data:
135+
telegram-session:

‎indexers/near-indexer/src/modules/venear_governance/processor.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -841,8 +841,10 @@ impl HosRuntimeState {
841841
}
842842

843843
fn evict_unreferenced_source_blocks(&mut self) -> RecentCacheEvictionStats {
844-
let mut stats = RecentCacheEvictionStats::default();
845-
stats.retained_block_count = self.recent_block_ref_counts.len();
844+
let mut stats = RecentCacheEvictionStats {
845+
retained_block_count: self.recent_block_ref_counts.len(),
846+
..RecentCacheEvictionStats::default()
847+
};
846848

847849
let stale_heights = self
848850
.recent_blocks

0 commit comments

Comments
 (0)