search: logsim search daemon + IR API + TUI + --to local forwarding

[hyfather]

Summary

logsim search starts an in-process HTTP daemon on :3700 that ingests Splunk/Cribl HEC and exposes a small set of information-retrieval functions over HTTP, plus a bubbletea TUI for spot-checking what's been ingested.

• Daemon: chi router on 127.0.0.1:3700, CORS open for the future frontend, ctrl-c shuts down + deletes every db.
• DBs: 6-char [a-z0-9] codes (auto-generated or caller-supplied), backed by in-memory DuckDB. Each db is one DuckDB connection with a fixed schema modelled on Splunk HEC: (time, host, source, sourcetype, index, raw, fields JSON).
• Ingest: POST /dbs/{code}/services/collector/event (NDJSON HEC) and /raw (line-by-line). HEC ingest auto-creates the target db.
• IR API (room to grow): get_raw, get_summary (count/sum/avg/min/max/distinct_count, optional group_by), get_distribution (time-bucketed counts), get_top_values. The function set is intentionally narrow — every operation is expressible in Splunk SPL / Cribl Search / Datadog / Quickwit, so future backends can implement them natively.
• TUI: live db list (code · events · oldest · newest · age), Enter to spot-check the first 100 raw lines of any db.
• Forwarding: logsim run <scenario> --to local derives a db code from the scenario slug (e.g. web-service → webser) and forwards via the existing CriblSink. --to local:<code> pins to a specific code; --to local:my-cache-test slugifies.

$ logsim search --no-tui &
$ logsim run web-service --ticks 5 --to local --quiet --force
logsim: sent 183 events to local:webser in 2 batches
$ curl -s http://127.0.0.1:3700/dbs | jq '.dbs[0].event_count'
183
$ curl -s -X POST http://127.0.0.1:3700/dbs/webser/get_summary \
       -H 'content-type: application/json' \
       -d '{"agg_fn":"count","group_by":"sourcetype"}'
{"rows":[{"group":"nodejs","value":56},{"group":"mysql:query","value":56},{"group":"nginx:access","value":56},{"group":"aws:vpcflow","value":15}]}

Architecture

• pkg/search/backend.go — Backend interface (Ingest, GetRaw, GetSummary, GetDistribution, GetTopValues, Stats, Close) + canonical Event type. Swappable seam for Splunk/Cribl/Datadog/Quickwit later.
• pkg/search/duckdb.go — DuckDBBackend (CGO; //go:build cgo).
• pkg/search/duckdb_nocgo.go — stub returning ErrNoCGO so the rest of the module still builds with CGO_ENABLED=0.
• pkg/search/registry.go — process-wide map of code → backend, lifecycle, GetOrCreate for HEC auto-create.
• pkg/search/hec.go + raw_lines.go — HEC envelope and raw-line parsers.
• pkg/search/server.go — chi handlers.
• pkg/search/tui.go — bubbletea program.
• cmd/logsim/search.go — wires it all together.

CGO / Vercel

The duckdb-go binding requires CGO. To keep the Vercel functions in api/ unaffected:

• pkg/search/duckdb.go is gated //go:build cgo; a !cgo stub satisfies the Backend interface with ErrNoCGO.
• api/ does not import pkg/search, so CGO_ENABLED=0 go build ./api/... builds and runs as before.
• CGO_ENABLED=0 go test ./... is also clean (the duckdb-specific test file is gated too).

Follow-up: release.yml currently builds with CGO_ENABLED=0, which means binaries from GitHub Releases will return ErrNoCGO if the user runs logsim search. Enabling CGO in the release matrix needs cross-compile toolchains (e.g. gcc-aarch64-linux-gnu for linux/arm64, native macos runners for darwin) — punted to a separate PR so this one stays focused. go install from source works today.

Test plan

• CGO_ENABLED=1 go test ./... — passes
• CGO_ENABLED=0 go test ./... — passes (duckdb tests gated)
• CGO_ENABLED=0 go build ./api/... — passes (Vercel-safe)
• End-to-end smoke: logsim search --no-tui + logsim run web-service --to local + curl /dbs/webser/get_summary round-trips
• HEC parser handles NDJSON, object events, missing time, mixed empty lines, malformed lines
• Registry race test confirms GetOrCreate is single-flight

Out of scope (called out for the next iteration)

• Auth/CSRF on :3700 (deferred per request)
• Frontend wiring against the IR API
• More IR functions (substring search, get_correlated, get_patterns)
• Persistent dbs / on-disk DuckDB
• Splunk HEC ack endpoints (?channel=..., /ack)
• release.yml CGO build (so release binaries support logsim search)

---

Generated by Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
logsim2	Ready	Preview, Comment	May 6, 2026 10:46pm