gnata-sqlite

Author: rbbydotdev

.gitignore

@@ -1,3 +1,10 @@
 *.wasm
+*.dylib
+*.so
+gnata_jsonata.h
+.agents
+skills-lock.json
 .secret
 .DS_Store
+node_modules/
+editor/codemirror/dist/

AGENTS.md

@@ -2,50 +2,123 @@
 
 This file provides guidance to AI agents (Claude Code, GitHub Copilot, Cursor, etc.) when working with code in this repository.
 
-## What Is Gnata
-
-Gnata is a full JSONata 2.x implementation in Go, built for production streaming workloads. JSONata is a query and transformation language for JSON data. Gnata provides two-tier evaluation: fast-path (GJSON zero-copy) for simple expressions and full AST evaluation for complex ones, plus a lock-free `StreamEvaluator` for high-throughput batched evaluation.
+## Project Overview
+
+gnata-sqlite is a fork of [RecoLabs/gnata](https://github.com/RecoLabs/gnata) — a full JSONata 2.x implementation in Go. This fork extends it with:
+
+- **SQLite C extension** (`sqlite/`) — registers `jsonata()`, `jsonata_query()`, and `jsonata_each` as SQL functions/virtual tables via CGo
+- **Query planner** (`internal/planner/`) — decomposes JSONata expressions for streaming SQL aggregation
+- **Editor/LSP** (`editor/`) — language server and WASM entry point for browser-based editing
+- **CodeMirror plugin** (`editor/codemirror/`) — TypeScript CodeMirror 6 language support
+
+Module path: `github.com/rbbydotdev/gnata-sqlite`
+
+## Package Map
+
+| Package | Purpose |
+|---------|---------|
+| Root (`gnata`) | Core JSONata 2.x engine — lexer, parser, evaluator, streaming. Entry points: `gnata.go`, `stream.go` |
+| `functions/` | 55+ built-in JSONata functions (string, array, numeric, datetime, etc.). Registered via `functions.RegisterAll` |
+| `internal/evaluator/` | AST evaluation dispatch — binary ops, functions, chains, transforms. One file per eval category (`eval_binary.go`, `eval_function.go`, `eval_chain.go`, etc.) |
+| `internal/parser/` | Pratt parser, AST nodes, fast-path analysis (`parser.AnalyzeFastPath`) |
+| `internal/lexer/` | Tokenizer for JSONata expression strings |
+| `internal/planner/` | Query planner — decomposes JSONata for streaming SQL aggregation. Extracts paths, predicates, accumulators |
+| `sqlite/` | SQLite C extension via CGo — registers `jsonata()`, `jsonata_query()`, `jsonata_each` virtual table. Routes aggregates through planner |
+| `sqlite/tinygo/` | TinyGo-compatible eval subset for WASM builds |
+| `sqlite/benchmarks/` | SQL benchmark files for SQLite extension performance testing |
+| `editor/` | LSP server (native) + TinyGo WASM entry point — completions, hover, diagnostics via JSON-RPC 2.0 over stdin/stdout |
+| `editor/codemirror/` | npm package — CodeMirror 6 language support (TypeScript) |
+| `wasm/` | WASM entry point for browser playground. Exports `gnataEval`, `gnataCompile`, `gnataEvalHandle` |
 
 ## Development Commands
 
 ```sh
-# Lint (from package root)
+# Run all tests (includes CGo sqlite tests)
+go test ./...
+
+# Run tests excluding sqlite (no CGo needed)
+go test $(go list ./... | grep -v sqlite)
+
+# Lint
 golangci-lint run
 
-# Run all tests (1,273 JSON test cases from official jsonata-js suite + unit tests)
-go test ./...
+# Build sqlite extension (macOS)
+go build -buildmode=c-shared -o gnata_jsonata.dylib ./sqlite
+
+# Build sqlite extension (Linux)
+go build -buildmode=c-shared -o gnata_jsonata.so ./sqlite
 
-# Run a single test
-go test -run TestName
+# Build WASM LSP
+tinygo build -o gnata-lsp.wasm -target wasm ./editor
+
+# Build WASM playground
+GOOS=js GOARCH=wasm go build -ldflags="-s -w" -trimpath -o gnata.wasm ./wasm
+
+# Build CodeMirror package
+cd editor/codemirror && npm install && npm run build
+
+# Build native LSP server
+go build -o gnata-lsp ./editor/
 
 # Run benchmarks
 go test -bench=. -benchmem
 ```
 
+## CI
+
+GitHub Actions (`.github/workflows/ci.yml`) runs on push/PR to `main`:
+- `go test -race -count=1 ./...`
+- `golangci-lint` via `golangci/golangci-lint-action@v9`
+
 ## Architecture
 
 ### Compilation Pipeline
 
-Lexer → Parser → AST Processing → Fast-Path Analysis → Expression
+Lexer -> Parser -> AST Processing -> Fast-Path Analysis -> Expression
 
 1. **Lexer** (`internal/lexer/`) — Tokenizes JSONata expression strings
 2. **Parser** (`internal/parser/`) — Pratt (top-down operator precedence) parser producing AST nodes
 3. **AST Processing** (`parser.ProcessAST`) — Normalizes and optimizes the AST
 4. **Fast-Path Analysis** (`parser.AnalyzeFastPath`) — Classifies expressions into:
    - Pure-path fast path (e.g., `Account.Name`) — uses GJSON zero-copy
    - Comparison fast path (e.g., `a.b = "x"`) — zero allocations
+   - Function fast path (e.g., `$exists(a.b)`) — direct GJSON evaluation
    - Full AST evaluation required
 
 ### Two-Tier Evaluation
 
 - `Eval(ctx, any)` — Evaluate against pre-parsed Go values via full AST walk
 - `EvalBytes(ctx, json.RawMessage)` — Fast-path expressions use GJSON directly on raw JSON bytes; full-path falls back to unmarshal + Eval
 
-### StreamEvaluator (stream.go)
+### StreamEvaluator (`stream.go`)
 
 Batch-evaluates multiple expressions against events. Schema-keyed `GroupPlan` caching deduplicates field extraction across expressions. Lock-free reads via `atomic.Pointer` snapshot; writes serialized by `sync.Mutex`. Single JSON scan per event via `gjson.GetManyBytes`.
 
-### Evaluator Dispatch (internal/evaluator/)
+### Query Planner (`internal/planner/`)
+
+Decomposes JSONata expressions into `QueryPlan` for SQL streaming aggregation. Each plan contains:
+- `Accumulators` — fed per row via `StepBatch` (single GJSON scan per row)
+- `FinalExpr` — evaluated once at finalization
+- `Predicates` — deduplicated, evaluated once per row; accumulators reference by index
+- Used by `jsonata_query()` SQL aggregate function in the SQLite extension
+
+### SQLite Bridge (`sqlite/`)
+
+CGo extension that registers SQL functions with SQLite:
+- `jsonata(expr, json [, bindings])` — scalar function, evaluates JSONata expression against JSON
+- `jsonata_query(expr, json)` — aggregate function, routes through query planner for streaming aggregation
+- `jsonata_each` — virtual table for iterating JSONata results as rows
+- Entry point: `extension.go` with C bridge in `bridge.c` / `bridge.h`
+
+### Editor/LSP (`editor/`)
+
+Shared Go code compiled to either:
+- Native LSP server (`main_lsp.go`, build tag `!js`) — JSON-RPC 2.0 over stdin/stdout
+- TinyGo WASM (`main_wasm.go`, build tag `js`) — for browser integration
+
+Supports: `textDocument/didOpen`, `textDocument/didChange` (diagnostics), `textDocument/completion` (schema-aware)
+
+### Evaluator Dispatch (`internal/evaluator/`)
 
 `evaluator.Eval(node, input, env)` dispatches by `node.Type`. Each eval category is in its own file:
 - `eval_binary.go` — Binary operators, subscripts, filtering
@@ -58,32 +131,38 @@ Batch-evaluates multiple expressions against events. Schema-keyed `GroupPlan` ca
 - `eval_regex.go` — Regex compilation and matching
 - `eval_unary.go` — Unary operators (negation, array constructor)
 
-### Standard Library (functions/)
+### Fast-Path Byte Evaluation (`func_fast.go`)
 
-55+ built-in JSONata functions across categorized files: `string_funcs.go`, `string_match_replace.go`, `string_format_number.go`, `string_format_integer.go`, `string_encoding.go`, `numeric_funcs.go`, `array_funcs.go`, `object_funcs.go`, `hof_funcs.go`, `boolean_funcs.go`, `datetime_funcs.go`, `datetime_format.go`, `datetime_parse.go`. All registered via `functions.RegisterAll`.
+Dispatch-map of `funcFastHandlers` maps each `FuncFastKind` to a standalone handler function (e.g., `evalFuncContains`, `evalFuncString`). Each handler operates directly on `gjson.Result` for zero-copy evaluation.
 
-### Fast-Path Byte Evaluation (func_fast.go)
+## Key Types
 
-Dispatch-map of `funcFastHandlers` maps each `FuncFastKind` to a standalone handler function (e.g., `evalFuncContains`, `evalFuncString`). Each handler operates directly on `gjson.Result` for zero-copy evaluation. `FuncFastRound` is intentionally absent — it requires banker's rounding handled by the full evaluator.
+| Type | File | Description |
+|------|------|-------------|
+| `gnata.Expression` | `gnata.go` | Compiled, goroutine-safe JSONata expression with fast-path metadata |
+| `gnata.StreamEvaluator` | `stream.go` | Batch evaluator with copy-on-write expression slice + `BoundedCache` for schema plans |
+| `evaluator.Environment` | `internal/evaluator/env.go` | Lexical scope chain for variable bindings and function registry |
+| `parser.Node` | `internal/parser/node.go` | AST node types |
+| `planner.QueryPlan` | `internal/planner/planner.go` | Compiled execution plan for `jsonata_query` aggregate |
+| `BoundedCache` | `bounded_cache.go` | Lock-free FIFO ring-buffer cache (atomic pointer reads) |
+| `OrderedMap` | `internal/evaluator/ordered_map.go` | Insertion-ordered map preserving JSON field order |
 
-### Key Types
+## Testing
 
-- **Expression** (`gnata.go`) — Compiled, goroutine-safe JSONata expression with fast-path metadata
-- **StreamEvaluator** (`stream.go`) — Copy-on-write expression slice + `BoundedCache` for schema plans
-- **BoundedCache** (`bounded_cache.go`) — Lock-free FIFO ring-buffer cache (atomic pointer reads)
-- **OrderedMap** (`internal/evaluator/ordered_map.go`) — Insertion-ordered map preserving JSON field order
-- **Environment** (`internal/evaluator/env.go`) — Lexical scope chain for variable bindings
+- Tests use separate `_test` packages (`gnata_test`, `lexer_test`, `parser_test`)
+- `testdata/groups/` contains 100+ test case groups ported from the jsonata-js official suite
+- `testdata/datasets/` contains JSON test fixtures
+- `suite_test.go` loads 1,200+ JSON test cases — each `.json` file has `expr`, `dataset`, `bindings`, and `result` fields
+- Key test files: `gnata_test.go`, `stream_test.go`, `func_fast_test.go`, `evaluator_test.go`, `suite_test.go`, `lexer_test.go`, `parser_test.go`, `analysis_test.go`
+- SQLite benchmarks: `sqlite/benchmarks/*.sql`
+- CI runs tests with `-race` flag
 
 ## Dependencies
 
-Only one direct dependency (pure Go, no CGo):
+Only one direct dependency (pure Go, no CGo for core):
 - `tidwall/gjson` — Zero-copy JSON field extraction for fast-path byte-level evaluation
 
-Regex uses Go's standard `regexp` package (no external regex library).
-
-## Testing
-
-Tests use separate `_test` packages (`gnata_test`, `lexer_test`, `parser_test`). The primary test suite (`suite_test.go`) loads 1,200+ JSON test cases from `testdata/groups/` (100+ subdirectories) — each `.json` file is a case with `expr`, `dataset`, `bindings`, and `result` fields. Datasets live in `testdata/datasets/`. Additional unit tests in `evaluator_test.go` cover regression tests using table-driven test (TDT) style.
+The `sqlite/` package requires CGo (links against SQLite via `sqlite3ext.h`). Regex uses Go's standard `regexp` package.
 
 ## Custom Functions
 
@@ -94,7 +173,3 @@ customFuncs := map[string]gnata.CustomFunc{
 }
 se := gnata.NewStreamEvaluator(nil, gnata.WithCustomFunctions(customFuncs))
 ```
-
-## WASM
-
-`wasm/main.go` exports `gnataEval`, `gnataCompile`, `gnataEvalHandle` for browser use. Build with `GOOS=js GOARCH=wasm go build -ldflags="-s -w" -trimpath -o gnata.wasm ./wasm/`.

CODE_OF_CONDUCT.md

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2026 RecoLabs Inc.
+Copyright (c) 2026 Robert Polana
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal