Skip to content

Commit e346708

Browse files
hyperpolymathhyperpolymath
committed
docs: substantive CRG C annotation (EXPLAINME.adoc)
1 parent 5d9eb42 commit e346708

1 file changed

Lines changed: 220 additions & 6 deletions

File tree

‎EXPLAINME.adoc‎

Lines changed: 220 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,238 @@
11
// SPDX-License-Identifier: PMPL-1.0-or-later
2+
// Copyright (c) 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
23
= Next-Gen Databases — Show Me The Receipts
34
:toc:
5+
:toclevels: 3
46
:icons: font
7+
:author: Jonathan D.A. Jewell
58

6-
The README makes claims. This file backs them up.
9+
The README makes claims. This file backs them up with architectural context,
10+
code paths for the primary database, and honest priority rankings so an
11+
external reviewer can distinguish active engineering from research exploration.
12+
13+
IMPORTANT: This is a *parent tracking repository* — no implementation code
14+
lives here. All databases are in subdirectories (`verisimdb/`, `quandledb/`,
15+
`lithoglyph/`) which are also mirrored as standalone canonical repos.
16+
17+
== Claim 1: "Coordinated effort to build specialised databases, each with its own query language, storage engine, and web interface"
718

819
[quote, README]
920
____
10-
This repository serves as the central tracking hub for the Next-Gen Databases initiative — a coordinated effort to build specialised database applications across different domains, each with its own query language, storage engine, and web interface.
21+
This repository serves as the central tracking hub for the Next-Gen Databases
22+
initiative — a coordinated effort to build specialised database applications
23+
across different domains, each with its own query language, storage engine,
24+
and web interface.
1125
____
1226

27+
=== How it works
28+
29+
The architecture shared across all three databases follows a layered stack:
30+
31+
[source]
32+
----
33+
Query Language (DSL)
34+
↓
35+
Database Engine (storage, indexing, computation)
36+
↓
37+
HTTP API Server (JSON endpoints)
38+
↓
39+
Web Frontend (ReScript SPA)
40+
----
41+
42+
Each layer is domain-specific. VeriSimDB uses VQL over a Rust/Elixir engine.
43+
QuandleDB uses KQL over a Julia (Skein.jl) engine. LithoGlyph uses GQL over
44+
a Forth/Factor/Zig engine with Idris2 proofs.
45+
46+
=== The primary path: VeriSimDB drift detection
47+
48+
VeriSimDB is the primary engineering focus. Each entity exists across up to
49+
8 simultaneous representations (the *octad*): Graph, Vector, Tensor, Semantic,
50+
Document, Temporal, Provenance, Spatial.
51+
52+
The drift detection demo entry point is:
53+
54+
[source,bash]
55+
----
56+
cd verisimdb/elixir-orchestration && mix run ../demos/drift-detection/run_demo.exs
57+
----
58+
59+
This demo creates 1000 entities, corrupts 50, and verifies that the drift
60+
detector identifies all 50 with zero false positives (in the sample output).
61+
The Elixir orchestration layer (`verisimdb/elixir-orchestration/`) coordinates
62+
the Rust core (`verisimdb/src/`) and the VQL parser (`verisimdb/src/vql/`).
63+
64+
The ABI layer at `verisimdb/src/abi/` (Idris2: `Types.idr`, `Layout.idr`,
65+
`Foreign.idr`) provides machine-checked memory layout proofs for the C-ABI
66+
bridge between Rust and Elixir. The capability registry at
67+
`verisimdb/src/registry/` tracks which modalities are active per entity.
68+
69+
=== Honest caveat
70+
71+
VeriSimDB is at beta. The drift detection loop works (demo is live). The VQL
72+
query language is approximately 40% complete — basic entity CRUD and
73+
cross-modal consistency queries work; advanced dependent-type queries (VQL-DT)
74+
are planned. The "formally verified queries" claim in the README refers to
75+
the Idris2 ABI proofs, not end-to-end query verification (which is roadmap).
76+
77+
== Claim 2: "VeriSimDB detects drift before it causes damage; QuandleDB and LithoGlyph cover specialist domains"
78+
79+
[quote, README]
80+
____
81+
Project priorities: VeriSimDB is the primary database engineering project.
82+
LithoGlyph is working towards production for specialist use in journalism and
83+
narrative arts. QuandleDB is a research exploration — a thought experiment made
84+
concrete.
85+
____
86+
87+
=== Priority breakdown
88+
89+
[cols="1,1,3"]
90+
|===
91+
| Database | Priority | Honest Status
92+
93+
| *VeriSimDB*
94+
| PRIMARY — active development
95+
| Beta. Drift detection, octad entity model, Elixir orchestration, VQL parser
96+
(~40% complete), Idris2 ABI proofs. Deployed with per-project instances
97+
(per VeriSimDB policy: one instance per consumer repo, never shared).
98+
99+
| *LithoGlyph*
100+
| Secondary — on backburner
101+
| Active development paused while VeriSimDB matures. Architecture is defined:
102+
Forth storage engine, Zig bridge, Factor for GQL, Idris2 proofs, Lean 4
103+
normaliser. The reversibility design (GQL operations have machine-checked
104+
inverses) will benefit from MAA Framework lessons.
105+
106+
| *QuandleDB*
107+
| Exploratory — research only
108+
| Not intended as a production tool. Wraps Skein.jl (Julia knot database)
109+
as a web app. 36 knots from the Rolfsen table, Jones polynomial display.
110+
Thought experiment in applying knot theory to database design.
111+
|===
112+
113+
=== How VeriSimDB's cross-modal consistency actually works
114+
115+
When an entity's vector embedding is stored, VeriSimDB also records a
116+
content hash of the source document used to generate it. On each read
117+
(configurable: also on write), the engine recomputes the content hash,
118+
compares it to the stored hash, and flags divergence as `VectorDocumentDrift`.
119+
If the spatial coordinates stored for an entity no longer appear in its text
120+
representation, that is `SpatialSemanticDrift`. The drift type taxonomy in
121+
`verisimdb/src/` determines the repair strategy: re-embed, re-index, or
122+
escalate to human review.
123+
124+
=== Honest caveat on QuandleDB
125+
126+
The README is explicit: QuandleDB is not production-ready. An external
127+
reviewer should not treat it as evidence of a production knot-theory database.
128+
It is a proof-of-concept that knot-theoretic invariants can be stored and
129+
queried as first-class database attributes. If you need a production database,
130+
use VeriSimDB.
131+
132+
== Dogfooded Across The Account
133+
134+
[cols="1,2,2"]
135+
|===
136+
| Technology | Role in Next-Gen Databases | Also Used In
137+
138+
| *Rust* (VeriSimDB core)
139+
| Storage engine, VQL parser, drift detection algorithms
140+
| ephapax, gitbot-fleet, protocol-squisher
141+
142+
| *Elixir* (VeriSimDB orchestration)
143+
| Distributed coordination, Phoenix API, BEAM fault tolerance
144+
| burble (voice platform), gossamer (service discovery target)
145+
146+
| *ReScript* (all frontends)
147+
| VQL query UI, QuandleDB frontend, planned LithoGlyph frontend
148+
| idaptik, panll, gossamer-powered SPAs
149+
150+
| *Idris2 ABI* (VeriSimDB + planned LithoGlyph)
151+
| Machine-checked memory layout proofs for cross-language FFI
152+
| Gossamer, Groove, Stapeln — every repo with Idris2 ABI layer
153+
154+
| *Zig FFI* (LithoGlyph bridge, VeriSimDB FFI)
155+
| C-compatible bridge between storage engine and query layer
156+
| game-server-admin, Stapeln, Gossamer
157+
158+
| *Julia* (Skein.jl, QuandleDB engine)
159+
| Knot invariant computation; Julia batch scripting convention
160+
| developer-ecosystem, data processing scripts
161+
162+
| *VeriSimDB itself*
163+
| Per-project instances deployed across every significant hyperpolymath
164+
repo for cross-modal entity consistency
165+
| Mandatory everywhere per VeriSimDB policy
166+
167+
| *Stapeln containers*
168+
| `stapeln.toml` for all three databases
169+
| All containerised services in the account
170+
171+
| *Hypatia scanning*
172+
| `.hypatia/` CI workflows
173+
| Every RSR repo in the account
174+
|===
175+
13176
== File Map
14177

15-
[cols="1,2"]
178+
[cols="1,3"]
16179
|===
17-
| Path | What's There
180+
| Path | What It Proves
181+
182+
| `verisimdb/`
183+
| Primary database. `src/` — Rust core (storage, drift detection, VQL parser).
184+
`src/abi/` — Idris2 ABI proofs (Types.idr, Layout.idr, Foreign.idr).
185+
`src/registry/` — modality capability registry.
186+
`src/vql/` — VQL query language parser and evaluator.
187+
`elixir-orchestration/` — Elixir/Phoenix distributed coordinator.
188+
`api/` — JSON HTTP API surface.
189+
`demos/` — drift-detection demo (the live proof that octad consistency works).
190+
`connectors/` — adapters for PostgreSQL, ArangoDB, Elasticsearch federation.
191+
`debugger/` — query debugger and drift inspector.
192+
`admin/` — admin tooling.
193+
194+
| `verisimdb/src/abi/`
195+
| Machine-checked proofs of memory layout correctness for the Rust/Elixir
196+
ABI boundary. Types.idr, Layout.idr, Foreign.idr.
197+
198+
| `quandledb/`
199+
| Knot-theory database (exploratory). `frontend/` — ReScript web app.
200+
`public/` — static assets. `docs/` — design documents. Wraps the external
201+
Skein.jl library rather than implementing knot algorithms directly.
202+
203+
| `lithoglyph/`
204+
| Narrative-first reversible database (backburner). Architecture defined:
205+
Forth storage, Zig bridge, Factor GQL, Idris2 proofs, Lean 4 normaliser.
206+
Current state: skeleton structure and architectural documentation.
207+
208+
| `nqc/`
209+
| NQC — experimental TypeQL-adjacent query language research, tracked here
210+
as part of the database query language ecosystem.
211+
212+
| `typeql-experimental/`
213+
| Experimental TypeQL integration work.
214+
215+
| `tests/`
216+
| Cross-database integration tests.
217+
218+
| `.machine_readable/`
219+
| A2ML checkpoint files. Canonical AI session state.
220+
221+
| `TOPOLOGY.md`
222+
| Visual architecture map and completion dashboard for all three databases.
223+
224+
| `TOOLING-STATUS.adoc`
225+
| Current tooling readiness — which query languages have parsers, which have
226+
evaluators, which are stub-only.
18227

19-
| `test(s)/` | Test suite
228+
| `stapeln.toml`
229+
| Stapeln container configuration (notably large: 19K — covers multi-database
230+
deployment configurations).
20231
|===
21232

22233
== Questions?
23234

24-
Open an issue or reach out directly — happy to explain anything in more detail.
235+
For the primary project, start with `verisimdb/README.adoc` and run the drift
236+
detection demo. For the full ecosystem picture, see `TOPOLOGY.md`. For
237+
LithoGlyph architecture intent (even though implementation is on backburner),
238+
see `lithoglyph/ARCHITECTURE.adoc`.

0 commit comments

Comments
 (0)