Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions 0-AI-MANIFEST.a2ml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,13 @@ This is the AI manifest for **verisimiser**. It declares:
- Critical invariants (rules that must NEVER be violated)
- Repository structure and organization

**verisimiser** augments existing databases with VeriSimDB octad capabilities.
It wraps PostgreSQL, SQLite, MongoDB, Redis, and other backends to add eight
octad dimensions (data, metadata, provenance, lineage, constraints, access
control, temporal, simulation) without requiring database migration.
**verisimiser** augments existing databases with its fixed eight-concern
Octad (ADR-0004/ADR-0014), drawing capability from VeriSimDB's upstream
eight-modality Octad model. It wraps PostgreSQL, SQLite, MongoDB, Redis, and
other backends to add those eight concerns (data, metadata, provenance,
lineage, constraints, access control, temporal, simulation) without requiring
database migration. "Octad" unqualified means the concern Octad; upstream's
is written *VeriSimDB's Octad*.

## CANONICAL LOCATIONS (UNIVERSAL RULE)

Expand Down Expand Up @@ -93,9 +96,9 @@ Policy enforcement contracts (k9, dust, lust, must, trust).

## VERISIMISER-SPECIFIC INVARIANTS

- **Octad dimensions**: data, metadata, provenance, lineage, constraints, access-control, temporal, simulation
- **Concern Octad** (ADR-0004): data, metadata, provenance, lineage, constraints, access-control, temporal, simulation -- fixed arity, not the upstream modality Octad
- **Tier 1 (piggybacks)**: drift detection, provenance tracking, temporal versioning -- sidecar-only
- **Tier 2 (overlays)**: graph, vector, tensor, semantic, document, spatial -- additional storage
- **Tier 2 (overlays)**: graph, vector, tensor, semantic, document, spatial -- six-member overlay modality set (not an octad); per-manifest enabled-overlay selection (ADR-0014)
- **Supported backends**: PostgreSQL, SQLite, MongoDB, Redis (extensible)
- **VQL-UT**: Type-safe query interface for octad queries

Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
== Changelog

All notable changes to `+verisimiser+` will be documented in this file.
Expand Down Expand Up @@ -70,6 +70,15 @@

==== Documentation

* docs(adr): 0014-modality-vocabulary — unqualified *Octad* is
VeriSimiser's concern Octad; upstream is always *VeriSimDB's Octad*;
Tier 2 is a six-member overlay modality set with per-manifest enabled
overlays, not a "modality profile" (closes #213)
* docs(readme): vocabulary table for the three octad/modality models;
manifest section corrected to the canonical `[project]`/`[database]`/
`[octad]`/`[sidecar]`/`[retention]` schema, with the legacy
`[verisimiser]`/`[tier1]`/`[tier2]` shape documented as parsed-but-inert;
Tier 2 subset and stub status called out (#213)
* docs(readme): rewrite around concerns octad (V-L1-A2) (#99)
* docs(adr): 0009-build-path — cargo + Containerfile canonical; rest
experimental (#97)
Expand Down
23 changes: 18 additions & 5 deletions CONTRIBUTING-DEV.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,11 @@ threshold.

== Adding a Tier 2 overlay

Tier 2 overlays require additional storage alongside the target database
(graph, vector, tensor, semantic, …). New overlays are not the same as
new backends.
Tier 2 overlays require additional storage alongside the target database.
The overlay modality set is fixed at six (graph, vector, tensor,
semantic, document, spatial) per ADR-0014; a seventh is an ontology
change, not a config edit. New overlays are not the same as new
backends.

[%interactive]
* [ ] Add a boolean field to `Tier2Config` in `src/manifest/mod.rs` (and
Expand All @@ -100,11 +102,22 @@ new backends.
queries to the overlay.
* [ ] Add a round-trip test in `src/codegen/overlay.rs::tests` covering
the flag both on and off.
* [ ] Update `README.adoc`'s *Octad: Eight Modalities* table to reflect
the new overlay's tier and storage.
* [ ] Update the `README.adoc` <<tier-2-augmentation-layer,Tier 2>>
section (overlay modality set, storage, status) and the vocabulary
table in the README (anchor `octad-vocabulary`). The README no
longer has an "Octad: Eight Modalities" table. Note that
a *new* overlay changes the set's membership and so needs an ADR,
not just a README edit (ADR-0014 §14.3).

== Style notes

* **Vocabulary** (ADR-0014): unqualified `Octad` means VeriSimiser's
fixed eight-concern model; write `VeriSimDB's Octad` for the upstream
eight-modality API/model. Tier 2 uses *overlay modality set* for the
fixed six and *enabled overlays* for a manifest's selection — not
"modality profile". Do not rename schema, CLI, or manifest fields to
match a doc change; that needs its own compatibility decision.

* Treat `cargo clippy --all-targets -- -D warnings` as part of `cargo
build`. The crate has no blanket `#![allow(...)]` blocks
(`docs/decisions/0001` and 0002 keep the surface clean).
Expand Down
209 changes: 162 additions & 47 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,11 @@ Each entity in a VeriSimiser-augmented database is enriched along
**eight concerns** — properties you want to know or enforce about the
data, independent of how it is stored. The concerns octad is canonical
per `docs/decisions/0004-octad-ontology.adoc` (ADR-0004). The earlier
"Eight Modalities" framing (graph, vector, tensor, …) is reframed as a
set of **Tier 2 overlay representations**, not the top-level identity;
see <<tier-2-augmentation-layer,Tier 2>> below.
"Eight Modalities" framing (graph, vector, tensor, …) is reframed as the
six-member **Tier 2 overlay modality set**, not the top-level identity;
see <<tier-2-augmentation-layer,Tier 2>> below and
<<octad-vocabulary,the vocabulary note>> for how that differs from
VeriSimDB's Octad.

[cols="1,2,1,1,1"]
|===
Expand Down Expand Up @@ -94,6 +96,46 @@ see <<tier-2-augmentation-layer,Tier 2>> below.
The canonical Rust enumeration of these is `abi::OctadDimension`. The
CLI's `verisimiser octad` subcommand prints this set with descriptions.

[[octad-vocabulary]]
=== Three things called "octad"

"Octad" names two different eight-element models in two different
projects, and VeriSimiser's Tier 2 overlays are a third, non-octad
thing. Per `docs/decisions/0014-modality-vocabulary.adoc` (ADR-0014):

[cols="1,3,3",options="header"]
|===
| Term | What it denotes | What it does *not* denote

| **Octad** (unqualified, in this repo)
| VeriSimiser's fixed eight-**concern** model — the table above.
| Never the upstream modality list; never the Tier 2 overlays.

| **`OctadDimension`**
| The Rust enum with exactly eight concern variants (Data … Simulation).
| It has no `Graph`, `Vector`, `Tensor`, `Semantic`, `Document`, or `Spatial` variant.

| **VeriSimDB's Octad** (upstream)
| VeriSimDB's named eight-**modality** API and model — graph, vector, tensor, semantic, document, temporal, provenance, spatial.
| Not a VeriSimiser type, field, or enum; VeriSimiser does not implement all eight of its modalities natively.

| **Tier 2 overlay modality set**
| The six overlay modalities an entity can additionally be projected into: graph, vector, tensor, semantic, document, spatial.
| Not "the Octad" — it has six members, not eight. Upstream's Temporal and Provenance members are Tier 1 *concerns* here.

| **Enabled overlays**
| Per-manifest selection from that six-member set (any subset, 0–6 members; all six default to `false`).
| Not a versioned "modality profile" — see below.
|===

VeriSimiser docs do not call an enabled-overlay selection a
**modality profile**: upstream
https://github.com/hyperpolymath/verisimdb/issues/299[verisimdb#299]
proposes that term for a *versioned, implementation-specific* selection
of supported modalities, and `[tier2]` flags are unversioned enablement
booleans with derived, undeclared cardinality. Adopting the term here
would need a manifest-schema change first (ADR-0014 §14.5).

== Tier 1: True Piggybacks

These work like PostGIS — they add capability without replacing
Expand Down Expand Up @@ -150,13 +192,13 @@ Prefix-typed principals (`user:`, `group:`, `role:`) and an
against the same row that flows through the read-path. See ADR-0007 for
the model.

=== Drift detection: symptoms of Constraints violation across modalities
=== Drift detection: symptoms of Constraints violation across representations

Drift detection is **what the Constraints concern observes** when the
same entity is represented in more than one form and the forms disagree.
Once a Tier 2 overlay (graph, vector, tensor, …) is enabled alongside
the primary Data, the representations can fall out of sync — at which
point the Constraints concern reports the symptom.
Once a Tier 2 overlay modality (graph, vector, tensor, …) is enabled
alongside the primary Data, the representations can fall out of sync —
at which point the Constraints concern reports the symptom.

ADR-0003 fixes each drift category as a triple of (input, distance
function, default threshold). All categories produce a score in `[0, 1]`
Expand All @@ -167,7 +209,7 @@ categories:
|===
| Category | What it compares | Default threshold

| **Temporal** | `MAX(version)` per modality for the same entity | `0.10`
| **Temporal** | `MAX(version)` per representation for the same entity | `0.10`
| **Structural** | Schema fingerprint (sorted `(field, type)` pairs) | `0.05`
| **Semantic** | Free-text labels via sentence embeddings (cosine) | `0.20`
| **Statistical** | Distributions over the same field (1-Wasserstein) | `0.15`
Expand All @@ -191,12 +233,15 @@ doesn't modify them. Safe to add, safe to remove, no data dependency.

Tier 2 capabilities require additional storage alongside your database.
They are honest about being "VeriSimDB modalities with your database as
the primary store." This is still valuable — it's how you get octad
capabilities incrementally — but it's not a bolt-on.

Tier 2 modalities are **overlay representations** — alternative shapes
the same entity is projected into for a specific query workload. A user
enables `vector` because they want similarity search; they enable
the primary store" — but only a six-member subset of VeriSimDB's Octad
(graph, vector, tensor, semantic, document, spatial). Upstream's Octad
also lists Temporal and Provenance; here those are Tier 1 *concerns*
(see above), not overlays. So Tier 2 is not "the Octad": it is the
**Tier 2 overlay modality set**.

Tier 2 overlay modalities are **overlay representations** — alternative
shapes the same entity is projected into for a specific query workload.
A user enables `vector` because they want similarity search; they enable
`spatial` because they want geofencing. Enabling an overlay is
independent of which concerns are active.

Expand All @@ -212,60 +257,124 @@ independent of which concerns are active.
* **Spatial overlay** — geospatial coordinates. Stored in an R-tree
sidecar.

Each Tier 2 overlay has its own storage and can be enabled
independently via `[tier2]` in the manifest. Your primary database
remains the source of truth for its native data. When an overlay
diverges from the primary Data, the Constraints concern reports the
symptom via the relevant drift category (see above).
The six-member set is fixed by `Tier2Config` in `src/manifest/mod.rs`;
what a manifest chooses is an **enabled-overlay selection** — any
subset of the six, cardinality 0–6, all defaulting to `false`. Your
primary database remains the source of truth for its native data. When
an overlay diverges from the primary Data, the Constraints concern
reports the symptom via the relevant drift category (see above).

**Status**: the Tier 2 modules are stubs today (`src/tier2/mod.rs`).
The `[tier2]` flags are parsed but not yet acted on — no overlay index,
overlay DDL, or query routing ships yet, so enabling an overlay is a
declaration of intent rather than an active capability. Overlay codegen
is tracked as V-L1-F/G/H.

The selection is not a versioned "modality profile" in the sense of
verisimdb#299; see <<octad-vocabulary,the vocabulary note>> and
ADR-0014 before introducing that term anywhere.

== The Manifest

The canonical (Phase 1) shape is `[project]`, `[database]`, `[octad]`,
`[sidecar]`, `[retention]`. `[octad]` configures the concern Octad
above; only its six togglable concerns appear, because Data and
Metadata are always on:

[source,toml]
----
[verisimiser]
[project]
name = "my-augmented-db"
version = "0.1.0"

[database]
target-db = "postgresql"
connection-string = "postgres://localhost/mydb"
backend = "postgresql" # postgresql | sqlite | mongodb
connection-string-env = "DATABASE_URL"
# schema-source = "schema.sql"

# The concern Octad (ADR-0004) — six togglable concerns, not modalities
[octad]
enable-provenance = true # SHA-256 hash-chain audit trail
enable-lineage = true # acyclic derivation DAG (ADR-0005)
enable-constraints = true # invariant enforcement / drift reports
enable-access-control = true # row/column policies (ADR-0007)
enable-temporal = true # automatic version history
enable-simulation = false # what-if branches (ADR-0006)

# Tier 1: concerns layered onto your DB without altering its storage
[tier1]
provenance = true # SHA-256 hash-chain audit trail
lineage = true # acyclic derivation DAG (ADR-0005)
constraints = true # invariant enforcement / drift reports
access-control = true # row/column policies (ADR-0007)
temporal-versioning = true # automatic version history
drift-detection = true # cross-modal observer (Constraints symptom)
[sidecar]
storage = "sqlite" # sqlite | postgres | json (ADR-0012)
path = ".verisim/sidecar.db"

[tier1.provenance]
sidecar = "sqlite" # sqlite | postgres | json (V-L2-F3 / ADR-0012)
sidecar-path = ".verisimiser/provenance.db"
[retention]
temporal-days = 90 # purged by `verisimiser gc`
# provenance-days = 0 # 0 = keep forever
# lineage-days = 0
----

[tier1.temporal]
sidecar = "sqlite"
`[retention]` keys are `provenance-days`, `temporal-days`, and
`lineage-days` (`0` = keep forever). Sidecar backend and json encoding
are covered in <<sidecar-storage,sidecar storage backends>>.

[retention]
temporal-days = 90 # purged by `verisimiser gc`
=== Legacy manifest sections

Pre-Phase-1 manifests used `[verisimiser]`, `[tier1]`, and `[tier2]`.
They still parse, so existing files keep working:

[source,toml]
----
[verisimiser]
name = "my-augmented-db"

[database]
target-db = "postgresql" # legacy alias of `backend`
connection-string = "postgres://localhost/mydb"

[tier1]
provenance = true
temporal-versioning = true
drift-detection = true

# Tier 2: overlay representations (additional storage alongside your DB)
[tier2]
graph = false
vector = false
tensor = false
semantic = false
document = false
spatial = false

[tier2.vector]
# model = "sentence-transformers/all-MiniLM-L6-v2"
# dimensions = 384
----

Read this carefully before copying it:

* **`[tier1]` accepts exactly three keys** — `provenance`,
`temporal-versioning`, `drift-detection`. Lineage, Constraints, and
AccessControl toggles exist only in `[octad]`. Unknown keys inside a
section are ignored on parse, so a legacy manifest that set
`lineage = true` under `[tier1]` enabled nothing.
* **Nested tables under `[tier1]`/`[tier2]` are a parse error** —
their keys are booleans, so `[tier1.provenance]` or
`[tier2.vector]` fails with a type error. Overlay options such as
embedding model or dimensions are not expressible yet.
* **The legacy sections do not drive behaviour**: the runtime reads
`[octad]`/`[sidecar]`/`[retention]` only. Set those, even if a legacy
section is also present.
* **`backend` and `target-db` disagreeing is an error**, not a silent
tie-break (V-L2-E1). `connection-string` is likewise a legacy
alternative to `connection-string-env`; the env-var form is preferred
so secrets stay out of the file.
* **`[tier2]` is the enabled-overlay selection** over the six-member
Tier 2 overlay modality set, not a modality profile and not part of
the Octad (ADR-0014). The flags are currently inert; see
<<tier-2-augmentation-layer,Tier 2>>.

Field names are stable: this section documents them, and does not
rename any of them. A rename would be a compatibility decision of its
own (ADR-0014 §14.6).

The `verisimiser octad` subcommand prints the active concerns from your
manifest; `verisimiser doctor` checks that sidecars, thresholds, and
retention bounds are configured consistently.

[[sidecar-storage]]
=== Sidecar storage backends (Phase 1 schema)

The Phase 1 manifest uses a single `[sidecar]` section. All octad data
Expand Down Expand Up @@ -342,7 +451,7 @@ are not yet implemented and refuse explicitly rather than silently no-op.
└───────────────────────────────────┼──────────────────────┘
│
┌───── optional, per-overlay ───────┘
│ Tier 2 overlays (modalities)
│ Tier 2 overlay modalities (six-member set)
│
│ ┌───────┐ ┌────────┐ ┌────────┐ ┌──────────┐ ┌────────┐ ┌────────┐
│ │ graph │ │ vector │ │ tensor │ │ semantic │ │ docs │ │ spatial│
Expand All @@ -368,12 +477,18 @@ VeriSimiser is NOT a replacement for VeriSimDB. It is a gateway drug.
(Provenance, Lineage, Constraints, AccessControl, Temporal — plus
Data and Metadata which are always-on) on your existing database.
Zero commitment.
* **VeriSimiser Tier 2** gives you the modality overlays (graph,
* **VeriSimiser Tier 2** gives you the six overlay modalities (graph,
vector, tensor, semantic, document, spatial) and the Simulation
concern as sidecars. Incremental adoption.
* **Full VeriSimDB** gives you the complete octad with native
cross-modal querying, VCL, and built-in drift normalisation. Full
commitment.
* **Full VeriSimDB** gives you VeriSimDB's complete Octad — its
eight-modality API and model — with native cross-modal querying, VCL,
and built-in drift normalisation. Full commitment.

Note the asymmetry: VeriSimiser Tier 2 ships six of VeriSimDB's eight
modalities, because Temporal and Provenance are Tier 1 concerns here.
"Octad" in this README always means the concern Octad unless it is
written "VeriSimDB's Octad"; see <<octad-vocabulary,the vocabulary
note>>.

The migration path is Tier 1 → Tier 2 → full VeriSimDB (if you want
it). Most users will be happy at Tier 1 or Tier 2.
Expand Down
3 changes: 2 additions & 1 deletion ROADMAP.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,13 @@
== Phase 0: Scaffold (COMPLETE)
* [x] RSR template with full CI/CD
* [x] CLI with subcommands (init, start, drift, provenance, history, status, octad, doctor, gc, validate, version)
* [x] Manifest parser (verisimiser.toml with `[tier1]`/`[tier2]`/`[retention]` config)
* [x] Manifest parser (verisimiser.toml: `[project]`/`[database]`/`[octad]`/`[sidecar]`/`[retention]`, plus legacy `[verisimiser]`/`[tier1]`/`[tier2]`)
* [x] Concerns octad fixed canonical (ADR-0004): Data, Metadata, Provenance, Lineage, Constraints, AccessControl, Temporal, Simulation
* [x] Tier 1 data types per concern (ProvenanceEntry, TemporalVersion, DriftReport, AccessPredicate)
* [x] ABI module stubs (Idris2 + Zig FFI) and domain-tagged `compute_hash`
* [x] Drift categories pinned (ADR-0003): input / distance / threshold per category
* [x] README rewritten around concerns octad (V-L1-A2, #20)
* [x] Vocabulary fixed (ADR-0014, #213): unqualified *Octad* = the concern Octad; *VeriSimDB's Octad* = upstream eight-modality API/model; Tier 2 = six-member overlay modality set with per-manifest enabled overlays (no "modality profile")

== Phase 1: PostgreSQL Tier 1 MVP — concerns layered onto your DB
* [ ] PostgreSQL logical replication interception (Data / Metadata feed)
Expand Down
Loading
Loading