Skip to content

feat(search)!: engine- and domain-agnostic query model, Typesense compiler, and GraphQL surface#529

Open
ddeboer wants to merge 13 commits into
mainfrom
feat/search-core
Open

feat(search)!: engine- and domain-agnostic query model, Typesense compiler, and GraphQL surface#529
ddeboer wants to merge 13 commits into
mainfrom
feat/search-core

Conversation

@ddeboer

@ddeboer ddeboer commented Jun 28, 2026

Copy link
Copy Markdown
Member

What

Reworks @lde/search and @lde/search-typesense into a unified, engine- and
domain-agnostic
search API, and adds @lde/search-api-graphql (the new GraphQL surface).
@lde/search and @lde/search-typesense already existed in the repo; this reworks them
(breaking), it does not introduce them. One declarative search schema drives projection, the
engine collection schema, the query semantics, and the GraphQL surface – so they cannot
drift. The domain type (Dataset, Person, …) and the engine choice (Typesense, …) are the
consumer’s, configured at the seams; the libraries never name a domain.

Terminology

The model has three levels (see the new Terminology section in the @lde/search README):

Term What it is SHACL GraphQL
SearchField One queryable field: kind, IR path, capability flags property shape field
SearchType One root type’s declaration: type IRI + fields + derivations NodeShape object type
SearchSchema Every SearchType, keyed by type IRI; built with searchSchema(…) shapes graph schema

projectGraph consumes a SearchSchema; the engine port and the GraphQL surface operate on
one SearchType at a time.

Review guide

Three tiers by stability; spend review effort accordingly.

1. Stable API Contract (the emitted GraphQL SDL): highest scrutiny. The consumer-facing
surface a Presentation Layer couples to. Its stability is independent of @lde package versions
and is guarded by the printGraphQLSchema SDL snapshot, so this is the one part that must stay
right.

  • @lde/search-api-graphql build-schema.ts: output types, where/orderBy/facet inputs,
    named reference types, nullability.

2. @lde library API (0.x, still stabilizing): review the design, not for permanence.
Developer-facing package APIs; pre-1.0 a breaking change is a routine minor bump (our nx
adjustSemverBumpsForZeroMajorVersion), so review for correctness and shape, not as frozen.

  • @lde/search engine.ts (SearchEngine port + result types), query.ts (SearchQuery IR /
    filter operators), schema.ts (SearchField/SearchType/SearchSchema model).

3. Internal / swappable: lower scrutiny. Behind the port; changeable without consumer impact
(ADR 0003).

  • @lde/search project.ts / frame-by-type.ts; @lde/search-typesense query-compiler.ts /
    collection-schema.ts / search.ts.

The neutral-fixture snapshot tests pin each generator; a snapshot diff flags a generated-shape
change, so start there.

Not in this PR: the consumer (Dataset Register) side, including the hand-written dr:*
CONSTRUCTs, lands in a separate DR PR after these packages publish. Those CONSTRUCTs are
provisional (slated for SHACL-driven replacement) and will be guarded there by a
schema/CONSTRUCT contract test, so none of that review burden is here.

Packages

@lde/search (core) — breaking

  • Unifies the field model: one SearchField / SearchType replaces the projection
    FieldSpec/Projection and the discriminated FieldKind.
  • Names the whole declaration: SearchSchema is the map of SearchTypes keyed by type IRI,
    built with the new searchSchema() factory; projectGraph(quads, schema) consumes it.
  • Adds the neutral query IR (SearchQuery / Filter / Sort) and filter-operator semantics.
  • Adds the SearchEngine port and logical result types (SearchResult / SearchHit /
    ResultDocument / Reference / LocalizedValue / FacetBucket).
  • Adds physicalFields (the shared physical-fanout convention) and field selectors
    (searchableFields, facetableFields, filterableFields, sortableFields, outputFields).
  • Rewrites projectDocument/projectGraph onto the unified model; projection output is
    unchanged — the guardrail test was ported field-for-field.
  • BREAKING: FieldSpec, Projection, and the discriminated FieldKind are removed. The
    per-type declaration is SearchType (formerly named SearchSchema).

@lde/search-typesense (engine adapter) — additive

  • buildCollectionSchema derives a Typesense collection from the field model (kind→type, the
    physical fanout via physicalFields, per-locale stemming, required / default-sorting-field).
  • buildSearchParams compiles SearchQuery into Typesense params — filter_by / sort_by /
    facet_by / query_by with active-locale weighting and exact membership for non-facet
    fields (grouped facets are ordinary denormalized values, not a special clause).
  • createTypesenseSearchEngine implements the SearchEngine port end to end: it reconstructs
    logical documents and resolves reference (and reference-facet) labels from the sidecar
    labels collection in a single lookup.
  • Covered by unit tests plus a Typesense testcontainer integration test.

@lde/search-api-graphql (GraphQL surface) — new

  • buildGraphQLSchema(searchType, { typeName }) builds an executable GraphQLSchema at
    runtime from any SearchType (no codegen, no SDL artifact), served by one generic resolver
    over any SearchEngine.
  • Derives output types, where/orderBy/facet inputs, named reference types, and nullability
    (from required / array / kind) from the field model; best-first Accept-Language
    output ordering; a nullable facet label resolved for reference facets only.
  • Exports printGraphQLSchema for a consumer-side SDL snapshot guard.

Notes

  • ADRs 0003/0004 updated: the unified model and its terminology (SearchType per NodeShape,
    SearchSchema as the type-keyed map), the SearchEngine rename (the interface is the port;
    the Typesense class is the adapter), sizeFloat (int64 overflow), the typed-surface
    design, and facet labels.
  • Root README: the packages table gains the @lde/search-api-graphql row and the architecture
    diagram gains the search family (search, search-typesense, search-api-graphql,
    text-normalization).
  • Each generator ships a neutral-fixture snapshot to pin its output across versions.
  • Deferred: the idOnly/inline reference strategies, the OutputOf<S> typed-surface
    overlay, and a REST surface.

ddeboer added 3 commits June 28, 2026 20:26
…d result types

- replace FieldSpec and Projection with one SearchField/SearchSchema model

- add SearchQuery, Filter, Sort and the filter-operator semantics

- add the SearchEngine port and result types (SearchResult/SearchHit/ResultDocument/Reference)

- add physicalFields (the shared fanout convention) and schema selectors

- rewrite projectDocument and projectGraph onto the unified model; projection output unchanged

- remove FieldSpec, Projection and the discriminated FieldKind (breaking)
… and SearchEngine

- buildCollectionSchema derives a Typesense collection from the unified SearchField model

- buildSearchParams compiles SearchQuery into Typesense params (filter_by/sort_by/facet_by/query_by)

- createTypesenseSearchEngine implements the SearchEngine port: compile, search, reconstruct

- resolve reference and reference-facet labels from the sidecar labels collection in one lookup

- add a testcontainer integration test and a generator-stability snapshot
- buildSearchSchema builds an executable GraphQLSchema from any SearchSchema at runtime (no codegen)

- one generic resolver maps args to SearchQuery, calls the engine, and maps the result back

- derive output, where, orderBy and facet types plus nullability from the field model

- best-first Accept-Language output ordering; nullable facet label for reference facets

- add printSearchSchema for a consumer SDL snapshot, plus a generator-stability snapshot
@ddeboer ddeboer force-pushed the feat/search-core branch from ae639ea to 66969a2 Compare June 28, 2026 18:51
ddeboer added 2 commits June 29, 2026 09:38
- state the decisions directly as the reconciled architecture, not deviations from a draft

- remove the deviation/reconcile framing and the deviations-to-reconcile lists

- align wording with the stack platform layer
- number fields now project as floats (not truncated like integer)

- closes the step-1 gap so an int64-magnitude field mapped to number (Float) indexes
Replace the repo-path breadcrumb with a direct link to the docs site, so the
status note points readers at the rendered page rather than a source file path.
@ddeboer ddeboer changed the title feat: unified engine- and domain-agnostic search API (@lde/search* family) feat(search)!: engine- and domain-agnostic query model, Typesense compiler, and GraphQL surface Jul 1, 2026
ddeboer added 4 commits July 1, 2026 09:47
… the group companion

- Keyed per-type facets object on the GraphQL surface (ValueBucket / RangeBucket),
  selection-is-the-request with skip-own-filter.
- Numeric range facets and an opt-in label cache in the Typesense adapter.
- Reconcile ADRs 0003 and 0004 with the implementation.

BREAKING CHANGE: remove SearchField.group and its *_group companion field, collection
column and query split. Deployments denormalize group tokens into the field values
instead, so a group is an ordinary facet value with no engine mechanism.
…@lde/* pins

npm ci failed because the lockfile lacked the new @lde/search-api-graphql workspace.
Regenerating against npmjs adds it and brings ~24 @lde/* internal deps up to their latest
in-range patches; no third-party or duplicate-version changes.
… search-engine test

`result.facets` is a `Partial` record, so a facet is `FacetBucket[] | undefined`; guard the two
spreads with `?? []` so the `typecheck` target passes (it never ran in CI before the lockfile fix).
…ations

Fold the unified-field-model blockquote and the dated Consequences bullet into running
text, so the ADR reads as the current design rather than a change log.
ddeboer added 3 commits July 2, 2026 19:46
- SearchType is one root type declaration (one SHACL NodeShape, one GraphQL
  object type); SearchSchema now names the whole search declaration: a
  ReadonlyMap of SearchTypes keyed by type IRI, built with the new
  searchSchema() factory
- projectGraph now consumes a SearchSchema instead of a SearchType array
- rename buildSearchSchema / printSearchSchema / BuildSearchSchemaOptions to
  buildGraphQLSchema / printGraphQLSchema / BuildGraphQLSchemaOptions: they
  construct a GraphQLSchema rather than the SearchSchema the old names implied
- rename schema parameters to searchType where they take one type, and the
  FacetFieldsOf/OutputFieldsOf/EngineFor/ResultFor generic from Schema to Type
- add a Terminology section to the @lde/search README mapping SearchField /
  SearchType / SearchSchema onto SHACL and GraphQL; update ADRs 3 and 4, the
  package READMEs and npm descriptions
- drop section-divider comments in build-schema.ts and stale grouped-facet
  mentions in the READMEs

BREAKING CHANGE: the per-type interface SearchSchema is renamed to SearchType,
and SearchSchema now denotes the type-keyed map built with searchSchema().
projectGraph(quads, types[]) becomes projectGraph(quads, searchSchema(...types)).
In @lde/search-api-graphql, buildSearchSchema, printSearchSchema and
BuildSearchSchemaOptions are renamed to buildGraphQLSchema, printGraphQLSchema
and BuildGraphQLSchemaOptions.
- add the missing @lde/search-api-graphql row to the packages table
- add the search, search-typesense, search-api-graphql and text-normalization
  dependency edges to the architecture diagram, which lacked the search family
  entirely
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant