feat(search)!: engine- and domain-agnostic query model, Typesense compiler, and GraphQL surface#529
Open
ddeboer wants to merge 13 commits into
Open
feat(search)!: engine- and domain-agnostic query model, Typesense compiler, and GraphQL surface#529ddeboer wants to merge 13 commits into
ddeboer wants to merge 13 commits into
Conversation
…d result types - replace FieldSpec and Projection with one SearchField/SearchSchema model - add SearchQuery, Filter, Sort and the filter-operator semantics - add the SearchEngine port and result types (SearchResult/SearchHit/ResultDocument/Reference) - add physicalFields (the shared fanout convention) and schema selectors - rewrite projectDocument and projectGraph onto the unified model; projection output unchanged - remove FieldSpec, Projection and the discriminated FieldKind (breaking)
… and SearchEngine - buildCollectionSchema derives a Typesense collection from the unified SearchField model - buildSearchParams compiles SearchQuery into Typesense params (filter_by/sort_by/facet_by/query_by) - createTypesenseSearchEngine implements the SearchEngine port: compile, search, reconstruct - resolve reference and reference-facet labels from the sidecar labels collection in one lookup - add a testcontainer integration test and a generator-stability snapshot
- buildSearchSchema builds an executable GraphQLSchema from any SearchSchema at runtime (no codegen) - one generic resolver maps args to SearchQuery, calls the engine, and maps the result back - derive output, where, orderBy and facet types plus nullability from the field model - best-first Accept-Language output ordering; nullable facet label for reference facets - add printSearchSchema for a consumer SDL snapshot, plus a generator-stability snapshot
ae639ea to
66969a2
Compare
- state the decisions directly as the reconciled architecture, not deviations from a draft - remove the deviation/reconcile framing and the deviations-to-reconcile lists - align wording with the stack platform layer
- number fields now project as floats (not truncated like integer) - closes the step-1 gap so an int64-magnitude field mapped to number (Float) indexes
Replace the repo-path breadcrumb with a direct link to the docs site, so the status note points readers at the rendered page rather than a source file path.
… the group companion - Keyed per-type facets object on the GraphQL surface (ValueBucket / RangeBucket), selection-is-the-request with skip-own-filter. - Numeric range facets and an opt-in label cache in the Typesense adapter. - Reconcile ADRs 0003 and 0004 with the implementation. BREAKING CHANGE: remove SearchField.group and its *_group companion field, collection column and query split. Deployments denormalize group tokens into the field values instead, so a group is an ordinary facet value with no engine mechanism.
…@lde/* pins npm ci failed because the lockfile lacked the new @lde/search-api-graphql workspace. Regenerating against npmjs adds it and brings ~24 @lde/* internal deps up to their latest in-range patches; no third-party or duplicate-version changes.
… search-engine test `result.facets` is a `Partial` record, so a facet is `FacetBucket[] | undefined`; guard the two spreads with `?? []` so the `typecheck` target passes (it never ran in CI before the lockfile fix).
…ations Fold the unified-field-model blockquote and the dated Consequences bullet into running text, so the ADR reads as the current design rather than a change log.
- SearchType is one root type declaration (one SHACL NodeShape, one GraphQL object type); SearchSchema now names the whole search declaration: a ReadonlyMap of SearchTypes keyed by type IRI, built with the new searchSchema() factory - projectGraph now consumes a SearchSchema instead of a SearchType array - rename buildSearchSchema / printSearchSchema / BuildSearchSchemaOptions to buildGraphQLSchema / printGraphQLSchema / BuildGraphQLSchemaOptions: they construct a GraphQLSchema rather than the SearchSchema the old names implied - rename schema parameters to searchType where they take one type, and the FacetFieldsOf/OutputFieldsOf/EngineFor/ResultFor generic from Schema to Type - add a Terminology section to the @lde/search README mapping SearchField / SearchType / SearchSchema onto SHACL and GraphQL; update ADRs 3 and 4, the package READMEs and npm descriptions - drop section-divider comments in build-schema.ts and stale grouped-facet mentions in the READMEs BREAKING CHANGE: the per-type interface SearchSchema is renamed to SearchType, and SearchSchema now denotes the type-keyed map built with searchSchema(). projectGraph(quads, types[]) becomes projectGraph(quads, searchSchema(...types)). In @lde/search-api-graphql, buildSearchSchema, printSearchSchema and BuildSearchSchemaOptions are renamed to buildGraphQLSchema, printGraphQLSchema and BuildGraphQLSchemaOptions.
- add the missing @lde/search-api-graphql row to the packages table - add the search, search-typesense, search-api-graphql and text-normalization dependency edges to the architecture diagram, which lacked the search family entirely
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Reworks
@lde/searchand@lde/search-typesenseinto a unified, engine- anddomain-agnostic search API, and adds
@lde/search-api-graphql(the new GraphQL surface).@lde/searchand@lde/search-typesensealready existed in the repo; this reworks them(breaking), it does not introduce them. One declarative search schema drives projection, the
engine collection schema, the query semantics, and the GraphQL surface – so they cannot
drift. The domain type (
Dataset,Person, …) and the engine choice (Typesense, …) are theconsumer’s, configured at the seams; the libraries never name a domain.
Terminology
The model has three levels (see the new Terminology section in the
@lde/searchREADME):SearchFieldkind, IRpath, capability flagsSearchTypetypeIRI + fields + derivationsSearchSchemaSearchType, keyed bytypeIRI; built withsearchSchema(…)projectGraphconsumes aSearchSchema; the engine port and the GraphQL surface operate onone
SearchTypeat a time.Review guide
Three tiers by stability; spend review effort accordingly.
1. Stable API Contract (the emitted GraphQL SDL): highest scrutiny. The consumer-facing
surface a Presentation Layer couples to. Its stability is independent of
@ldepackage versionsand is guarded by the
printGraphQLSchemaSDL snapshot, so this is the one part that must stayright.
@lde/search-api-graphqlbuild-schema.ts: output types,where/orderBy/facet inputs,named reference types, nullability.
2.
@ldelibrary API (0.x, still stabilizing): review the design, not for permanence.Developer-facing package APIs; pre-1.0 a breaking change is a routine minor bump (our nx
adjustSemverBumpsForZeroMajorVersion), so review for correctness and shape, not as frozen.@lde/searchengine.ts(SearchEngineport + result types),query.ts(SearchQueryIR /filter operators),
schema.ts(SearchField/SearchType/SearchSchemamodel).3. Internal / swappable: lower scrutiny. Behind the port; changeable without consumer impact
(ADR 0003).
@lde/searchproject.ts/frame-by-type.ts;@lde/search-typesensequery-compiler.ts/collection-schema.ts/search.ts.The neutral-fixture snapshot tests pin each generator; a snapshot diff flags a generated-shape
change, so start there.
Not in this PR: the consumer (Dataset Register) side, including the hand-written
dr:*CONSTRUCTs, lands in a separate DR PR after these packages publish. Those CONSTRUCTs are
provisional (slated for SHACL-driven replacement) and will be guarded there by a
schema/CONSTRUCT contract test, so none of that review burden is here.
Packages
@lde/search(core) — breakingSearchField/SearchTypereplaces the projectionFieldSpec/Projectionand the discriminatedFieldKind.SearchSchemais the map ofSearchTypes keyed bytypeIRI,built with the new
searchSchema()factory;projectGraph(quads, schema)consumes it.SearchQuery/Filter/Sort) and filter-operator semantics.SearchEngineport and logical result types (SearchResult/SearchHit/ResultDocument/Reference/LocalizedValue/FacetBucket).physicalFields(the shared physical-fanout convention) and field selectors(
searchableFields,facetableFields,filterableFields,sortableFields,outputFields).projectDocument/projectGraphonto the unified model; projection output isunchanged — the guardrail test was ported field-for-field.
FieldSpec,Projection, and the discriminatedFieldKindare removed. Theper-type declaration is
SearchType(formerly namedSearchSchema).@lde/search-typesense(engine adapter) — additivebuildCollectionSchemaderives a Typesense collection from the field model (kind→type, thephysical fanout via
physicalFields, per-locale stemming, required / default-sorting-field).buildSearchParamscompilesSearchQueryinto Typesense params —filter_by/sort_by/facet_by/query_bywith active-locale weighting and exact membership for non-facetfields (grouped facets are ordinary denormalized values, not a special clause).
createTypesenseSearchEngineimplements theSearchEngineport end to end: it reconstructslogical documents and resolves reference (and reference-facet) labels from the sidecar
labelscollection in a single lookup.@lde/search-api-graphql(GraphQL surface) — newbuildGraphQLSchema(searchType, { typeName })builds an executableGraphQLSchemaatruntime from any
SearchType(no codegen, no SDL artifact), served by one generic resolverover any
SearchEngine.where/orderBy/facet inputs, named reference types, and nullability(from
required/array/kind) from the field model; best-firstAccept-Languageoutput ordering; a nullable facet
labelresolved for reference facets only.printGraphQLSchemafor a consumer-side SDL snapshot guard.Notes
SearchTypeper NodeShape,SearchSchemaas the type-keyed map), theSearchEnginerename (the interface is the port;the Typesense class is the adapter),
size→Float(int64 overflow), the typed-surfacedesign, and facet labels.
@lde/search-api-graphqlrow and the architecturediagram gains the search family (
search,search-typesense,search-api-graphql,text-normalization).idOnly/inlinereference strategies, theOutputOf<S>typed-surfaceoverlay, and a REST surface.