Skip to content

Expose per-record delete/forget, external-id lookup, and relationships via the REST server (API parity) #75

Description

@dcfocus

Motivation

The REST server added in #66 exposes only a subset of what the core library and
the Python Context API already support. Applications that adopt the server as
their integration surface cannot perform several first-class record operations
and must fall back to the in-process API or open the Lance dataset directly.

Current server routes

GET    /api/v1/health
POST   /api/v1/contexts                      # create
GET    /api/v1/contexts                      # list
GET    /api/v1/contexts/{name}
DELETE /api/v1/contexts/{name}               # delete whole context
POST   /api/v1/contexts/{name}/records       # add batch
GET    /api/v1/contexts/{name}/records       # list
GET    /api/v1/contexts/{name}/records/{id}  # get by internal id
POST   /api/v1/contexts/{name}/search
POST   /api/v1/contexts/{name}/retrieve
GET    /api/v1/contexts/{name}/version
POST   /api/v1/contexts/{name}/checkout
POST   /api/v1/contexts/{name}/compact
GET    /api/v1/contexts/{name}/compact/stats

Capabilities present in the library but missing from the server

The core ContextStore / Python Context provide these, with no equivalent
server route:

  • Per-record delete / forget: delete_by_id, delete_by_external_id, and
    Python forget(...) logically delete a record by writing a tombstone. The
    server can delete an entire context, but not a single record.
  • Get by external id: get_by_external_id(...) resolves the caller-supplied
    stable identifier documented in the Python API and README. The server's
    records/{id} route resolves internal ids only.
  • Relationship traversal: list_related(...) / Python related(...) from
    the relationships feature is not exposed through REST.

The same gap exists at the lance-context-api trait level, which currently
declares only add / get / list / search / retrieve / version / checkout / compact / compaction_stats.

Proposal

Bring the server, remote client, and API trait to parity with the core library
for these existing record-level operations:

  • DELETE /api/v1/contexts/{name}/records/{id}: logically delete a record by
    internal id.
  • DELETE /api/v1/contexts/{name}/records?external_id=...: logically delete a
    record by caller-supplied external id.
  • GET /api/v1/contexts/{name}/records/by-external-id?external_id=...: fetch a
    record by external id.
  • GET /api/v1/contexts/{name}/records/related?target_id=...&relation=...&limit=...:
    list records with relationships pointing at a target, honoring lifecycle
    visibility options.

Query parameters are preferable for external_id and relationship targets
because these caller-supplied identifiers often look like source URIs or paths
and may contain /, #, or ?.

Scope for a first PR

  • Add the routes above with request/response DTOs consistent with existing
    records/search handlers.
  • Extend the lance-context-api trait and remote client so callers can use the
    same operations through either local or remote stores.
  • Add focused tests for delete by id, delete by external id, external-id lookup,
    and related-record traversal.

Non-goals

Backward compatibility

Purely additive; existing routes and payloads are unchanged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions