plans: cross-cache-identity architecture pivot decision record#800
Merged
Merged
Conversation
Captures the architectural reasoning + 4-wave action sequence before
any artifacts are closed or filed. The pivot moves Backend out of the
metadata-resolution business: LML composes identity end-to-end,
Backend stores LML's verdict via HTTP only.
Sections:
- Why a pivot (the muddled current state, the two findings during
sub-PR 2.1 design that made it concrete)
- What we're going for (Backend = catalog + cache; LML = sole
resolution authority; HTTP-only contract surface)
- Backend read performance under the pivot (cache contents stay,
only the population mechanism changes)
- What changes in LML (new bulk-resolve endpoint; composition rules
in Python; release-level matching ports over)
- Migration phases 0-7
- Costs and benefits
- What we keep vs throw away
- Action plan with parallelization (Wave 1: gh state changes; Wave
2: new tickets; Wave 3: wiki amendment; Wave 4: close out)
- Open decisions for execution
- Reversibility analysis
- Success criteria
This is a decision record, not yet an implementation. Once Wave 4
lands the pivot is in motion across BS/LML/wxyc-shared/wiki.
This was referenced May 10, 2026
jakebromberg
added a commit
that referenced
this pull request
May 10, 2026
The pre-pivot plan committed Backend to a five-source backfill (S1=canonical_entity_id, S2=LML entity.identity, S3=flowsheet_match, S4=fuzzy_resolved, S5=semantic-index reconciliation_log). On 2026-05-09 the project pivoted: Backend stops reading LML's PG and consumes a new POST /api/v1/identity/bulk-resolve-libraries endpoint instead. Replaces the 277-line draft plan with a stub redirecting to the decision record (#800), the wiki amendment (WXYC/wiki#25), and the new tickets (wxyc-shared#103, library-metadata-lookup#272, #802). Captures what survives (substrate from #790, S1 self-migration leg, 2.2 spike memo) and what's discarded (dispatch.ts, lml-provenance-index.ts, resolve-s2.ts). Wave 4 of the cross-cache-identity architecture pivot.
This was referenced May 10, 2026
Open
Open
jakebromberg
added a commit
to WXYC/wxyc-shared
that referenced
this pull request
May 10, 2026
…identity pivot Per the cross-cache-identity architecture pivot (WXYC/Backend-Service#800, 2026-05-09), Backend stops composing cross-cache identity from multiple PG schemas; LML becomes the sole composer. This commit locks the HTTP contract for the new seam: POST /api/v1/identity/bulk-resolve-libraries. Adds 7 schemas (BulkResolveLibrariesRequest, BulkResolveInput, BulkResolveResultKind, BulkResolveProvenanceEntry, BulkResolveTrackIdentity, BulkResolveResult, BulkResolveLibrariesResponse), 1 endpoint, 1 security scheme (LMLBearerAuth, distinct from BearerAuth which is Better Auth JWT). Reuses ReconciledIdentity, IdentitySource, IdentityMethod, CacheStats, ApiErrorResponse for cross-spec consistency. Request shape is uniform per #103: {library_id, artist_name, album_title}. LML auto-detects V/A from library_id and returns a kind discriminator on the response (single_artist | compilation | unresolved). Provenance always returned (empty array if no source resolved); no optional knobs. Compilation results carry per-track identity in tracks[]; full V/A matcher behaviour ships under WXYC/library-metadata-lookup#271. info.version bumped 1.1.0 to 1.2.0. No breaking changes (additions only).
8 tasks
This was referenced May 11, 2026
jakebromberg
added a commit
that referenced
this pull request
May 11, 2026
Implements BS#802 under the post-#800 cross-cache-identity pivot: Backend is now a thin writer; LML is the sole composer of cross-cache identity. The new `jobs/library-identity-consumer/` package consumes LML's `POST /api/v1/identity/bulk-resolve-libraries` endpoint (api.yaml v1.2.0, wxyc-shared#104; deployed via LML#272 / PR #273) and UPSERTs the verdicts into `library_identity` + `library_identity_source` atomically per library_id. The SELECT predicate picks libraries needing identity refresh: `library.canonical_entity_id IS NOT NULL OR library.id IN (SELECT library_id FROM library_identity WHERE last_verified_at < NOW() - interval '7 days')`. Note: BS#802's body wrote `last_refreshed_at`, but the actual column on `library_identity` is `last_verified_at`; the code uses the real column name. Batches up to 500 inputs per LML call (LML caps at 1000). For each `BulkResolveResult`: `single_artist` → write per-source rows + main row in `db.transaction()`; `unresolved` → counted, no write; `compilation` → counted as `rows_skipped { compilation }` and deferred to BS#801 (per-track identity writes for V/A rows). A batch-level LML error counts every input as `rows_skipped { lml_error }` and continues — the next run re-picks failed rows via the SELECT predicate, so retry is free. Sentry-traced metrics (`rows_resolved`, `rows_unresolved`, `rows_skipped`, `lml_total_calls`, `lml_total_latency_ms`) land as attributes on a top-level `library-identity-consumer.run` span. The per-batch LML POST is wrapped in `lml.bulk_resolve_libraries` (`http.client`); LML's `cache_stats` projects onto the same span as `lml.cache.*` attributes (LML#229 pattern). DRY_RUN still calls LML so the resolve/unresolved/error counts are honest predictions; only DB writes are suppressed, and a locked-schema JSON object is emitted on stdout. Deletes the predecessor `jobs/library-identity-backfill/` and `Dockerfile.library-identity-backfill` in the same change (BS#802 acceptance). The deploy-base.yml matrix discovers targets dynamically via `jobs/$TARGET_APP/package.json`'s `job-type` field, so registering the new one-shot target requires no workflow edit. CLAUDE.md and docs/env-vars.md are updated to reflect the rename and the new env vars (`LIBRARY_METADATA_URL`, `LML_API_KEY`, `STALE_THRESHOLD_DAYS`). Known scope cuts called out for follow-up: `library_identity` has no main-row destinations for `discogs_artist_id`, `musicbrainz_artist_id`, or `bandcamp_id` from LML's `ReconciledIdentity` payload — those values flow through `library_identity_source.external_id` (text) via provenance rows so no data is dropped, but the main row is a partial denormalised view until a follow-up migration adds artist-id columns (deliberately out of scope here). Compilation handling is BS#801's scope. Test coverage (34 tests across select / writer / orchestrate): SELECT predicate honors the post-#800 disjunction and env var validation; writer is transactional with per-source-then-main ordering and null-confidence skipping; orchestrator dispatches by kind, counts LML and writer errors without aborting the run, paginates by id-cursor, and emits the locked DRY_RUN JSON schema.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Decision record capturing the architectural reasoning behind pivoting cross-cache-identity §4 step 2. The pivot moves Backend out of the metadata-resolution business: LML composes identity end-to-end via a new bulk-resolve endpoint; Backend stores LML's verdict locally for read performance via HTTP only, no cross-DB reads.
This PR is the decision record itself. It does not execute the pivot. Wave 1 (closures + reframes) follows in separate PRs/issue actions linked from this doc.
Why now
While building sub-PR 2.1 (the S2 LML reader), two findings made the cost of the original architecture concrete:
flowsheet_matchandfuzzy_resolvedaren't part of LML's discogs-cache schema — Backend's own scripts materialize them inside LML's PG. So Backend wasn't reading LML's data; it was reading data Backend's scripts wrote into LML's PG.entity.reconciliation_logalready carries per-row(method, confidence). The original "blanket alias_match 0.85" interim was unnecessary — but reaching this conclusion required Backend to read yet another LML schema (reconciliation_log) directly.Each new source leg added another LML schema Backend depended on. The architectural debt was compounding fast enough that catching it pre-2.1-merge avoids ~1500 LOC of unwritten work that would have to be undone later.
What the doc covers
Action sequence after this lands
This doc is a precondition for Wave 1. Once it's reviewable:
Each wave's specific actions are listed in the doc.
Review notes
This is a planning artifact, not code. Looking for:
If everything looks right, merging this as-is gives Wave 1 a public URL to cite when closing #797 et al.
Refs #663