feat: decode inline Arrow IPC + warehouse-compat fallback

[jamesbroadhead]

Summary

Some warehouses return ARROW_STREAM + INLINE results as base64 Arrow IPC in result.attachment rather than result.data_array. Some refuse JSON_ARRAY + INLINE entirely and only support ARROW_STREAM for INLINE. Others (most classic + some serverless) do the opposite — they refuse ARROW_STREAM + INLINE and require EXTERNAL_LINKS. The previous code path silently returned empty results in all of these cases.

This PR makes the analytics plugin work across all three warehouse shapes by:

• handling both INLINE and EXTERNAL_LINKS dispositions for ARROW_STREAM,
• stashing inline Arrow IPC server-side and serving it via the existing /arrow-result route,
• falling back between dispositions based on what the warehouse signals it accepts,
• preserving the caller's requested format end-to-end (a JSON_ARRAY caller always gets JSON rows; an ARROW_STREAM caller always gets Arrow bytes), regardless of which disposition the warehouse actually accepted.

Design

Earlier iterations of this PR sent inline Arrow bytes over SSE (base64 inside JSON inside SSE framing) using an arrow_inline message type. That was reworked: inline payloads are now stashed server-side and delivered through the same /arrow-result/<jobId> endpoint that already serves EXTERNAL_LINKS results. The SSE control channel only ever carries { type: "arrow", statement_id }; the id is either a real warehouse statement id or a synthetic inline-<uuid>, and the server demultiplexes.

Wins from the redesign:

• Keeps multi-MiB blobs off SSE
• One client code path for both INLINE and EXTERNAL_LINKS
• Real binary content-type (application/vnd.apache.arrow.stream) instead of base64-in-JSON-in-SSE
• The retired arrow_inline wire message is gone — the discriminated union rejects it as schema-invalid

Stash properties (see inline-arrow-stash.ts):

• Drain-on-read (no replay path)
• TTL-bounded, evicted on put/take
• Per-user keyed
• Memory-bounded with rejection on overflow — callers fall back to EXTERNAL_LINKS rather than evicting in-flight entries
• Caveat: process-local. Multi-replica deployments need sticky sessions or a shared store (out of scope)

Disposition fallback

The plugin classifies an inline rejection into one of two signals (case-insensitive, requires INVALID_PARAMETER_VALUE / NOT_IMPLEMENTED plus mention of "inline" — unrelated errors aren't reinterpreted as disposition mismatches):

• needs-arrow: warehouse says it only accepts ARROW_STREAM for INLINE ("Inline disposition only supports ARROW_STREAM format."). If the caller asked for JSON_ARRAY, retry as ARROW_STREAM + INLINE and decode the Arrow IPC attachment to plain row objects server-side — the caller's JSON contract is preserved. Scalar values are stringified to match the warehouse's native JSON_ARRAY shape.
• needs-json: warehouse says it only accepts JSON_ARRAY for INLINE ("The format field must be JSON_ARRAY when the disposition field is INLINE.", "ARROW_STREAM is not supported with INLINE disposition"). If the caller asked for ARROW_STREAM, retry as ARROW_STREAM + EXTERNAL_LINKS.

For ARROW_STREAM callers, the stash-full case also falls back to EXTERNAL_LINKS. Explicit format requests are honored — no auto-downgrade across format boundaries (a JSON caller never gets Arrow bytes; an Arrow caller never gets JSON rows).

Changes

• Server-side Arrow IPC decoding (connectors/sql-warehouse/arrow-schema.ts + client.ts): detects result.attachment and decodes via apache-arrow's tableFromIPC. apache-arrow@21.1.0 added as a server dependency.
• Inline Arrow stash (plugins/analytics/inline-arrow-stash.ts): the bounded, TTL'd, drain-on-read, per-user keyed store described above.
• Unified /arrow-result route (plugins/analytics/analytics.ts): single endpoint serves both warehouse statement ids and synthetic inline-* ids; differentiation is by id prefix.
• Disposition classifier + bidirectional fallback (plugins/analytics/analytics.ts): replaces the prior heuristic, which case-sensitively required both INLINE and ARROW_STREAM in the error message and never matched the actual wire wordings.
• JSON_ARRAY server-side Arrow→rows decoder: when retrying with ARROW_STREAM, decode and stringify so the SSE payload still looks like {type: "result", data: [...rows]}.
• Zod-validated SSE wire protocol (shared/src/sse/analytics.ts): single source of truth between server and client. Default format remains JSON_ARRAY.

Tests

• connectors/sql-warehouse/tests/arrow-schema.test.ts (new, 514 lines)
• connectors/sql-warehouse/tests/client.test.ts (new, 383 lines) — real base64 Arrow IPC captured from a live serverless warehouse
• plugins/analytics/tests/inline-arrow-stash.test.ts (new, 116 lines)
• plugins/analytics/tests/analytics.test.ts (+770) — integration coverage for both fallback directions (warehouse refuses JSON_ARRAY + INLINE → server retries as ARROW_STREAM + INLINE and decodes; warehouse refuses ARROW_STREAM + INLINE → falls back to EXTERNAL_LINKS), plus stash-full fallback and the no-fallback safety net
• appkit-ui/src/react/hooks/__tests__/use-analytics-query.test.ts — arrow-IPC SSE tests (this PR) merged with the stable-params tests already on main from fix(appkit-ui): stabilize useAnalyticsQuery params reference to avoid infinite refetch #321
• shared/src/sse/analytics.test.ts (new, 87 lines)

Full suite: 2,674 tests, all green on the current head.

Stack

This was the third PR in a stack; both predecessors are now on main:

• test: backfill coverage for genie connector, service context, stream registry #327 — coverage backfill (merged)
• feat: rename AnalyticsFormat to API enum names with legacy aliases #328 — AnalyticsFormat rename to API enum names (merged)
• (this PR) — the actual fix

Test plan

Verified via automated tests on the current head:

• useAnalyticsQuery reads ARROW_STREAM results via /arrow-result for both real warehouse statement ids and synthetic inline-* ids
• JSON_ARRAY direct path: warehouse accepts INLINE + JSON_ARRAY → row data returned unchanged
• JSON_ARRAY fallback path: warehouse refuses INLINE + JSON_ARRAY but accepts INLINE + ARROW_STREAM → server decodes Arrow attachment to JSON rows; SSE wire carries {type: "result"}, not {type: "arrow"}
• JSON_ARRAY no-fallback safety: rejection without a needs-arrow signal does NOT trigger a retry
• ARROW_STREAM + INLINE → EXTERNAL_LINKS fallback when the warehouse rejects INLINE
• ARROW_STREAM + INLINE → EXTERNAL_LINKS fallback when the inline stash is full
• Inline-stash invariants: drain-on-read, TTL eviction, per-user keying, reject-on-overflow leaves prior entries intact
• Retired arrow_inline SSE message is rejected as schema-invalid and never triggers a local decode

Live smoke test against three e2-dogfood warehouses (SELECT 1 AS one, 'hello' UNION ALL SELECT 2, 'world'):

Warehouse	JSON_ARRAY caller (default)	ARROW_STREAM caller
serverless A (refuses JSON_ARRAY + INLINE)	✅ rejected on first attempt, fallback retries as ARROW_STREAM + INLINE, decoded to [{"one":"1","greeting":"hello"},{"one":"2","greeting":"world"}]	✅ INLINE + ARROW_STREAM accepted directly, attachment decoded
serverless B	✅ INLINE + JSON_ARRAY accepted directly	✅ INLINE rejected with needs-json, falls back to EXTERNAL_LINKS
classic	✅ INLINE + JSON_ARRAY accepted directly	✅ INLINE rejected with needs-json, falls back to EXTERNAL_LINKS

All three warehouses now return identical row shapes for the same default useAnalyticsQuery call.

Fixes #242. Replaces #256.

This pull request was AI-assisted by Isaac.

[MarioCadenas]

Some agentic review @jamesbroadhead Please take a look

Xavier Review (v3 @ 0f5022fb) — request changes

Three reviewer personas (correctness, security, performance) all returned request changes. Findings re-verified against the live worktree at PR head 0f5022fb. The critical stash-full double-execution issue from the v2 review (2026-05-13) persists into v3 and is now pinned by an explicit test.

Reviewer	Verdict	Findings
correctness	request changes	5 (1 high, 3 medium, 1 low)
security	request changes	8 (1 high, 5 medium, 2 low)
performance	request changes	8 (1 critical, 3 high, 2 medium, 2 low)

Blockers

🔴 [critical] Stash-full path doubles warehouse work AND can return a divergent result — pinned by test as intentional

File: packages/appkit/src/plugins/analytics/analytics.ts:391-396, 408-425, 452-458
Test asserting current behavior: packages/appkit/src/plugins/analytics/tests/analytics.test.ts:1074-1081

Flagged by all three reviewers. In the cap-overflow path:

1. INLINE + ARROW_STREAM executor.query completes and is billed (:391-396).
2. Multi-MiB Buffer.from(attachment, "base64") runs synchronously on the Node event loop (:408).
3. inlineArrowStash.put() returns null (:417) — the decoded bytes are thrown away.
4. A second executor.query runs with EXTERNAL_LINKS (:452-458) — billed again, and for any non-deterministic SQL (CURRENT_TIMESTAMP, RAND, late-arriving rows) the second result can differ from the discarded first one.

The new test pins this as deliberate — so this needs an explicit product decision before merging.

Suggestion (any of):

• Reserve stash capacity before executing.
• Reuse the already-decoded buffer for this request via a one-shot sidestep endpoint.
• Switch from reject-on-overflow to safe LRU eviction with in-flight refcount.
• Route to EXTERNAL_LINKS directly when stash utilization is near cap.

If divergent-result + double-billing is the accepted tradeoff, please document it on the SSE/route contract and add a counter so it's monitorable.

🟠 [high] Warehouse / SDK error text still reflected verbatim to clients (RECURRING — 5 PRs deep: #310, #197, #255, #329 v1/v2/v3)

Files:

• packages/appkit/src/errors/execution.ts:48-50 — Statement failed: ${errorMessage} body
• packages/appkit/src/stream/stream-manager.ts:288-308 — SSE serializer forwards error.message verbatim
• packages/appkit/src/plugins/analytics/analytics.ts:170-172 — /arrow-result warehouse-path 404 returns error.message (inline misses use a fixed string at :133-135)

Anyone reading the SSE stream or hitting /arrow-result for a missing warehouse id sees raw Databricks wording — statement fragments, object names, correlation IDs, internal paths. CWE-209.

v3 added structured errorCode propagation (good — already consumed by _classifyInlineRejection), but the human message still embeds raw upstream text.

Suggestion: Map upstream failures to a small set of stable, client-safe messages; keep full detail server-side only. UI should branch on errorCode, not message.

🟠 [high] JSON_ARRAY "needs-arrow" fallback may diverge from native JSON_ARRAY shape + heavy server-side materialization

File: packages/appkit/src/plugins/analytics/analytics.ts:373-382 (retry+decode), :591-625 (decodeArrowAttachmentToRows); shape comment at :607-620

Flagged by both correctness AND performance:

• Correctness: warehouse-native JSON_ARRAY stringifies scalars and represents nested fields as JSON strings; the fallback path stringifies scalars but explicitly leaves nested values "as-is" (acknowledged by code comment at :607-620). Callers see different shapes depending on which path executed.
• Performance: per-row Record build is O(rows × cols) CPU + allocation on the Node main thread. A 100k×50 fallback can cost hundreds of ms of blocking time + GC churn on top of the first failed INLINE attempt.

Suggestion: Align serialization with _transformDataArray / documented JSON_ARRAY semantics (e.g. JSON.stringify nested values), and cap row count for the fallback materializer with an explicit error past the cap.

🟠 [high] Client safeParse keeps O(rows × keys) Zod validation for large JSON results (RECURRING from v1/v2)

Files:

• packages/shared/src/sse/analytics.ts:30 — data: z.array(z.record(z.string(), z.unknown())).optional()
• packages/appkit-ui/src/react/hooks/use-analytics-query.ts:189 — AnalyticsSseMessage.safeParse runs on every SSE message after JSON.parse

10k–1M rows = hundreds of ms to seconds of main-thread TBT after JSON.parse.

Suggestion: Loose schema for result.data (z.array(z.unknown())), or branch on type === "result" before deep validation.

Other notable findings

• [medium] JSON.parse failure still leaves useAnalyticsQuery stuck in loading (RECURRING from v1/v2) — outer catch at `packages/appkit-ui

[jamesbroadhead]

👋 Hi Mario — Claude here, replying on James's behalf.

Pushed 4afce1f6 addressing every blocker in your v3 review. Closed #390 as superseded — the overflow pool subsumes that PR's putWait half-measure, so there's no value in keeping it open.

🔴 critical — stash-full double-execution + divergent result

Fixed at the root, not papered over. InlineArrowStash now has a second pool ("overflow") sized at maxOverflowBytes (default same as maxBytes). When the regular pool is at cap, the already-decoded bytes spill into overflow rather than being thrown away. Memory is bounded above by maxBytes + maxOverflowBytes.

• ✅ Single executor.query call — never re-runs on EXTERNAL_LINKS.
• ✅ No discarded multi-MiB decode work.
• ✅ No double-billing.
• ✅ No divergent-result possibility (the bytes the client gets are the bytes the warehouse returned the first time).
• ✅ Only when both pools hit cap does the route surface INLINE_ARROW_STASH_EXHAUSTED — clean error, never silent re-execution.

The pinning test … falls back to EXTERNAL_LINKS when the inline stash is full from v3 has been replaced with two tests:

• … spills the already-decoded bytes into the stash overflow pool when the regular pool is full — single execution, no double-billing
• … surfaces a stable error when both regular and overflow pools are exhausted — never silently double-bills

Files: packages/appkit/src/plugins/analytics/inline-arrow-stash.ts, analytics.ts:391-430, tests/inline-arrow-stash.test.ts, tests/analytics.test.ts.

🟠 high — raw warehouse / SDK error text reflected to clients (CWE-209)

Done. Added clientMessage to AppKitError (sanitized default per subclass) and a new errorCode field on the SSE error payload (SSEError interface). The stream-manager broadcaster now sends clientMessage over the wire — raw error.message stays in server logs only. Same treatment for the /arrow-result 410/404 paths. UI branches on errorCode (INLINE_ARROW_STASH_EXHAUSTED, NOT_IMPLEMENTED, etc.).

Test pin: should send a sanitized error event when handler throws — raw error text is kept server-side only asserts the raw Error.message (with a fake /internal/path + corrId=…) never appears anywhere on the SSE wire.

Files: packages/appkit/src/errors/base.ts, errors/execution.ts, stream/stream-manager.ts:287-340, stream/sse-writer.ts, stream/types.ts, plugins/analytics/analytics.ts:170-180.

🟠 high — JSON_ARRAY fallback shape + heavy materialization

Both legs fixed:

• Shape: nested values (List / Struct / Map) now go through JSON.stringify(v). apache-arrow's typed Vector values expose toJSON(), so the output matches the warehouse-native JSON_ARRAY shape (nested-as-JSON-string). Previous comment about "round-tripping would mismatch" is no longer accurate now that we lean on toJSON().
• Performance: added a hard JSON_ARRAY_FALLBACK_MAX_ROWS = 100_000 cap. Past the cap, the materializer throws RESULT_TOO_LARGE_FOR_JSON_FALLBACK with guidance to re-issue with format: "ARROW_STREAM" — instead of blocking the Node event loop for hundreds of ms.

File: packages/appkit/src/plugins/analytics/analytics.ts:591-660.

🟠 high — client safeParse O(rows × keys) Zod validation

Loosened AnalyticsResultMessage.data from z.array(z.record(z.string(), z.unknown())) to z.array(z.unknown()). The per-row shape is already enforced at the source by the typed makeResultMessage builder, so the deep client-side validation was pure cost. TS-level type narrows back to Record<string, unknown>[] for consumers via an Omit + intersection so call sites are unchanged.

File: packages/shared/src/sse/analytics.ts.

🟢 medium — JSON.parse failure stranded useAnalyticsQuery in loading=true

Fixed. The outer catch in the SSE message handler now sets loading=false and surfaces "Unable to load data, please try again" so consumers can render a retry affordance. New test pins the behavior.

Files: packages/appkit-ui/src/react/hooks/use-analytics-query.ts:260-269, __tests__/use-analytics-query.test.ts.

---

Truncated review tail: your comment ends mid-sentence at "outer catch at packages/appkit-ui" — looks like Xavier's output hit a size limit. The tally in the header table (correctness 5, security 8, performance 8 = 21 total) implies ~16 more medium/low findings beyond the blockers detailed above. Could you re-post the rest, or kick off a Xavier v4 run against 4afce1f` so the remaining items show up? Happy to roll a follow-up commit on top of this one as soon as I can see them.

Validation on this commit: full appkit suite (2,096 tests) + full appkit-ui suite (292 tests) green; tsc --noEmit clean on appkit, appkit-ui, shared; biome check clean on changed files.

[jamesbroadhead]

Self-review (Xavier multi-model: GPT-5.4 + Gemini-3.1) on 4afce1f6 surfaced 11 follow-ups against my own commit. All addressed in 28fdb928.

#	Severity	Finding	Fix
1	🔴 critical	decodeArrowAttachmentToRows regression: bare JSON.stringify throws on bigint and emits broken shapes for Uint8Array / Date	New nestedJsonReplacer (bigint → string, Uint8Array → base64, Date → ISO); top-level Date / Uint8Array handled explicitly so they never reach the nested fallback
2	🟠 high	Single-payload throw used maxBytes + maxOverflowBytes, but a payload only lands in one pool	Threshold is now Math.max(maxBytes, maxOverflowBytes)
3	🟠 high	Overflow inherited the 10-min regular TTL — long-lived cross-user memory pressure	Separate overflowTtlMs (default 30s); overflow exists only to bridge the immediate /arrow-result fetch
4	🟠 high	Overflow defaulted to maxBytes, doubling worst-case memory headroom	Default dropped to maxBytes / 4
5	🟠 high	JSON_ARRAY fallback row cap ignored cell size (10 × 10MB rows = OOM)	Added 64MB byte cap on the decoded attachment, checked before tableFromIPC
6	🟢 medium	gc() ran O(N) on every put	Throttled to gcMinIntervalMs (default 5s); tests opt out with 0
7	🟢 medium	table.getChild(name) walked schema for every cell	Hoisted into [name, vector] pairs once before the row loop
8	🟢 medium	Hook dropped the new structured errorCode from SSE error payloads	Added errorCode: string | null to UseAnalyticsQueryResult; populated from parsed.errorCode
9	🟢 medium	Outer SSE catch cleared loading but left the stream open	Now also calls abortController.abort()

Validation: 2,097 appkit + 292 appkit-ui tests pass; tsc clean on appkit / appkit-ui / shared; biome clean on changed files.