perf(core): smooth cut-boundary playback — mount-time audio pre-decode, upcoming-clip readiness, edge gain ramps - #2895
Conversation
…p in WebAudio transport - decodeAudioElement: first fetch per src uses the normal HTTP cache (media URLs are stable; edited assets ship under new URLs). no-store is applied only on a RETRY after a failed attempt, preserving the late-asset self-heal semantics from heygen-com#1602 without re-downloading identical bytes on every fresh document. - schedulePlayback: ~8ms linear gain ramps at each source's effective start and (for bounded clips) into its scheduled end, so adjacent trimmed clips hand off with a micro-crossfade instead of a waveform-discontinuity pop. - stopAll: fade to zero and defer the stop ~20ms instead of hard-cutting mid-waveform; node teardown moves to the ended listener.
… telemetry Cut-boundary playback smoothness for segment-heavy compositions (an editor emitting one media element per cut): - init: decode-only warm pass over audio[data-start] as soon as the WebAudio transport exists (and periodically for late-loaded sub-compositions), so first play after a document (re)load doesn't run full-file decodeAudioData lazily while sound falls back to the buffering HTMLMedia element. - init: staged preload — when a composition carries >6 timed media elements, far-future segments (start >10s) begin preload=metadata; the sync layer upgrades them as the playhead approaches. Stops the mount-time fetch stampede that leaves EARLY segments buffering late. - media: two readiness stages for upcoming clips while playing: pre-seek to the clip's media offset ~3s ahead (buffer the RIGHT region), and a muted video pre-roll ~350ms ahead so activation is a visibility flip instead of a cold seek+play (decoder reset ~150ms freeze). - init: bounded runtime_media_stall diagnostics (waiting/stalled while playing) + freeze the transport clock when a buffering ACTIVE video has no audio mastering the clock (video-only compositions), mirroring the existing audio-buffering freeze.
…on pagehide The mount-time warm pass fetched every timed audio source's WHOLE file and held its decoded PCM (~23MB per stereo minute) — on every document load. In an editor that reloads the preview iframe per edit (and keeps a standby iframe), a multi-clip composition multiplied that into gigabytes and could OOM the tab. Pre-decode now warms only clips near the current time (the periodic re-run slides the window with the playhead; play() decodes exactly what it needs as before), embedders can opt a never-played document out via window.__HF_AUDIO_PREDECODE_DISABLED, and teardown (which closes the AudioContext and drops the buffer cache) also runs on pagehide — the dependable unload signal for iframes whose document is replaced.
Two independent OOM paths in the WebAudio decode pipeline, both hit by a composition whose audio rides inside large camera-video containers: - No in-flight dedup: decoding a long source takes seconds, while the warm pass / play() / rate changes may re-request the same src many times a second. Each re-request launched ANOTHER whole-file fetch — dozens of concurrent copies of the same source could stack up before the first decode ever reached the buffer cache. decodeAudioElement now coalesces concurrent requests per src onto one fetch+decode promise. - No size bound: WebAudio decode requires the ENTIRE file in an ArrayBuffer. Sources beyond 96MB are now permanently skipped for the session (content-length short-circuit before any bytes buffer, plus a capped stream read when length is undeclared) and keep playing through the streamed HTMLMediaElement path instead.
There was a problem hiding this comment.
The direction is strong: the near-playhead warm window in packages/core/src/runtime/init.ts:1923-1967 bounds speculative decode work, and the per-source pending-decode map in packages/core/src/runtime/webAudioTransport.ts:202-216 correctly coalesces the most expensive concurrent path. The two issues below can leave playback incorrect, though.
blocker — packages/core/src/runtime/media.ts:511-523: pre-seek is marked complete before a seek actually occurs. When the upcoming element has not reached HAVE_METADATA, this adds it to upcomingPreseeked and skips the assignment. Later ticks see the set membership and never retry, so the clip can still hit its boundary at the wrong offset and pay the cold seek this PR is meant to remove. Only mark the element pre-seeked after metadata exists and currentTime was successfully assigned (or attach a one-shot metadata handler); add a regression where the first readiness tick is below HAVE_METADATA and a later tick becomes seekable.
blocker — packages/core/src/runtime/init.ts:3439-3446: unconditional pagehide teardown breaks BFCache restores. pagehide also fires with PageTransitionEvent.persisted === true when the page enters the back-forward cache. This permanently sets state.tornDown, removes listeners/styles, and closes the audio context; returning via pageshow does not rerun initialization, leaving the restored player dead. Keep the editor/srcdoc cleanup for non-persisted page hides, but skip destructive teardown for BFCache (or suspend/resume via paired pagehide/pageshow handlers), with a persisted-pagehide regression test.
Non-blocking, low-risk next gains
packages/core/src/runtime/webAudioTransport.ts:278-292: when a validContent-Lengthis available, fill one capped/preallocatedUint8Arraywhile streaming instead of retaining chunks and then allocating/copying a second full buffer. Keep the same overflow cancellation check. This lowers peak compressed-byte memory and removes a full-file copy without changing scheduling behavior.packages/core/src/runtime/init.ts:1947-1967,2989-2994: avoid a document-widequerySelectorAllevery 30 transport ticks. Maintain a media registry or a MutationObserver-backed dirty flag and rescan only after DOM changes / WebAudio readiness or when the playhead advances enough to move the 45-second warm horizon. Same warm coverage, less steady-state main-thread work.- Deduplicate the warm pass by resolved
srcbefore callingdecodeAudioElement. The pending/cache maps already protect fetch/decode, but this also avoids repeated source resolution and promise churn for segment-heavy edits where many elements point at one asset. - Add sampled timings for fetch, decode, pre-seek-to-ready, and boundary stall outcome. That will show whether the next bottleneck is network,
decodeAudioData, or video frame readiness and lets the windows be tuned from evidence rather than widened (which would trade memory/network for latency).
Verdict: REQUEST CHANGES
Reasoning: The performance strategy is sensible and bounded, but the readiness latch can silently skip its only seek and unconditional BFCache teardown can leave restored playback permanently inert.
…ock-freeze Under memory pressure the browser can evict a media element's resource with no error and no event — readyState collapses to HAVE_NOTHING, videoWidth to 0, and an active clip goes permanently blank. syncRuntimeMedia now detects that shape on active clips (resource present, not fetching, rate-limited) and load()s + reseeks it, letting the normal play path re-issue play(). The video-buffering clock freeze from the previous commit is removed: on an evicted element it never released — a permanent playback hang worse than the transient lag it smoothed. Video stalls surface via runtime_media_stall diagnostics and the eviction recovery instead.
2 participants
Summary
Smooths playback across cut boundaries in segment-heavy compositions (an editor that removes silences / cuts a source video emits one
<video>/<audio>element per segment, all pointing at the same source with differentdata-playback-startoffsets). Today each boundary pays a coldcurrentTimeseek +play()on the incoming element ("Seeking a playing video resets the decoder, causing a ~150ms freeze" — media.ts's own comment), and the first play after a document load runs a lazy full-filedecodeAudioDataper audio source while sound falls back to the still-buffering HTMLMedia element (heard as a dropout).Two commits:
webAudioTransport(audio)decodeAudioElement: the first fetch per src now uses the normal HTTP cache — media URLs are stable (an edited asset ships under a new URL), sono-storeon first touch just re-downloads identical bytes on every fresh document.no-storeis kept for retries after a failed attempt, fully preserving the late-asset self-heal semantics introduced in fix(core): mute preview audio per-element so a slow-decoding track isn't silenced #1602 (a retry must not replay a cached 404).stopAll()fades to zero and defers the stop ~20ms instead of hard-cutting mid-waveform; node teardown moves to theendedlistener.init/media(decode timing + video boundaries)audio[data-start]as soon as the WebAudio transport exists (and periodically, for late-loaded sub-compositions), so decode overlaps document load instead of blocking the firstplay(). Skipped in render-capture mode (audio is mixed by ffmpeg).preload="metadata"; the sync layer upgrades them as the playhead approaches. Stops the mount-time fetch stampede that leaves EARLY segments buffering late.syncRuntimeMedia: pre-seek an upcoming clip to its media offset ~3s ahead (buffer the RIGHT region of the source, not offset 0), and a muted video pre-roll ~350ms ahead — the incoming element rolls in frommediaStart − lead·rate, so it crosses its boundary at exactlymediaStart, already playing: activation becomes a pure visibility flip (no cold seek, no decoder reset, noplay()latency). Pre-roll is only entered when the element is (or would be) muted, so audio can never leak.runtime_media_stalldiagnostics (waiting/stalledwhile playing), and the transport clock now freezes on a buffering ACTIVE video when no audio is mastering the clock (video-only compositions) — mirroring the existing audio-buffering freeze.Testing
media.test.ts/webAudioTransport.test.ts/init.test.ts(12 new: fetch cache policy, edge ramps, fade-out stop, pre-seek/pre-roll windows and their guard rails, mount-time pre-decode).src/runtimesuite: 42 files / 830 tests green;typecheck:runtimeclean;build:hyperframes-runtimeproduces a working IIFE.No public API changes; render/export paths untouched (pre-decode and pre-roll are preview-transport-only behaviors, guarded off in render-capture mode where relevant).