Phase 1: WebGL engine — GPU compositor, shader effects, real crossfades

[drftstatic]

What

Phase 1 (The Engine) — complete. Five milestones: the CSS-filter era is over, the app has a musical clock, a GPU effect rack, and a real output pipeline.

1. WebGL2 compositor (client/src/engine/)

Portable engine module — no React, no store, no DOM beyond the canvas (PRD §5.5: carries unchanged into the Electron/iPad shells). Pipeline: MIX (two video decks as GPU textures, contain-fit, geometry transforms, A/B crossfader) → FX über-shader (full color grade, glitch, chroma, pixelate, blur/sharpen, noise, vignette, scanlines — identity at defaults, the Radiance contract) → rack chain → FEEDBACK (ping-pong Trails/Warp). The Player renders through the compositor canvas; CSS filter strings and overlay-div hacks are deleted.

2. Real crossfades (engine/deck-transition.ts)

Advances arm a hidden transition deck, mix A→B on the GPU with an eased audio ramp, then hand the clip back to the program deck invisibly. crossfade: true → 1.5s mix; hard cuts get a 0.35s micro-mix — the black load-gap between clips is gone. Error-safe fallback, clean cancellation.

3. Beat clock + quantized triggering (engine/beat-clock.ts)

Onset detection, confidence-tracked BPM estimation (60–180), drift-corrected phase, tap tempo. Compositor samples it per frame (uBeatPhase); nextTrack/previousTrack route through a quantize scheduler (rAF-accurate + setTimeout backstop). BeatMeter in the status bar: pulsing dot, BPM, TAP, Q toggle.

4. Effect rack (engine/effects.ts + RackPanel)

Six chainable single-pass GPU effects — Kaleidoscope, Mirror, Posterize, Wave, Zoom Pulse (beat-driven), VHS — one intensity knob each, identity at zero, beat-phase aware. The panel UI is generated from the engine registry; the registry shape is ISF-adjacent so a community-shader importer can populate it later.

5. Program output + set recording (lib/program-output.ts)

The compositor canvas renders once; captureStream feeds everything. A minimal same-origin projector window (double-click fullscreen) replaces the 734-line dual-player PopOutPlayer and its postMessage sync. SetRecorder captures the actual program mix (video + WebAudio tap) to a downloadable WebM, replacing the getDisplayMedia screen-picker.

Verified live (not just typechecked)

• 60 fps with glitch + chroma + trails + scanlines stacked; clean identity passthrough
• Mid-crossfade frame shows both clips blending; handback leaves deck A owning the clip
• Synthetic 500 ms taps → 120 BPM, msToNextBeat = 500; quantized advance defers and lands the cut
• Rack stack (kaleido + VHS + wave + zoom-pulse) rendering on playing video
• 19.6 s program recording started/stopped cleanly; program stream exposes its video track
• tsc and both builds pass

Reviewer notes

• engine/params.ts mirrors the store's CSS-era VideoEffects shape so existing panels drive the GPU pipeline unchanged.
• Dev builds expose window.__store, window.__beatClock, window.__programOutput for console-driven testing.
• Two things the embedded preview pane could not verify (environment limits, not code paths): exact beat-landing latency (pane throttles rAF/timers when idle) and the projector window (pane blocks popups even on real clicks — it's standard window.open + srcObject). Please try both in a real browser: tap a tempo, flip Q, mash next; and hit the output-window button with a video playing.
• Phase 1.5 next per the PRD: the Electron shell (Syphon/Spout, offline set cache) — plus ISF import and MIDI learn in Phase 2.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request replaces the CSS-based video filters and overlays with a high-performance WebGL2 compositor pipeline (Compositor) and a deck-to-deck crossfade transition engine (deck-transition.ts). It also introduces new temporal effects (trails and warp) to the UI, store, and types. The review feedback highlights several critical areas for improvement: ensuring proper audio and resource cleanup when a crossfade is cancelled, preventing a WebGL resource leak by deleting the Vertex Array Object on destruction, optimizing GPU texture uploads by caching video frame times, and capping the maximum rendering resolution to prevent performance degradation on high-DPI displays.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

If a crossfade is cancelled (e.g., when skipping tracks rapidly or stopping), deckA (the main program deck) can be left in a muted state (deckA.muted = true) or with a reduced volume level from the incomplete audio ramp. To ensure audio playback is correctly restored and resources are cleaned up, the cancel handler should restore deckA's volume and unmuted state, and clean up deckB's source.

    cancel: () => {
      signal.cancelled = true;
      setCrossfade(0);
      deckB.pause();
      deckB.removeAttribute('src');
      deckB.load();
      deckA.muted = false;
      deckA.volume = volume;
    },

[gemini-code-assist]

The destroy() method does not delete the WebGL Vertex Array Object (this.quad) created in the constructor. This will cause a WebGL resource leak every time the compositor is recreated (which happens whenever currentVideo flips). We should explicitly delete the VAO using gl.deleteVertexArray(this.quad).

Suggested change

	destroy() {
	this.stop();
	const gl = this.gl;
	for (const t of this.targets) destroyTarget(gl, t);
	if (this.feedback) { destroyTarget(gl, this.feedback[0]); destroyTarget(gl, this.feedback[1]); }
	gl.deleteTexture(this.deckA.texture);
	gl.deleteTexture(this.deckB.texture);
	gl.deleteProgram(this.progMix);
	gl.deleteProgram(this.progFx);
	gl.deleteProgram(this.progFeedback);
	gl.deleteProgram(this.progBlit);
	}
	destroy() {
	this.stop();
	const gl = this.gl;
	for (const t of this.targets) destroyTarget(gl, t);
	if (this.feedback) { destroyTarget(gl, this.feedback[0]); destroyTarget(gl, this.feedback[1]); }
	gl.deleteVertexArray(this.quad);
	gl.deleteTexture(this.deckA.texture);
	gl.deleteTexture(this.deckB.texture);
	gl.deleteProgram(this.progMix);
	gl.deleteProgram(this.progFx);
	gl.deleteProgram(this.progFeedback);
	gl.deleteProgram(this.progBlit);
	}

[gemini-code-assist]

Currently, uploadDeck uploads the video texture via gl.texImage2D on every single frame, even if the video is paused or the playback time hasn't advanced. Uploading video textures to the GPU is a highly expensive operation. We can optimize this by caching the currentTime of the last uploaded frame and skipping the upload if the time hasn't changed (except for live streams where srcObject is active).

  private uploadDeck(deck: Deck & { lastTime?: number }) {
    const gl = this.gl;
    const v = deck.video;
    if (!v || v.readyState < 2 || v.videoWidth === 0) return;
    if (v.currentTime === deck.lastTime && !v.srcObject) return;
    gl.bindTexture(gl.TEXTURE_2D, deck.texture);
    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA8, gl.RGBA, gl.UNSIGNED_BYTE, v);
    deck.width = v.videoWidth;
    deck.height = v.videoHeight;
    deck.lastTime = v.currentTime;
  }

[gemini-code-assist]

On high-resolution displays (like 4K monitors or Retina screens), the canvas drawing buffer size can become extremely large (e.g., 7680x4320 with DPR 2). Running a 3-pass WebGL pipeline at such resolutions will severely degrade performance and can cause GPU crashes. It is highly recommended to cap the maximum rendering resolution to a sensible limit (such as 1920x1080) and let the browser upscale the canvas using CSS.

Suggested change

	// Size canvas to display size (device-pixel aware, capped for perf)
	const dpr = Math.min(window.devicePixelRatio || 1, 2);
	const w = Math.max(2, Math.floor(canvas.clientWidth * dpr));
	const h = Math.max(2, Math.floor(canvas.clientHeight * dpr));
	if (canvas.width !== w || canvas.height !== h) {
	canvas.width = w;
	canvas.height = h;
	}
	const dpr = Math.min(window.devicePixelRatio || 1, 2);
	let w = Math.max(2, Math.floor(canvas.clientWidth * dpr));
	let h = Math.max(2, Math.floor(canvas.clientHeight * dpr));
	const MAX_WIDTH = 1920;
	const MAX_HEIGHT = 1080;
	if (w > MAX_WIDTH || h > MAX_HEIGHT) {
	const aspect = w / h;
	if (aspect > MAX_WIDTH / MAX_HEIGHT) {
	w = MAX_WIDTH;
	h = Math.round(MAX_WIDTH / aspect);
	} else {
	h = MAX_HEIGHT;
	w = Math.round(MAX_HEIGHT * aspect);
	}
	}
	if (canvas.width !== w || canvas.height !== h) {
	canvas.width = w;
	canvas.height = h;
	}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
staticbuffet	Canceled		Jul 20, 2026 1:28am