GPU compositor refactor with masks, effects, and preview fixes#118
Merged
GPU compositor refactor with masks, effects, and preview fixes#118
Conversation
#98) * fix(preview): prevent quality dropdown from capturing Space play/pause Block Space key on the quality SelectTrigger so it doesn't open the dropdown, and use capture-phase event listener for the play/pause hotkey to ensure it fires before Radix UI consumes the event. Also defer blur calls via requestAnimationFrame to avoid focus races when the dropdown closes. * fix(preview): apply same Space key and blur fixes to zoom dropdown Rename handler to handleSelectTriggerKeyDown and reuse it on the zoom SelectTrigger. Defer zoom blur calls via requestAnimationFrame and add onCloseAutoFocus to the zoom SelectContent to match the quality dropdown fixes.
…#99) * fix(export): align IO trim conversion with preview timing * fix(export): route source frame conversion through deps adapter
…heels (#102) * feat(text): add italic and underline style support * feat(effects): add color grading system with LUT, curves, and wheels Introduce a color-grading effect type with three variants: - LUT: built-in preset looks (cinematic, teal-orange, film stocks, etc.) with intensity control - Curves: per-channel tone curves with interactive control points and slider-based shadows/midtones/highlights/contrast adjustments - Wheels: 3-way color wheels (shadows/midtones/highlights) with temperature, tint, and saturation global controls Also adds a color scopes panel toggle in the preview area and upgrades the frame capture pipeline to support scopes and thumbnail workflows. * feat(effects): add custom .cube LUT import with OPFS library and canvas rendering Support importing, saving, and applying custom .cube LUT files in the color grading pipeline. LUTs are persisted in OPFS via a library manager, parsed with a dedicated .cube parser, and rendered through both GPU (WebGL) and CPU fallback paths during export. The preview pipeline forces the fast-scrub overlay when custom cube LUTs are active to ensure accurate per-frame color grading. * fix(effects): route media-library access through deps seam * fix: address PR review findings across color grading and effects UI - Add aria-label and aria-pressed to text style toggle buttons - Guard loadSavedCubeLuts with try/catch to prevent unhandled rejections - Fix curves add-point using true geometric midpoint instead of endpoint - Add fallback for unknown LUT preset IDs from persisted data - Add native disabled, aria-disabled, and touchAction to color wheel - Move vectorscope graticule drawing after putImageData so it renders - Serialize color scopes interval to prevent overlapping async captures - Align WHEELS_CONFIG amount ranges to 0..1 matching the interface
…xt clipping (#104) * fix(preview): keep gizmo transforms live during fast-scrub LUT overlay Pass live gizmo preview transforms into the fast-scrub composition renderer so dragged items reflect transform changes with LUT preview applied instead of freezing at committed values. Subscribe to gizmo store changes to pump the render loop during forced custom-CUBE overlays. Preserve the viewed frame when clearing preview mode to avoid one-frame render source jumps during gizmo interactions. * fix(preview): sync gizmo transforms with fast-scrub output and fix text clipping Route gizmo drag interactions through the fast-scrub render path so transform previews stay consistent with the displayed canvas output. Add `displayedFrame` to playback state so UI components (gizmo overlay, keyframe transforms) resolve against the frame actually rendered on screen rather than a stale preview/current frame. Lift transform computation out of SelectableItem into GizmoOverlay via useVisualTransforms, which merges keyframe animations with gizmo and property-panel preview overrides. Apply text layout expansion for preview so text items aren't hard-clipped to their bounding box during editing, matching the live DOM preview behavior. Update fast-scrub overlay guard to allow the overlay during gizmo interactions when the rendered frame matches the current frame, removing the blanket gizmo-interaction bail-out that caused frame jumps. * refactor(preview): clean up transform gizmo deps and extract text expansion helper Remove redundant item.id from TransformGizmo useMemo dependency array since item is already tracked. Extract applyTextExpansion helper in useVisualTransforms to centralize the three inline text preview expansion call sites. Add comment documenting default values in getTextRequiredHeight that mirror TextItem type defaults. * fix(preview): break overlong first word in text wrap layout The word-wrapping logic skipped character-breaking when the first word in a paragraph exceeded maxWidth because the condition was guarded by `&& currentLine`. Remove the guard so overlong words are broken via breakWord regardless of their position in the line.
* Lock preview to full resolution and remove quality selectors * Remove unused preview quality state from video preview
…109) * feat(export): support masks inside pre-compositions and fix source frame math Add mask collection and application for sub-composition items in both preview (composition-content) and canvas export (canvas-item-renderer) paths. Replace manual Math.round(duration * speed) calculations in canvas-audio with proper timelineToSourceFrames/sourceToTimelineFrames conversions for correct mixed-FPS handling. * refactor(preview): move color scopes from preview overlay to editor panel tab (#107) Relocate the color scopes panel from a floating overlay in the preview area to a dedicated tab in the bottom editor panel alongside keyframes. Add a raw ImageData capture path (captureFrameImageData) to avoid the encode/decode overhead of the data URL path used for thumbnails. Extract the scopes view into a shared component for reuse across both locations. * fix: prevent vectorscope Uint16 overflow and fix conditional hook calls Change vectorscope density buffer from Uint16Array to Uint32Array to handle paused sample sizes (384x216 = 82,944 pixels) that can overflow a single bin. Move visibleTrackIds and activeMaskInfos useMemo hooks before the early return in CompositionContent so hooks run unconditionally.
* fix(precomp): apply nested keyframes during live playback * test(precomp): remove unused React binding in keyframes regression test
Run boundary checks, deps contracts, legacy lib imports, deps wrapper health, edge budgets, and lint before every push to catch CI failures locally.
…shaders Replace Canvas 2D scope rendering with WebGPU compute+render pipeline for histogram, waveform, vectorscope, and RGB parade. Adds phosphor bloom, Gaussian trace spread, bilinear sampling, anti-aliased graticule with color targets and skin tone line. Near-zero-copy frame capture via copyExternalImageToTexture bypasses getImageData CPU readback. Falls back to Canvas 2D when WebGPU is unavailable. Adds RGB/R/G/B/Y view mode toggles for waveform and histogram scopes.
Port GPU effects system from MasterSelects with standalone EffectsPipeline that manages its own GPUDevice. Includes 30 WGSL fragment shader effects across 5 categories: color (9), blur (5), distort (7), stylize (8), and keying (1). Pipeline supports both canvas output (preview) and ImageData readback (export) via ping-pong rendering with blit passthrough. Also wires up canvas source capture in video-preview for GPU effect integration and adds @webgpu/types for TypeScript support.
…eview Add GpuEffect type to VisualEffect union and wire GPU shader effects through all three rendering paths: - UI: GPU effect categories in Add Effect dropdown with generic GpuEffectPanel supporting number/boolean/select params - Export: lazy EffectsPipeline init in client-render-engine with async GPU readback via applyAllEffectsAsync for both item and transition rendering - Preview: WebGPU overlay canvas driven by useGpuEffectsOverlay hook that captures frames via captureCanvasSource and processes at ~20fps
Wire the standalone WebGPU effects pipeline into the editor: - Add GpuEffect type to VisualEffect union with per-effect params - Add GpuEffectPanel component and GPU effect categories in Add Effect dropdown - Add WebGPU overlay canvas for event-driven GPU effects preview during scrubbing - Integrate applyAllEffectsAsync with lazy EffectsPipeline in export renderer - Fix TypeScript narrowing for color-grading effects after union expansion
Integrate GPU shader effects into the editor UI, preview overlay, and export pipeline. Effects run through a WebGPU EffectsPipeline with ping-pong textures, cached uniforms, and GPU backpressure. Preview uses a dedicated overlay canvas with rAF loop for scrubbing/playback; export uses async applyEffectsToImageData path with proper GPU readback.
…nders Adopt the masterselects WebGPU approach: capture directly from <video> elements via copyExternalImageToTexture (near-zero-copy GPU DMA) instead of running full CPU composition renders through captureCanvasSource. - EffectsPipeline.applyEffects now accepts HTMLVideoElement as source - Hook finds <video> in Player DOM and reads every rAF frame on GPU - Eliminates captureCanvasSource async chain for seeking/playback - Add cached uniform buffers, texture views, and GPU backpressure - Scrubbing still uses offscreen canvas dirty flag (already rendered)
Filter collectGpuEffects by item time range so effects from one clip don't bleed onto other clips. Clear the overlay with a transparent render pass when no effects are active at the current frame.
…r preview - Parallelize video decoding across items via Promise.all in preview mode, cutting multi-video render time roughly in half - Add GPU effects pipeline pool mode (beginBatch/endBatch) with per-item output canvases for deferred compositing, allowing GPU work pipelining - Add rendered frame ImageBitmap cache (120 frames LRU) for instant backward scrubbing without mediabunny stream restarts - Add forward jump restart threshold (>3s) to avoid reading through hundreds of samples on large forward seeks - Update canvas-effects to return deferred GPU canvas in pool mode for z-order-correct deferred compositing - Clean up batch pool resources in EffectsPipeline.destroy()
…layback During playback, use the Remotion Player's DOM <video> elements instead of mediabunny decode, and feed them directly to the GPU via importExternalTexture for zero-copy effect processing. - Add domVideoElementProvider to ItemRenderContext for querying Player video elements by item ID during playback - Add IMPORT_EXTERNAL_SHADER with texture_external binding and destRect uniform for positioned video-to-GPU blit - Add applyEffectsToVideo() to EffectsPipeline: imports video as GPUExternalTexture, blits with positioning to ping texture, runs effect chain, outputs to canvas — zero pixel copies - Direct video→GPU path in renderItemWithEffects when video has GPU-only effects, simple transform, and DOM video is available - Export getGpuEffectInstances for use by render engine - Graceful fallback to standard canvas 2D path when importExternalTexture is unsupported or fails
Use full renderFrame for prewarm instead of lightweight prewarmFrame so prewarmed frames populate the frame cache as ImageBitmaps. Subsequent scrubs to those frames are instant cache hits (~0.1ms vs 5-80ms). - Prewarm uses renderFrame to fill frame cache ahead of scrub position - Forward prewarm steps increased from 1 to 3 frames - Frame cache increased from 120 to 240 frames (8s at 30fps) - 16ms time budget caps prewarm work per cycle to keep scrubbing responsive
…in cache to 300 - Add 60-frame ImageBitmap LRU cache to StrictDecodedVideoFrame for instant revisits during drag direction reversal in 2-up/4-up overlays - Increase main composition renderer frame cache from 240 to 300 frames - Cache keyed by quantized source time (~1/60s resolution) - Properly clean up ImageBitmaps on source change and unmount
Export with GPU effects was slow due to 4 pixel copies per item per frame (getImageData → writeTexture → GPU → readback → putImageData). Now uses the same zero-copy path as preview (copyExternalImageToTexture → GPU → canvas output → drawImage). Also enables parallel video decode and GPU batch/pool mode during export, matching the preview optimizations for multi-clip compositions.
Overlap JPEG encoding of frame N with video decode of frame N+1. Previously, the worker blocked on convertToBlob before starting the next decode. Now: snapshot canvas to ImageBitmap immediately (frees canvas pool slot), flush previous encode, kick off new encode async, then resume the decode generator.
Filmstrip thumbnails now appear instantly during extraction by transferring ImageBitmaps from the worker (zero-copy) instead of waiting for JPEG encode + blob URL creation. Two parallel pipelines per decoded frame: 1. FAST: createImageBitmap → transfer to main thread → canvas render 2. SLOW: JPEG encode → OPFS save (persistence, runs in background) The FilmstripTile component renders from bitmap via canvas when available, falling back to <img> for OPFS-loaded frames. When the JPEG blob arrives later, it upgrades the frame with a proper URL and the bitmap is closed.
Remove all legacy effect systems (CSS filters, glitch, halftone, vignette, LUT, color grading) and consolidate to GPU shader effects. Add v6 migration to convert existing projects. Restore specialized UI panels for curves (interactive SVG tone curve editor) and color wheels (circular hue/amount pickers) adapted for GPU effect params, with batch param updates for atomic multi-param changes.
…tter WASM decode prewarm renders (40-80ms each) were blocking the fast scrub render loop from processing priority frames during playback, causing the GPU effects overlay to fall ~30 frames behind and show stale content.
…w, and thumbnails Implement 5 GPU rendering subsystems and integrate blend modes into preview/export paths: - GPU compositor pipeline with 25 Photoshop-compatible blend modes (WGSL) - Bezier mask renderer (CPU path → GPU texture upload) - Corner pin perspective warp via 16×16 subdivided mesh - Optical flow analyzer with 3-level Gaussian pyramid (Lucas-Kanade) - Scene detection service and frame sampling strategies - GPU thumbnail renderer with low-power device context Integration: - Blend mode dropdown in Fill section (grouped by category) - CSS mix-blend-mode on preview items via item-visual-wrapper - Canvas2D globalCompositeOperation for export compositing - GPU effects overlay detection extended for non-normal blend modes - Occlusion culling updated to skip items with blend modes - Schema v7 migration for blendMode, masks, cornerPin fields
GPU/canvas transitions fully handle all transition rendering now. The CSS-based TransitionOverlay, NativeTransitionVideo, and OptimizedEffectsBasedTransitionsLayer are no longer needed.
copyExternalImageToTexture requires BOTH COPY_DST and RENDER_ATTACHMENT usage flags on destination textures (WebGPU spec). Input textures were missing RENDER_ATTACHMENT, causing Dawn validation errors and black output for all GPU transitions (dissolve, glitch, light leak). Also refactors pipeline to two-pass architecture (transition→rgba8unorm→ blit→canvas) matching EffectsPipeline, rewrites glitch shader with multi-scale displacement and proper smoothstep args, and fixes writeBuffer to pass typed array directly instead of underlying ArrayBuffer.
…le-pass GPU render - Replace hard-seek RVFC drift correction with rate-based correction (±2-5% playbackRate adjustment) to eliminate visible "drift then jump" jitter - Pre-mount shadow video elements ~0.5s before transition start so pool elements are loaded when the overlap begins (avoids mediabunny fallback) - Simplify GPU transition pipeline to single-pass (remove blit shader and intermediate rgba8unorm texture) - Fix writeBuffer TypeScript signature for typed arrays - Align render engine drift threshold (200ms) with RVFC correction - Remove per-frame diagnostic logging from transition hot path - Deduplicate getAdjustmentLayerEffects call in transition renderer
- Lower play gate from readyState >= 3 to >= 2 (saves 100-300ms waiting for HAVE_FUTURE_DATA when HAVE_CURRENT_DATA is sufficient) - Seek and play immediately on mount when playback is active instead of pausing and waiting for sync effect next frame (~16-50ms savings) - Skip redundant seeks when element is already at target position (avoids readyState drop that delays play start) - Pre-resume shared AudioContext on playback start to eliminate 50-100ms audio delay from suspended context on cold resume - Skip force-render timeout during playback to prevent play→pause race
Replace flat 300-frame ImageBitmap LRU cache with a 3-tier architecture: T1 (VRAM GPUTexture, 300 frames), T2 (per-video last-frame), T3 (RAM ImageBitmap, 900 frames/8GB). Skip cache writes during sequential forward scrubbing where mediabunny decode (~1ms) is faster than write overhead (~2-5ms createImageBitmap + GPU upload).
…r GPU transitions Three new WebGPU shader transitions: pixelate (mosaic dissolve with CPU-side block computation), chromatic (directional RGB split with radial lens distortion), and radial blur (zoom + spin blur with vignette). All include CSS and Canvas 2D fallbacks.
Replace NumberInput with SliderInput across properties panel for bounded-range controls (volume, fades, speed, opacity, rotation, mask params, effects). SliderInput features spring-animated click-to-snap, rubber-band overflow, handle dodge, hash marks, and inline click-to-edit value input. Add searchable full-width effect picker replacing the DropdownMenu, with real-time filtering of effects and presets. Add scroll wheel support to NumberInput. Move SliderInput to shared/ui for cross-feature reuse.
…ider drag Replace SVG feGaussianBlur (CPU-rendered, laggy) with Canvas2D blur via Skia (GPU-accelerated) at reduced resolution for clip mask feather. Add preview store pattern (like corner pin) so feather/opacity sliders write directly to a lightweight Zustand store that ItemVisualWrapper subscribes to, bypassing the slow React prop chain through tracks → composition → StableVideoSequence. Also fix mask gizmo pointer capture, hide transform gizmo during mask editing, and add masks to StableVideoSequence's areGroupPropsEqual whitelist.
…ayback Canvas data URL masks caused Chrome to re-decode PNG each paint cycle, forcing software compositing and audio jitter during playback. Switch to SVG-only approach: clip-path for simple masks (GPU geometry), SVG mask for complex cases (feather/opacity/modes). Add preview store integration for live slider drag feedback.
… animation Add 'path' shape type with bezier vertices drawn via pen tool in the preview overlay. Refactor mask rendering from global SVG mask approach to per-item resolution via useActiveMasks hook, enabling keyframe animation on shape masks in both preview and export pipelines.
Create GPU and analysis facade modules in src/infrastructure/ to eliminate 21 direct @/lib/* imports flagged by the legacy-lib-imports check. Whitelist facade files in the check script.
* refactor: unify composition render scene planning * refactor: tighten preview and timeline rendering * Add editable shape and mask pen paths * Fix timeline navigator drag behavior * Address PR feedback and CI budgets
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Too many files changed for review. ( |
Contributor
|
Important Review skippedToo many files! This PR contains 219 files, which is 69 over the limit of 150. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (219)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99c533ae31
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+218
to
220
| const visualEffectSchema = z.union([ | ||
| cssFilterEffectSchema, | ||
| glitchEffectSchema, |
There was a problem hiding this comment.
The migration path now converts effects to type: 'gpu-effect' (migrate in src/domain/projects/migrations/migrations.ts), but visualEffectSchema here still only accepts legacy effect shapes, so validateSnapshotData() rejects snapshots that contain any migrated/new GPU effect and import fails outright. This blocks round-tripping projects once users add or migrate effects under the new GPU-only model.
Useful? React with 👍 / 👎.
| @@ -87,7 +98,7 @@ export type ImageItem = BaseTimelineItem & { | |||
| sourceHeight?: number; | |||
| }; | |||
|
|
|||
| export type ShapeType = 'rectangle' | 'circle' | 'triangle' | 'ellipse' | 'star' | 'polygon' | 'heart'; | |||
| export type ShapeType = 'rectangle' | 'circle' | 'triangle' | 'ellipse' | 'star' | 'polygon' | 'heart' | 'path'; | |||
There was a problem hiding this comment.
This commit introduces 'path' as a valid ShapeType, but the snapshot validator still uses a shapeTypeSchema enum without 'path', so any project containing a path shape fails schema validation during import (validateSnapshotData) even though the runtime/editor now supports it. That makes newly created path-based shapes non-portable through JSON import/export workflows.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Testing
npm run test:run -- src/features/preview/components/mask-editor-overlay.test.tsx