Skip to content

feat: mask editing, keyframe editor, transcription, and editor UX improvements#124

Merged
walterlow merged 20 commits intomainfrom
develop
Mar 18, 2026
Merged

feat: mask editing, keyframe editor, transcription, and editor UX improvements#124
walterlow merged 20 commits intomainfrom
develop

Conversation

@walterlow
Copy link
Copy Markdown
Owner

@walterlow walterlow commented Mar 18, 2026

Summary

  • Mask editing: scoped masks to lower tracks only, full path edit mode with vertex selection/marquee, live preview during edits, corner/bezier knot conversion, Delete key vertex removal, pen tool toolbar in preview area, and interaction lock during all mask editing modes
  • Keyframe editor: per-property row controls (prev/next nav, toggle, auto-key, inline value input), local/global frame readout inputs, deferred drag commits, graph clip-path and focus-aware zoom, K hotkey scoped to active editor property
  • Transcription: local Whisper-based media transcription with browser worker pipeline, caption item generation, and transcript persistence in IndexedDB
  • Effects: new distort/stylize GPU effects, color params, halftone edge artifact fixes
  • Preview: frame capture button, color scopes monitor, canvas media drop, configurable editor density presets
  • Editor UX: clip inspector tab persistence, even canvas dimension enforcement, media sidebar and toolbar improvements, settings dialog enhancements

Test plan

  • Verify mask scoping — masks only clip content on lower tracks in both preview and export
  • Test path edit mode: drag vertices/handles, marquee select, body drag, double-click edge insert, right-click delete, corner/bezier conversion
  • Confirm Delete/Backspace removes selected mask vertices without deleting the item
  • Test dopesheet inline value editing with Enter commit, Escape cancel, blur commit
  • Verify keyframe drag preview shows positions during drag, commits on release
  • Test local/global frame inputs in dopesheet header
  • Confirm K hotkey adds keyframe for the active editor property
  • Verify graph zoom keeps keyframes visible
  • Test transcription workflow end-to-end
  • Verify halftone and distort/stylize effects render correctly

Summary by CodeRabbit

Release Notes

  • New Features

    • Path-based mask editing with vertex selection, dragging, and bezier/corner conversion
    • Enhanced keyframe editor with per-property value inputs and auto-key toggle
    • Track-based mask scoping—masks now only affect content on lower tracks
    • Multi-vertex selection for masks with drag preview feedback
    • Improved mask editor UI with better controls and visual feedback
  • Bug Fixes

    • Corrected mask application logic to respect track ordering

…ion generation

Add end-to-end local speech-to-text pipeline using browser-based Whisper
inference with SharedArrayBuffer support (COEP/COOP headers). Includes
transcript storage (IndexedDB v10), settings for default model/quantization/
language, per-clip caption generation from timeline context menu with progress
overlays, and media library transcription controls. Replaces landing page
YouTube iframe with a static thumbnail link for cross-origin isolation
compatibility.
Move direct cross-feature import of settings-store through a deps/*
adapter module to satisfy the boundary check.
Add a centralized editor layout system with compact and default density
presets, driven by CSS custom properties. Compact (the new default) fits
more of the editor on a 1080p screen by shrinking toolbar, sidebars,
preview controls, timeline headers, track heights, and clip row sizes.
Users can switch between presets in Settings > Interface. Also adds
@vercel/analytics and tightens various UI element sizes for the denser
layout.
Move color scopes out of the keyframe graph panel into a dedicated
resizable side panel in the preview area. The scopes toggle now opens a
ColorScopesMonitor alongside the program monitor (right side), with
independent drag-to-resize handles that respect min/max constraints when
both source monitor and scopes are open simultaneously.
- Extract renderOffscreenFrame to deduplicate frame rendering across capture methods
- Add in-flight guards for captureCanvasSource and GPU scope rendering
- Replace decay compute shader with encoder.clearBuffer for accumulator clearing
- Split waveform render into accumulate + renderAccumulated with renderBatch API
- Add configurable 3-up/all stack layout for embedded scopes panel
- Use aspect-ratio-based canvas resize instead of square constraint
Add a camera button to playback controls that captures the current
preview frame, downloads it as PNG, and imports it into the media
library. Simplify timecode display layout by using a single reserved
width for the entire component instead of per-span sizing. Add
importGeneratedImage method to media library service for editor-
generated assets with rollback on failure.
…errides

- Fix halftone shader bright edge by replacing outOfFrame geometric bounds
  check with alpha-based source coverage and clamped UV sampling
- Snap halftone grid to whole cells to prevent fractional border columns
- Disable importExternalTexture fast path which produced different edge
  pixel values than canvas 2D due to YUV→RGB conversion differences
- Use VideoFrame.visibleRect cropping in video extractor and tier-2
  frame drawing to avoid codec padding artifacts
- Pass preview effects overrides through to adjustment layer resolution
  so live-edited adjustment effects are reflected during preview
… stamp

Even cell counts land texture edges at cell boundaries (fract=0), exposing
paper/background color as a rectangular border. Forcing odd counts centers
the grid so edges land at fract=0.5 (dot centers), making the pattern
bleed seamlessly to the boundary.
…zmo cache invalidation

- Canvas panel snaps width/height to nearest even on commit and steps by 2,
  preventing odd dimensions that video codecs reject
- LinkedDimensions accepts a step prop for callers that need non-default steps
- Export validateSettings auto-rounds odd dimensions as a safety net instead
  of rejecting with an error
- Fix stale fast-scrub cache during single-item gizmo transforms by
  invalidating the current frame when previewTransform changes
Store the active clip inspector tab (transform/effects/media) in the
editor store so it survives selection changes. Falls back gracefully
when the remembered tab is unavailable for the current item type.
… utilities

Enable drag-and-drop media placement directly onto the preview canvas,
positioning items at the drop point with smart track selection. Refactor
timeline track drop logic to share item-building and file-extraction
utilities, reducing duplication across canvas and timeline drop targets.
Lock non-canvas UI regions (toolbar, sidebars, timeline, controls) during
pen mode via InteractionLockRegion overlay. Add a Drawing Mode HUD with
progress hints and finish/cancel buttons. Place new mask shapes at the
playhead on the best available track using shared drop-placement logic,
and prevent timeline overlaps in addItem/addItems via collision detection.
Also swallow all non-modifier keyboard shortcuts while pen mode is active
and fix vectorscope overflow in color scopes.
…keyframe graph panel

Add min-h-0 and h-full to InteractionLockRegion to prevent flex overflow,
and move KeyframeGraphPanel above the bottom status bar so it renders in
the correct stacking position.
… pen placement

When completing a pen-mode mask shape and all existing tracks are occupied
at the playhead, create a new track above to preserve the intended time
position instead of failing silently.
Masks now only clip items on tracks below the mask track (higher order),
matching standard NLE behavior. Extracts shared doesMaskAffectTrack
utility and threads track order through preview, export, and composition
runtime pipelines.
…d toolbar controls

Add a full path edit mode for mask shapes: vertex/handle dragging with
live clip-path preview, marquee box-selection, shape body translation
via gizmo, double-click edge insertion, right-click vertex deletion,
and corner/bezier knot conversion from toolbar buttons.

Thread preview path vertex overrides through the composition runtime,
export canvas renderer, and fast-scrub pipeline so mask geometry updates
are visible in real time during editing. Move pen/edit mode HUD from the
mask overlay into the preview-area toolbar and broaden interaction locks
to cover all mask editing modes, not just pen mode.
Intercept Delete and Backspace in path edit mode to remove selected
vertices instead of letting the event bubble to the timeline's item
deletion handler. Supports single and multi-vertex deletion while
preserving the minimum 3-vertex constraint.
… deferred drag commits

Redesign the dopesheet editor with per-row prev/next navigation,
keyframe toggle, auto-key toggle, and inline value input fields that
commit on Enter or blur. Add local/global frame readout inputs in the
header for repositioning selected keyframes by typing exact frames.

Defer keyframe drag commits to pointer-up (preview positions via direct
DOM manipulation during drag). Extract layout, row-controls, and
property-value-ranges into shared modules. Move the K hotkey from
timeline editing shortcuts into the keyframe graph panel so it targets
the active editor property. Clip value graph content to the graph area
and zoom around the keyframe cluster centroid.
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Mar 18, 2026

Too many files changed for review. (169 files found, 100 file limit)

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Mar 18, 2026

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR introduces per-track mask ordering and scoping semantics, enabling masks to affect only lower tracks. It adds preview-based path-vertex editing for masks, significantly expands the dopesheet editor with property-value editing and auto-key toggles, and updates rendering pipelines to support track-scoped mask application and path-vertex overrides.

Changes

Cohort / File(s) Summary
Edge Budget
scripts/check-feature-edge-budgets.mjs
Increased maxImports for 'editor → preview' edge from 11 to 12.
Mask Ordering Infrastructure
src/shared/utils/mask-scope.*, src/features/composition-runtime/utils/preview-path-override.*
Introduced doesMaskAffectTrack to determine track-scoped mask applicability; added PreviewPathVerticesOverride type and utility functions (applyPreviewPathVerticesToShape, applyPreviewPathVerticesToItem) to override shape paths during preview edits.
Mask Info & Frame Scene Resolution
src/features/composition-runtime/utils/mask-info.*, src/features/composition-runtime/utils/frame-scene.*
Extended MaskInfo and ResolvedShapeMask with trackOrder: number; added getMasksForTrackOrder helper; updated resolveActiveShapeMasksAtFrame to accept preview path vertex overrides and track ordering.
Composition Runtime Masking
src/features/composition-runtime/components/composition-content.tsx, src/features/composition-runtime/components/item.tsx, src/features/composition-runtime/components/hooks/use-item-visual-state.ts
Refactored active mask selection from ID-based to per-track ordered collection; implemented per-track mask scoping via getMasksForTrackOrder; added preview vertex editing context in visual state computation.
Main Composition & Scene Assembly
src/features/composition-runtime/compositions/main-composition.tsx
Updated FrameActiveMasksProvider to use ShapeMaskWithTrackOrder instead of ShapeItem; added itemTrackOrder prop to MaskedItem and integrated per-track mask filtering.
Export/Render Pipeline Updates
src/features/export/utils/canvas-item-renderer.ts, src/features/export/utils/canvas-masks.ts, src/features/export/utils/client-render-engine.ts
Integrated doesMaskAffectTrack for per-track masking; threaded PreviewPathVerticesOverride through rendering paths; added track-scoped mask application via applyTrackScopedMasks; extended public signatures to accept getPreviewPathVerticesOverride and itemTrackOrder.
Mask Editor Store & State
src/features/preview/stores/mask-editor-store.ts
Added vertex selection state (selectedVertexIndices, selectedVertexIndex); introduced pen-mode request actions (requestFinishPenMode, requestCancelPenMode, requestConvertSelectedVertex) with version counters; propagated vertex selection in drag operations.
Preview Area & Path Editing UI
src/features/editor/components/preview-area.tsx, src/features/editor/components/preview-area.test.tsx
Expanded mask editing UI with separate headers for pen-tool and path-edit modes; added controls for vertex conversion (Corner/Bezier), vertex count display, and finish/cancel actions; replaced isPenModeActive checks with isMaskEditingActive.
Mask Editor Overlay & Path Rendering
src/features/preview/components/mask-editor-overlay.tsx, src/features/preview/components/mask-editor-overlay.test.tsx
Major expansion: added marquee-based multi-vertex selection; integrated gizmo interactions for shape translation; implemented vertex conversion (corner ↔ bezier); added hit-testing for curved segments; extended test coverage for complex editing scenarios.
Mask Path Utilities
src/features/preview/utils/mask-path-utils.ts
Added public exports convertVertexToCorner and convertVertexToBezier to convert vertices between sharp and smooth knots with proper handle synthesis.
Video Preview Integration
src/features/preview/components/video-preview.tsx, src/features/preview/components/video-preview.sync.test.tsx
Introduced getPreviewPathVerticesOverride hook to read mask editor preview vertices; wired store subscription to invalidate scrub cache on preview vertex changes; replaced pen-mode guard with mask-editing guard.
Editor & Interaction UI
src/features/editor/components/editor.tsx, src/features/editor/components/interaction-lock-region.tsx
Replaced isPenModeActive with isMaskEditingActive in editing guards; updated lock-region UI comments and class structure to reflect mask-editing semantics.
Dopesheet Editor Major Revision
src/features/keyframes/components/dopesheet-editor/index.tsx, src/features/keyframes/components/dopesheet-editor/layout.ts, src/features/keyframes/components/dopesheet-editor/row-controls.ts
Expanded with property-value editing, auto-key toggles per row, global frame tracking, drag-preview for in-progress moves, and enhanced playhead rendering; introduced getVisibleKeyframeX layout utility; added getDopesheetRowControlState for keyframe navigation; increased visual density (PROPERTY_COLUMN_WIDTH 140→290, ROW_HEIGHT 28→36).
Dopesheet Tests
src/features/keyframes/components/dopesheet-editor/*.test.tsx
Added comprehensive test coverage: drag preview flow, header frame readout editing, value input commit behavior (blur/Enter/auto-key), playhead positioning, and row control state resolution.
Value Graph Editor
src/features/keyframes/components/value-graph-editor/index.tsx, src/features/keyframes/components/value-graph-editor/use-graph-interaction.ts, src/features/keyframes/components/value-graph-editor/types.ts
Added SVG clipping to constrain graph rendering; refactored use-graph-interaction to compute preview values during drag, defer onKeyframeMove to drag end, and integrate viewport clamping; extracted PROPERTY_VALUE_RANGES to dedicated module.
Value Graph & Interaction Tests
src/features/keyframes/components/value-graph-editor/*.test.tsx
Added tests for graph clipping, preview value rendering during drag, zoom focus behavior, and zoom-constrained viewport updates.
Keyframe Graph Panel Integration
src/features/timeline/components/keyframe-graph-panel.tsx, src/features/timeline/components/keyframe-editor-hotkey.ts
Introduced activeDopesheetProperty tracking; added editorHotkeyProperty resolution via new resolveKeyframeEditorHotkeyProperty helper; implemented per-property value commit via handlePropertyValueCommit; added ADD_KEYFRAME hotkey support; wired property-value editing UI to graph/dopesheet editors.
Keyframe Hotkey & Timeline
src/features/timeline/hooks/shortcuts/use-editing-shortcuts.ts, src/features/timeline/stores/actions/keyframe-actions.ts, src/features/timeline/stores/keyframes-store.ts, src/features/timeline/components/timeline.tsx
Removed per-item ADD_KEYFRAME hotkey logic (now handled in keyframe-graph-panel); minor comment updates; reorganized KeyframeGraphPanel placement in Timeline render order.
Contract Exports
src/features/editor/deps/preview-contract.ts, src/features/export/deps/composition-runtime-contract.ts
Re-exported useItemsStore and preview-path-override utilities (applyPreviewPathVerticesToItem, applyPreviewPathVerticesToShape, PreviewPathVerticesOverride) to expand public API surfaces.
Test Infrastructure
src/features/composition-runtime/components/composition-content.masks.test.tsx, src/features/export/utils/canvas-item-renderer.composition-masks.test.ts, src/features/export/utils/canvas-masks.test.ts
Updated mask resolution tests to reflect track ordering; added regression test ensuring masks do not clip content above mask track; extended mock signatures for preview vertex and track-order handling.
Property Value Ranges
src/features/keyframes/property-value-ranges.ts
New module defining PropertyValueRange interface and PROPERTY_VALUE_RANGES mapping with min/max bounds, units, and decimals for animatable properties (x, y, width, height, rotation, opacity, cornerRadius, volume).

Sequence Diagrams

sequenceDiagram
    participant Client as Render Client
    participant FrameRes as Frame Resolution
    participant MaskFilter as Mask Filtering
    participant MaskApply as Track-Scoped Apply
    participant Canvas as Canvas Renderer

    Client->>FrameRes: resolveFrameCompositionScene(scene, masks, getPreviewPathVertices)
    FrameRes->>FrameRes: resolveActiveShapeMasksAtFrame(masks, getPreviewPathVertices)
    FrameRes->>FrameRes: Apply preview vertex overrides to shapes
    FrameRes->>Client: Return resolved masks with trackOrder

    Client->>MaskFilter: Collect visible tracks with trackOrder
    loop For each track
        MaskFilter->>MaskFilter: getMasksForTrackOrder(activeMasks, trackOrder)
        MaskFilter->>MaskFilter: Filter via doesMaskAffectTrack(maskTrackOrder < itemTrackOrder)
    end
    MaskFilter->>Client: Return scoped masks per track

    Client->>MaskApply: applyTrackScopedMasks(itemCanvas, scopedMasks)
    MaskApply->>MaskApply: For each mask, apply clip-path to canvas
    MaskApply->>Canvas: Draw masked result to output
    Canvas->>Client: Composite final rendered frame
Loading
sequenceDiagram
    participant Editor as Mask Editor
    participant Store as Mask Editor Store
    participant PathUtil as Path Utils
    participant Preview as Preview Renderer
    participant Cache as Frame Cache

    Editor->>Editor: User drags path vertex
    Editor->>Store: updatePreviewVertices(vertices)
    Store->>Store: Store preview vertex state (previewVertices, editingItemId)

    Preview->>Store: getPreviewPathVerticesOverride()
    Store->>Preview: Return getPreviewPathVertices callback

    Preview->>PathUtil: applyPreviewPathVerticesToShape(shape, getPreviewPathVertices)
    PathUtil->>PathUtil: Override shape.pathVertices with preview
    PathUtil->>Preview: Return shape with preview vertices

    Preview->>Cache: invalidateFrameCache([currentFrame])
    Cache->>Preview: Re-render frame with preview geometry
    Preview->>Editor: Display updated preview rendering

    Editor->>Editor: User finishes edit
    Editor->>Store: stopEditing() or confirm changes
    Store->>Preview: Clear preview state
    Preview->>Cache: Invalidate cache again (commit preview)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

  • PR #117: Both PRs directly modify the same mask-resolution code paths (resolveActiveShapeMasksAtFrame, trackOrder, getMasksForTrackOrder, doesMaskAffectTrack) and preview path-vertex override utilities, making them closely coupled changes.
  • PR #109: Both PRs touch the same mask/pre-composition rendering code paths (composition-content, per-track mask scoping, canvas rendering) and implement similar per-track masking semantics.
  • PR #104: Both PRs modify preview rendering plumbing and canvas-item rendering (client-render-engine, canvas-item-renderer) to thread preview overrides through the rendering pipeline.

Poem

🐰 Masks now know their tracks, vertices dance in preview!
Dopesheet's grown bright with properties and auto-keys,
Per-track scoping keeps chaos at bay,
While path editors sketch and refine with glee.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 27.78% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main changes: mask editing improvements, keyframe editor enhancements, transcription features, and editor UX improvements. It clearly summarizes the primary features added in the changeset.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch develop
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Tip

You can make CodeRabbit's review stricter and more nitpicky using the `assertive` profile, if that's what you prefer.

Change the reviews.profile setting to assertive to make CodeRabbit's nitpick more issues in your PRs.

@vercel
Copy link
Copy Markdown

vercel Bot commented Mar 18, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
freecut Ready Ready Preview, Comment Mar 20, 2026 9:27am

# Conflicts:
#	src/features/editor/components/editor.tsx
#	src/features/editor/components/interaction-lock-region.tsx
#	src/features/editor/components/preview-area.tsx
#	src/features/export/utils/canvas-item-renderer.ts
#	src/features/export/utils/client-render-engine.ts
#	src/features/preview/components/mask-editor-overlay.test.tsx
#	src/features/preview/components/mask-editor-overlay.tsx
#	src/features/preview/components/video-preview.sync.test.tsx
#	src/features/preview/components/video-preview.tsx
#	src/features/timeline/components/keyframe-graph-panel.tsx
#	src/features/timeline/components/timeline.tsx
@walterlow walterlow merged commit 0efaec2 into main Mar 18, 2026
4 of 5 checks passed
@chatgpt-codex-connector
Copy link
Copy Markdown

💡 Codex Review

if (hasWebCodecs) {
this.decoderWorker.postMessage({ type: 'init', file });
return;

P1 Badge Fall back to AudioContext when WebCodecs path fails

This branch unconditionally commits to the WebCodecs worker path whenever AudioDecoder exists, and immediately returns; if that worker later reports a decoder error (unsupported codec/config), transcription is terminated instead of retrying with the existing decodeWithAudioContext fallback. In browsers with partial WebCodecs support, this turns otherwise-transcribable files into hard failures, so the decoder error path should trigger an AudioContext retry before failing.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant