Skip to content

Pen mode interaction lock, canvas drop, scopes, transcription, and effects#123

Merged
walterlow merged 13 commits intomainfrom
develop
Mar 17, 2026
Merged

Pen mode interaction lock, canvas drop, scopes, transcription, and effects#123
walterlow merged 13 commits intomainfrom
develop

Conversation

@walterlow
Copy link
Copy Markdown
Owner

@walterlow walterlow commented Mar 17, 2026

Summary

  • Pen mode interaction lock: Lock non-canvas UI regions (toolbar, sidebars, timeline) during mask pen mode with Drawing Mode HUD and keyboard shortcut capture
  • Canvas media drop: Drop media directly onto the preview canvas with shared drop-placement utilities and overlap prevention in addItem/addItems
  • Color scopes monitor: Standalone color scopes panel in preview area with batched waveform GPU renders
  • Local Whisper transcription: Browser-based media transcription and caption generation via Web Workers
  • New GPU effects: Color params and new distort/stylize effects with even canvas dimension enforcement
  • Editor density presets: Configurable compact/default editor layouts
  • Clip inspector: Persist tab selection across item changes
  • Structured logging: Wide event pattern and console migration
  • Halftone fixes: Edge artifacts and adjustment layer preview overrides

Test plan

  • Verify pen mode locks toolbar, sidebars, timeline and shows Drawing Mode HUD
  • Drop media onto preview canvas and confirm correct track/frame placement
  • Open color scopes monitor and verify waveform/vectorscope rendering
  • Test local Whisper transcription on an audio/video file
  • Apply new distort/stylize effects and verify preview + export parity
  • Toggle editor density between compact and default
  • Run npm run test:run to confirm all tests pass

Summary by CodeRabbit

  • New Features

    • Added media transcription and automatic caption generation with language and model selection.
    • Added editor interface density settings for compact or default layouts.
    • Added local inference runtime management for on-device AI model execution.
    • Added canvas drag-and-drop support for media placement.
    • Added frame capture to save preview frames as images.
    • Added pen mode for shape drawing with interaction locking.
    • Enhanced color scopes visualization with stack layout support.
  • Improvements

    • Improved responsive editor layout and sizing across sidebars and timeline.
    • Enhanced timeline with collision-aware item placement.
  • Bug Fixes

    • Fixed export validation to auto-round odd dimensions to even values.

…ion generation

Add end-to-end local speech-to-text pipeline using browser-based Whisper
inference with SharedArrayBuffer support (COEP/COOP headers). Includes
transcript storage (IndexedDB v10), settings for default model/quantization/
language, per-clip caption generation from timeline context menu with progress
overlays, and media library transcription controls. Replaces landing page
YouTube iframe with a static thumbnail link for cross-origin isolation
compatibility.
Move direct cross-feature import of settings-store through a deps/*
adapter module to satisfy the boundary check.
Add a centralized editor layout system with compact and default density
presets, driven by CSS custom properties. Compact (the new default) fits
more of the editor on a 1080p screen by shrinking toolbar, sidebars,
preview controls, timeline headers, track heights, and clip row sizes.
Users can switch between presets in Settings > Interface. Also adds
@vercel/analytics and tightens various UI element sizes for the denser
layout.
Move color scopes out of the keyframe graph panel into a dedicated
resizable side panel in the preview area. The scopes toggle now opens a
ColorScopesMonitor alongside the program monitor (right side), with
independent drag-to-resize handles that respect min/max constraints when
both source monitor and scopes are open simultaneously.
- Extract renderOffscreenFrame to deduplicate frame rendering across capture methods
- Add in-flight guards for captureCanvasSource and GPU scope rendering
- Replace decay compute shader with encoder.clearBuffer for accumulator clearing
- Split waveform render into accumulate + renderAccumulated with renderBatch API
- Add configurable 3-up/all stack layout for embedded scopes panel
- Use aspect-ratio-based canvas resize instead of square constraint
Add a camera button to playback controls that captures the current
preview frame, downloads it as PNG, and imports it into the media
library. Simplify timecode display layout by using a single reserved
width for the entire component instead of per-span sizing. Add
importGeneratedImage method to media library service for editor-
generated assets with rollback on failure.
…errides

- Fix halftone shader bright edge by replacing outOfFrame geometric bounds
  check with alpha-based source coverage and clamped UV sampling
- Snap halftone grid to whole cells to prevent fractional border columns
- Disable importExternalTexture fast path which produced different edge
  pixel values than canvas 2D due to YUV→RGB conversion differences
- Use VideoFrame.visibleRect cropping in video extractor and tier-2
  frame drawing to avoid codec padding artifacts
- Pass preview effects overrides through to adjustment layer resolution
  so live-edited adjustment effects are reflected during preview
… stamp

Even cell counts land texture edges at cell boundaries (fract=0), exposing
paper/background color as a rectangular border. Forcing odd counts centers
the grid so edges land at fract=0.5 (dot centers), making the pattern
bleed seamlessly to the boundary.
…zmo cache invalidation

- Canvas panel snaps width/height to nearest even on commit and steps by 2,
  preventing odd dimensions that video codecs reject
- LinkedDimensions accepts a step prop for callers that need non-default steps
- Export validateSettings auto-rounds odd dimensions as a safety net instead
  of rejecting with an error
- Fix stale fast-scrub cache during single-item gizmo transforms by
  invalidating the current frame when previewTransform changes
Store the active clip inspector tab (transform/effects/media) in the
editor store so it survives selection changes. Falls back gracefully
when the remembered tab is unavailable for the current item type.
… utilities

Enable drag-and-drop media placement directly onto the preview canvas,
positioning items at the drop point with smart track selection. Refactor
timeline track drop logic to share item-building and file-extraction
utilities, reducing duplication across canvas and timeline drop targets.
Lock non-canvas UI regions (toolbar, sidebars, timeline, controls) during
pen mode via InteractionLockRegion overlay. Add a Drawing Mode HUD with
progress hints and finish/cancel buttons. Place new mask shapes at the
playhead on the best available track using shared drop-placement logic,
and prevent timeline overlaps in addItem/addItems via collision detection.
Also swallow all non-modifier keyboard shortcuts while pen mode is active
and fix vectorscope overflow in color scopes.
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Mar 17, 2026

Too many files changed for review. (129 files found, 100 file limit)

@vercel
Copy link
Copy Markdown

vercel Bot commented Mar 17, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
freecut Ready Ready Preview, Comment Mar 17, 2026 2:17pm

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Mar 17, 2026

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This pull request introduces a comprehensive editor layout density system, browser-based media transcription with caption generation, multi-panel preview layout with draggable splits, collision-aware timeline item placement, and extensive UI refinements across the editor, settings, and preview components. New features include settings for editor density and Whisper transcription defaults, Popover and Combobox UI components, and support for media frame capture to the library.

Changes

Cohort / File(s) Summary
Configuration & Environment
.gitignore, eslint.config.js, index.html, vite.config.ts, vercel.json
Added tmp ignore entry, ESLint ignore pattern, crossorigin attribute to Google Fonts link, and COEP/COOP security headers for dev/preview servers.
Package Dependencies
package.json
Added @radix-ui/react-popover and @vercel/analytics dependencies.
Core Layout System
src/shared/ui/editor-layout.ts
New comprehensive editor layout system with density presets (default/compact) defining dimensions for toolbar, sidebars, preview, timeline, and video waveform; includes CSS variable integration and width clamping utilities.
Local Inference Runtime Management
src/shared/state/local-inference/...
New store, registry, types, and utilities for managing local inference runtimes; includes state tracking, unload lifecycle, summary aggregation, and formatted byte/progress estimates.
Transcription System
src/features/media-library/transcription/...
Complete browser-based transcription pipeline via Web Workers (Whisper, decoder, bridge orchestration), types, runtime estimates, and streaming iteration support.
Media Transcription Service
src/features/media-library/services/media-transcription-service.ts
High-level transcription and caption-insertion workflow; resolves clips, manages caption tracks, and handles existing caption replacement.
Caption Utilities
src/features/media-library/utils/caption-items...
Comprehensive caption generation, framing, track selection, and text-item building utilities with normalization and styling template support.
Media Library Drop Handling
src/features/media-library/utils/file-drop.ts, src/features/media-library/stores/media-import-actions.ts
File System Access API drag-drop validation, media-file entry extraction, and refactored import task orchestration with optimistic UI and deduplication support.
Media Library Service Extensions
src/features/media-library/services/media-library-service...
Added image import, transcript cleanup, and supporting utilities (dimensions, blob conversion, thumbnail generation).
Media Library State & Types
src/features/media-library/stores/media-library-store.ts, src/features/media-library/types.ts
New transcript status/progress state maps and actions; transcript result types and action exports.
Settings & Storage Schema
src/features/settings/..., src/infrastructure/storage/indexeddb/...
Added editor density and Whisper defaults to settings store; new transcripts object store with media/creation indexes; new Transcript type and CRUD accessors.
UI Components
src/components/ui/popover.tsx, src/components/ui/combobox.tsx
New Radix-based Popover wrapper and custom Combobox with search, filtering, keyboard navigation, and ResizeObserver width-matching.
Editor Store & State
src/shared/state/editor/..., src/features/editor/stores/editor-store.ts
Added clipInspectorTab, colorScopesOpen, and sidebar layout synchronization; layout-aware width normalization and loading logic.
Editor Layout Integration
src/features/editor/...
Updated toolbar, media sidebar, properties sidebar, and preview area to use dynamic layout values from editor density; added pen-mode interaction locking via InteractionLockRegion component.
Preview Area Enhancement
src/features/editor/components/preview-area.tsx
Restructured to support draggable source-monitor and color-scopes splits with responsive sizing, pen-mode locking, and dynamic layout-driven padding.
Pen Mode & Mask Editor
src/features/preview/components/mask-editor-overlay..., src/features/editor/components/interaction-lock-region.tsx
Added pen-mode HUD with Finish/Cancel buttons, keyboard shortcuts (Escape, Backspace), and InteractionLockRegion component for disabling UI during drawing.
Canvas Media Drop
src/features/preview/hooks/use-canvas-media-drop.ts, src/features/preview/components/gizmo-overlay.tsx
New drag-drop hook for canvas media placement with state-tracking, coordinate conversion, bounds clamping, and multi-source handling (library/external files); integrated into GizmoOverlay.
Timeline Drop Placement
src/features/timeline/utils/drop-placement..., src/features/timeline/components/timeline-track.tsx
New drop placement utilities for best-fit track/frame selection; refactored TimelineTrack drop logic to support external files and concurrent item resolution with collision detection.
Timeline Item Placement & Overlays
src/features/timeline/stores/actions/item-actions.ts, src/features/timeline/stores/timeline-item-overlay-store.ts
Added collision-aware placement on tracks and new overlay store for tracking segment status (progress, tone) per item; used for caption generation progress display.
Timeline Item Enhancements
src/features/timeline/components/timeline-item/...
Added caption generation support with context menu actions, progress tracking, segment overlays, and integration with transcription service.
Timeline Layout & Constants
src/features/timeline/constants.ts, src/features/timeline/components/timeline-markers.tsx, src/features/timeline/theme.css
Replaced hard-coded dimensions with editor-layout-driven values; CSS vars for track height and sidebar width with fallbacks.
Export Pipeline Enhancement
src/features/export/utils/canvas-...
Added preview effects override callback to adjustment-layer effects retrieval; integrated visibleRect source-region drawing in video frame extraction with fallback handling.
Playback Controls & Frame Capture
src/features/preview/components/playback-controls...
Added Save Frame button to capture current frame as PNG and import to media library with metadata (dimensions, tags, codec).
Settings Dialog
src/features/editor/components/settings-dialog.tsx, src/routes/settings.tsx
New Interface section with editor density selector; Whisper defaults section with model, quantization, language selectors; LocalInferenceUnloadControl integration.
Local Inference Control
src/features/settings/components/local-inference-unload-control.tsx
New component for UI-driven unloading of all local inference runtimes with status feedback and toast notifications.
Color Scopes Refactor
src/shared/components/color-scopes-view.tsx, src/features/preview/components/color-scopes-monitor.tsx
ColorScopesView now supports 'stack' layout mode with per-section headers and layout controls; new ColorScopesMonitor wrapper component for embedded preview panels.
GPU Effects & Scopes
src/lib/gpu-effects/effects/stylize.ts, src/lib/gpu-scopes/...
Tightened halftone sampling boundaries with safe UV clamping and alpha early-exit; added renderBatch for waveform scope to support concurrent multi-canvas rendering; removed decay compute pipeline.
Whisper Settings Utilities
src/shared/utils/whisper-settings.ts
Comprehensive Whisper configuration with model/quantization/language defaults, options arrays, normalization, and formatter helpers for UI consumption.
Transcription Progress Utilities
src/shared/utils/transcription-progress.ts
New progress tracking utilities mapping stages (loading/decoding/transcribing) to overall percent, merging snapshots, and providing readable labels.
Type Definitions
src/types/storage.ts, src/types/timeline.ts, src/types/project.ts
New MediaTranscript, MediaTranscriptModel, MediaTranscriptQuantization, MediaTranscriptSegment types; added optional captionSource to TextItem and ProjectTimeline items.
Cross-Feature Adapter Contracts
src/features/.../deps/...
New and extended adapter modules for preview, timeline, and editor to re-export transcription, settings, drop-placement, and file utilities across feature boundaries.
Tests & Documentation
src/features/.../...test.ts, src/features/timeline/deps/README.md
Comprehensive test suites for transcription, caption utilities, playback controls, timeline placement, and color scopes; documentation update for media-transcription-service dependency.
Routes & Landing Page
src/routes/index.tsx, src/routes/settings.tsx
Replaced embedded YouTube iframe with clickable link card; added full settings UI for editor density and Whisper transcription configuration.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • PR #102: Introduces ColorScopesPanel component and ColorScopesMonitor wrapper referenced in this PR's preview monitor integration.
  • PR #77: Modifies timeline drop/placement and batch item-addition collision logic; directly overlaps with the new drop-placement utilities and TimelineTrack refactoring.
  • PR #114: Updates ClipPanel tab handling with controlled state and availability synchronization; aligns with the new clipInspectorTab editor store field and related logic.

Suggested labels

codex


🐰 Transcription streams through workers,
Caption captions dance on tracks,
Density bends the layout tight,
Colors scoped in stacks—hooray!

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch develop

@walterlow walterlow merged commit fc7d7ce into main Mar 17, 2026
4 of 5 checks passed
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b921dbbb0a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +73 to +75
if (hasWebCodecs) {
this.decoderWorker.postMessage({ type: 'init', file });
return;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fall back to AudioContext when WebCodecs decode is unsupported

When AudioDecoder exists, start() always takes the WebCodecs path and returns immediately, but the decoder worker can reject specific codecs (AudioDecoder.isConfigSupported(...)), which then hard-fails transcription instead of using the existing decodeWithAudioContext fallback. In browsers that expose WebCodecs but lack support for a given audio codec, users will be unable to transcribe files that the AudioContext path could still decode; this regression is introduced by the unconditional early return in the WebCodecs branch.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant