feat(star6e): digital image stabilization (DIS) on VPE by snokvist · Pull Request #81 · OpenIPC/waybeam_venc

snokvist · 2026-05-21T18:20:55Z

Adds opt-in digital image stabilization for the Star6E backend, exposed through a unified video0.framing preset:

stab: off | low | medium | high
zoom: zoom-1.25x | zoom-1.50x | zoom-1.75x | zoom-2x

(mutually exclusive; zoom works on both backends, stab is Star6E-only.)

How it works

Hardware-crop path: VPE port0 hardware-crops the stab window via MI_VPE_SetPortCrop straight into a VENC bind (zero-copy, no manual drain, torn down by the standard bound-port path). A tiny port1 256×256 detector tap is manually drained and disabled via a pause/park + pthread_join quiesce on stop. AE meter follows the stabilized crop window; source clamps to ≤1920×1080 when stab is active to avoid the high-res fps regression. Builds on the v0.11.0 zoom/pan-ramp base.

Validation

make verify clean (both backends) + host unit tests pass.

Device-validated on ssc338q/imx335 (v0.12.0), on top of master with the debug_osd_destroy MMU-race fix (#79) + cold-vif (#80):

18 stab-path respawns → 0 client-0x15 MMU faults, 0 oops, 0 reboots, uptime continuous, ~3.9 MB/3s RTP egress every transition:

4× bare same-mode /api/v1/restart with stab active (the exact historic ~40%-wedge scenario)
all 4 stab framing presets + stab on↔off cycling
both zoom presets (zoom-1.50x / zoom-2x)
4× rapid soak with stab active (cumulative-degradation risk)

The teardown wedge that previously hit ~40% of stab restarts was the same debug_osd_destroy client-0x15 race, now fixed at the root (#79) underneath this feature.

Follow-up fix — `framing=off` stab leak (commit `6ae9db4`)

framing="off" still produced stabilized frames whenever a non-zero stabCropPct was left in the config from a prior stab session: load_video0 applied the stabCropPct/stabRecenterSpeed overrides after the framing preset unconditionally, overwriting the off preset's stab_crop_pct=0, and star6e_stab_enabled() gates stab solely on stab_crop_pct >= 50 (never checks framing). On cold boot this also drove the fragile stab IVE init at framing=off. Fix: honor the two overrides only when framing=="stab"; off/zoom keep the cleared 0/0. Self-heals on next save and covers the MUT_RESTART HTTP path. Regression tests added. Device-verified on ssc338q/imx335: framing=off → no stab/ive threads, full 2560×1920 ~58 fps, clean RTP egress.

Known limitations (separate, not addressed here)

Stab cold-init IVE hang on a watchdog cold-boot with framing="stab" persisted — the normal SET/respawn apply path is clean. (With the follow-up fix above, a stale stabCropPct at framing=off no longer triggers this path.)
image.flip encoder stall on imx335 (sensor-specific).

🤖 Generated with Claude Code

Adds opt-in digital image stabilization for the Star6E backend, exposed through a unified video0.framing preset (off | low | medium | high for stab; zoom-1.25x | zoom-1.50x | zoom-1.75x | zoom-2x for zoom — mutually exclusive). Stab uses a hardware-crop path: VPE port0 hardware-crops the stab window via MI_VPE_SetPortCrop straight into a VENC bind (zero-copy, no manual drain), with a tiny port1 256x256 detector tap manually drained and torn down via a pause/park + pthread_join quiesce on stop. AE meter follows the stabilized crop window; source clamps to <=1920x1080 when stab is active to avoid the high-res fps regression. Builds on the v0.11.0 zoom/pan-ramp base. make verify clean (both backends) + host unit tests pass. Device-validated on 192.168.1.13 (ssc338q/imx335, v0.12.0), rebased onto master after the debug_osd_destroy MMU-race fix (#79) + cold-vif (#80): 18 stab-path respawns (bare same-mode restart with stab active x4, all framing presets, on<->off cycling, both zoom presets, 4x rapid soak) => 0 client-0x15 MMU faults, 0 oops, 0 reboots, uptime continuous, ~3.9 MB/3s RTP egress every transition. The teardown wedge that previously hit ~40% of stab restarts was the same debug_osd_destroy client-0x15 race, now fixed at the root underneath the feature. Known limitations (separate, not addressed here): the stab cold-init IVE hang on a watchdog cold-boot with framing!=off persisted (the normal SET/respawn apply path is clean), and an image.flip encoder stall on imx335. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

zoomX/zoomY are the zoom-mode pan and were also being applied as the stab crop window's center. An off-center pan (e.g. a stale zoomX=0.7 left from a prior zoom session) positions the stab window against the frame edge, where SetPortCrop clamps it — leaving the accumulator no room to move, so stabilization silently does nothing (observed on device: framing=medium with zoomX/zoomY=0.7 produced acc that never shifted the crop). Stabilization must always be centered so the accumulator has symmetric ±max headroom on both axes. Force 0.5/0.5 at stab start and ignore live zoomX/zoomY updates while stabilizing; zoom modes are unchanged and still pan by zoomX/zoomY. The saved zoomX/zoomY are preserved (not clobbered), so switching back to a zoom preset restores the user's pan. Device-verified (192.168.1.13, imx335): with zoomX/zoomY=0.7 on disk, framing=medium now reports pan=(500,500); a live zoom_x=0.8 SET during stab leaves pan at 500; framing=zoom-2x still zooms/pans by zoomX. make verify clean (both backends) + host tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The HW-crop refactor cheapened the Shift_Detector geometry (search crop 384→256, correlation box 256→128, pyramid 3→2) to buy fps. That made the per-frame motion estimates noticeably noisier; since the offset is applied raw every frame (no temporal smoothing), the noise shows up as visibly shaky/jittery stabilization. User confirmed it "used to be more smooth." Restore the full 384/256/3 detector. On this SoC the cost is hidden: the stab presets clamp source to ≤1080p, so the heavier detector still fits the frame budget — device-measured 89fps at off/low/medium/high (192.168.1.13, imx335), identical to the cheap detector, 0 client-0x15 faults. The ~40fps figure that motivated the cheapening was at un-clamped 2048×1536. The stab tick now tracks real 2D motion (meas both axes, acc builds to meaningful offsets) instead of the small/noisy values the cheap detector produced. DETECT_EVERY stays 1 (detect+correct every frame). make verify clean (both backends) + host tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…m/high The three stab presets traded crop-border against magnification and only the middle one felt right on device: 'low' (90% crop) saturated its tiny border and juddered, 'high' (65% crop) magnified per-frame jitter ~1.5× and juddered, while 'medium' (80%) sat in the sweet spot. Collapse them into one tuned mode. Framing values are now: off | stab | zoom-1.25x | zoom-1.50x | zoom-1.75x | zoom-2x. 'stab' = the former medium: 80% centered crop, recenter tau 180. Smoothness work folded in (the reason medium alone was clean is now applied universally): - EMA low-pass on the applied crop offset (STAB_OUTPUT_SMOOTH_ALPHA 0.30, ~3-frame time constant) so per-frame detector/quantization jitter doesn't reach the crop as judder — the raw-offset apply was the root of the shake. - Stiffer recenter hold: STILL_FRAMES 30→60, EDGE_PCT 70→88, so the stabilized view holds its position longer instead of creeping back ("locked scene" feel the user asked for). Backward compat: retired low/medium/high config values migrate to "stab" on load (load_video0) so existing configs keep stabilization rather than silently falling back to off; SET only accepts the new value set. Schema touch points updated: preset table + migration (venc_config.c), validator message (venc_api.c), webui enum/help (dashboard.html + regenerated venc_webui.c), HTTP_API_CONTRACT, and the config unit test. Device-verified (192.168.1.13, imx335): SET stab → centered 80% crop, smooth, 0 client-0x15 faults; legacy "medium" on disk migrates to stab; SET high/medium rejected with the updated message; zoom presets unchanged. make verify clean (both backends) + host tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Extend the framing preset enum with two deeper Approach-C zoom levels: zoom-3x zoom_pct 0.3333 1080p -> 640x352 zoom-4x zoom_pct 0.25 1080p -> 480x256 Approach-C shrinks crop+output 1:1 (no upscale), so the deep crops are not bound by the ch1 ~2x SCL upscale ceiling; both stay above the 256-px floor. Live zoomX/zoomY panning works on both, same as the existing zoom presets. Device-verified on Star6E (ssc338q/imx335, 192.168.1.13): all framing modes apply with correct encode dims (off/stab/1.25x..4x), pan sweeps at 3x/4x clean, 0 client-0x15 faults across ~18 respawns. Docs: README "Digital Zoom" section rewritten as "Framing: Stabilization & Digital Zoom" — replaces the removed settable zoom_pct API with the framing preset table (incl. stab + migration) and adds zoom pan curl examples. HTTP_API_CONTRACT framing enum + semantics + changelog updated. - src/venc_config.c: preset table + comment - src/venc_api.c: validator message + field comment - web/dashboard.html: framing enum + tooltip (venc_webui.c regenerated) - tests/test_venc_config.c: zoom-3x/4x load assertions (1609 pass) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

snokvist · 2026-05-21T20:02:05Z

Added a follow-on commit: zoom-3x / zoom-4x digital zoom presets (mirrors fork PR #130).

Extends video0.framing with two deeper Approach-C zoom levels on top of the stab refactor in this PR:

`framing`	zoom_pct	encode dim @1080p
`zoom-3x`	0.3333	640×352
`zoom-4x`	0.25	480×256

Approach-C shrinks crop and output 1:1 (no upscale), so the deep crops are not bound by the ch1 ~2× SCL upscale ceiling; both stay above the 256-px floor. Live zoomX/zoomY panning works identically to the existing zoom presets.

Device-verified on Star6E (ssc338q/imx335): all framing modes apply with correct encode dims (off/stab/1.25x..4x), pan sweeps at 3×/4× clean, 0 client-0x15 faults across ~18 framing respawns. make verify clean on both backends, 1609/1609 host tests.

…ed) (#131) The single-stab-mode change added a load-time migration mapping the retired low/medium/high framing presets to "stab". That binary was never shipped, so no config in the wild carries those values — the migration is dead code. Remove it; low/medium/high now fall back to "off" on load like any other unknown framing value (with the existing WARNING). - src/venc_config.c: drop the migration block + update comment - tests/test_venc_config.c: assert retired names fall back to off - README / HTTP_API_CONTRACT: drop the migration note + changelog fix make verify clean on both backends; 1608/1608 host tests. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…nobs (#132) The single-stab-preset refactor baked crop% and recenter speed at 80/180. Per field testing, re-expose them as advanced overrides so the strength and behavior can be tuned (notably recenter=0 "stick to patch", which makes stabilization most visible for demos — though it drifts to the border in real use). Semantics: - video0.stab_crop_pct (0 = preset default 80; else 50..100): kept-frame %. Smaller % = bigger dead border = more motion headroom, tighter frame. - video0.stab_recenter_speed (0..3600): glide-back decay constant in frames. 0 = stick (no recenter); higher = slower; preset default 180. Both restart-required, aliases stabCropPct/stabRecenterSpeed. Read AFTER framing-preset expansion so a plain framing=stab keeps 80/180 while explicit values win; re-selecting framing=stab resets them. Inert under off/zoom. Persisted, so overrides survive the reinit respawn. Device-verified (Star6E 192.168.1.13): framing=stab → 80/180; SET stabRecenterSpeed=0 → "recenter=0 (stick)" in stab thread + persisted; SET stabCropPct=60 → out 1152x648 (60% crop); out-of-range rejected; 0 client-0x15 faults across reinits. - venc_api.c: g_fields + aliases + range validation - venc_config.c: load override-after-preset, pretty-print, JSON serialize - config/waybeam.default.json: stabCropPct/stabRecenterSpeed defaults (0/0) - web/dashboard.html: fields + tooltips (venc_webui.c regenerated) - README / HTTP_API_CONTRACT: advanced-tuning docs + examples - tests: override-wins + absent-keeps-default (1612 pass) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

snokvist · 2026-05-21T20:51:55Z

Added a follow-on commit: re-exposed stab_crop_pct + stab_recenter_speed as advanced tuning knobs (mirrors fork PR #132).

The single-stab preset baked these at 80/180; this lets them be overridden (notably stabRecenterSpeed=0 = stick-to-patch for demos). Read after preset expansion so framing=stab alone keeps 80/180; explicit values win and persist across the reinit respawn. Both restart_required, inert under off/zoom-*.

Device-verified on Star6E: framing=stab→80/180; stabRecenterSpeed=0→recenter=0 (stick) + persisted + survives respawn; stabCropPct=60→out 1152×648; out-of-range rejected; 0 client-0x15 faults. make verify clean, 1612/1612 host tests.

framing="off" still produced stabilized frames whenever a non-zero stabCropPct was left in the config from a prior stab session. load_video0 applied the stabCropPct/stabRecenterSpeed overrides AFTER the framing preset *unconditionally*, overwriting the off preset's stab_crop_pct=0; star6e_stab_enabled() then gates stab solely on stab_crop_pct>=50 (it never checks framing), so stab ran. On cold boot this also drove the fragile stab IVE init at framing=off, which can wedge the daemon (mi_log D-state / VPE teardown). Honor the two overrides only when framing=="stab" so off/zoom keep the preset's cleared 0/0. Self-heals on next save and covers the HTTP path too (stabCropPct is MUT_RESTART -> respawn re-reads JSON). Add regression tests for framing=off and zoom with a stale override. Device-verified on Star6E 192.168.1.13 (imx335): framing=off -> no stab threads, full 2560x1920 ~58fps, clean RTP egress. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

framing=stab previously exposed only stabCropPct + stabRecenterSpeed; the remaining feel parameters were compile-time constants. Surface four more as video0 fields (MUT_RESTART, scoped to framing=stab, mirroring the existing override plumbing) so the smoothness/lock/sensitivity can be tuned without a rebuild: stabSmoothPct EMA low-pass on the applied offset (5..100; the primary judder/smoothness knob; default 30) stabStillFrames stillness frames before recenter starts (0..600; 60) stabEdgePct dead-border % used before margin reclaim (50..100; 88) stabMotionThresh px shift counted as "moving" (0..16; 1) The stab thread now reads these from globals seeded in star6e_stab_configure (clamped, with compile-time-default fallbacks for hand-edited configs); the stab start log echoes the active values. README gains a calibration guide (using the stab tick log as the instrument); HTTP_API_CONTRACT documents the fields; +16 host test assertions. Device-verified on imx335 (192.168.1.13): overrides flow through, warm restart clean, 60fps. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

snokvist · 2026-05-23T22:13:36Z

Added one commit (e888e14) extending the stab preset with four advanced "feel" tuning knobs (video0.stab_smooth_pct / stab_still_frames / stab_edge_pct / stab_motion_thresh), following the same MUT_RESTART + framing-scoped override pattern as the existing stab_crop_pct / stab_recenter_speed. README gains a calibration guide; HTTP_API_CONTRACT documents the fields; +16 host-test assertions. Both backends cross-compile and 1627/1627 host tests pass locally; device-verified on imx335 (overrides flow through, warm restart clean, 60 fps).

🤖 Generated with Claude Code

…libs (#134) The Star6E binary dlopens libmi_ive.so (image stabilization) and libmi_rgn.so (debug OSD) at runtime, but neither was present in libs/star6e/ — the firmware's /rom/usr/lib provided them implicitly, so the feature breaks on any firmware drop that lacks libmi_ive.so. Add both libs (pulled from a verified imx335 device's /rom/usr/lib, symbol-checked against the dlsym call sites), plus a README + MD5SUMS documenting the set (mirrors vendor-libs/maruko), and a `push-libs` command in scripts/star6e_direct_deploy.sh to deploy them to /usr/lib. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

snokvist · 2026-05-23T22:16:23Z

Added the runtime libs needed for stabilization to this PR (commit b57c4d0):

libs/star6e/libmi_ive.so — software IVE, dlopened by the stab motion detector (MI_IVE_Shift_Detector). Without it framing=stab fails to start.
libs/star6e/libmi_rgn.so — debug-OSD overlay.
libs/star6e/README.md + MD5SUMS documenting the owned 13-lib reference set.
push-libs command in scripts/star6e_direct_deploy.sh to stream libs/star6e/*.so → device /usr/lib.

These belong here because the stabilization path dlopens libmi_ive.so at runtime and OpenIPC firmware drops don't always ship it — so the feature is incomplete without the lib bundled alongside the code that uses it. libmi_ive.so is self-contained (only needs libc/libgcc). md5sums verified against the committed MD5SUMS.

snokvist and others added 5 commits May 21, 2026 20:20

snokvist and others added 2 commits May 21, 2026 22:11

snokvist and others added 2 commits May 23, 2026 19:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(star6e): digital image stabilization (DIS) on VPE#81

feat(star6e): digital image stabilization (DIS) on VPE#81
snokvist wants to merge 10 commits into
OpenIPC:masterfrom
snokvist:feature/star6e-stabilization

snokvist commented May 21, 2026 •

edited

Loading

Uh oh!

snokvist commented May 21, 2026

Uh oh!

snokvist commented May 21, 2026

Uh oh!

snokvist commented May 23, 2026

Uh oh!

snokvist commented May 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

snokvist commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works

Validation

Follow-up fix — framing=off stab leak (commit 6ae9db4)

Known limitations (separate, not addressed here)

Uh oh!

snokvist commented May 21, 2026

Uh oh!

snokvist commented May 21, 2026

Uh oh!

snokvist commented May 23, 2026

Uh oh!

snokvist commented May 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

snokvist commented May 21, 2026 •

edited

Loading

Follow-up fix — `framing=off` stab leak (commit `6ae9db4`)