Skip to content

fix: sanitize terminal app accessibility context to prevent garbled transcription #99

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MundusCaeli wants to merge 4 commits into
base: main
Choose a base branch
from
Open

fix: sanitize terminal app accessibility context to prevent garbled transcription #99

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
506c5fd
fix: sanitize terminal app accessibility context to prevent garbled t…
MundusCaeli Feb 14, 2026
38d0498
fix: send appType as 'default' for terminal apps to avoid server vali…
MundusCaeli Feb 14, 2026
e2b5c5c
refactor: strip ANSI sequences instead of removing entire terminal co…
MundusCaeli Feb 14, 2026
71f20da
fix: address review feedback - strip ANSI from selectedText, add comm…
MundusCaeli Feb 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion apps/desktop/src/pipeline/providers/formatting/formatter-prompt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { GetAccessibilityContextResult } from "@amical/types";
/**
* Application type for formatting context
*/
export type AppType = "email" | "chat" | "notes" | "amical-notes" | "default";
export type AppType = "email" | "chat" | "notes" | "amical-notes" | "terminal" | "default";

/**
* App-type specific formatting rules inserted into the system prompt
Expand Down Expand Up @@ -38,6 +38,8 @@ const APP_TYPE_RULES: Record<AppType, string> = {
- Use code blocks (\`\`\`) for technical content, commands, or code snippets
- Keep formatting minimal and purposeful - don't over-format simple content
- Preserve natural speech flow while adding structure where it improves clarity`,
// Terminal apps use universal rules only — no app-specific overrides.
terminal: "",
default: "",
Comment on lines +42 to 43
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Empty-string entries silently suppress default rules/examples for terminal apps.

APP_TYPE_RULES.terminal and APP_TYPE_EXAMPLES.terminal are "". Because ?? only falls through on null/undefined (not empty string), the template on lines 226 and 231 will inject an empty string rather than the default rules/examples:

${APP_TYPE_RULES[appType] ?? APP_TYPE_RULES.default ?? ""}
//                 ""     → "" (not null, so ?? doesn't trigger)

If the intent is for terminal apps to use no app-specific rules (relying on universal rules only), this is fine but worth a brief comment. If the intent is to fall back to default rules, change the values to undefined or use || instead of ??.

Also applies to: 110-111

🤖 Prompt for AI Agents 
In `@apps/desktop/src/pipeline/providers/formatting/formatter-prompt.ts` around
lines 41 - 42, APP_TYPE_RULES.terminal and APP_TYPE_EXAMPLES.terminal are set to
empty strings which prevents the nullish-coalescing (??) in the template from
falling back to defaults; either change those terminal values to undefined (so
APP_TYPE_RULES[appType] ?? APP_TYPE_RULES.default works as intended) or update
the template to use logical OR (||) instead of ?? to treat empty string as
falsy; if empty string is intentional (meaning no app-specific rules/examples),
add a clarifying comment next to
APP_TYPE_RULES.terminal/APP_TYPE_EXAMPLES.terminal indicating that behavior.

};

Expand Down Expand Up @@ -106,6 +108,8 @@ We decided to ship on Friday.

- Benchmark performance
- Update docs</formatted_text>`,
// Terminal apps use universal examples only — no app-specific examples.
terminal: "",
default: `### Filler removal + grammar fix:
<input>so the main issue is that um we need more time</input>
<formatted_text>The main issue is that we need more time.</formatted_text>
Expand Down Expand Up @@ -284,6 +288,14 @@ const BUNDLE_TO_TYPE: Record<string, AppType> = {
"com.evernote.Evernote": "notes",
"notion.id": "notes",
"com.agiletortoise.Drafts-OSX": "notes",
"com.apple.Terminal": "terminal",
"com.googlecode.iterm2": "terminal",
"com.mitchellh.ghostty": "terminal",
"dev.warp.Warp-Stable": "terminal",
"net.kovidgoyal.kitty": "terminal",
"com.github.wez.wezterm": "terminal",
"io.alacritty": "terminal",
"co.zeit.hyper": "terminal",
};

// Browser bundle identifiers
Expand Down
61 changes: 40 additions & 21 deletions apps/desktop/src/pipeline/providers/transcription/amical-cloud-provider.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,18 @@ import {
type CloudErrorResponse,
} from "../../../types/error";

// Strip ANSI escape sequences and control characters from accessibility text.
// Terminal apps expose raw terminal output via the accessibility API which
// contains escape codes that degrade transcription quality.
const ANSI_REGEX =
// eslint-disable-next-line no-control-regex
/[\u001b\u009b][[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><~]/g;

function stripAnsiSequences(text: string | undefined | null): string | undefined {
if (!text) return undefined;
return text.replace(ANSI_REGEX, "");
}
Comment on lines +25 to +28
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

stripAnsiSequences("") returns undefined instead of "".

The !text guard is true for empty strings, so an input of "" is coerced to undefined. If the server or any downstream consumer distinguishes between an absent field and an empty string, this could cause subtle differences in behavior.

If this is intentional (empty and absent are equivalent), this is fine. Otherwise, a narrower null-check would preserve empty strings:

♻️ Suggested fix 
 function stripAnsiSequences(text: string | undefined | null): string | undefined {
-  if (!text) return undefined;
+  if (text == null) return undefined;
   return text.replace(ANSI_REGEX, "");
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
function stripAnsiSequences(text: string | undefined | null): string | undefined {
if (!text) return undefined;
return text.replace(ANSI_REGEX, "");
}
function stripAnsiSequences(text: string | undefined | null): string | undefined {
if (text == null) return undefined;
return text.replace(ANSI_REGEX, "");
}
🤖 Prompt for AI Agents 
In `@apps/desktop/src/pipeline/providers/transcription/amical-cloud-provider.ts`
around lines 25 - 28, The current stripAnsiSequences function treats empty
string as absent because it uses a truthy check; change the guard to only return
undefined for null/undefined (e.g., check text == null or text === undefined ||
text === null) so that stripAnsiSequences("") returns "" and non-null empty
strings are preserved, then continue to call text.replace(ANSI_REGEX, "") for
other inputs; reference the stripAnsiSequences function and ANSI_REGEX when
making this change.


// Type guard to validate error codes from server
const isValidErrorCode = (code: string | undefined): code is ErrorCode =>
code !== undefined && Object.values(ErrorCodes).includes(code as ErrorCode);
Expand Down Expand Up @@ -283,28 +295,35 @@ export class AmicalCloudProvider implements TranscriptionProvider {
enabled: enableFormatting,
},
sharedContext: this.currentAccessibilityContext
? {
selectedText:
this.currentAccessibilityContext.context?.textSelection
?.selectedText,
beforeText:
this.currentAccessibilityContext.context?.textSelection
?.preSelectionText,
afterText:
this.currentAccessibilityContext.context?.textSelection
?.postSelectionText,
appType: detectApplicationType(
? (() => {
const appType = detectApplicationType(
this.currentAccessibilityContext,
),
appBundleId:
this.currentAccessibilityContext.context?.application
?.bundleIdentifier,
appName:
this.currentAccessibilityContext.context?.application?.name,
appUrl:
this.currentAccessibilityContext.context?.windowInfo?.url,
surroundingContext: "", // Empty for now, future enhancement
}
);
const isTerminal = appType === "terminal";
return {
selectedText: stripAnsiSequences(
this.currentAccessibilityContext!.context?.textSelection
?.selectedText,
),
beforeText: stripAnsiSequences(
this.currentAccessibilityContext!.context?.textSelection
?.preSelectionText,
),
afterText: stripAnsiSequences(
this.currentAccessibilityContext!.context?.textSelection
?.postSelectionText,
),
appType: isTerminal ? "default" : appType,
appBundleId:
this.currentAccessibilityContext!.context?.application
?.bundleIdentifier,
appName:
this.currentAccessibilityContext!.context?.application?.name,
appUrl:
this.currentAccessibilityContext!.context?.windowInfo?.url,
surroundingContext: "",
};
})()
Comment thread
coderabbitai[bot] marked this conversation as resolved.
: undefined,
}),
});
Expand Down
7 changes: 6 additions & 1 deletion apps/desktop/src/pipeline/providers/transcription/whisper-provider.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,8 +277,13 @@ export class WhisperProvider implements TranscriptionProvider {
return aggregatedTranscription;
}

const beforeText =
const rawBeforeText =
accessibilityContext?.context?.textSelection?.preSelectionText;
// Strip ANSI escape sequences that terminal apps expose via accessibility
// context — they degrade Whisper recognition quality.
const beforeText = rawBeforeText
// eslint-disable-next-line no-control-regex
?.replace(/[\u001b\u009b][[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><~]/g, "");
if (beforeText && beforeText.trim().length > 0) {
logger.transcription.debug(
`Generated initial prompt from before text: "${beforeText}"`,
Expand Down