Abstraction layer for LLM with Windows-based applications

[gnh1201]

Summary

There is a growing need to integrate LLMs with various Windows-based applications and APIs, including but not limited to:

• Adobe Photoshop
• Autodesk AutoCAD
• hwp.exe (the de facto word processor in Korea)
• Stock market APIs commonly used in Korea, typically exposed as Windows COM/DLL modules [lib/stock-kr] Korean Stock Market #113

Currently, controlling each of these applications requires writing application-specific glue code. This approach does not scale and quickly becomes difficult to maintain.

What is needed is an abstraction layer that allows the capabilities, interfaces, and constraints of each target application to be defined declaratively, rather than by implementing custom logic for every application. The goal is to describe what an application can do and how it can be driven, without rewriting control code each time.

As a candidate for the declarative layer, Cucumber / Gherkin-style specifications are being considered. A behavior-driven, human-readable format could make it easier to:

• Describe application actions, states, and preconditions
• Map high-level LLM intents to concrete Windows automation steps
• Lower the learning curve for users defining new integrations

WelsonJS already supports automation of certain Windows applications (e.g., Microsoft Office), so the technical feasibility is proven. However, from a user perspective, the current model still exposes too much complexity. Reducing this complexity is essential for wider adoption.

Supporting MCP (Model Context Protocol) or a similar standardized interface may also be a viable approach, especially if combined with a declarative specification layer such as Cucumber, to provide a common contract between LLMs and Windows-based software. #233

Expected outcome:

• A declarative specification language for Windows application control
• Reduced per-application custom automation code
• Improved usability and extensibility for LLM ↔ Windows application integration
• Evaluation of Cucumber/Gherkin and MCP as complementary or alternative solutions

[linear]

CAT-98

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• [Roadmap] Add support MCP #233

👤 Suggested Assignees

• gnh1201

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Warning

A plan generation is already in progress. Please wait for it to complete before requesting a new plan.

[coderabbitai]

Coding Plan

Approach

Create an abstraction layer that enables LLMs to control Windows applications through declarative capability definitions rather than application-specific code, reducing complexity and improving scalability for integrating diverse applications (Office, Adobe, AutoCAD, hwp.exe, stock APIs) with LLM agents.

Observations

WelsonJS is a Windows automation framework built on COM object abstraction with JavaScript wrappers. The project follows consistent patterns: constructor-based module exports with create() factories, registry-based dispatchers (testloader.js's test_implements), JSON-driven configuration (routing, test specs, proxy capabilities), and a router pattern for content-type based dispatch. LLM integrations (chatgpt.js, anthropic.js, etc.) currently support only basic chat with no function calling. Automation modules (msoffice.js, shell.js, wmi.js, autoit.js) wrap COM objects with try-catch error handling and method chaining. The architecture emphasizes declarative JSON specifications over code-based configuration, making it well-suited for a capability registry approach.

Assumptions

Assumption 1: Capability Definition Format

Options Considered:

• Gherkin feature files with step definitions: BDD-style syntax requires parser and step definition maintenance overhead
• JSON schemas with descriptive metadata: Aligns with existing codebase patterns (test-*.json, available_proxies.json) and requires no new parser
• MCP protocol schemas: Adds protocol complexity before validating abstraction layer design

Chosen Option: JSON schemas with descriptive metadata: Aligns with existing codebase patterns (test-*.json, available_proxies.json) and requires no new parser

Rationale: This extends the existing test specification pattern (test-misc.json structure) and aligns with how the codebase already defines capabilities (available_proxies.json). Gherkin's value is readability, which JSON achieves through good field names and descriptions. This keeps implementation simple while providing a path to MCP protocol adaptation later.

Assumption 2: LLM Integration Approach

Options Considered:

• Implement full MCP server with JSON-RPC 2.0 transport: Standard protocol but significant upfront complexity
• Extend existing chat() functions with function calling support: Simpler but requires per-LLM-provider implementation
• Create generic function calling bridge that wraps all LLM modules: Single integration point, reusable across providers

Chosen Option: Create generic function calling bridge that wraps all LLM modules: Single integration point, reusable across providers

Rationale: This avoids modifying each LLM module individually and provides a single point for capability discovery, execution coordination, and result formatting. Existing prompt_function_calling_*.json files can inform the prompting strategy without requiring full integration initially.

Assumption 3: Execution Model for Application Control

Options Considered:

• Require each capability definition to specify executor type: Forces all capability authors to understand execution models
• Auto-detect executor based on application name/pattern: Too much implicit behavior, hard to debug
• Capability definitions reference automation modules, execution dispatched to appropriate module: Clear separation of concerns, reuses existing module architecture

Chosen Option: Capability definitions reference automation modules, execution dispatched to appropriate module: Clear separation of concerns, reuses existing module architecture

Rationale: This keeps capability definitions simple while maintaining flexibility and aligning with the existing module structure. For applications without existing modules, developers create new ones following established patterns (create() factory, method chaining).

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Capability Definition and Registry System

Establish the declarative specification format and runtime capability management infrastructure that will serve as the foundation for LLM-driven automation.

Task 1: Define Capability Schema and Example Definitions

Create the JSON schema structure for describing application capabilities, their parameters, constraints, and execution requirements.

• Define schema in data/schemas/capability.schema.json with fields for capability ID, human description, parameters (with types/constraints), executor module reference, and optional preconditions/postconditions
• Create example capability definitions in data/capabilities/ directory: excel-capabilities.json (open, close, write cell, read cell), autoit-capabilities.json (mouse click, keyboard send), generic template
• Include metadata fields for versioning, tags, and documentation links similar to test-*.json structure
• Use parameter schemas that specify type (string, number, boolean, object), required/optional flags, default values, and validation rules (min/max, regex patterns, allowed values)

Task 2: Implement Capability Registry

Build the runtime system for loading, validating, and querying capability definitions.

• Create lib/capability-registry.js following the constructor + create() factory pattern used by other lib/ modules
• Implement registry methods: loadCapabilities(directory), getCapability(id), listCapabilities(filter), validateCapabilitySchema(definition)
• Maintain in-memory capability index with fast lookup by ID and metadata-based filtering (by tag, executor, application)
• Include error handling for schema validation failures, duplicate capability IDs, and missing executor modules
• Support dynamic capability registration via addCapability(definition) for programmatic capability additions

Task 3: Create Capability Executor Dispatcher

Build the execution layer that routes capability invocations to the appropriate automation modules.

• Create lib/capability-executor.js with execute(capabilityId, parameters, context) as primary method
• Implement executor module loading using existing require() pattern: capability specifies executor name, dispatcher loads lib/[executor].js module
• Define standard executor interface contract: executor modules expose executeCapability(action, parameters) method
• Add parameter validation against capability schema before execution (type checking, constraint enforcement)
• Include result normalization: convert automation module responses to standard format (success/failure, data, error message)
• Implement try-catch error recovery pattern consistent with existing automation modules

🤖 Prompt for AI agents

Phase 1 establishes the foundation for declarative application control through a
capability registry system.

Create data/schemas/capability.schema.json defining the structure for capability
definitions with fields for ID, description, parameters (types, constraints,
validation rules), executor module reference, preconditions/postconditions, and
metadata (versioning, tags, docs).

Create example capability files in data/capabilities/: excel-capabilities.json
and autoit-capabilities.json with representative operations.

Implement lib/capability-registry.js following constructor + create() factory
pattern:
- loadCapabilities(directory) to load capability definitions from directory
- getCapability(id) for capability lookup by ID
- listCapabilities(filter) for metadata-based filtering
- validateCapabilitySchema(definition) for schema validation
- addCapability(definition) for dynamic registration
- Maintain in-memory index with fast lookups
- Handle errors for validation failures, duplicate IDs, missing executors

Implement lib/capability-executor.js as execution dispatcher:
- execute(capabilityId, parameters, context) as primary method
- Load executor modules dynamically using require() based on capability's
executor field
- Define executeCapability(action, parameters) as standard executor interface
- Validate parameters against capability schema before execution
- Normalize results to standard format (success/failure, data, error message)
- Use try-catch error handling consistent with existing modules

Phase 2: LLM Integration and Function Calling Bridge

Extend LLM integrations to support function calling and create the bridge between LLM requests and capability execution.

Task 1: Design Function Calling Message Protocol

Establish the communication format between LLMs and the capability system, including capability discovery and invocation.

• Define standard message schemas for: capability query (list available capabilities with optional filtering), capability invocation (capability ID + parameters), execution result (success/failure + data)
• Create prompt templates in data/prompts/ that instruct LLMs how to discover and invoke capabilities using the message protocol
• Reference prompt_function_calling_en.json structure but adapt for capability registry context instead of fixed get_search method
• Include system prompt sections for capability discovery flow, parameter construction, and error handling guidance
• Design result interpretation guidelines for LLMs (how to parse success/failure, extract data, retry on errors)

Task 2: Implement Function Calling Bridge

Create the coordination layer that manages LLM conversation flow with capability execution.

• Create lib/function-calling-bridge.js that wraps existing LLM chat() calls with capability execution loop
• Implement conversation handler: send initial prompt with available capabilities → process LLM response for function calls → execute requested capabilities → append results to conversation → continue until completion
• Add capability discovery endpoint: when LLM requests available actions, return filtered list from registry with descriptions and parameter schemas
• Implement execution coordinator: parse LLM function call requests, validate against capability schemas, delegate to capability-executor, format results for LLM consumption
• Include conversation history management with configurable depth limit and context pruning strategies
• Add tracing/logging for debugging LLM requests and capability execution flow

Task 3: Extend LLM Modules for Function Calling

Modify existing LLM integration modules to support the function calling protocol.

• Update lib/chatgpt.js, lib/anthropic.js, lib/groq.js to add functionCall(content, availableCapabilities) method alongside existing chat()
• Implement capability injection into system prompts using templates from data/prompts/
• Parse LLM responses for function call indicators (JSON blocks, specific markers) based on prompt instructions
• Handle multi-turn conversations where LLM alternates between function calls and text responses
• Add configuration for function calling vs. pure chat mode (backward compatibility with existing usage)
• Update module exports to expose both chat() for legacy usage and functionCall() for capability-driven automation

🤖 Prompt for AI agents

Phase 2 creates the bridge between LLM interactions and capability execution.

Define message schemas for LLM-capability communication:
- Capability query format (list capabilities with filtering)
- Capability invocation format (capability ID + parameters)
- Execution result format (success/failure + data)

Create prompt templates in data/prompts/ instructing LLMs on capability
discovery and invocation, adapting prompt_function_calling_en.json structure for
capability registry context. Include guidance for parameter construction, error
handling, and result interpretation.

Implement lib/function-calling-bridge.js as coordination layer:
- Wrap existing LLM chat() calls with capability execution loop
- Implement conversation handler managing flow: send prompt with capabilities →
process LLM response → execute capabilities → append results → continue
- Add capability discovery endpoint returning filtered capability lists with
schemas
- Implement execution coordinator parsing function calls, validating parameters,
delegating to capability-executor, formatting results
- Manage conversation history with configurable depth and pruning
- Add tracing/logging for debugging

Extend lib/chatgpt.js, lib/anthropic.js, lib/groq.js:
- Add functionCall(content, availableCapabilities) method alongside existing
chat()
- Inject capabilities into system prompts using templates
- Parse LLM responses for function call indicators
- Handle multi-turn conversations with alternating function calls and text
- Support configuration for function calling vs chat mode
- Export both chat() and functionCall() methods

Phase 3: Application Adapters and Documentation

Create capability definitions for existing automation modules and provide patterns for integrating new applications.

Task 1: Implement Capability Adapters for Existing Modules

Build the executor implementations for currently supported automation targets.

• Create executor adapters in lib/executors/ directory: excel-executor.js, autoit-executor.js following the executeCapability(action, parameters) interface contract
• Excel executor: map capabilities (open, close, writeCell, readCell, saveAs) to msoffice.js Excel methods, handle workbook/worksheet state management
• AutoIt executor: map capabilities (mouseClick, keyboardSend, mouseMove) to autoit.js methods, implement coordinate/window parameter validation
• Each executor implements parameter transformation: capability generic parameters → module-specific method arguments
• Include state tracking where needed (currently open workbook in Excel executor, active window in AutoIt executor)
• Add error translation: convert module-specific errors to standard capability error format

Task 2: Create Comprehensive Capability Definitions

Define complete capability libraries for the adapters implemented in Task 3.1.

• Create data/capabilities/excel.json with full suite of Excel operations: file operations (open, close, save, saveAs), cell operations (read, write, format), sheet operations (add, delete, rename, copy)
• Create data/capabilities/autoit.json with GUI automation operations: mouse (move, click, drag), keyboard (send keys, press combinations), window (activate, close, wait)
• Include rich parameter schemas: Excel cell references with validation patterns, AutoIt window selectors with description, coordinate constraints
• Add prerequisite declarations where capabilities depend on prior state (e.g., writeCell requires open workbook)
• Include usage examples in capability descriptions showing common parameter combinations

Task 3: Document Integration Patterns and Create Template

Provide guidance for developers adding new application integrations.

• Create docs/capability-integration-guide.md with sections: capability definition structure, executor module contract, parameter schema design, state management patterns, error handling conventions
• Include complete worked example: adding a new application (e.g., Notepad) with capability definitions, executor module, registration in capability registry
• Create data/capabilities/template.json as starting point for new capability definitions with annotated fields and common patterns
• Create lib/executors/template-executor.js showing minimal executor implementation with parameter validation, error handling, and result formatting
• Document testing approach: how to validate capability definitions, test executor implementations in isolation, debug LLM integration issues
• Include troubleshooting section: common issues (parameter validation failures, executor module not found, state inconsistencies) with resolution steps

🤖 Prompt for AI agents

Phase 3 implements adapters for existing automation modules and provides
integration patterns for new applications.

Create executor adapters in lib/executors/:
- excel-executor.js mapping capabilities (open, close, writeCell, readCell,
saveAs) to msoffice.js methods with workbook/worksheet state management
- autoit-executor.js mapping capabilities (mouseClick, keyboardSend, mouseMove)
to autoit.js methods with coordinate/window validation
- Both following executeCapability(action, parameters) interface
- Transform generic capability parameters to module-specific arguments
- Track state where needed (open workbooks, active windows)
- Translate module errors to standard capability error format

Create comprehensive capability definitions:
- data/capabilities/excel.json with file operations (open, close, save, saveAs),
cell operations (read, write, format), sheet operations (add, delete, rename,
copy)
- data/capabilities/autoit.json with mouse operations (move, click, drag),
keyboard operations (send keys, press combinations), window operations
(activate, close, wait)
- Include rich parameter schemas with validation patterns, constraints, and
usage examples
- Add prerequisite declarations for state-dependent capabilities

Create integration documentation and templates:
- docs/capability-integration-guide.md covering capability definition structure,
executor contract, parameter schema design, state management, error handling
- Include worked example adding new application (e.g., Notepad)
- data/capabilities/template.json as annotated starting point
- lib/executors/template-executor.js showing minimal implementation with
validation, error handling, result formatting
- Document testing approach and troubleshooting for common issues

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Phase 1 establishes the foundation for declarative application control through a
capability registry system.

Create data/schemas/capability.schema.json defining the structure for capability
definitions with fields for ID, description, parameters (types, constraints,
validation rules), executor module reference, preconditions/postconditions, and
metadata (versioning, tags, docs).

Create example capability files in data/capabilities/: excel-capabilities.json
and autoit-capabilities.json with representative operations.

Implement lib/capability-registry.js following constructor + create() factory
pattern:
- loadCapabilities(directory) to load capability definitions from directory
- getCapability(id) for capability lookup by ID
- listCapabilities(filter) for metadata-based filtering
- validateCapabilitySchema(definition) for schema validation
- addCapability(definition) for dynamic registration
- Maintain in-memory index with fast lookups
- Handle errors for validation failures, duplicate IDs, missing executors

Implement lib/capability-executor.js as execution dispatcher:
- execute(capabilityId, parameters, context) as primary method
- Load executor modules dynamically using require() based on capability's
executor field
- Define executeCapability(action, parameters) as standard executor interface
- Validate parameters against capability schema before execution
- Normalize results to standard format (success/failure, data, error message)
- Use try-catch error handling consistent with existing modules
===============================================================================

Task: 2

Phase 2 creates the bridge between LLM interactions and capability execution.

Define message schemas for LLM-capability communication:
- Capability query format (list capabilities with filtering)
- Capability invocation format (capability ID + parameters)
- Execution result format (success/failure + data)

Create prompt templates in data/prompts/ instructing LLMs on capability
discovery and invocation, adapting prompt_function_calling_en.json structure for
capability registry context. Include guidance for parameter construction, error
handling, and result interpretation.

Implement lib/function-calling-bridge.js as coordination layer:
- Wrap existing LLM chat() calls with capability execution loop
- Implement conversation handler managing flow: send prompt with capabilities →
process LLM response → execute capabilities → append results → continue
- Add capability discovery endpoint returning filtered capability lists with
schemas
- Implement execution coordinator parsing function calls, validating parameters,
delegating to capability-executor, formatting results
- Manage conversation history with configurable depth and pruning
- Add tracing/logging for debugging

Extend lib/chatgpt.js, lib/anthropic.js, lib/groq.js:
- Add functionCall(content, availableCapabilities) method alongside existing
chat()
- Inject capabilities into system prompts using templates
- Parse LLM responses for function call indicators
- Handle multi-turn conversations with alternating function calls and text
- Support configuration for function calling vs chat mode
- Export both chat() and functionCall() methods
===============================================================================

Task: 3

Phase 3 implements adapters for existing automation modules and provides
integration patterns for new applications.

Create executor adapters in lib/executors/:
- excel-executor.js mapping capabilities (open, close, writeCell, readCell,
saveAs) to msoffice.js methods with workbook/worksheet state management
- autoit-executor.js mapping capabilities (mouseClick, keyboardSend, mouseMove)
to autoit.js methods with coordinate/window validation
- Both following executeCapability(action, parameters) interface
- Transform generic capability parameters to module-specific arguments
- Track state where needed (open workbooks, active windows)
- Translate module errors to standard capability error format

Create comprehensive capability definitions:
- data/capabilities/excel.json with file operations (open, close, save, saveAs),
cell operations (read, write, format), sheet operations (add, delete, rename,
copy)
- data/capabilities/autoit.json with mouse operations (move, click, drag),
keyboard operations (send keys, press combinations), window operations
(activate, close, wait)
- Include rich parameter schemas with validation patterns, constraints, and
usage examples
- Add prerequisite declarations for state-dependent capabilities

Create integration documentation and templates:
- docs/capability-integration-guide.md covering capability definition structure,
executor contract, parameter schema design, state management, error handling
- Include worked example adding new application (e.g., Notepad)
- data/capabilities/template.json as annotated starting point
- lib/executors/template-executor.js showing minimal implementation with
validation, error handling, result formatting
- Document testing approach and troubleshooting for common issues

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[gnh1201]

For example:

English

Feature: File automation using FSO-compatible API

  Background:
    Given the working directory is "sandbox"

  Scenario: Create a text file, write content, then read it back
    When I create a text file "hello.txt" with overwrite enabled
    And I write line "Hello from FSO scenario"
    And I close the text stream
    Then the file "hello.txt" should contain "Hello from FSO"

  Scenario: Append a line to an existing file
    Given the file "log.txt" exists
    When I open the text file "log.txt" for appending
    And I write line "new entry"
    And I close the text stream
    Then the file "log.txt" should contain "new entry"

  Scenario: BuildPath should join paths correctly
    When I build path from "." and "a.txt" as "resultPath"
    Then the variable "resultPath" should end with "a.txt"

Korean

# language: ko
기능: FSO 호환 API로 파일 자동화

  배경:
    조건 작업 디렉터리는 "sandbox" 이다

  시나리오: 텍스트 파일을 만들고 내용을 쓴 뒤 다시 읽기
    만약 "hello.txt" 텍스트 파일을 덮어쓰기 허용으로 생성한다
    그리고 "Hello from FSO scenario" 한 줄을 쓴다
    그리고 텍스트 스트림을 닫는다
    그러면 "hello.txt" 파일에는 "Hello from FSO" 가 포함되어야 한다

  시나리오: 기존 파일에 한 줄 추가하기
    조건 "log.txt" 파일이 존재한다
    만약 "log.txt" 텍스트 파일을 추가 모드로 연다
    그리고 "new entry" 한 줄을 쓴다
    그리고 텍스트 스트림을 닫는다
    그러면 "log.txt" 파일에는 "new entry" 가 포함되어야 한다