SDK 0.3.0 does not support images in tool results' `binaryResultsForLlm`

[connor4312]

1. Define a tool that provides an image to the SDK
2. Have the agent call it in a model that supports it (e.g. Haiku)
3. 🐛 the agent can't see the image

Specifically for us this happens here: https://github.com/microsoft/vscode/blob/a1803b2a0c88b4a22ff83262b132cca1befc0886/src/vs/platform/agentHost/node/copilot/copilotAgentSession.ts#L592

This is the image being returned: image.zip

Here is our log dump which includes the copilot CLI events.jsonl. You can see the binary results in that log:

ah-logs-Open-apple.com,-take-a-screenshot,-and-t.zip

And just for edification, here is the exact image that was returned, a valid jpeg: image.zip

Upstream issue: microsoft/vscode#315947

[github-actions]

Investigation findings — confirmed bug 🐛

What I investigated

• nodejs/src/types.ts — the SDK's public ToolResultObject and ToolBinaryResult types, and convertMcpCallToolResult
• nodejs/src/session.ts — _executeToolAndRespond, the path that runs a registered tool handler and sends the result back to the CLI
• nodejs/src/generated/rpc.ts — the generated CLI protocol schema, specifically ExternalToolTextResultForLlm and ExternalToolTextResultForLlmContentImage

Root cause

There is a format mismatch between the SDK's public API and what the CLI protocol actually consumes for rich binary content.

SDK's ToolResultObject (public API):

{
  textResultForLlm: string;
  binaryResultsForLlm?: ToolBinaryResult[];  // e.g. [{type:"image", (redacted) mimeType:"image/jpeg"}]
}

CLI protocol's ExternalToolTextResultForLlm (what the CLI reads for images):

{
  textResultForLlm: string;
  contents?: ExternalToolTextResultForLlmContent[];  // e.g. [{type:"image", (redacted) mimeType:"image/jpeg"}]
}

The contents field with typed image/audio/resource blocks was added to the generated RPC schema in the @github/copilot 1.0.48 update (landed today in #1292). The CLI uses contents to pass images to the LLM — not binaryResultsForLlm.

In session.ts, _executeToolAndRespond passes the ToolResultObject directly to handlePendingToolCall with no conversion:

await this.rpc.tools.handlePendingToolCall({ requestId, result });

So binaryResultsForLlm is sent as an extra/unknown JSON field that the CLI silently ignores. The images never reach the LLM.

The same gap exists in the v2 path (normalizeToolResultV2 in client.ts) and across all SDKs (Python, Go, .NET, Rust all have binaryResultsForLlm in their ToolResultObject equivalents).

Fix needed

Before calling handlePendingToolCall, the SDK should convert binaryResultsForLlm entries into contents items of the appropriate type (type: "image", type: "resource") so the CLI receives the standard format it expects. The ToolBinaryResult struct maps cleanly to ExternalToolTextResultForLlmContentImage — the fields (data, mimeType) are the same.

Generated by Bug Handler for issue #1298 · ● 2.6M · ◷