feat: add mid-execution cancellation#781
Open
notowen333 wants to merge 1 commit intostrands-agents:mainfrom
Open
feat: add mid-execution cancellation#781notowen333 wants to merge 1 commit intostrands-agents:mainfrom
notowen333 wants to merge 1 commit intostrands-agents:mainfrom
Conversation
|
Assessment: Comment This is an excellent implementation of cooperative cancellation for the TypeScript SDK. The code is well-structured, follows existing patterns, and includes comprehensive testing. Review Summary
Documentation PR: This PR introduces new public API surface ( Nice work aligning this with the Python SDK's cancellation capabilities! 🎉 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Motivation
We have cancellations in sdk-python but not in TS.
The SDK currently provides no way to stop an agent mid-execution. Once
agent.invoke()oragent.stream()is called, the caller must wait for the model to reach a natural stop. This is a problem for real-world applications where cancellation is routine: HTTP servers need to abort work when clients disconnect, UIs need a "stop generating" button, and batch pipelines need per-task timeouts. Without SDK-level support, developers are forced to build fragile workarounds or let orphaned work run to completion.Public API Changes
LocalAgentinterface (public contract for tool authors and framework integrators)New
cancellationSignalreadonly property -- exposes the cancellation signal for the current invocation, orundefinedwhen idle. This is the primary surface tools use to participate in cooperative cancellation:Agentclass (concrete implementation)New
cancel()method -- cooperatively cancels the current invocation:New
isCancelledgetter -- convenience boolean for checking cancellation state. Returnsfalsewhen idle.New
cancellationSignalgetter -- implements theLocalAgentproperty, backed by an internalAbortControllercomposed (viaAbortSignal.any()) with any external signal passed throughInvokeOptions.InvokeOptions(options forinvoke()/stream())New
cancellationSignaloption -- accepts an externalAbortSignalfor framework-driven cancellation. The agent composes this with its own internal controller, so bothagent.cancel()and the external signal can trigger cancellation independently:StopReasontypeNew
'cancelled'variant -- returned onAgentResult.stopReasonwhen an invocation is cancelled.StreamOptions(model layer)New
cancellationSignalfield -- forwarded from the agent to the model stream call. Model implementations do not need to handle it; the agent checks the signal between streaming events.All changes are backward compatible. Existing code that does not use cancellation is unaffected.
Use Cases
AbortSignal.timeout()to cap execution time for batch or latency-sensitive pipelinesagent.cancel()from a UI event handler to let users stop generation on demandCancellation Flows
Cancellation is cooperative. The agent checks the composed signal at specific points and takes a different code path depending on where in the loop cancellation is detected. There are four distinct flows:
Flow 1: Top of the agent loop cycle
_throwIfCancelled()runs at thewhile (true)entry in_stream(). If already aborted,AgentCancelInterruptis thrown. The catch block appends a synthetic"Cancelled by user"assistant message and returnsstopReason: 'cancelled'.Flow 2: During model response streaming
_throwIfCancelled()runs between each streamed event in_consumeModelStream(). This is the tightest checkpoint -- responsiveness is bounded by the gap between two consecutive stream chunks. The thrownAgentCancelInterruptpropagates through_callModel()(bypassing retry logic andAfterModelCallEvent) into_stream(), handled as in Flow 1. No partial assistant message is appended because the agent uses a deferred-append pattern.Flow 3: Before tool execution
After the model returns tool_use blocks but before any tool runs,
_stream()checksthis.isCancelleddirectly (no throw needed -- no nested generator to unwind). If cancelled:ToolResultBlocks ("Tool execution cancelled") are created for every pending tool and appended as a user message.AgentResultis returned.This preserves a valid assistant/user message sequence so the conversation can be resumed.
Flow 4: Between sequential tool executions
Inside the sequential loop in
executeTools(),this.isCancelledis checked before each tool. Already-completed tools keep their real results; remaining tools get synthetic error results. Control returns to_stream(), which exits via Flow 1 or 3 on the next iteration.Running tools are never forcibly interrupted -- they run to completion unless the tool checks
context.agent.cancellationSignalitself.Finally-block fallback
The
_stream()finally block checksthis.isCancelledto handle the edge case where the generator is terminated externally via.return()(e.g., a consumer breaking out offor-await-of). If cancelled but no result was produced, it appends the cancel message so the agent can be reinvoked.AgentCancelInterrupt(internal)An internal (not exported)
Errorsubclass used to unwind nestedyield*generator chains. The agent's streaming path is_stream()->_callModel()->_consumeModelStream(), each connected viayield*. A normalreturnfrom the innermost generator wouldn't propagate up -- throwingAgentCancelInterruptdoes. It is caught exclusively in_stream()and converted to a cancelledAgentResult. In_callModel(), it bypasses retry logic and skipsAfterModelCallEventsince there is no meaningful model result to report.Flows 3 and 4 don't use
AgentCancelInterruptbecause they checkthis.isCancelledin_stream()/executeTools()directly -- there is no nested generator to unwind.Vended tool update:
httpRequestThe
httpRequesttool was updated to demonstrate cancellation-aware tool design. Before this change, it managed its ownAbortControllerfor timeout handling. After:This is simpler (no manual controller/timer cleanup) and cancellation-aware. On abort, it distinguishes the cause:
Related Issues
#769
Documentation PR
Type of Change
New feature
Testing
Add unit and integ tests
npm run checkChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.