Skip to content

Commit 38b0fbf

Browse files
[INC-285] Inner workflows (#2230)
* Add basic subworkflow components * Add WORKFLOWS_MAX_SUBWORKFLOW_DEPTH environment variable to configure maximum subworkflow depth * Add UseSubworkflowBlockV1 for nested workflow execution - Introduced UseSubworkflowBlockV1 to enable execution of embedded workflows with parameter mapping from the parent workflow. - Updated loader.py to include the new block in the workflow loading process. - Added constants for the new block type in constants.py. * Add LocalSubworkflowRunner initialization in ExecutionEngineV1 - Updated ExecutionEngineV1 to initialize LocalSubworkflowRunner if not already provided in init_parameters. - Enhanced LocalSubworkflowRunner to execute nested workflows using the v1 workflow executor. - Improved documentation for subworkflow components to clarify usage and implementation details. * Implement nested workflow compilation and validation - Introduced new functions for compiling and validating nested workflows within the execution engine. - Added `GraphCompilationResult` class to encapsulate compilation results. - Enhanced `compile_workflow` to handle nested workflows and their parameters. - Created `compiler_bridge.py` for subworkflow composition validation and output projection. - Updated existing workflow compilation logic to integrate subworkflow handling. * Add subworkflow step execution handling - Introduced new functions for executing `use_subworkflow` steps, including `run_use_subworkflow_simd` and `run_use_subworkflow_non_simd`. - Added utility functions to determine if a step is a subworkflow and to manage runtime parameters. - Enhanced the main execution engine to support error handling for subworkflow steps. - Created a new file for subworkflow step execution logic, improving modularity and maintainability. * Add integration tests for nested workflow execution with use_subworkflow - Introduced end-to-end tests for the `use_subworkflow` functionality, validating parameter mapping between parent and child workflows. - Implemented tests for both single-level and multi-level nested workflows to ensure correct execution and output. - Enhanced test coverage for the execution engine's handling of subworkflows, contributing to improved reliability and maintainability of the workflow system. * Add nested output dimensionality handling for subworkflows - Introduced `nested_output_dimensionality_lift` to `BlockManifest` for tracking output dimensionality from child steps in embedded workflows. - Updated `denote_data_flow_for_step` to incorporate the new lift value when determining output dimensionality offsets. - Implemented `max_projection_output_lift_from_embedded_workflow` to calculate the maximum lift from child steps. - Enhanced the `LocalSubworkflowRunner` and related execution functions to utilize the new dimensionality handling. - Added utility functions for reshaping results based on the nested output dimensionality lift during execution. - Expanded integration tests to validate the new functionality and ensure correct behavior in nested workflows. * Enhance use_subworkflow functionality with reference resolution - Added support for resolving `use_subworkflow` steps that reference saved workflows by ID into inline `embedded_workflow` definitions. - Introduced a new module for reference resolution, including a default resolver for fetching workflow specifications. - Updated the compilation process to normalize references before parsing and validation, ensuring mutual exclusivity between embedded workflows and reference fields. - Expanded integration and unit tests to cover new reference resolution features, validating correct behavior in various scenarios. * Refactor subworkflow to inner workflow terminology and enhance functionality - Renamed `use_subworkflow` to `use_inner_workflow` throughout the codebase for consistency. - Introduced a new document detailing the design and goals of inner workflow composition. - Updated compilation and execution logic to support inner workflows, including validation of composition and nesting depth. - Enhanced error handling for inner workflow composition issues, including cycle detection and nesting depth limits. - Added new modules for inner workflow management, including reference resolution and execution strategies. - Updated environment variables to reflect the new inner workflow terminology, ensuring backward compatibility with legacy subworkflow settings. * Refactor legacy subworkflow references and improve documentation - Removed legacy references to `workflows_core.subworkflow_spec_resolver` in favor of the updated `workflows_core.inner_workflow_spec_resolver`. - Enhanced documentation for the `use_subworkflow` functionality, clarifying the usage of inline definitions and workspace configurations. - Streamlined error messages related to workflow resolution, ensuring clarity in user feedback. * Refactor and implement inner workflow functionality - Renamed `use_subworkflow` to `inner_workflow` throughout the codebase for consistency and clarity. - Introduced a new `InnerWorkflowBlockV1` class to handle execution of nested workflows with parameter mapping. - Updated documentation to reflect changes in terminology and provide clear guidelines on inner workflow composition. - Enhanced compilation and execution logic to support inner workflows, including validation of parameter bindings and nesting depth. - Improved error handling for inner workflow references and added integration tests to validate functionality. * Refactor inner workflow implementation and remove legacy references - Updated documentation to clarify the `max_nesting_depth` environment variable, removing references to the legacy `WORKFLOWS_MAX_SUBWORKFLOW_DEPTH`. - Streamlined the execution engine to exclusively utilize `workflows_core.inner_workflow_runner`, eliminating fallback to the legacy runner. - Enhanced the inner workflow step execution logic by removing deprecated references and ensuring consistency in the handling of nested workflows. - Improved code clarity and maintainability by consolidating inner workflow runner logic. * Add end-to-end tests for inner workflow execution - Introduced a new integration test file for `roboflow_core/inner_workflow@v1`, validating nested workflow execution. - Implemented tests to ensure correct parameter mapping and output resolution between parent and child workflows. - Enhanced documentation to include the new test file in the inner workflow design overview. * Refactor inner workflow terminology and update parameter handling - Renamed `embedded_workflow` references to `workflow` throughout the codebase for consistency and clarity. - Updated documentation to reflect changes in parameter names and improve clarity on workflow execution. - Enhanced validation logic to ensure correct handling of workflow references and inline definitions. - Adjusted integration and unit tests to align with the new terminology and validate functionality of inner workflows. * Update inner workflow terminology from `workflow` to `workflow_definition` - Renamed all instances of `workflow` to `workflow_definition` across the codebase for consistency and clarity. - Adjusted validation logic and error messages to reflect the new terminology. - Updated integration and unit tests to ensure proper functionality with the new parameter name. - Enhanced documentation to clarify the changes in parameter handling for inner workflows. * Refactor variable naming in inner workflow tests - Updated variable names from `embedded` to `inner` for clarity and consistency in the inner workflow test cases. - Ensured that all references to workflow definitions in the tests align with the updated terminology. - Improved readability of the test code by standardizing naming conventions across multiple test functions. * Add integration tests for inner workflow inlining - Introduced multiple test cases to validate the equivalence between nested inner workflows and their inlined counterparts. - Implemented tests for various scenarios, including parameter passing, output resolution, and handling of batch inputs. - Ensured consistency in behavior between inner workflows and inlined steps, enhancing the robustness of the integration testing framework. - Updated common helper functions to support the new test cases and improve code reusability. * Remove integration test for inner workflow execution - Deleted the test file `test_workflow_with_inner_workflow.py`, which contained end-to-end tests for nested workflow execution. - This removal is part of a cleanup effort to streamline the testing framework and eliminate redundant or outdated test cases. - Future tests will focus on more relevant scenarios and improve overall test coverage. * Remove inner workflow design document - Deleted the `inner_workflow_design.md` file, which outlined the design and goals for nested workflows. - This removal is part of a broader effort to streamline documentation and focus on more relevant and updated resources. - Future documentation will prioritize clarity and conciseness in conveying workflow functionality. * Remove inner workflow composition section from documentation - Deleted the section on inner workflow composition from the `workflows_compiler.md` file to streamline documentation. - This change aligns with recent efforts to focus on more relevant and updated resources regarding workflow functionality. * Add init file for inner workflow core step * Refactor inner workflow test assertions for consistency - Updated assertions in multiple inner workflow integration tests to compare results against `nested_result` instead of `flat_result`. - This change enhances clarity and ensures that the tests accurately reflect the expected behavior of inlined inner workflows. - Adjusted related test cases to maintain consistency in output validation across various scenarios. * Add integration test for inner workflow continue_if with crop batch lineage - Introduced a new test case to validate the behavior of the inner workflow when using `continue_if` and `first_non_empty_or_default` steps while the parent workflow processes detection, cropping, and classification. - This test ensures that the inner workflow correctly handles input parameters and outputs, enhancing the robustness of the integration testing framework for nested workflows. * Refactor inner workflow steps to use scalar_only_echo - Updated multiple inner workflow integration tests to replace `first_non_empty_or_default` steps with `scalar_only_echo`, enhancing consistency in step types. - Introduced a new `conftest.py` file to register the stub plugin required for the `echo_child_workflow`. - Improved documentation in the `echo_child_workflow` function to clarify usage and requirements for the stub plugin. * Remove integration test for inner workflow with list-valued workflow parameter - Deleted the test file `test_inner_workflow_with_list_valued_workflow_parameter.py`, which contained tests for validating the equivalence of nested inner workflows and their inlined counterparts with list-valued parameters. - This removal is part of a cleanup effort to streamline the testing framework and eliminate outdated or redundant test cases. * Refactor inner workflow tests to consistently use scalar_only_echo - Updated multiple integration tests to replace instances of `first_non_empty_or_default` with `scalar_only_echo`, ensuring uniformity in the workflow steps. - Adjusted test documentation to reflect the changes in step types and their expected behavior. - This refactor enhances the clarity and maintainability of the test suite for inner workflows. * Remove integration test for inner workflow continue_if with parent first non-empty - Deleted the test file `test_inner_workflow_continue_if_last_in_inner_with_parent_first_non_empty.py`, which validated the behavior of the inner workflow when using `continue_if` with a parent workflow's `first_non_empty_or_default` step. - This removal is part of ongoing efforts to streamline the testing framework and eliminate outdated or redundant test cases. * Refactor inner workflow tests to replace first_non_empty with echo - Updated integration tests to change references from `first_non_empty` to `echo` in both the workflow definitions and output selectors. - This change enhances consistency across the test cases and aligns with recent refactoring efforts to standardize step types in inner workflows. * Add inner workflow detection test for parent dynamic crop - Introduced a new test case to validate the behavior of the inner workflow when the parent runs detection and applies dynamic cropping on the inner model's predictions. - Added a new function `child_detection_only_for_parent_dynamic_crop` to define the inner workflow structure for this scenario. - This enhancement improves the integration testing framework by ensuring the correctness of nested workflows involving detection and cropping operations. * Refactor inner workflow execution to support compile-time inlining - Updated the inner workflow implementation to validate parameter bindings and inline child workflows at compile time, enhancing performance and simplifying execution. - Removed the `InnerWorkflowRunner` and related execution logic, as inner workflows are now expanded into ordinary steps during compilation. - Adjusted documentation and comments to reflect the new inlining behavior and updated parameter binding requirements. - Added tests to validate the new parameter binding rules for child workflows, ensuring robustness in the compilation process. * Enhance inner workflow detection tests for dynamic cropping - Refactored the test suite to introduce a new mock function for object detection inference, improving the handling of single and batch image requests. - Added a new test case to validate the behavior of the inner workflow when processing a list of images, ensuring consistent results with the single-image case. - Updated assertions to verify the correct number of calls and parameters in the mock, enhancing the robustness of the integration tests for nested workflows. * Add new tests for inner workflow continue_if behavior with outer echo - Introduced two new test cases to validate the behavior of the inner workflow when using `continue_if` with the outer workflow's `echo` step. - Enhanced the `_inner_continue_if_only_outer_echo_name_matches_parent` and `_nested_workflow_continue_if_inner_echo_outer` functions to support the new test scenarios. - Updated assertions to ensure the correctness of the integration tests, verifying that the nested and flat workflow results match as expected. * Add integration test for inner workflow dimension collapse on inner detections - Introduced a new test case to validate the behavior of the inner workflow when applying dimension collapse on object detection results from the inner model. - Implemented mock functions to simulate object detection responses and ensure consistent results when processing single and batch images. - Enhanced assertions to verify that the nested and flat workflow results match, confirming the correctness of the integration tests for dimension collapse functionality. * Add child dimension collapse workflow and corresponding integration test - Introduced a new inner workflow function `child_dimension_collapse_from_parent_detections` to handle dimension collapse using predictions from the parent workflow. - Added a comprehensive integration test to validate the behavior of the inner workflow when applying dimension collapse on object detection results. - Implemented mock functions to simulate object detection responses and ensure consistent results for both single and batch image processing. - Enhanced assertions to confirm that the results from the nested and flat workflows match, ensuring the correctness of the integration tests for dimension collapse functionality. * Refactor inner workflow validation and error handling - Updated the inner workflow validation logic to improve error handling for invalid step entries and parameter bindings. - Introduced new error classes for better granularity in error reporting, including `InnerWorkflowInvalidStepEntryError` and `InnerWorkflowParameterBindingsError`. - Refactored the `validate_inner_workflow_composition_from_raw_workflow_definition` function to enhance clarity and maintainability. - Adjusted the maximum nesting depth configuration for inner workflows and updated related validation checks. - Added unit tests to ensure robust validation of inner workflow definitions and parameter bindings. * Enhance documentation for inner workflow functions - Added detailed docstrings to several functions in the inline workflow module, improving clarity on their purpose, parameters, and error handling. - Updated function descriptions to include information about exceptions raised, enhancing the usability and maintainability of the code. - This change aims to provide better guidance for developers working with inner workflows and facilitate easier debugging and integration. * Add unit tests for inline inner workflow functionality - Introduced new unit tests for the `inline_inner_workflow_steps` function, focusing on its behavior with minimal workflow graphs. - Added comprehensive test cases to validate the handling of inner workflow steps, ensuring correct inlining and reference resolution. - Enhanced the test suite for the inner workflow module to improve coverage and maintainability, facilitating easier debugging and integration in future developments. - This addition aims to strengthen the testing framework for inner workflows and ensure robust functionality across various scenarios. * Refactor inner workflow step validation logic - Updated the `_contains_inner_workflow_step` function to return `False` for invalid input types instead of raising exceptions, improving usability and error handling. - Modified corresponding unit tests to reflect the new behavior, ensuring that non-list and non-dict entries are correctly handled without exceptions. - This change enhances the robustness of the inner workflow validation process and simplifies error management for users. * Refactor inner workflow code for improved readability and consistency - Reformatted several code sections across multiple files to enhance readability, including consistent line breaks and indentation. - Updated the `validate_workflow_or_reference` method to improve clarity in the inline workflow validation process. - Refactored function calls to maintain a consistent style, ensuring better maintainability and understanding of the codebase. - These changes aim to streamline the inner workflow implementation and facilitate easier future modifications. * Add inner workflow total count validation and error handling - Introduced a new environment variable `WORKFLOWS_MAX_INNER_WORKFLOW_COUNT` to set the maximum allowed inner workflow steps. - Added `InnerWorkflowTotalCountError` to handle cases where the inner workflow step count exceeds the defined limit. - Updated the `validate_inner_workflow_composition` function to include validation for the total count of inner workflow steps. - Enhanced unit tests to cover scenarios for total count validation, ensuring robust error handling and compliance with the new limits. - These changes improve the overall validation logic for inner workflows, enhancing usability and error management. * Add support for inner workflows and update documentation - Introduced the `roboflow_core/inner_workflow@v1` block, allowing workflows to embed other workflows inline. - Updated the execution engine to version `1.9.0`, incorporating compile-time inlining of nested workflows. - Added a new documentation page detailing inner workflows, including usage, parameter bindings, and compile-time validation. - Enhanced the workflows compiler to handle inner workflow steps, ensuring proper validation and inlining during compilation. - Updated existing documentation to reference the new inner workflow design and its capabilities. --------- Co-authored-by: Paweł Pęczek <146137186+PawelPeczek-Roboflow@users.noreply.github.com>
1 parent 64c5c27 commit 38b0fbf

42 files changed

Lines changed: 5632 additions & 20 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/workflows/execution_engine_changelog.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,37 @@
22

33
Below you can find the changelog for Execution Engine.
44

5+
## Execution Engine `v1.9.0` | inference `v1.2.0`
6+
7+
!!! Note "New feature: nested workflows via compile-time inlining"
8+
9+
This release adds the **`roboflow_core/inner_workflow@v1`** block so a workflow can embed another
10+
workflow definition (inline JSON or resolved from `workflow_workspace_id` / `workflow_id` /
11+
optional `workflow_version_id`). Child inputs are wired from the parent with
12+
**`parameter_bindings`** (child `inputs[].name` → parent selectors). At compile time the
13+
engine **inlines** nested steps into the parent graph; execution uses the same path as ordinary
14+
steps (no separate nested runtime).
15+
16+
**What changed**
17+
18+
* **Inner workflow block** — New flow-control block type `roboflow_core/inner_workflow@v1` registered
19+
in `roboflow_core`. Parent outputs may reference child workflow JsonField outputs as
20+
`$steps.<inner_step_name>.<child_output_name>` until inlining rewrites selectors.
21+
22+
* **Compile pipeline** — Before parsing the root definition, the compiler: (1) **normalizes**
23+
references (default: Roboflow API + `workflows_core.api_key`, or custom
24+
`workflows_core.inner_workflow_spec_resolver`), (2) **validates composition** (acyclicity, max
25+
nesting depth, max inner-workflow count), (3) **inlines** all inner workflow steps into ordinary
26+
steps, then continues with parse, workflow specification validation, and execution graph
27+
construction.
28+
29+
* **Limits (environment variables)**`WORKFLOWS_MAX_INNER_WORKFLOW_DEPTH` (default `4`) caps
30+
containment depth from the root; `WORKFLOWS_MAX_INNER_WORKFLOW_COUNT` (default `32`) caps the
31+
total number of `inner_workflow` steps in the nested definition.
32+
33+
* **Documentation** — See [Inner workflows (nested definitions)](./inner_workflow_design.md) for usage,
34+
bindings, limits, and an example.
35+
536
## Execution Engine `v1.8.0` | inference `v1.1.1`
637

738
!!! Note "Additive change + one breaking change due to bug fix with minimal expected impact"
Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# Inner workflows (nested definitions)
2+
3+
The **Inner workflow** block (`roboflow_core/inner_workflow@v1`) lets you embed one workflow definition inside another. At **compile time**, the engine resolves any saved-workflow references, validates **composition** (nesting limits and cycles), validates **parameter bindings**, then **inlines** the child’s steps into the parent. After compilation there is no separate “nested run”: the graph is the same as if you had written those steps at the parent level.
4+
5+
This page describes the block, the compile-time pipeline, limits, and a minimal Python example. For general compilation stages, see [Compilation of Workflow Definition](./workflows_compiler.md).
6+
7+
!!! Note
8+
9+
This feature is implemented in Execution Engine **v1**. The inner workflow block’s `run()` method is never used at runtime; the step is removed during compilation.
10+
11+
## Inner workflow block
12+
13+
Each inner workflow step is a JSON object in the parent’s `steps` list with `type: "roboflow_core/inner_workflow@v1"`.
14+
15+
### How you supply the child definition
16+
17+
You must provide **either**:
18+
19+
- **`workflow_definition`**: a full nested workflow JSON object (same shape as a root workflow: `version`, `inputs`, `steps`, `outputs`), **or**
20+
- **`workflow_workspace_id`** and **`workflow_id`**, with optional **`workflow_version_id`**, to load a saved workflow spec at compile time.
21+
22+
You must **not** set both an inline `workflow_definition` and the reference fields on the same step.
23+
24+
### `parameter_bindings`
25+
26+
`parameter_bindings` is an object whose **keys** are the **names** of the **child workflow’s** entries in its `inputs` array. Each **value** is a **selector** (or value the engine can coerce) from the **parent** scope, typically:
27+
28+
- `$inputs.<parent_input_name>` for parent workflow inputs, or
29+
- `$steps.<parent_step_name>.<output_property>` for data produced by earlier parent steps.
30+
31+
**Rules:**
32+
33+
- Every child input that **requires** a value from the parent must appear in `parameter_bindings`, **except** inputs of type `WorkflowParameter` / `InferenceParameter` that declare a **non-null** `default_value` in the child definition. Those may be omitted; the child’s default is applied when the definition is inlined.
34+
- Keys that are not child input names are rejected at compile time.
35+
- Child steps should consume parent data through **`$inputs.<child_input_name>`** in the nested definition; the compiler replaces those references with the bound parent selectors (or injected defaults) during inlining.
36+
37+
### Referencing child outputs from the parent
38+
39+
The nested workflow’s `outputs` array defines **JsonField** entries with a `name` and `selector`. After compilation, the parent treats the inner step like a logical block with outputs named after those JsonField **`name`** values.
40+
41+
From the parent you reference them as:
42+
43+
```text
44+
$steps.<inner_step_name>.<child_output_name>
45+
```
46+
47+
where `<child_output_name>` is the `name` field of a JsonField in the child’s `outputs`, not necessarily the last step’s name.
48+
49+
## Compile-time pipeline (Execution Engine v1)
50+
51+
When `compile_workflow_graph` runs, inner workflows go through the following **before** the main “parse workflow definition” step:
52+
53+
1. **Reference resolution (normalization)**
54+
Any step that uses `workflow_workspace_id` / `workflow_id` (and optional `workflow_version_id`) is resolved to an inline `workflow_definition`. This happens recursively inside nested definitions.
55+
- Default resolver uses the Roboflow API and **`workflows_core.api_key`** in workflow init parameters (unless the workspace is `"local"` or you supply a custom resolver).
56+
- Override with init parameter **`workflows_core.inner_workflow_spec_resolver`**: a callable `(workspace_id, workflow_id, workflow_version_id, init_parameters) -> dict` returning the child workflow JSON.
57+
58+
2. **Composition validation**
59+
The engine builds a **composition graph**: one edge per `inner_workflow` step from the parent workflow’s fingerprint to the child definition’s fingerprint. It then checks:
60+
- the graph is **acyclic** (no A → B → … → A),
61+
- **nesting depth** from the root is within **`WORKFLOWS_MAX_INNER_WORKFLOW_DEPTH`**,
62+
- the **total number** of inner-workflow steps (edges) is within **`WORKFLOWS_MAX_INNER_WORKFLOW_COUNT`**.
63+
See [Limits and environment variables](#limits-and-environment-variables) below.
64+
65+
3. **Inlining**
66+
Each `inner_workflow` step is expanded into ordinary steps: child step names become **`{inner_step_name}__{child_step_name}`** (with collision handling), selectors are rewritten (`$inputs` / `$steps` in the child, and parent references to `$steps.<inner_step_name>…`), then the inner step is removed. The rest of compilation (parse, workflow specification validation, execution graph construction, step initialization) sees only a flat workflow.
67+
68+
4. **Parsing and validation**
69+
The flattened JSON is parsed with block manifests, `validate_workflow_specification` runs, and the execution graph is built like any other workflow.
70+
71+
## Example (Python)
72+
73+
The following pattern matches the `examples/workflows/inner_workflows/main.py` example in this repository: resolve a saved workflow by id, bind the parent image into the child’s expected input name, then consume a child output from a downstream parent step.
74+
75+
```python
76+
import json
77+
import os
78+
79+
from inference.core.managers.base import ModelManager
80+
from inference.core.registries.roboflow import RoboflowModelRegistry
81+
from inference.core.workflows.core_steps.common.entities import StepExecutionMode
82+
from inference.core.workflows.execution_engine.core import ExecutionEngine
83+
from inference.models.utils import ROBOFLOW_MODEL_TYPES
84+
85+
WORKFLOW_DEFINITION = {
86+
"version": "1.0",
87+
"inputs": [
88+
{"type": "WorkflowImage", "name": "image"},
89+
],
90+
"steps": [
91+
{
92+
"type": "roboflow_core/inner_workflow@v1",
93+
"name": "inner",
94+
"workflow_workspace_id": "your-workspace",
95+
"workflow_id": "your-workflow-id",
96+
"workflow_version_id": "optional-version-id",
97+
"parameter_bindings": {
98+
"image": "$inputs.image",
99+
},
100+
},
101+
{
102+
"type": "roboflow_core/roboflow_classification_model@v2",
103+
"name": "classification",
104+
"images": "$steps.inner.dynamic_crop_output",
105+
"model_id": "resnet50",
106+
},
107+
],
108+
"outputs": [
109+
{
110+
"type": "JsonField",
111+
"name": "predictions",
112+
"selector": "$steps.classification.predictions",
113+
},
114+
],
115+
}
116+
117+
if __name__ == "__main__":
118+
model_registry = RoboflowModelRegistry(ROBOFLOW_MODEL_TYPES)
119+
model_manager = ModelManager(model_registry=model_registry)
120+
121+
execution_engine = ExecutionEngine.init(
122+
workflow_definition=WORKFLOW_DEFINITION,
123+
init_parameters={
124+
"workflows_core.model_manager": model_manager,
125+
"workflows_core.api_key": os.getenv("ROBOFLOW_API_KEY"),
126+
"workflows_core.step_execution_mode": StepExecutionMode.LOCAL,
127+
},
128+
)
129+
130+
result = execution_engine.run(
131+
runtime_parameters={
132+
"image": {
133+
"type": "file",
134+
"value": "/path/to/your/image.jpg",
135+
},
136+
},
137+
)
138+
139+
print(json.dumps(result, indent=2, default=str))
140+
```
141+
142+
Replace workspace, workflow, version, model, output selectors, and image path with values that match your saved workflow and parent graph. The key requirement is that **`parameter_bindings`** keys match the **child** workflow’s `inputs[].name` fields.
143+
144+
## Limits and environment variables
145+
146+
| Variable | Default | Meaning |
147+
|----------|---------|--------|
148+
| `WORKFLOWS_MAX_INNER_WORKFLOW_DEPTH` | `4` | Maximum **depth** of the composition graph from the root workflow: each direct `inner_workflow` child counts as one level along a path. |
149+
| `WORKFLOWS_MAX_INNER_WORKFLOW_COUNT` | `32` | Maximum **number** of `inner_workflow` steps across the whole nested definition (each inner step is one edge in the composition graph). |
150+
151+
**Cycles:** The composition graph must be a **DAG**. You cannot have a cycle of nested references (for example, workflow A embedding B embedding A), even if the per-step execution graph of each workflow is acyclic.
152+
153+
Violations raise compile-time errors (`InnerWorkflowNestingDepthError`, `InnerWorkflowTotalCountError`, `InnerWorkflowCompositionCycleError`, etc.) with messages describing depth, count, or cycle involvement.
154+
155+
## Related reading
156+
157+
- [Workflows definitions](./definitions.md) — JSON shape, inputs, steps, outputs
158+
- [Compiler](./workflows_compiler.md) — overall compilation stages
159+
- [Workflow execution](./workflow_execution.md) — runtime behavior after compilation

docs/workflows/workflows_compiler.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@ during execution and verifying Workflow integrity
3535
5. Initializing Workflow steps from blocks: Setting up the individual workflow steps based on the available blocks,
3636
steps definitions and configuration of execution environment.
3737

38+
If the definition contains [`roboflow_core/inner_workflow@v1`](./inner_workflow_design.md) steps, the compiler **first** resolves saved-workflow references, validates nested **composition** (depth, total count, acyclicity), and **inlines** child steps into the parent **before** parsing and building the execution graph. See [Inner workflows (nested definitions)](./inner_workflow_design.md) for the full pipeline, `parameter_bindings`, and environment limits.
39+
3840
Let's take a closer look at each of the workflow compilation steps.
3941

4042
### Workflows blocks loading

inference/core/env.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -639,6 +639,12 @@
639639
WORKFLOWS_STEP_EXECUTION_MODE = os.getenv("WORKFLOWS_STEP_EXECUTION_MODE", "local")
640640
WORKFLOWS_REMOTE_API_TARGET = os.getenv("WORKFLOWS_REMOTE_API_TARGET", "hosted")
641641
WORKFLOWS_MAX_CONCURRENT_STEPS = int(os.getenv("WORKFLOWS_MAX_CONCURRENT_STEPS", "8"))
642+
WORKFLOWS_MAX_INNER_WORKFLOW_DEPTH = int(
643+
os.getenv("WORKFLOWS_MAX_INNER_WORKFLOW_DEPTH", "4")
644+
)
645+
WORKFLOWS_MAX_INNER_WORKFLOW_COUNT = int(
646+
os.getenv("WORKFLOWS_MAX_INNER_WORKFLOW_COUNT", "32")
647+
)
642648
WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE = int(
643649
os.getenv("WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE", "1")
644650
)

inference/core/workflows/core_steps/flow_control/inner_workflow/__init__.py

Whitespace-only changes.
Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
from typing import Any, Dict, List, Literal, Optional, Type
2+
3+
from pydantic import ConfigDict, Field, model_validator
4+
5+
from inference.core.workflows.execution_engine.entities.base import OutputDefinition
6+
from inference.core.workflows.execution_engine.entities.types import (
7+
WILDCARD_KIND,
8+
Selector,
9+
)
10+
from inference.core.workflows.execution_engine.v1.inner_workflow.errors import (
11+
InnerWorkflowRunNotSupportedError,
12+
)
13+
from inference.core.workflows.prototypes.block import (
14+
BlockResult,
15+
WorkflowBlock,
16+
WorkflowBlockManifest,
17+
)
18+
19+
SHORT_DESCRIPTION = (
20+
"Run a nested workflow definition with parameters mapped from the parent workflow."
21+
)
22+
23+
LONG_DESCRIPTION = """
24+
Execute a **nested workflow** while mapping parent data into the child's inputs via `parameter_bindings`.
25+
26+
Provide either a full inline definition in `workflow_definition`, or resolve a saved workflow using
27+
`workflow_workspace_id` and `workflow_id` (optional `workflow_version_id`).
28+
Reference fields are expanded at compile time via `workflows_core.inner_workflow_spec_resolver`
29+
(default: Roboflow API using `workflows_core.api_key`, or local definitions when workspace is
30+
`"local"`).
31+
32+
At compile time the engine validates composition (acyclicity, max depth) and `parameter_bindings`,
33+
then **inlines** the child's steps into the parent workflow graph (same execution path as ordinary
34+
steps).
35+
36+
The block's `run()` method is not used at runtime; do not call it directly.
37+
"""
38+
39+
40+
class BlockManifest(WorkflowBlockManifest):
41+
model_config = ConfigDict(
42+
json_schema_extra={
43+
"name": "Inner Workflow",
44+
"version": "v1",
45+
"short_description": SHORT_DESCRIPTION,
46+
"long_description": LONG_DESCRIPTION,
47+
"license": "Apache-2.0",
48+
"block_type": "flow_control",
49+
"ui_manifest": {
50+
"section": "flow_control",
51+
"icon": "fak fa-diagram-nested",
52+
"blockPriority": 2,
53+
},
54+
}
55+
)
56+
type: Literal["roboflow_core/inner_workflow@v1"]
57+
workflow_definition: Optional[Dict[str, Any]] = Field(
58+
default=None,
59+
description=(
60+
"Full nested workflow definition (same JSON shape as a root workflow: version, inputs, "
61+
"steps, outputs). Required unless `workflow_workspace_id` and `workflow_id` are set; "
62+
"mutually exclusive with those reference fields."
63+
),
64+
)
65+
workflow_workspace_id: Optional[str] = Field(
66+
default=None,
67+
description=(
68+
'Workspace id for a saved workflow to load (Roboflow slug or `"local"` for on-disk '
69+
"definitions). Use with `workflow_id`; mutually exclusive with a non-empty "
70+
"`workflow_definition`."
71+
),
72+
)
73+
workflow_id: Optional[str] = Field(
74+
default=None,
75+
description="Saved workflow id to fetch. Use with `workflow_workspace_id`.",
76+
)
77+
workflow_version_id: Optional[str] = Field(
78+
default=None,
79+
description="Optional pinned workflow version when resolving by id.",
80+
)
81+
parameter_bindings: Dict[str, Selector()] = Field(
82+
description=(
83+
"Maps **child** workflow input names to a selector (or literal coerced by the engine) "
84+
"from the parent. Required for every child input except `WorkflowParameter` / "
85+
"`InferenceParameter` entries that declare a non-null `default_value` (those may be "
86+
"omitted and the child's default is used during compilation inlining)."
87+
),
88+
json_schema_extra={
89+
"keys_bound_in": "parameter_bindings",
90+
},
91+
)
92+
93+
@model_validator(mode="after")
94+
def validate_workflow_or_reference(self) -> "BlockManifest":
95+
has_inline = (
96+
isinstance(self.workflow_definition, dict)
97+
and len(self.workflow_definition) > 0
98+
)
99+
100+
workspace_id = (self.workflow_workspace_id or "").strip()
101+
workflow_id = (self.workflow_id or "").strip()
102+
has_ref = bool(workspace_id and workflow_id)
103+
104+
if has_inline and has_ref:
105+
raise ValueError(
106+
"Provide either `workflow_definition` or workflow reference fields "
107+
"(`workflow_workspace_id` and `workflow_id`), not both."
108+
)
109+
110+
if has_inline or has_ref:
111+
return self
112+
113+
raise ValueError(
114+
"inner_workflow requires a non-empty `workflow_definition` object or both "
115+
"`workflow_workspace_id` and `workflow_id`."
116+
)
117+
118+
@classmethod
119+
def describe_outputs(cls) -> List[OutputDefinition]:
120+
return [OutputDefinition(name="*", kind=[WILDCARD_KIND])]
121+
122+
@classmethod
123+
def accepts_batch_input(cls) -> bool:
124+
return False
125+
126+
@classmethod
127+
def get_execution_engine_compatibility(cls) -> Optional[str]:
128+
return ">=1.4.0,<2.0.0"
129+
130+
131+
class InnerWorkflowBlockV1(WorkflowBlock):
132+
"""Placeholder block; inner workflows are expanded at compile time and never executed as a unit."""
133+
134+
@classmethod
135+
def get_manifest(cls) -> Type[WorkflowBlockManifest]:
136+
return BlockManifest
137+
138+
def run(self, *args, **kwargs) -> BlockResult:
139+
raise InnerWorkflowRunNotSupportedError(
140+
"inner_workflow steps are compiled away into ordinary steps; block.run() must not be called."
141+
)

inference/core/workflows/core_steps/loader.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,9 @@
137137
from inference.core.workflows.core_steps.flow_control.delta_filter.v1 import (
138138
DeltaFilterBlockV1,
139139
)
140+
from inference.core.workflows.core_steps.flow_control.inner_workflow.v1 import (
141+
InnerWorkflowBlockV1,
142+
)
140143
from inference.core.workflows.core_steps.flow_control.rate_limiter.v1 import (
141144
RateLimiterBlockV1,
142145
)
@@ -714,6 +717,7 @@ def load_blocks() -> List[Type[WorkflowBlock]]:
714717
DetectionsTransformationBlockV1,
715718
RoboflowDatasetUploadBlockV1,
716719
ContinueIfBlockV1,
720+
InnerWorkflowBlockV1,
717721
RateLimiterBlockV1,
718722
PerspectiveCorrectionBlockV1,
719723
DeltaFilterBlockV1,

0 commit comments

Comments
 (0)