Skip to content

[runtime] CDC YAML supports inline Python UDF definition#4395

Open
yuxiqian wants to merge 1 commit into
apache:masterfrom
yuxiqian:feat/python-udf
Open

[runtime] CDC YAML supports inline Python UDF definition#4395
yuxiqian wants to merge 1 commit into
apache:masterfrom
yuxiqian:feat/python-udf

Conversation

@yuxiqian
Copy link
Copy Markdown
Member

@yuxiqian yuxiqian commented May 13, 2026

This PR provides the ability to write inline Python UDFs with YAML pipeline jobs. It could be used like this:

transform:
  - source-table: db.users
    projection: ID, py_normalize(EMAIL) AS EMAIL_NORM, py_accumulate(AGE) AS acc

pipeline:
  user-defined-function:
    - name: py_normalize
      classpath: org.apache.flink.cdc.python.PythonUdf
      options:
        python-executable: /usr/bin/python3
        source: |
          def eval(s: str) -> str:
            return s.strip().lower()
    - name: py_accumulate
      classpath: org.apache.flink.cdc.python.PythonUdf
      options:
        python-executable: /usr/bin/python3
        source: |
          total = 0
          def eval(x: int) -> int:
            global total
            total += x
            return total

The wrapper itself is implemented as a Java UDF as well. No changes are made in the existing framework except the following:

  • Runtime UDF binding names are slightly changed to allow defining multiple UDFs with the same class.
  • Added an overload function for UserDefinedFunction#getReturnType to pass extra context info.

@github-actions github-actions Bot added docs Improvements or additions to documentation common runtime build e2e-tests labels May 13, 2026
@yuxiqian yuxiqian force-pushed the feat/python-udf branch 2 times, most recently from 94f43dd to 73f04b2 Compare May 13, 2026 14:15
@leonardBang leonardBang self-requested a review May 14, 2026 12:39
@yuxiqian yuxiqian added this to the V3.7.0 milestone May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build common docs Improvements or additions to documentation e2e-tests runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant