cel: add secret_state for encrypted credentials in CEL programs

[efd6]

Proposed commit message

cel: add secret_state for encrypted credentials in CEL programs

Add a secret_state variable (type: textarea, secret: true) that
lets users store API keys and credentials encrypted by Fleet and
reference them in CEL programs as state.secret.<key>. Wire the
variable into the agent input template and add a system test that
validates the secret value reaches the CEL program via a header
check against the mock server.

Requires the matching beats change to accept string values for
secret_state, since Fleet resolves secrets to strings. Bumps
kibana.version to the versions that will include the Fleet fix.

Note

Depends on a workaround in beats for a fleet bug.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Ref type: yaml variables with secret: true are silently corrupted during policy compilation kibana#267859

Screenshots

[macroscopeapp]

🟠 High cel/changelog.yml:5

The changelog entry at line 5 uses https://github.com/elastic/integrations/pull/XXXXX as a placeholder PR link, which will be published as a broken link that 404s. Consider replacing XXXXX with the actual pull request number before merging.

-      link: https://github.com/elastic/integrations/pull/XXXXX
+      link: https://github.com/elastic/integrations/pull/XXXXX

🤖 Copy this AI Prompt to have your agent fix this:

In file packages/cel/changelog.yml around line 5:

The changelog entry at line 5 uses `https://github.com/elastic/integrations/pull/XXXXX` as a placeholder PR link, which will be published as a broken link that 404s. Consider replacing `XXXXX` with the actual pull request number before merging.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 0f575d0

Failed CI Steps

• Check integrations cel

History

cc @efd6

[github-actions]

TL;DR

Check integrations cel failed before tests ran because the stack bootstrap selected version 8.19.17, but Docker images for that tag are not available (manifest unknown).

Remediation

• Re-run this job with an explicitly available stack version (for example a current 8.19.x-SNAPSHOT) via STACK_VERSION, or temporarily relax the CEL package lower bound until 8.19.17 images are published.
• Validate by re-running Check integrations cel and confirming elastic-package stack up succeeds before package tests start.

Investigation details

Root Cause

prepare_stack falls back to the package’s oldest supported Kibana version when STACK_VERSION is unset (.buildkite/scripts/common.sh, around lines 475-481). In this PR, CEL’s manifest now requires ^8.19.17 || ^9.3.6 || ^9.4.1 (packages/cel/manifest.yml:11), so the selected version becomes 8.19.17.

That selected version cannot be pulled in CI right now, so stack startup fails before running CEL tests.

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/42381
• Job/step: Check integrations cel
• Version selected from manifest:
  • cd packages/cel && python3 ../../.buildkite/scripts/find_oldest_supported_version.py --manifest-path manifest.yml
  • Output: 8.19.17
• Key log excerpts (/tmp/gh-aw/buildkite-logs/integrations-check-integrations-cel.txt):
  • Error response from daemon: manifest for docker.elastic.co/elasticsearch/elasticsearch:8.19.17 not found: manifest unknown
  • Error response from daemon: manifest for docker.elastic.co/elastic-agent/elastic-agent-wolfi:8.19.17 not found: manifest unknown

Verification

• Confirmed locally in the repo that this PR updates CEL Kibana constraints to ^8.19.17 || ^9.3.6 || ^9.4.1 and that Buildkite’s version resolution script returns 8.19.17 for this manifest.

Follow-up

If this constraint bump is required for the Fleet fix, keeping the manifest as-is is fine; CI just needs to target an available stack tag (or wait for 8.19.17 image publication).

Note

🔒 Integrity filter blocked 3 items

The following items were blocked because they don't meet the GitHub integrity level.

• #18834 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• cel: add secret_state for encrypted credentials in CEL programs #18834 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #18834 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.