[safe-output-health] Safe Output Health Report — 2026-05-11

[github-actions[bot]]

Executive Summary

• Period: Last 24 hours (2026-05-10 → 2026-05-11)
• Runs analyzed: 88
• Runs that emitted safe outputs: 24
• Safe-output items processed: 70
• Hard safe-output job failures: 2 (in 2 distinct runs)
• Soft failures with graceful fallback: 2
• Error clusters identified: 4

Overall safe-output health is good — only 2 of 70 (~2.9%) processed messages produced a hard failure, and both surfaced clean diagnostic messages. Two further events were soft failures that the handlers already handle gracefully; one of those (the empty submit_pull_request_review) silently drops the action and warrants a small fix.

Safe Output Job Statistics

Tool	Items	Hard Failures	Soft / Silent Failures
add_comment	~14	0	0
create_issue	~3	0	0
create_discussion	~2	0	0
create_pull_request	1	0	1 (workflows-perm push → fallback issue)
create_pull_request_review_comment	~5	0	0
submit_pull_request_review	3	0	1 (empty body+comments → 422)
resolve_pull_request_review_thread	1	1	0
set_issue_field	1	1	0
push_to_pull_request_branch	2	0	0
add_labels, remove_labels, add_reviewer, upload_artifact, upload_asset, update_pull_request, dispatch_workflow, create_code_scanning_alert, post_slack_message, set_issue_type, missing_tool, missing_data, noop	rest	0	0

Error Clusters

Cluster 1 — resolve_pull_request_review_thread: Resource not accessible by integration

• Count: 1 occurrence

• Affected workflow: Smoke Claude (run §25649467832)

• Sample error:

  ##[error]Failed to resolve review thread: Request failed due to following response errors:
   - Resource not accessible by integration
  ##[error]✗ Message 7 (resolve_pull_request_review_thread) failed
  

• Root cause: The default GITHUB_TOKEN issued for the smoke workflow does not have permission to call the GraphQL resolveReviewThread mutation on the target PR. The mutation requires pull-requests: write plus that the token's app is allowed to mutate that thread; the smoke workflow's token scope is missing the latter.

• Impact: Smoke Claude marks the run as failed at the job level even though 13/14 messages succeeded. Real workflows are unlikely to hit this often because most don't try to resolve threads, but the smoke matrix exercises it intentionally.

Cluster 2 — set_issue_field: No issue fields discovered

• Count: 1 occurrence

• Affected workflow: Smoke Codex (run §25649467850)

• Sample error:

  ##[error]No issue fields were discovered for this repository.
  Verify issue fields are enabled and visible to this token.
  ##[error]✗ Message 4 (set_issue_field) failed
  

• Root cause: The repository does not have any custom Issue Field schemas exposed to the smoke workflow's token, but the agent emitted a set_issue_field request anyway. The handler correctly errors out, but the diagnostic is not actionable for the agent.

• Impact: The Codex smoke matrix step that exercises set_issue_field always fails on github/gh-aw because the feature is not configured here. This is a known smoke-test coverage gap, not a regression.

Cluster 3 — create_pull_request: workflows-permission rejection (graceful fallback)

• Count: 1 occurrence

• Affected workflow: Q (run §25650191988)

• Sample error:

  ! [remote rejected] q/blog-post-writer-agent-dedup-... ->
    (refusing to allow a GitHub App to create or update workflow
    `.github/workflows/weekly-blog-post-writer.md` without `workflows` permission)
  ##[error]Git push failed: ... exit code 1
  ##[warning]Git push operation failed - creating fallback issue instead of pull request
  Created fallback issue #31412
  

• Root cause: The patch produced by the Q workflow touched .github/workflows/weekly-blog-post-writer.md, but the GitHub App token doing the push does not have workflows: write. The push is rejected by GitHub's protected-paths rule.

• Impact: Already handled — the safe-output handler falls back to opening a review issue ([q] fix(weekly-blog-post-writer): grep blog posts to avoid duplicate Agent of the Week #31412) so no work is lost. Final processing summary reports Failed: 0. This is the system working as designed.

Cluster 4 — submit_pull_request_review: 422 on empty review (silent drop)

• Count: 1 occurrence

• Affected workflow: Test Quality Sentinel (run §25650796757)

• Sample error:

  Submitting PR review on github/gh-aw#31418: event=COMMENT, comments=0, bodyLength=0
  ##[error]Failed to submit PR review: Unprocessable Entity
  ##[warning]✗ Failed to submit PR review: Unprocessable Entity
  

• Root cause: The agent emitted two submit_pull_request_review messages — one with no body/event and one with body+event=APPROVE. The handler merged review state and submitted the first (empty) one. GitHub REST returns 422 when event=COMMENT is paired with empty body and no comments.

• Subtle bug: After the 422, the handler logs the error and the warning, but the processing summary still reports Successful: 2 / Failed: 0 because the per-message success was set when the message was accepted, before the deferred submit. The actual PR review is not created.

Root Cause Analysis

Permission / Scope Issues (2 of 4)

• Smoke Claude resolve_pull_request_review_thread — token scope.
• Q create_pull_request — workflow path push protection (handled gracefully).

Configuration Issues (1 of 4)

• Smoke Codex set_issue_field — feature not enabled for this repo.

Validation / Handler Bugs (1 of 4)

• Test Quality Sentinel submit_pull_request_review — handler accepts an empty review and submits it without validation; the eventual 422 is not counted as a failure.

Recommendations

Critical Issues (Immediate Action)

None. No safe-output regression is causing widespread failures.

View Bug Fixes Required

1. submit_pull_request_review should validate before submit and count 422 as a failure

   • Priority: Medium
   • Root cause: The handler sets ✓ Message N (submit_pull_request_review) completed successfully at acceptance time, then submits in a finalization pass. A failure at finalization is logged but not reflected in the Failed: counter.
   • Fix: (a) Reject event=COMMENT with empty body and zero inline comments before queuing; (b) when the deferred submit fails, set a non-zero failure exit and include it in the safe output(s) failed: list.
   • Affected: any workflow that submits PR reviews (Test Quality Sentinel, PR Code Quality Reviewer, Grumpy Code Reviewer, Smoke matrix, etc.).

2. set_issue_field should differentiate "feature disabled" from "transient failure"

   • Priority: Low
   • Root cause: Error message "No issue fields were discovered" reads like an environmental problem but is really a configuration prereq.
   • Fix: Emit a structured hint (e.g. "Issue type fields are not enabled in this repository; configure them under Settings → Issues or skip this safe-output.") and consider downgrading to a warning when the feature is simply unavailable.

View Configuration Changes

3. Smoke matrix: skip resolve_pull_request_review_thread and set_issue_field on environments where they are guaranteed to fail
   • Current: Smoke workflows unconditionally exercise both, causing one expected failure per run.
   • Recommended: Gate those messages behind a capability probe (or move them to an opt-in smoke variant).
   • Reason: Today every Smoke Claude and Smoke Codex run reports a misleading 1-message failure, which dilutes signal on real regressions.

View Process Improvements

4. Add a dashboard tile for submit_pull_request_review finalization failures
   • Currently these are only visible in the per-job log; surfacing them in the audit/health monitor would catch silent drops.

Work Item Plans

Work Item 1 — Fix submit_pull_request_review silent-failure accounting

• Type: Bug Fix
• Priority: Medium
• Description: Empty submit_pull_request_review messages get accepted, then fail at finalization with HTTP 422, but the processing summary still reports Failed: 0. PR review is not created.
• Acceptance criteria:
  • Handler validates event + body + comments before queueing and returns a clear error for event=COMMENT with no content.
  • Finalization failure is recorded in the safe output(s) failed: list and reflected in the Failed: counter.
  • Job step exits non-zero when finalization fails.
  • Unit test covering the empty-review case.
• Technical approach: In the submit_pull_request_review queueing path, reject event=COMMENT when body==='' && comments.length===0. In the finalization helper, on error update the per-message status from success → failure and increment the failure counter.
• Estimated effort: Small

Work Item 2 — Improve set_issue_field diagnostic when feature is disabled

• Type: Enhancement
• Priority: Low
• Description: The current error wording ("No issue fields were discovered") sounds like a transient failure but is actually a configuration prereq.
• Acceptance criteria:
  • Probe issue-types/issue-fields availability once at handler start.
  • When unavailable, emit a one-line warning with a remediation link and treat the safe-output as a no-op instead of a hard failure.
• Technical approach: Add a preflight listIssueTypes/listProjectFields call; if empty, route subsequent set_issue_field messages to a graceful-skip path.
• Estimated effort: Small

Work Item 3 — Make smoke workflows capability-aware

• Type: Enhancement
• Priority: Low
• Description: Smoke Claude / Smoke Codex always fail one message per run on resolve_pull_request_review_thread (token scope) and set_issue_field (feature off), which masks real failures in those engines.
• Acceptance criteria:
  • Smoke matrix probes for the feature/scope and skips with a missing_tool entry rather than failing.
  • Smoke runs show Failed: 0 on a fully-green path.
• Estimated effort: Small

Historical Context

This is the first audit captured in safe-output-health/ cache memory, so no trend comparison is available. Subsequent audits will compare against 2026-05-11.json.

Metrics and KPIs

• Overall safe-output success rate: 68/70 messages = 97.1%
• Most reliable tools (last 24h): add_comment, add_labels, create_issue, create_discussion, push_to_pull_request_branch — 100% success.
• Most problematic tools (last 24h): resolve_pull_request_review_thread (1/1 failed), set_issue_field (1/1 failed), submit_pull_request_review (1/3 silently failed).
• Soft-failure resilience: create_pull_request correctly fell back to an issue when the workflows-permission rule rejected the push.

Next Steps

• Implement Work Item 1 (validate + accurately count submit_pull_request_review failures).
• Implement Work Item 2 (set_issue_field graceful skip when feature disabled).
• Implement Work Item 3 (smoke workflows capability probes).
• Re-run health monitor in 24h and compare against 2026-05-11.json to confirm the silent submit_pull_request_review failure is now counted.

References:

• §25649467832 — Smoke Claude
• §25649467850 — Smoke Codex
• §25650796757 — Test Quality Sentinel

Generated by Safe Output Health Monitor · ● 23.2M · ◷

• expires on May 12, 2026, 5:42 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #31638.