[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-05-12

[github-actions[bot]]

Summary

• Date: 2026-05-12
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: As a brand-new user, the home page is welcoming and the Quick Start guide is well-structured, but there are a few jargon bumps and small inconsistencies that could trip up total beginners.

---

🔴 Critical Issues Found

1. Gemini not listed in Prerequisites (Quick Start)

The Prerequisites section on the Quick Start page lists only three AI providers — GitHub Copilot, Anthropic Claude, and OpenAI Codex. However, Step 2 (add-wizard) mentions Google Gemini as a fourth choice ("Choose between Copilot, Claude, Codex, or Gemini") and lists GEMINI_API_KEY as a valid secret. A newcomer picking Gemini won't find it in the prerequisites and may not know it's supported.

Recommendation: Add `Google Gemini ((ai.google.dev/redacted) to the Prerequisites AI Account bullet.

---

🟡 Confusing Areas

2. "Frontmatter" and "lock file" jargon appear before they are explained

In Step 3 of the Quick Start, the note box mentions .lock.yml files without explanation. Then Step 4 finally explains the two-file system. A beginner reading Step 3 first will be confused. The term "frontmatter" also appears without explanation before the note in Step 4 defines it.

Recommendation: Move the two-file explanation note to just before Step 3 (or add a brief one-liner in Step 3: "The .lock.yml file is GitHub Actions YAML auto-generated from your .md file — you don't edit it directly.").

3. COPILOT_GITHUB_TOKEN setup is complex and non-obvious

The callout box explaining how to create a COPILOT_GITHUB_TOKEN requires users to navigate to a specific GitHub settings page, find "Account permissions → Copilot Requests", and set it to "Read". For a beginner, the path Settings → Personal Access Tokens → New (fine-grained) → Account permissions → Copilot Requests is not intuitive. "Account permissions" (as opposed to "Repository permissions") is easy to miss.

Recommendation: Add a direct link to the exact settings page (https://github.com/settings/personal-access-tokens/new) and note explicitly: "Look for 'Account permissions' (not 'Repository permissions') and expand it to find 'Copilot Requests'".

4. CLI page opens with enterprise/advanced content too early

The CLI Commands page starts with installation, then immediately dives into Pinning to a Specific Version, Alternative Standalone Installer, GitHub Actions Setup Action, and GitHub Enterprise Server Support — all before a beginner has even run their first command. This is overwhelming for new users.

Recommendation: Add a "If you're just getting started, skip to Commands" note at the top, or reorganize so advanced installation options are in a collapsible <details> block.

5. Early development warning may discourage beginners

The home page has a prominent warning note: "GitHub Agentic Workflows is in early development and may change significantly... Use it with caution, and at your own risk." While honest, placing this before any getting-started content could scare away new users before they even try.

Recommendation: Move this warning to a sidebar or a dedicated "Known Limitations" section, or soften the placement — keeping it honest but not front-and-center on the first impression.

---

🟢 What Worked Well

• ✅ Home page hero is clear: The tagline "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" immediately conveys the value proposition.
• ✅ Quick Start has a time estimate (⏱️ 10 minutes): Great for setting expectations.
• ✅ Prerequisites are well-structured: Checkboxes make it easy to verify readiness.
• ✅ Step-by-step flow is logical: Steps 1-4 have a natural progression.
• ✅ "Most Common Commands" table on CLI page: Excellent quick reference for beginners.
• ✅ Video on Quick Start page: Having a walkthrough video is very helpful for visual learners.
• ✅ "What's next?" section at end of Quick Start: Good call-to-action to continue learning.
• ✅ "By the Numbers" table on home page: Gives instant credibility metrics.

---

Recommendations

Quick wins (high-impact, low-effort):

1. Add Google Gemini to the Prerequisites list on Quick Start
2. Add a one-liner about .lock.yml before Step 3
3. Link directly to the GitHub PAT settings page for COPILOT_GITHUB_TOKEN setup

Longer-term improvements:
4. Reorganize CLI page to put advanced installation options in collapsible sections
5. Move or soften the early-development warning on the home page
6. Consider adding a glossary for terms like "frontmatter", "lock file", "activation job" that appear throughout

---

Screenshots

📎 home.png — Home page hero and navigation

📎 quick-start.png — Quick Start guide (top of page)

📎 cli.png — CLI Commands page (top of page)

---

References:

• §25714139139

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by Documentation Noob Tester · ● 21M · ◷

• expires on May 13, 2026, 5:01 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #31864.