[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-05-11

[github-actions[bot]]

Summary

• Date of test: 2026-05-11
• Pages visited:
  • Home: `(localhost/redacted)
  • Quick Start: `(localhost/redacted)
  • CLI Commands: `(localhost/redacted)
• Overall impression: The landing page clearly conveys what this tool does, and the Quick Start guide is well-structured and focused. However, there are several pain points for a true beginner, particularly around prerequisite knowledge (PATs, frontmatter, GitHub Actions concepts) and the need to understand multiple token types before even running the first command.

---

🔴 Critical Issues Found

1. Gemini mentioned in Quick Start but not supported in the engine list

Step 2 of the Quick Start says "Choose between Copilot, Claude, Codex, or Gemini" but the frontmatter reference and other docs only mention copilot, claude, codex, and custom. A beginner selecting Gemini may get confused or encounter errors. If Gemini is supported, it needs consistent documentation; if not, it should be removed.

2. COPILOT_GITHUB_TOKEN is confusing — it's not the GITHUB_TOKEN

The note explaining that COPILOT_GITHUB_TOKEN is "distinct from the default GITHUB_TOKEN" is buried in a collapsed note block. This is the #1 place beginners will get stuck. A new user will likely think their normal GitHub login is sufficient. The distinction needs to be front-and-center, not in a note.

3. gh aw add-wizard requires gh auth login --scopes repo,workflow

The prerequisite says to run gh auth login --scopes repo,workflow if needed, but there's no guidance on when it's needed or what error to look for. First-timers won't know if they already have this scope.

4. No explicit mention of what happens if GitHub Actions is disabled

The prerequisite says "GitHub Actions enabled — Check in Settings → Actions" but gives no guidance on what to do if it's disabled (enterprise restrictions, etc.).

---

🟡 Confusing Areas

1. "Frontmatter" jargon not explained

The Quick Start and CLI pages use the term "frontmatter" without defining it. A beginner who doesn't know YAML or static site generators won't know what this means. A one-sentence explanation ("the configuration block between --- markers") should be added the first time it appears — it actually does appear later in the page but only in a note box, not upfront.

2. Two-file system (.md + .lock.yml) is confusing

The page explains that "workflows have two files" but this explanation appears late in Step 4 (customize). A beginner will encounter the .lock.yml file before seeing this explanation and wonder what it is. This explanation should be earlier, perhaps right after installation.

3. gh aw add-wizard vs gh aw add — unclear difference

Both commands appear (wizard in Quick Start, non-wizard in CLI page) without a clear comparison. Beginners may not know which to use for their situation.

4. "PAT" acronym used without definition

The note says "Create a fine-grained PAT" — Personal Access Token should be spelled out on first use for absolute beginners.

5. CLI Commands page is extremely long

The CLI Commands page is very comprehensive (good!) but overwhelming as a starting point. The "Most Common Commands" table at the top helps, but there's no clear recommended learning path or "start here" guidance.

6. No "What is GitHub Actions?" link for total beginners

The tool relies heavily on GitHub Actions, but never links to GitHub's own documentation for readers who aren't familiar with it.

---

🟢 What Worked Well

• Home page is clear: The headline "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" immediately communicates the value proposition. ✅
• "By the Numbers" table gives a nice quick-glance summary of capabilities. ✅
• Quick Start has a time estimate ("⏱️ Estimated time: 10 minutes") — great for setting expectations. ✅
• Video in Quick Start — showing the install process visually is excellent for beginners. ✅
• CLI "Most Common Commands" table — immediately useful for orientation. ✅
• Navigation — "Quick Start", "Create", "Examples", "Docs", "FAQ" links in the top nav are well-organized. ✅
• Alternative installation methods are well-documented (curl script, pinned versions, GitHub Enterprise). ✅
• Collapsible note boxes for secondary info keep the main flow clean. ✅

---

Recommendations

Quick Wins 🚀

1. Add a glossary callout at the top of Quick Start defining key terms: frontmatter, GitHub Actions, lock file, PAT. Even a single sentence per term would help.
2. Move the two-file explanation earlier in the Quick Start (before Step 2, not in Step 4).
3. Make the COPILOT_GITHUB_TOKEN distinction prominent — add a warning/callout at the top of the token setup steps, not buried in a note.
4. Spell out acronyms: PAT → Personal Access Token (PAT).
5. Remove or document Gemini — either add engine documentation or remove from Step 2.

Longer-Term Improvements 📈

1. Add a "How GitHub Actions works" explanation or link for beginners unfamiliar with the platform.
2. Add a "Troubleshooting the Quick Start" section with common error messages and their fixes.
3. Create a visual diagram showing the relationship between .md workflow file, .lock.yml, and the GitHub Actions run.
4. Add a "Choose your engine" comparison page helping users pick between Copilot, Claude, Codex, and Gemini based on their situation.
5. Consider a step-by-step interactive tutorial for the CLI commands page, since it's currently a reference document that's hard to navigate as a beginner.

---

Screenshots

📎 [home.png] — Home page overview
asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/1f03139e4177e369a840deb22b9be6f9b1c7e091e6ed982b101e01cb2e40cee1.png?raw=true

📎 [quick-start.png] — Quick Start guide
asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/7f1394ed30af710d52aa1c8350d3ea326a271e2d7321f196a3809969eb1a6290.png?raw=true

📎 [cli.png] — CLI Commands page
asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/183a01b0106498148169fa2d193de9997f056ef78e37107cfff26a7f2f247d4f.png?raw=true

---

References: §25651330900

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by Documentation Noob Tester · ● 21.7M · ◷

• expires on May 12, 2026, 5:16 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #31635.