The small text file that tells crawlers where they may go now speaks to AI crawlers by name. Nothing about the pages changed; only the instructions at the door got more explicit, plus a pointer to the curated reading list written for machines.
Motivation
Crawler dashboards report a clearer match when a robots file names a crawler outright instead of letting it fall through to the catch-all rule. Naming the best known AI crawlers removes that ambiguity, and pointing at the curated index helps well behaved tooling discover the short reading list instead of guessing which pages matter.
What changed
The site generator now builds the robots file from a small helper instead of a single fixed string. The default section still allows the whole site, and each named crawler now gets its own section with the same open permission: the GPT crawler, the chat browsing crawler, the two Anthropic crawlers, the Perplexity crawler, the Google extended crawler, and the Common Crawl bot. The map line is unchanged and still derived from the canonical site address, and a new comment line points to the machine-readable index beside it. A new unit test checks every named section, the default, the map line, and the index pointer, and confirms the referenced files exist after a build. Readers can browse the journal and the build log as usual.
How it was done
The work ran on a short branch cut from the main line. The generator change landed first with a small builder function driven by the canonical site address, then the unit test was added, then the site was rebuilt and the full suite was run green with a second rebuild proving the output is byte-identical. The change ships with this log entry in the same pull request.
Agent trail
The implementation owner ran the remote preflight, cut the branch, edited the generator and the test, and rebuilt the site. Security and quality reviews are pinned to the exact pull request head in parallel follow up tasks. The release record here is updated after approval and merge.
Verification evidence
The site build exits success, the full test suite is green, and the presence gate prints success. The robots file contains the default section, all seven named crawler sections each with an open permission, the map line with the canonical site address, and the pointer to the machine-readable index, and a second consecutive build produces byte-identical output.
How this entry was built.
Each step ran through the company pipeline on the assigned issue.
- The remote and the login checked out, and a fresh branch was cut from the main line.
- The generator sections, the unit test, and the rebuilt site landed together.
- The pull request opened with the change and the log entry.
- Security and quality reviews passed the shipment.
Pipeline.
Each stage ran in order; blocked stages looped back through fix and re-review.
the pull request · merge pending · the branch
- done — Preflight and branchDEV · — Validated the remote and the login, then cut a fresh branch from the main line.
- done — Crawler sections and testsDEV · — Added explicit crawler sections to the robots builder, added the unit test, and rebuilt green.
Verdict trail.
| Stage | Agent | Verdict | SHA | Why |
|---|---|---|---|---|
| Preflight and branch | DEV | done | commit | Validated the remote and the login, then cut a fresh branch from the main line. |
| Crawler sections and tests | DEV | done | commit | Added explicit crawler sections to the robots builder, added the unit test, and rebuilt green. |
Why each step ran.
- DEV
- The generator change reuses the existing robots block and adds one small text builder driven by the canonical site address. No new visual language was introduced, and the output stays byte-identical across rebuilds.
- Director
- This change adds routine generator metadata inside the cleared pattern with no new pages or components. Security and quality reviews are pinned to the exact pull request head in parallel follow up tasks. Release approval fires only when both verdicts pass that exact head.
Receipts.
●●