Why Sitemap and robots.txt Bugs Survive for Months
Both files are generated at build time and nobody looks at them after the first deploy. When they break, the symptom is pages that never get crawled, which looks like a content problem.
The Repo Does Not Tell You What Ships
A Next.js app/sitemap.ts serves /sitemap.xml with no file on disk. Reading the repo, you conclude there is no sitemap; reading the live URL, there is. Judging from committed files is how a complete sitemap gets reported as empty.
Private Routes Leak Into the Sitemap
A generator that enumerates every route hands Google your /admin, /login and /api paths. Crawl budget goes to pages that should never be indexed, and some of them surface in search.
Robots Rules Get Quoted From Memory
An audit that claims robots.txt blocks /private/ when no such line exists sends you fixing a file that was fine. A robots finding is only useful if it quotes the file that was downloaded.
How SEOAgent Checks the Technical Layer
Generation edits your framework, so your coding agent does it. Validation is deterministic, so the CLI does it. Both halves work on the live site.
The Sitemap Is Fetched, Not Inferred
`seoagent sitemap` requests https://yourdomain/sitemap.xml and reports the HTTP status and URL count. An unreachable sitemap, a 4xx, invalid XML, or a 200 with zero URLs all fail the check. Pass `--url` to point at a sitemap index or a non-standard path.
Private Routes Are Flagged
Every URL in the sitemap is matched against private path patterns. Admin, login and API routes that made it into the generator come back as a warning, with the exact URLs listed.
Coverage Against Your Actual Pages
SEOAgent discovers the public routes and content files in your repo and compares them with the sitemap. Pages your site serves but the sitemap omits are reported as missing, so a new article does not sit undiscovered.
Honest lastmod Dates
A generator that stamps its own build time on every entry produces a lastmod field that carries no information. SEOAgent detects one uniform date across all URLs and warns. An absent lastmod is fine and is reported as info, not an error.
robots.txt Findings Quote the Real File
`seoagent crawl` downloads robots.txt and writes its contents into `.seoagent/audit/evidence.md`. Blocking rules are matched against your page list from the parsed file, and a rule that is not in it cannot be reported. A 200 that returns HTML is rejected as not a robots.txt.
Fixes Are Proposed, Then Approved
robots.txt optimization proposes the replacement file and waits. You read the diff and approve it before anything is written, and the change is reversible. Applying fixes is still partial: the proposal is the reliable half today.
How the Technical SEO Checks Run
Point SEOAgent at the Domain
`seoagent init` writes `.seoagent/project.md` with your domain. That is the origin the sitemap and robots checks fetch from, so the checks run against production rather than localhost.
Crawl and Validate
`seoagent crawl` fetches the live pages plus robots.txt into `.seoagent/audit/evidence.md`. `seoagent sitemap` fetches the sitemap and prints the report; `--json` gives the same result for scripting.
Fix the Generator, Not the Output
Findings point at the generator that produced them: drop the route from the URL list, add the content directory the generator misses, emit a real per-page date or omit lastmod. Your coding agent makes the edit in your repo.
Re-run, Then Deploy
Run `seoagent sitemap` again to confirm the fix landed. The command exits non-zero while the sitemap still fails, so it works as a CI check. You commit and deploy through your own pipeline.
Technical SEO Use Cases
New Pages Are Not Getting Crawled
Check whether the sitemap lists them at all. A generator with a hard-coded route list silently stops covering new content.
Post-Migration Sanity Check
After a framework change, confirm the new build still serves a valid sitemap and did not start leaking staging or preview routes.
Sitemap in CI
Run `seoagent sitemap` after deploy. A non-zero exit fails the pipeline when the sitemap becomes unreachable or empty.
Auditing a Client Site
Get a grounded robots.txt and sitemap report in one session, with the fetched file as evidence, instead of a checklist copied from another audit.
AI Crawler Policy
The audit reads the real robots.txt, so you can see which user-agents you actually allow before deciding whether to change that.
Documentation Sites
Keep a large docs tree covered as it grows, and catch the day the generator starts skipping a section.
Frequently asked questions
Does SEOAgent generate the sitemap for me?
Your coding agent edits the sitemap generator in your repo, because that is a code change specific to your framework. SEOAgent supplies the protocol for the common generators (Next.js app/sitemap.ts, next-sitemap, Astro, static files) and the deterministic check that tells you whether the result is correct. When your site is connected to SEOAgent Cloud, the cloud path can also build a sitemap and submit it to Google Search Console for you.
Do I need an account to check my sitemap?
No. `seoagent sitemap` fetches your live sitemap and compares it against the pages it finds in your repo, all locally, before the login check. Search Console submission is the part that needs a connected account.
Can it apply robots.txt changes?
It proposes them. The proposal is shown as a diff and applying it is approval-gated and reversible. Automatic application is still partial, so treat the proposal as the output you act on.
What does it check on robots.txt?
Whether the file returns 200, what its real Disallow rules are, whether any of them block pages in your inventory, and whether it carries a Sitemap: line. Claims are matched against the parsed file, so a rule that is not in your robots.txt is never reported as one that is.
My sitemap has no lastmod dates. Is that a problem?
No. A missing lastmod is reported as info, not an error. The warning is for the opposite case: one build timestamp repeated on every URL, which tells a crawler nothing and is worse than leaving the field out. Add a real per-page date where you have one, such as a file mtime or a CMS updatedAt.
Will it change files in my repo?
The sitemap and robots checks are read-only. Anything SEOAgent writes lands in your working tree as a diff you review, commit and deploy yourself.
More of the SEO engine
Every feature runs from the same Skill in your repo — install once, get all of it.
Find Out What Your Sitemap Actually Serves
Install the free SEOAgent Skill and run `seoagent sitemap` against your live site.