Why does robots.txt expose a preview sitemap host?
The robots metadata route often builds Sitemap and Host from a deployment URL. Use one validated public origin and keep preview environments protected or noindexed.
Written for next.js teams deploying the same application to production, branch previews, staging, and local environments.
Key facts
- Next.js supports a robots metadata file or route that can declare rules, sitemap, and host.
- A deployment URL environment variable may point to the current preview rather than the canonical production site.
- robots.txt is not a security boundary for preview deployments.
A useful rule: make each important claim understandable and verifiable without requiring the reader to reconstruct your meaning from the rest of the page.
How does the wrong host enter robots.txt?
Code often reads a platform-provided deployment URL and interpolates it into `sitemap` or `host`. On production that variable may be correct, but on branch and preview builds it naturally points to the temporary deployment. Another failure is a missing scheme or an environment fallback to localhost. Because robots output is plain text and may be cached, the mistake can survive after other metadata is fixed. Fetch `/robots.txt` from every environment and trace each absolute value to its configuration source.
- Inspect actual robots output, not only source code.
- Identify canonical and deployment URL variables separately.
- Clear edge caches after correcting the route.
What origin should production use?
Use one explicitly configured canonical public origin shared with metadata, sitemaps, structured-data IDs, and internal URL builders. Validate the protocol and hostname at build time. Production should fail loudly if the value is missing rather than inventing a host from the request. The sitemap declaration should point to the public XML sitemap served on that origin. Google's robots specification does not use a general `Host` directive as a canonical substitute, so rely on redirects and canonical signals for hostname preference.
- Centralize and validate the public origin.
- Keep sitemap location absolute and fetchable.
- Do not treat a Host line as canonical enforcement.
How should preview deployments behave?
The strongest preview control is access restriction when previews are not intended for public indexing. Authentication, platform deployment protection, or network controls prevent accidental discovery more reliably than robots.txt. If public preview access is required, add appropriate noindex responses and avoid linking previews from public surfaces. A robots disallow alone can stop crawling while leaving a discovered URL eligible to appear without content. Also prevent preview URLs from entering production sitemaps, canonical tags, and social metadata.
- Protect nonpublic previews with access control.
- Use noindex only where crawlers can fetch it.
- Scan production artifacts for preview hostnames.
Could cached robots rules affect a launch?
Yes. Crawlers cache robots.txt, and CDNs can cache the route too. A temporary production disallow or stale preview sitemap may persist after a code fix. Review response headers, edge cache configuration, and the live file before launch. Avoid changing from block-all to allow-all at the final minute when possible. Keep a release checklist that verifies robots, sitemap, canonical, and status on the production hostname. If access was mistakenly blocked, correct it, confirm 200 responses, and allow time for crawlers to refresh the file.
- Inspect cache headers and purge only the exact affected asset when needed.
- Verify the live file before announcing launch.
- Plan for crawler cache refresh time.
Which automated checks should run?
Request robots.txt in the built or deployed environment, parse every Sitemap value, and require the approved origin. Fetch each sitemap and inspect a sample of listed URLs. Scan generated HTML and JSON for known preview suffixes, localhost, and private IP addresses. Test that protected previews reject anonymous users and that production remains crawlable. Keep these checks separate from syntax validation because a perfectly formatted robots file can still advertise the wrong site or block the right one.
- Assert approved hosts in every discovery artifact.
- Fetch declared sitemaps and sampled pages.
- Test environment access policy independently.
Put it to work
Find the highest-impact fix on your site.
Find preview, local, and conflicting origins in robots.txt, sitemaps, canonicals, and metadata.
Check robots and sitemap hostsSources
- 1.Next.js documentation: robots.txtChecked 2026-07-26
- 2.Google Search Central: Introduction to robots.txtChecked 2026-07-26