Every Canonical on https://dil.dev, Including the Sitemap

The site has one canonical origin: https://dil.dev. Every tag, every sitemap entry and every open-graph URL must point there, and a postbuild script enforces that before any deploy can go out.

Why One Origin Matters

Search engines treat https://dil.dev and https://dil-dev.netlify.app as separate hosts unless you tell them otherwise. Duplicate content across two origins splits whatever signal the pages accumulate. It also means a crawler could index the Netlify subdomain directly, which is not the version you control the branding or caching on.

The fix is a 301 from dil-dev.netlify.app to https://dil.dev. A permanent redirect tells crawlers to follow, consolidate and forget the source host. It is not advisory the way robots.txt is. The redirect removes the subdomain from the picture entirely.

The Postbuild Script

npm run build runs scripts/verify-seo.mjs as a postbuild step. It runs after the static output is generated, so it reads the actual built HTML and XML rather than source templates.

The script checks:

• Every <link rel="canonical"> must have https://dil.dev as its origin.
• Every og:url and og:image must use the same apex host.
• Every <loc> in the sitemap must point to https://dil.dev.
• The Sitemap: line in robots.txt must reference the apex sitemap URL.
• Coverage: every indexable page must appear in the sitemap.
• Cleanliness: no page carrying a noindex directive may appear in the sitemap.

If any assertion fails, the script exits with a non-zero code. Netlify treats that as a failed build and does not deploy. The check is a hard gate, not a lint warning you can ignore.

Registering New Pages

New public pages do not appear in the sitemap automatically. You add the path to STATIC_PAGES in site/lib/seo.mjs. The postbuild script reads from that list when it validates coverage, so a page that exists in the build output but is absent from STATIC_PAGES will cause a failure.

This is intentional friction. It forces a deliberate decision: is this page meant to be indexed? If yes, register it. If not, make sure it carries a noindex directive so the cleanliness check does not flag it either.

The two states a page can be in:

• Indexable: listed in STATIC_PAGES, present in the sitemap, no noindex.
• Non-indexable: not listed, not in the sitemap, carries noindex if it exists in the build.

There is no third state where a page is in the sitemap but also noindexed. The script will catch that contradiction and fail the build.

Case Studies and the ready Flag

Case studies follow a slightly different path. A case study page can exist in the build without being indexable. The condition is ready: true in the case study data.

Until that flag is set:

• The page is excluded from the sitemap.
• The postbuild script treats it as non-indexable and does not expect it to appear in coverage.
• The page should carry noindex so that even if a crawler finds it via a link, it will not index it.

Setting ready: true moves the page into the indexable set. The next build will include it in the sitemap and the postbuild script will verify its canonical tags along with everything else.

This keeps draft work out of the index without requiring a separate branch or a password gate. The page can be reviewed at its real URL, shared internally, iterated on, and then flipped to ready when it is actually ready.

What the 301 Does That robots.txt Cannot

It is worth being precise here because the two mechanisms are often conflated.

• robots.txt disallow tells compliant crawlers not to fetch certain paths on a given host. It does not prevent the host from appearing as a separate origin. It does not consolidate link signals. A disallowed page can still appear in search results if other sites link to it.
• A 301 redirect tells every client, crawler or browser, that the resource has permanently moved. The destination gets the credit. The source host effectively disappears from the crawl graph for those URLs.

For dil-dev.netlify.app the 301 is the right tool. The goal is not to hide pages on that host. The goal is to make that host not exist as a separate origin in any meaningful sense. See how the Netlify config handles path vs redirect rules for the mechanics of setting that up.

Verifying It Yourself

After a build, the checks you can run manually:

• curl -I https://dil-dev.netlify.app should return 301 with Location: https://dil.dev.
• View source on any page and search for canonical. The href should start with https://dil.dev.
• Fetch https://dil.dev/sitemap.xml and grep for any loc that does not start with https://dil.dev. There should be none.
• Check that every URL in the sitemap returns a 200 and does not carry noindex in its response.

The postbuild script does all of this automatically, but running it manually after a config change is a reasonable sanity check before pushing.

What I'd do differently

The STATIC_PAGES list in site/lib/seo.mjs is a manual registry. It works, but it means adding a page in two places: wherever the route is defined, and in that list. A future version could derive the list from the router or from a filesystem glob over the built output, then diff against a known-noindex set. That would remove the manual step and make the coverage check self-maintaining. For now the manual list is small enough that the friction is acceptable, but it is the first thing I would revisit as the number of public pages grows.