# Every Canonical on https://dil.dev, Including the Sitemap

> Every URL on dil.dev points its canonical, og:url and sitemap loc at the apex host. A postbuild script enforces this so a stray Netlify subdomain never leaks into the index.

- Author: Dil Ahdan (https://dil.dev/)
- Published: 2026-10-01
- Updated: 2026-10-01
- Tags: seo, canonical, sitemap, netlify
- Canonical: https://dil.dev/notes/seo-every-url-on-the-apex-host/

## Key takeaways

- The Netlify preview host 301s to https://dil.dev so crawlers always land on the canonical origin.
- scripts/verify-seo.mjs runs as a postbuild step and fails the build if any canonical, og:url, og:image, sitemap loc or robots Sitemap line points off-apex.
- New public pages must be registered in STATIC_PAGES inside site/lib/seo.mjs or the script will flag them as missing.
- Case studies are excluded from the sitemap until ready: true is set in their data, keeping draft work out of the index.
- The sitemap must list every indexable page and nothing that carries a noindex directive, and the script checks both conditions.

The site has one canonical origin: `https://dil.dev`. Every tag, every sitemap entry and every open-graph URL must point there, and a postbuild script enforces that before any deploy can go out.

## Why One Origin Matters

Search engines treat `https://dil.dev` and `https://dil-dev.netlify.app` as separate hosts unless you tell them otherwise. Duplicate content across two origins splits whatever signal the pages accumulate. It also means a crawler could index the Netlify subdomain directly, which is not the version you control the branding or caching on.

The fix is a 301 from `dil-dev.netlify.app` to `https://dil.dev`. A permanent redirect tells crawlers to follow, consolidate and forget the source host. It is not advisory the way robots.txt is. The redirect removes the subdomain from the picture entirely.

## The Postbuild Script

`npm run build` runs `scripts/verify-seo.mjs` as a postbuild step. It runs after the static output is generated, so it reads the actual built HTML and XML rather than source templates.

The script checks:

- **Every `<link rel="canonical">`** must have `https://dil.dev` as its origin.
- **Every `og:url` and `og:image`** must use the same apex host.
- **Every `<loc>` in the sitemap** must point to `https://dil.dev`.
- **The `Sitemap:` line in robots.txt** must reference the apex sitemap URL.
- **Coverage:** every indexable page must appear in the sitemap.
- **Cleanliness:** no page carrying a `noindex` directive may appear in the sitemap.

If any assertion fails, the script exits with a non-zero code. Netlify treats that as a failed build and does not deploy. The check is a hard gate, not a lint warning you can ignore.

## Registering New Pages

New public pages do not appear in the sitemap automatically. You add the path to `STATIC_PAGES` in `site/lib/seo.mjs`. The postbuild script reads from that list when it validates coverage, so a page that exists in the build output but is absent from `STATIC_PAGES` will cause a failure.

This is intentional friction. It forces a deliberate decision: is this page meant to be indexed? If yes, register it. If not, make sure it carries a `noindex` directive so the cleanliness check does not flag it either.

The two states a page can be in:

- **Indexable:** listed in `STATIC_PAGES`, present in the sitemap, no `noindex`.
- **Non-indexable:** not listed, not in the sitemap, carries `noindex` if it exists in the build.

There is no third state where a page is in the sitemap but also noindexed. The script will catch that contradiction and fail the build.

## Case Studies and the `ready` Flag

Case studies follow a slightly different path. A case study page can exist in the build without being indexable. The condition is `ready: true` in the case study data.

Until that flag is set:

- **The page is excluded from the sitemap.**
- **The postbuild script treats it as non-indexable** and does not expect it to appear in coverage.
- **The page should carry `noindex`** so that even if a crawler finds it via a link, it will not index it.

Setting `ready: true` moves the page into the indexable set. The next build will include it in the sitemap and the postbuild script will verify its canonical tags along with everything else.

This keeps draft work out of the index without requiring a separate branch or a password gate. The page can be reviewed at its real URL, shared internally, iterated on, and then flipped to ready when it is actually ready.

## What the 301 Does That robots.txt Cannot

It is worth being precise here because the two mechanisms are often conflated.

- **robots.txt disallow** tells compliant crawlers not to fetch certain paths on a given host. It does not prevent the host from appearing as a separate origin. It does not consolidate link signals. A disallowed page can still appear in search results if other sites link to it.
- **A 301 redirect** tells every client, crawler or browser, that the resource has permanently moved. The destination gets the credit. The source host effectively disappears from the crawl graph for those URLs.

For `dil-dev.netlify.app` the 301 is the right tool. The goal is not to hide pages on that host. The goal is to make that host not exist as a separate origin in any meaningful sense. See [how the Netlify config handles path vs redirect rules](/notes/netlify-config-path-vs-redirect/) for the mechanics of setting that up.

## Verifying It Yourself

After a build, the checks you can run manually:

- `curl -I https://dil-dev.netlify.app` should return `301` with `Location: https://dil.dev`.
- View source on any page and search for `canonical`. The `href` should start with `https://dil.dev`.
- Fetch `https://dil.dev/sitemap.xml` and grep for any `loc` that does not start with `https://dil.dev`. There should be none.
- Check that every URL in the sitemap returns a 200 and does not carry `noindex` in its response.

The postbuild script does all of this automatically, but running it manually after a config change is a reasonable sanity check before pushing.

## What I'd do differently

The `STATIC_PAGES` list in `site/lib/seo.mjs` is a manual registry. It works, but it means adding a page in two places: wherever the route is defined, and in that list. A future version could derive the list from the router or from a filesystem glob over the built output, then diff against a known-noindex set. That would remove the manual step and make the coverage check self-maintaining. For now the manual list is small enough that the friction is acceptable, but it is the first thing I would revisit as the number of public pages grows.

## Quick answers

**Why does dil-dev.netlify.app redirect instead of just being blocked by robots.txt?**

A 301 redirect tells crawlers and link-equity that the canonical host is https://dil.dev, not just that the subdomain should be ignored. robots.txt disallow would still let the subdomain appear as a separate origin in some tools. The redirect removes the ambiguity entirely.

**What exactly does scripts/verify-seo.mjs check?**

It checks that every canonical tag, og:url, og:image and sitemap loc uses https://dil.dev as the origin. It also checks that the robots.txt Sitemap line points to the apex host, that every indexable page appears in the sitemap, and that no noindex page is listed there.

**How do I add a new public page so it appears in the sitemap?**

Add the path to STATIC_PAGES in site/lib/seo.mjs. The postbuild script reads from that list when it validates sitemap coverage, so omitting a page will cause a build failure.

**When does a case study become indexable?**

Only when ready: true is set in the case study data. Until that flag is present the page is excluded from the sitemap and the postbuild script treats it as non-indexable, so it will not appear in search results.

**What happens if the postbuild check fails?**

npm run build exits with a non-zero code, which fails the Netlify deploy. Nothing ships until every canonical and sitemap assertion passes. That makes the check a hard gate rather than an advisory warning.