Prerendering a React SPA for crawlers

I write my long-form essays into a database and read them back on a React front end. In a browser they look perfect: title, body, table of contents. So it took me embarrassingly long to check the one view that matters for search: the one a crawler sees when it does not run your JavaScript.

Every single essay was serving a blank. The check is one command. Ask for the page the way a crawler does, with scripts off, and read the title it gets back.

# Fetch the raw HTML a non-JS crawler receives,
# and read just the <title> it hands back.
curl -s https://your-site.com/some-page | grep -o '<title>[^<]*</title>'

The page fetches its own body. The component mounts, and inside an effect it calls the API for the content. That effect only runs in a real browser that executes JavaScript. A crawler that reads the raw HTML never runs it, so it receives the empty shell that was there before the fetch.

The confusing part was that some pages on the same site, built by the same prerender, were perfectly crawlable. The difference is where the content lives.

A page whose body is static data inside the component renders into the HTML at build time. A page whose body is fetched at runtime does not. Same site, same pipeline, two different outcomes, and the only difference is static versus fetched.

The fix is to make the body synchronously available at build time, without giving up the live fetch for real readers.

1. Snapshot the content at build time into a file the component can read synchronously.
2. Seed the component's initial state from that snapshot, so the server-rendered HTML already contains the real body.
3. Keep the live fetch on mount, so readers still get the freshest version after hydration.

Now the crawler gets a real page, the reader gets live content, and nothing about the browser experience changes.

This bug is a specific instance of a trap I keep relearning: you have to know exactly which layer a crawler actually sees, and verify that layer, not the one that is convenient.

It renders is not it is indexed. The browser shows you the best case. Verify the crawler's-eye view, page by page, or the serving layer will lie to you.

1. Curl one content page with scripts off and read the title and body it returns.
2. If the body is missing, find where that content is fetched instead of baked.
3. Snapshot it at build time and seed the initial render from the snapshot.
4. Re-check every page type. Static and fetched bodies behave differently.