Audit a WordPress network before every deploy, and after

A WordPress network with a main site and several brand sites has more ways to break than one site. I wrote a read-only audit that crawls every site before a deploy and again after it. Its first run found a site-wide broken link whose impact a full crawl had obscured, and two of its own checks were wrong.

By Shane Rounce · 5 minute read

One WordPress network I look after belongs to an industrial equipment supplier. It is a WordPress multisite: a main shop, a development site, and several brand sites that launch one at a time. Brand sites that have not launched must stay out of search. Production sites must never be held back. Links between them must not lead anywhere that is not ready.

After an SEO crawl came back with a long list of problems, I fixed them and then wrote something to stop them coming back.

What the WordPress network audit checks

It is a Python script that crawls every site in the network, read-only, with four workers, and exits with a failure if it finds any of these:

internal links that return 404 or a server error, or redirect to one;
redirect chains;
links from production sites into brand sites that have not launched;
reserved WordPress and WooCommerce query variables, such as ?product=, in generated links;
unlaunched brand sites that are indexable, and production sites that are not;
staging or development hostnames in rendered HTML;
sitemap entries that are noindexed or do not return 200;
brand and category archives with no og:description or og:image.

Each site has a kind (production, brand or development) that decides which rules apply, read from the same switch the theme uses to launch a brand. I run it against the local copy before a deploy and against live after one, and it can write a JSON report for anything that wants to read it.

What the first run found

One of the header's main product links pointed at a category address that had been retired when the category was renamed. Every page on the site carried that broken link. The earlier crawl had found it, but folded it into a total along with everything else, so it read as one problem among many rather than a broken link on every page. A redirect now maps the old address, and the saved menu item is corrected on live by the same guarded script that handles other retired routes.

It also found category archives that shared as bare cards because the category had no description. Those now get a short generated description.

Two checks that were wrong

An audit that cries wolf gets ignored, so its own mistakes matter. Two checks were wrong for a site still in preview:

An open robots.txt on a preview site is deliberate. The site is kept out of search with noindex. A Disallow in robots.txt would stop crawlers reading the pages, so they would never see the noindex and could keep old entries. The check is now a warning, not a failure.
Missing sitemaps on a preview site are expected, because WordPress removes them when search engines are discouraged.

Two multisite traps it helped explain

The audit did not catch these directly, but its reports made them easier to see.

Launching a brand site saved the wrong rewrite rules. The launch code switched to the brand site and called flush_rewrite_rules(). After switch_to_blog(), that rebuilds the rules from the code loaded for the site you started on, and saves them into the brand site. The fix is to delete the brand site's rewrite_rules option instead, so it rebuilds its own rules on its next request.

A menu rendered for a 404 was being kept. The mega menu is cached as a fragment. On a site with sitemaps switched off, a crawler asking for /wp-sitemap.xml got a 404 page, the menu's product queries returned the wrong thing on that page, and the stored menu was replaced by one showing a blog post. Nothing rendered for a 404 or a sitemap request is kept now, in the menu's own cache or the block cache.

Small details that made it trustworthy

Sitemap pages get the same checks as crawled pages, so a crawl limit cannot hide them.
Ace Crawl Enhancer, my SEO plugin, shows sitemaps as styled HTML to browsers, so the audit asks for XML explicitly with an Accept header.
It identifies itself with its own user agent, so its visits are easy to separate in logs.
It never writes anything.

Why before and after

Before a deploy, it catches what a change is about to break. After a deploy, it catches what differs between the copy I tested and the site people use: saved menus, content edited on live, settings that never left production. Both have found something.

If you look after more than one WordPress site under one roof, a script like this takes an afternoon. Start with the defects your last crawl found, make each one a check that fails loudly, and fix the checks that turn out to be wrong.