One of the WordPress sites I build and run for a client is a news aggregator. It reads feeds from a long list of outlets, groups stories about the same event, writes short summaries and shows readers how each story is covered. Every outlet is a publisher record with its own page, statistics and settings.
The interesting work is rarely a single story. It is managing the publishers in bulk without making the site slower or losing track of what changed.
A news aggregator's sources: a directory that cost 174 MB
The publisher directory lists every outlet with a few numbers: how many stories, how recent, which topics it covers. It was slow, and when I measured a cold load it was far worse than slow:
| Before | After | |
|---|---|---|
| Cold directory load | 9.8 seconds | 1.8 seconds |
| Database queries | 4,375 | 556 |
| Memory | 174 MB | 2 MB |
The statistics for each publisher loaded the metadata of every one of its stories into memory and then read five values from each, just to produce six numbers. It also fetched topic terms one story at a time. The directory did that for every publisher. Both loops became one grouped query each.
The memory figure was the real risk. A cold load only happens after the cache is cleared, and every deploy clears it. With up to 50 PHP workers, a handful of visitors arriving just after a deploy could each take 174 MB, which is a plausible route to running the server out of memory.
When the directory cache stampeded
With the directory cached, a different problem appeared. When the cache expired, every request that arrived in the next few seconds rebuilt it at once. The directory is now built in bulk in the background, stored, and its board is rendered as static HTML. Visitors read the stored copy, and only one process ever rebuilds it. I wrote about the general pattern in check-then-set is a race.
Adding outlets with a way back
When the client wanted better coverage of trending topics, I added about 30 new outlets in one change. Before touching the feed list I saved a copy of it as a named backup option, so the whole addition could be reversed in one step. New outlets were rated for the coverage balance view before their stories started arriving, so the balance never drifted while I caught up.
Retiring stale stories without deleting them
The same clean-up found tens of thousands of stories that had sat in a pending state for too long, waiting for summaries that would never be worth writing. Instead of deleting them, I moved them to draft with a marker saying why, and kept a log of every one. If the decision was wrong, the marker makes it a single query to reverse.
The rule I follow for bulk changes: take a backup that can be restored in one step, mark what you changed so it can be found again, log the list, and only then run it.
A long job that lost its folder
The backfill for the new outlets ran as a detached job for several hours. Partway through, it started producing empty batches with no errors. The job had started inside the site's current release folder, which is a symlink. The shell held on to the real folder it pointed at. Two deploys later, the deploy tool had pruned that old release, and every command the job ran was quietly failing in a folder that no longer existed.
The fix is small: re-enter current for each batch, and treat an empty batch as a failure rather than as "nothing to do". I now check any long-running job on a site with release folders.
What carries over
- Measure a cold load, not a warm one. Deploys make cold loads normal.
- Look for loops that run a query per item, then per item again.
- Let one process rebuild an expensive page, and serve the stored copy meanwhile.
- Make every bulk change reversible before you make it.
- Treat silence from a long job as a possible failure.