Check-then-set is a race: atomic locks in WordPress

Here is a pattern that appears in almost every WordPress codebase I look at, including my own:

if ( ! get_transient( 'rebuild_lock' ) ) {
    set_transient( 'rebuild_lock', 1, 60 );
    rebuild_expensive_thing();
    delete_transient( 'rebuild_lock' );
}

It reads well, and on a quiet site it works. Under load, two requests can both run the check before either has run the set. Both see "no lock", both take it, and both do the expensive work. The gap between the check and the set is small, but at a few hundred requests a second small gaps get hit all the time.

Where it bites

The same shape turns up in three places, each with its own symptom:

The balance case is the nastiest. On a WordPress site I run, a cached points balance was read, checked and written back. When I ran six workers spending from the same account at the same time, the balance ended at -190. Each worker had read a balance that looked healthy, passed its own check, and written its own result over the others'. No single request did anything wrong.

The fix: make the check and the set one operation

The cure is always the same idea. Stop asking "is it free?" and then claiming it. Ask the storage to claim it only if it is free, and tell you whether it worked. There are four ways I use, depending on what is available.

Redis SET with NX and EX

If you talk to Redis directly, SET key value NX EX seconds sets the key only if it does not exist, with an expiry, in one atomic command:

$token = wp_generate_uuid4();
if ( $redis->set( 'lock:rebuild', $token, [ 'nx', 'ex' => 60 ] ) ) {
    try {
        rebuild_expensive_thing();
    } finally {
        // Release only if it is still our lock.
        $redis->eval(
            "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) end return 0",
            [ 'lock:rebuild', $token ],
            1
        );
    }
}

The token matters. If the work runs past the expiry and another worker takes the lock, a plain DEL would release their lock, not yours.

wp_cache_add as an atomic add

Inside WordPress, wp_cache_add() only writes when the key does not already exist, and with a persistent object cache such as Redis it maps to an atomic add. It returns false if someone got there first:

if ( wp_cache_add( 'rebuild', 1, 'locks', 60 ) ) {
    rebuild_expensive_thing();
    wp_cache_delete( 'rebuild', 'locks' );
}

One caution: without a persistent object cache the cache lives only for one request, so this lock protects nothing between requests. Check that your drop-in is active with wp cache type before relying on it. Transients are not a substitute, because set_transient() overwrites rather than adds.

MySQL GET_LOCK

If you have no Redis, MySQL has named locks:

global $wpdb;
if ( '1' === $wpdb->get_var( "SELECT GET_LOCK('rebuild', 0)" ) ) {
    rebuild_expensive_thing();
    $wpdb->query( "SELECT RELEASE_LOCK('rebuild')" );
}

The lock belongs to the database connection and is released if the PHP process dies, which is a pleasant property. It does not help across separate database servers.

Recompute from the ledger

For balances, the better fix is to stop storing a running total as the truth. Record each spend and each credit as a row, and work out the balance from the rows. Guard the spend with a conditional write so it cannot overdraw:

INSERT INTO points_ledger (user_id, delta)
SELECT 42, -50
FROM DUAL
WHERE (SELECT COALESCE(SUM(delta), 0) FROM points_ledger WHERE user_id = 42) >= 50;

Run that inside a transaction with the user's rows locked (SELECT ... FOR UPDATE) on InnoDB, and treat any cached balance as a convenience you can always rebuild from the ledger. After this change the same six-worker test never went below zero.

Write the race test

Races do not show up in a normal test suite because a test runs one request at a time. You need real parallel processes. WP-CLI makes that easy. Put the operation in a small script:

<?php
// race.php: spend 50 points twenty times.
for ( $i = 0; $i < 20; $i++ ) {
    my_points_spend( 42, 50 );
}

Then reset the balance and run six copies at once:

wp eval 'my_points_set( 42, 1000 );'
for n in 1 2 3 4 5 6; do
  wp eval-file race.php &
done
wait
wp eval 'echo my_points_balance( 42 ), PHP_EOL;'

With 1,000 points and 50 per spend, exactly twenty spends should succeed and the balance should finish at zero. Anything below zero, or a count of successes above twenty, means a race. Run it several times, because a race that loses once in ten runs is still a race. Use the same harness for click dedupe (count the rows recorded) and stampede locks (count how many times the rebuild ran).

A quick checklist

• Search for get_transient, get_option or get_user_meta followed closely by a set of the same key. Each one is worth a look.
• Locks should be claimed with an atomic add and released only by their owner.
• Money, points and stock should be derived from a ledger, with the cache as a copy.
• Every fix should come with a parallel test that failed before it.

Further reading: the Redis documentation for SET with NX and EX.