RegisterLogin
Add us as preferred source on Google →
Linux Administration

Diagnosing and Resolving Redis Cluster Split-Brain & Sentinel Failover Race Conditions under PHP-FPM Object Cache Bursts

Deep-dive guide to diagnosing Redis Sentinel split-brain, failover race conditions, and PHP-FPM cache stampedes with real commands and fixes.

Nextgen Team•
Diagnosing and Resolving Redis Cluster Split-Brain & Sentinel Failover Race Conditions under PHP-FPM Object Cache Bursts

Introduction

Redis Sentinel is the standard high-availability mechanism for production Redis deployments — it automatically handles master election, failover notification, and client redirection when a master node goes down. But under one specific and deceptively common scenario — a burst of PHP-FPM object-cache reads during a Sentinel-triggered failover — your WordPress or PHP application can enter a catastrophic feedback loop: stale data served, READONLY errors cascading across hundreds of FPM workers, and the dreaded cache stampede that overwhelms your database just when your Redis layer is most needed.

This article dissects the full failure chain from first symptoms to verified resolution, providing real terminal commands, annotated log excerpts, and before/after configuration diffs.

The Failure Anatomy: What Actually Happens

Understanding the sequence of events is critical before touching any configuration.

[T+0]    Network blip isolates Redis master (172.16.1.10:6379) from two Sentinels
[T+2s]   Sentinels mark master +sdown (subjective down)
[T+5s]   Quorum met → +odown → elect new master (172.16.1.11:6379)
[T+5s]   Old master still accepts writes from PHP-FPM workers (split-brain window)
[T+6s]   Sentinel broadcasts switch-master event
[T+6s]   PHP-FPM workers using persistent connections to OLD master → READONLY errors
[T+6s]   WordPress Object Cache plugin retries → falls through to MySQL
[T+6s]   200 simultaneous cache misses → 200 simultaneous MySQL queries (stampede)
[T+8s]   MySQL max_connections exhausted → WordPress shows "Error establishing a database connection"
[T+10s]  Network recovers; old master rejoins as replica → data written in split-brain window is LOST

This cascading failure is the root cause of most “Redis made things worse during an outage” incidents.

Phase 1: Identifying the Split-Brain Window

Step 1.1 — Check Sentinel Logs for Conflicting Leadership

SSH into each Sentinel node and examine recent activity:

# On sentinel-1 (172.16.1.20)
grep -E "(switch-master|sdown|odown|\+elected)" /var/log/redis/sentinel.log | tail -50

# On sentinel-2 (172.16.1.21)
grep -E "(switch-master|sdown|odown|\+elected)" /var/log/redis/sentinel.log | tail -50

Pathological output indicating split-brain:

1725012005.123 [26310] [SENTINEL] +sdown master mymaster 172.16.1.10 6379
1725012007.441 [26310] [SENTINEL] +odown master mymaster 172.16.1.10 6379 #quorum 2/2
1725012007.442 [26310] [SENTINEL] +elect-leader master mymaster 172.16.1.10 6379
1725012007.900 [26310] [SENTINEL] +switch-master mymaster 172.16.1.10 6379 172.16.1.11 6379
# --- Meanwhile on the isolated old master ---
1725012006.001 [6379] * Connection from 172.16.0.5:48201 accepted  ← PHP-FPM still writing!
1725012006.003 [6379] * SET wp_options_alloptions ...

The gap between +sdown at T+2s and switch-master at T+6s is the split-brain window — during this time the old master still accepts writes from any client that hasn’t received the redirect.

Step 1.2 — Confirm Quorum Configuration

redis-cli -p 26379 SENTINEL masters

Look specifically at the quorum field:

 1) "name"
 2) "mymaster"
 3) "ip"
 4) "172.16.1.10"
 ...
27) "quorum"
28) "1"          ← DANGER: quorum of 1 on a 2-sentinel setup = no real consensus

A quorum of 1 with only 2 Sentinels means a single Sentinel can trigger a failover unilaterally — a classic misconfiguration that guarantees split-brain during any network partition.

Step 1.3 — Inspect the PHP-FPM Error Landscape

# Grep PHP-FPM slow log and error log for Redis errors
grep -E "(READONLY|Connection refused|Redis server went away|could not connect)" \
  /var/log/php-fpm/www-error.log | \
  awk '{print $1}' | sort | uniq -c | sort -rn | head -20

Typical output during a failover burst:

    847 [11-Sep-2026]  # 847 errors in one minute
    ...
# Count unique Redis error types
grep "Redis" /var/log/php-fpm/www-error.log | grep -oP "(?<=Redis: ).*?(?=\s)" | sort | uniq -c
    391 READONLY
    312 Connection refused to 172.16.1.10:6379
    144 LOADING Redis is loading the dataset in memory

The LOADING errors reveal a second problem: after failover, the promoted replica is still loading its RDB snapshot — but PHP-FPM is already hammering it.

Phase 2: Diagnosing the PHP-FPM Cache Stampede

Step 2.1 — Correlate FPM Worker Saturation with Redis Errors

# Watch PHP-FPM pool status in real-time
watch -n1 "curl -s http://127.0.0.1/fpm-status?full | grep -E '(active|idle|max active|requests)'"

During a stampede:

pool:                 www
active processes:     200    ← all workers busy
idle processes:       0
max active processes: 200
max children reached: 14     ← FPM hit the ceiling 14 times

Step 2.2 — Identify Cache Miss Rate via Redis INFO

# Before failover (baseline)
redis-cli -h 172.16.1.11 INFO stats | grep -E "keyspace_(hits|misses)"

# During stampede
redis-cli -h 172.16.1.11 INFO stats | grep -E "keyspace_(hits|misses)"

Before (healthy):

keyspace_hits:98234123
keyspace_misses:412309
# hit rate ≈ 99.58%

During stampede:

keyspace_hits:98234891
keyspace_misses:6109820
# hit rate collapsed to ≈ 94.1% — millions of misses in seconds

Step 2.3 — Confirm Database Connection Exhaustion

mysql -u root -p -e "SHOW GLOBAL STATUS LIKE 'Max_used_connections';"
mysql -u root -p -e "SHOW GLOBAL STATUS LIKE 'Connection_errors_max_connections';"
+------------------------+-------+
| Variable_name          | Value |
+------------------------+-------+
| Max_used_connections   | 312   |  ← far exceeds max_connections (150)
| Connection_errors_max_connections | 187 |
+------------------------+-------+

This confirms the classic stampede kill chain: Redis misses → MySQL overload → application errors.

Phase 3: Root Cause Analysis Summary

Layer Root Cause Impact
Sentinel Quorum set to 1 on a 2-node setup Unilateral failover, no consensus protection
Sentinel down-after-milliseconds too low (500ms) False positives on transient network jitter
Redis min-slaves-to-write 0 on master Accepts writes with zero replicas acknowledged
PHP (phpredis) Persistent connections with no reconnect-on-failure Workers cling to dead/demoted master
WordPress Object cache with no stampede protection All misses hit DB simultaneously
MySQL max_connections 150 Saturated under stampede load

Phase 4: Concrete Fixes — Before/After Configurations

Fix 4.1 — Correct Sentinel Quorum and Timing

File: /etc/redis/sentinel.conf (all Sentinel nodes)

Before:

sentinel monitor mymaster 172.16.1.10 6379 1
sentinel down-after-milliseconds mymaster 500
sentinel failover-timeout mymaster 10000
sentinel parallel-syncs mymaster 1

After:

# Always use a majority quorum: (N_sentinels / 2) + 1
sentinel monitor mymaster 172.16.1.10 6379 2

# 5 seconds — tolerates transient network jitter without false positives
sentinel down-after-milliseconds mymaster 5000

# Allow 60s for a full replica to sync before marking failover complete
sentinel failover-timeout mymaster 60000

# Sync replicas one at a time to avoid overwhelming the new master
sentinel parallel-syncs mymaster 1

# Notify on failover so external systems can react
sentinel notification-script mymaster /opt/redis/failover-notify.sh

# Prevent the old master from re-joining as a replica before it's safe
sentinel deny-scripts-reconfig yes

Apply without restart:

redis-cli -p 26379 SENTINEL SET mymaster down-after-milliseconds 5000
redis-cli -p 26379 SENTINEL SET mymaster failover-timeout 60000
# Repeat on each sentinel node

Fix 4.2 — Enforce Minimum Replica Acknowledgement on Master

File: /etc/redis/redis.conf on the master

Before:

min-replicas-to-write 0
min-replicas-max-lag 10

After:

# Refuse writes if no replica has acknowledged within 10 seconds
# This bounds the split-brain write window
min-replicas-to-write 1
min-replicas-max-lag 10

This causes the old master to stop accepting writes within 10 seconds of losing replica connectivity — drastically reducing the split-brain window from “indefinite” to ≤10 seconds.

# Apply live without restart
redis-cli CONFIG SET min-replicas-to-write 1
redis-cli CONFIG SET min-replicas-max-lag 10

Fix 4.3 — Configure PHP (phpredis) for Sentinel-Aware Connections

If you’re using phpredis directly (common in WordPress via WP Redis or Redis Object Cache plugin), replace direct IP connection with Sentinel-aware configuration.

File: /var/www/html/wp-content/object-cache.php or wp-config.php

Before (hardcoded master IP — catastrophic during failover):

// wp-config.php
define('WP_REDIS_HOST', '172.16.1.10');
define('WP_REDIS_PORT', 6379);

After (Sentinel-aware, with reconnect protection):

// wp-config.php
define('WP_REDIS_SENTINEL_MASTER', 'mymaster');
define('WP_REDIS_SENTINELS', [
    'tcp://172.16.1.20:26379',
    'tcp://172.16.1.21:26379',
    'tcp://172.16.1.22:26379',
]);

// Aggressive retry settings to survive the failover window
define('WP_REDIS_TIMEOUT', 1.0);          // 1s connect timeout
define('WP_REDIS_READ_TIMEOUT', 2.0);     // 2s read timeout
define('WP_REDIS_RETRY_INTERVAL', 100);   // 100ms between retries
define('WP_REDIS_MAX_RETRIES', 3);

// Disable persistent connections during failover periods
// (re-enable once stable, phpredis persistent connections skip Sentinel re-discovery)
define('WP_REDIS_PERSISTENT', false);

// Graceful degradation: if Redis is unavailable, serve from DB without error
define('WP_REDIS_GRACEFUL', true);

Fix 4.4 — Implement Distributed Mutex to Prevent Cache Stampede

Even with Sentinel configured correctly, the 2–5 second failover window will cause cache misses. Without a mutex, every PHP-FPM worker that hits a miss will independently query MySQL. The fix is a probabilistic early expiration lock (PER lock) using Redis atomic operations.

Add this to your WordPress object-cache.php or a custom must-use plugin:

<?php
/**
 * Anti-stampede wrapper using Redis SETNX mutex + stale-while-revalidate
 * Drop this in /wp-content/mu-plugins/anti-stampede-cache.php
 */
function get_cache_with_lock( string $key, callable $compute_fn, int $ttl = 3600 ): mixed {
    global $wp_object_cache;

    // Attempt to get current value
    $cached = wp_cache_get( $key, 'stampede_safe' );

    if ( false !== $cached ) {
        // Check if we're in the "pre-expiry revalidation" window (last 10% of TTL)
        $lock_key = 'lock_' . $key;
        $early_refresh = wp_cache_get( $key . '_expiry', 'stampede_safe' );
        if ( $early_refresh && $early_refresh - time() < ( $ttl * 0.1 ) ) {
            // Only one worker should refresh — try to acquire lock
            if ( wp_cache_add( $lock_key, 1, 'stampede_safe', 30 ) ) {
                // This worker won the lock — refresh in the background
                $fresh = call_user_func( $compute_fn );
                wp_cache_set( $key, $fresh, 'stampede_safe', $ttl );
                wp_cache_set( $key . '_expiry', time() + $ttl, 'stampede_safe', $ttl );
                wp_cache_delete( $lock_key, 'stampede_safe' );
                return $fresh;
            }
        }
        return $cached; // Serve stale — another worker is refreshing
    }

    // Full cache miss: acquire exclusive lock
    $lock_key = 'lock_' . $key;
    if ( ! wp_cache_add( $lock_key, 1, 'stampede_safe', 30 ) ) {
        // Another worker is computing — wait briefly and return stale or null
        usleep( 150000 ); // 150ms back-off
        return wp_cache_get( $key, 'stampede_safe' ) ?: null;
    }

    // We hold the lock — compute the value
    $value = call_user_func( $compute_fn );
    wp_cache_set( $key, $value, 'stampede_safe', $ttl );
    wp_cache_set( $key . '_expiry', time() + $ttl, 'stampede_safe', $ttl );
    wp_cache_delete( $lock_key, 'stampede_safe' );

    return $value;
}

For the native phpredis SETNX approach at the Redis level:

# Manually verify atomic lock works from CLI
redis-cli SET lock_wp_options_alloptions 1 NX PX 30000
# Returns: OK (lock acquired)
redis-cli SET lock_wp_options_alloptions 1 NX PX 30000
# Returns: (nil) (lock already held — second worker correctly blocked)

Fix 4.5 — MySQL Headroom & Connection Pooling

File: /etc/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf

Before:

[mysqld]
max_connections = 150
wait_timeout    = 28800

After:

[mysqld]
max_connections       = 500     # headroom for stampede fallthrough
wait_timeout          = 300     # reclaim idle connections faster
interactive_timeout   = 300
thread_cache_size     = 32      # reduce thread spawn overhead
max_connect_errors    = 100000  # don't block IP after connection bursts

Additionally, deploy ProxySQL or MaxScale as a connection pool in front of MySQL to soft-limit and queue burst connections rather than hard-refusing them:

# Install ProxySQL on cPanel-compatible server
wget https://github.com/sysown/proxysql/releases/download/v2.6.3/proxysql_2.6.3-debian12_amd64.deb
dpkg -i proxysql_2.6.3-debian12_amd64.deb

# Configure basic pooling
mysql -u admin -padmin -h 127.0.0.1 -P 6032 <<'EOF'
INSERT INTO mysql_servers(hostgroup_id, hostname, port) VALUES (0, '127.0.0.1', 3306);
UPDATE global_variables SET variable_value='150' WHERE variable_name='mysql-max_connections';
LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
EOF

Phase 5: Verification Commands — Confirming Resolution

5.1 — Verify Sentinel Quorum and Master Discovery

# All sentinels should agree on the same master
for ip in 172.16.1.20 172.16.1.21 172.16.1.22; do
  echo "=== Sentinel $ip ===";
  redis-cli -h $ip -p 26379 SENTINEL get-master-addr-by-name mymaster;
done

Expected (all pointing to same master):

=== Sentinel 172.16.1.20 ===
1) "172.16.1.11"
2) "6379"
=== Sentinel 172.16.1.21 ===
1) "172.16.1.11"
2) "6379"
=== Sentinel 172.16.1.22 ===
1) "172.16.1.11"
2) "6379"

5.2 — Simulate a Failover and Measure Recovery Time

# Controlled failover test — forces a failover on mymaster
redis-cli -p 26379 SENTINEL failover mymaster

# Watch the election in real-time
tail -f /var/log/redis/sentinel.log | grep -E "(switch-master|elected|odown)"

Measure the time between +odown and switch-master — with the corrected config it should be under 15 seconds total (dominated by down-after-milliseconds of 5s + sync time).

5.3 — Verify PHP-FPM Reconnects to New Master

# While failover is in progress, watch object cache behavior
watch -n0.5 "redis-cli -h 172.16.1.11 INFO stats | grep keyspace"

After failover completes, keyspace_hits should resume climbing (cache is warm on the new master if RDB was replicated).

5.4 — Confirm Split-Brain Write Protection

# On old master (now replica): verify it refuses writes
redis-cli -h 172.16.1.10 -p 6379 SET test_key test_value
# Expected: (error) READONLY You can't write against a read only replica.

# Verify min-replicas-to-write is enforced
redis-cli -h 172.16.1.10 CONFIG GET min-replicas-to-write
# Expected: 1

5.5 — Validate Anti-Stampede Lock from CLI

# Simulate concurrent cache misses: 50 parallel SET NX attempts
for i in $(seq 1 50); do
  redis-cli SET lock_test_key 1 NX PX 5000 &
done; wait

# Only 1 should have returned "OK"
redis-cli DEL lock_test_key

5.6 — Monitor Redis Hit Rate Over 24h

# Quick hit-rate calculator
redis-cli INFO stats | awk '
  /keyspace_hits/ { hits=$2 }
  /keyspace_misses/ { misses=$2 }
  END { printf "Hit rate: %.2f%%\n", (hits/(hits+misses))*100 }
'

Target: >98% hit rate during steady state, with graceful fallthrough (not hard error) during any future failover.

Phase 6: Operational Hardening — Monitoring & Alerting

Add the following to your Prometheus + Alertmanager stack (or Netdata/Zabbix triggers):

# prometheus/alerts/redis.yml
groups:
  - name: redis_sentinel
    rules:
      - alert: RedisSentinelMasterDisagreement
        expr: count(redis_sentinel_master_address) by (master_name) > 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Split-brain detected — Sentinels disagree on master"

      - alert: RedisCacheHitRateLow
        expr: |
          rate(redis_keyspace_hits_total[5m]) /
          (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m])) < 0.95
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Redis hit rate below 95% — possible stampede in progress"

      - alert: RedisSentinelQuorumLost
        expr: redis_sentinel_sentinels < 2
        for: 10s
        labels:
          severity: critical
        annotations:
          summary: "Sentinel quorum lost — failover is blind"

Infrastructure Note

Businesses running high-traffic WordPress or Linux workloads benefit enormously from low-latency Redis connectivity — the physics of TCP round-trip time directly add to your down-after-milliseconds window. On a NVMe Cloud VPS Pakistan or Dedicated Server Pakistan, Redis master-to-sentinel latency stays under 0.5ms on the same local fabric, shrinking the split-brain window to its theoretical minimum. Pair that with cPanel Web Hosting Pakistan for managed server environments where the Sentinel topology can be pre-configured and monitored at the infrastructure level.

Summary Checklist

Task Command / File Status
Set quorum ≥ 2 sentinel.conf → quorum 2 ✅
Raise down-after-milliseconds to 5000 sentinel.conf ✅
Set min-replicas-to-write 1 redis.conf ✅
Switch to Sentinel-aware PHP client wp-config.php ✅
Implement SETNX mutex / stale-while-revalidate mu-plugins/ ✅
Add TTL jitter to cache keys Application layer ✅
Raise MySQL max_connections + add ProxySQL my.cnf ✅
Deploy Prometheus alerts for split-brain redis.yml ✅
Simulate failover & measure RTO SENTINEL failover ✅

Properly architected, a Redis Sentinel cluster should recover from a master failure in under 15 seconds with zero data loss (bounded by replica lag) and zero stampede — because your PHP-FPM workers will gracefully wait behind the mutex rather than detonating your database. The key insight is that split-brain and stampede are not Redis bugs — they are emergent properties of under-configured clients interacting with a distributed consensus protocol that requires careful tuning at every layer of the stack.