The Problem
On Black Friday 2024, a US-based WooCommerce store handling over $50,000 per hour in sales went completely down at 11:42 AM EST — peak traffic period. The site was returning a 500 error and the admin panel was unreachable. Every minute of downtime cost approximately $833.
Our Diagnosis
Within 4 minutes of the emergency ticket being raised, our engineer had SSH access and identified the root cause:
- MySQL had hit the
max_connectionslimit of 100 — all connections exhausted - PHP-FPM had spawned 240 worker processes (max_children set to 250) all waiting for MySQL
- A WooCommerce plugin was running an unoptimised ORDER BY query on the wp_postmeta table — no index — causing each query to take 8-12 seconds
- With 150+ concurrent users and 8-second queries, connections stacked up and MySQL collapsed
The Fix — Step by Step
Our engineer took the following actions in sequence:
- Increased
max_connectionsto 500 temporarily via MySQL global variable — this immediately allowed the site to start recovering - Identified and killed the 47 sleeping connections holding locks
- Added a composite index on
wp_postmeta (post_id, meta_key)— the offending query dropped from 8.2s to 0.04s - Reduced PHP-FPM max_children to 60 with pm.max_requests=500 to prevent memory exhaustion
- Enabled MySQL slow query log and set long_query_time=1 for ongoing monitoring
- Added Redis object caching to prevent repeat database hammering
Results
Full service was restored 18 minutes after the initial emergency ticket. The store processed a record number of Black Friday orders that afternoon. Post-incident, the server has maintained 99.9% uptime through two subsequent peak sale events with zero issues.