Why peaks fail
Most commerce platforms handle normal traffic well. The trouble starts on the few days a year when traffic jumps several times over in minutes: a holiday weekend, a product drop, a campaign that works better than expected. Revenue on those days can be a large share of the year, and an outage or a slow checkout costs more than on any other day.
Peaks rarely fail because the cloud runs out of capacity. They fail on limits nobody checked: a service quota, a database connection pool, a third-party payment or tax API, a cache that was cold, a scaling policy that reacts in five minutes when the traffic arrived in one. Each of those can be found and fixed in advance. That is what peak readiness is.
A 12-week timeline
Twelve weeks is comfortable; eight is workable; four means choosing only the highest risks. The order matters more than the exact dates.
| When | Focus | Output |
|---|---|---|
| Weeks 12–10 | Peak forecast, order path map, known limits and dependencies | One agreed target and a risk list |
| Weeks 9–6 | Load tests, quota increases, architecture fixes for the top risks | Test results against the target; fixes in progress |
| Weeks 5–3 | Second round of load tests, capacity reservations, runbooks, game day | Path proven at target plus margin |
| Weeks 2–0 | Change freeze, pre-warming, event staffing and dashboards | A team ready to run the event |
| After | Review what happened and scale capacity back down | Lessons and a cost back to normal |
Step 1: agree the number you are preparing for
"Be ready for Black Friday" is not a target. A target is a number engineering can test against: peak orders per minute, peak sessions, peak requests per second on the busiest endpoints, and the expected ratio between them.
- Start from last year's peak and the commercial plan for this year: growth, campaigns, new markets, new channels.
- Add a safety margin, commonly 1.5 to 2 times the forecast, because campaigns and press coverage are hard to predict.
- Get it signed off by the commercial owner. It turns a technical exercise into a business decision with an owner.
Step 2: map the order path and its limits
Draw the path a customer takes from landing page to order confirmation, including every service and third party involved. For each component, write down its known limit and how it scales.
| Component | Typical limit to check | How it usually fails |
|---|---|---|
| CDN and static content | Cache hit ratio for product and category pages | Origin overloaded by cache misses |
| Load balancers | Capacity scaling speed for sudden spikes | Errors in the first minutes of a spike |
| Compute | Auto Scaling speed, minimum capacity, account service quotas | New instances or tasks arrive after the spike |
| Databases | Connections, write throughput, read replica lag | Connection exhaustion; slow checkout queries |
| Search and catalog | Query throughput, index refresh | Slow search drives abandonment before checkout |
| Payments, tax, fraud and shipping APIs | Contracted rate limits and timeouts | Third-party throttling surfaces as checkout errors |
| Inventory and order systems | Write contention on popular items | Overselling or locked rows on hot products |
Check account-level limits early with Service Quotas and request increases weeks ahead; some take time to approve. Ask each third party for its limit in writing, and for its own peak plan.
Step 3: load test the whole path
Component tests are useful, but the failures that matter appear when everything runs together. Test the full order path, with realistic behavior: browsing, search, cart, checkout, in the proportions you expect on the day.
- Ramp, then spike. Test a gradual ramp to the target, and a sudden jump from normal to target within a minute or two.
- Hold at peak long enough to see slow problems: connection leaks, queue backlogs, growing latency.
- Use production-like data and a production-like environment, or test production itself in a quiet window with care.
- Stub or coordinate third parties. Payment providers usually have sandboxes with their own limits; agree test windows with them.
- Measure what the customer sees: checkout latency and error rate, not only CPU.
AWS publishes a Distributed Load Testing solution that can generate this kind of traffic from your own account. Whatever the tool, run the test twice: once to find the limits, and again after the fixes to prove them.
Step 4: reserve and pre-warm capacity
Autoscaling handles growth over minutes. A peak that arrives in seconds needs capacity already in place.
- Raise minimum capacity for compute ahead of known peaks with scheduled scaling, then lower it afterwards.
- Reserve load balancer capacity. Since November 2024, Application and Network Load Balancers support Load Balancer Capacity Unit Reservation, which sets a minimum capacity for an expected spike.
- Pre-warm DynamoDB tables. DynamoDB warm throughput lets you raise the throughput a table can handle instantly, so on-demand tables do not have to scale up during the spike.
- Warm the caches. Pre-load popular product, category and search results so the first wave of traffic does not hit the database.
- Size database connections. Use connection pooling, for example Amazon RDS Proxy, and check that the maximum fits the peak number of application instances.
Step 5: protect the order path
Even a well-tested platform should degrade gracefully if the forecast is wrong. Decide in advance what gives way first, so the order path keeps working.
| Technique | What it protects |
|---|---|
| Queue non-urgent work | Confirmation emails, loyalty updates and analytics go to a queue such as Amazon SQS instead of slowing checkout |
| Feature switches | Recommendations, reviews or personalization can be turned off to free capacity for checkout |
| Timeouts and fallbacks for third parties | A slow fraud or tax check falls back to an agreed default instead of blocking the order |
| Rate limits and bot protection | Scalpers and scrapers do not consume capacity meant for customers |
| Virtual waiting room | For extreme drops, customers queue politely instead of hitting errors |
Payments deserve special care. A checkout that takes the order but loses the payment confirmation creates reconciliation work for weeks. Make payment calls idempotent, record every attempt, and reconcile orders against processor settlements daily; our reconciliation guide covers the pattern.
Step 6: run the event
- Freeze changes one to two weeks before, with an emergency path for real fixes.
- Run a game day a few weeks before: simulate a component failure at peak load and practice the response.
- Prepare runbooks for the likely failures: who decides, what to switch off, how to scale, how to communicate.
- Staff the event with named people on duty, a shared dashboard of business and technical metrics, and contacts for every critical third party.
- Involve AWS early. AWS Support offers AWS Countdown for planned high-traffic events; check what your support plan includes.
- Review afterwards: what reached its limit, what was over-provisioned, what to change before the next peak.
The checklist on one page
Print it, assign an owner to each line and review it weekly from week twelve.
- Peak target agreed and signed off by the commercial owner, with margin.
- Order path mapped, with the limit and scaling behavior of every component.
- Service quota increases requested and confirmed.
- Third-party limits confirmed in writing; their peak plans received.
- Full-path load test passed at target plus margin, including a sudden spike.
- Minimum capacity scheduled; load balancer and database capacity reserved or pre-warmed; caches warmed.
- Non-urgent work queued; feature switches tested; third-party timeouts and fallbacks in place.
- Payment calls idempotent; reconciliation of orders against settlements running daily.
- Runbooks written; game day completed; on-call rota and contacts published.
- Change freeze dates agreed; scale-down dates set for every temporary increase.
Keeping cost under control
Peak readiness should not mean paying for peak capacity all year. A common cost problem in commerce is infrastructure that was scaled up for the holidays and never scaled down. Put a date on every temporary increase, scale back after the event, and compare the month after with the month before. Our note on why cloud cost is an architecture problem explains why scaling on real demand matters more than discounts.
Questions we hear
Is it too late to prepare for this year's peak?
Not if you narrow the scope. With four weeks, agree the target, load test the order path once, fix the top three risks, raise minimum capacity and prepare runbooks. Leave larger architecture changes for next year.
Should we test in production?
A production-like environment is safer and finds most problems. Some teams also run a carefully controlled test in production during a quiet window, because it is the only environment with real dependencies. Agree it with every third party first.
Can serverless remove the need for this work?
It removes some capacity planning, but not limits: account concurrency, downstream databases and third-party APIs still need checking and testing.
How much extra capacity is enough?
Enough to handle the signed-off target with margin, proven by a load test. Without a test, any number is a guess.