The scaling answer I hear most often when coaching engineers through system design interviews is:

We'd add more servers to handle the load.

More servers than what? Handling the load while meeting what? Without a declared acceptance criterion, "add more servers" is underspecified. Engineers can still evaluate a change against an observed workload or before-and-after measurements, but the tactic alone does not say what outcome would count as an improvement.

For an online request-serving system, a capacity benchmark measures how much representative traffic a configuration can handle within defined performance and correctness limits. A deployment-sizing recommendation applies that benchmark to a workload and fault model. A complete scaling plan goes further, covering concerns such as state growth, durability, consistency, dependencies, deployment behavior, and recovery.

This article works through the narrower deployment-sizing problem. Latency, throughput, availability, fault model, and cost provide a useful baseline for that example. They are neither necessary for every isolated scaling claim nor sufficient for every production plan.

Establish enough targets before defending a sizing claim

A deployment-sizing recommendation for this example becomes evaluable when you establish five things:

• Latency: State the request population, measurement boundary, evaluation window, and a user-facing percentile target such as p99. For Little's Law, separately state the achieved or assumed mean application-tier residence time over the calculation boundary. A p99 SLO describes the slow tail; Little's Law operates on the mean.
• Throughput: Define the offered load the system must sustain and for how long, separated by request class when operations have materially different resource profiles. Track completed throughput and successful goodput separately so fast rejection does not look like capacity.
• Availability: Define which attempts enter the denominator and which outcomes count as successful over a specific window. A tighter availability target changes the redundancy, deployment, and recovery strategy.
• Fault model: State which failures the system must continue operating through. "99.99% available" and "survives the loss of one availability zone" are related requirements, but they are not the same requirement.
• Cost: Set a monthly budget or a cost per thousand externally observed attempts or successful operations, with the included infrastructure clearly scoped.

Correctness requirements belong beside these targets when they change the work being measured. A write acknowledged before durable commit is not equivalent to one acknowledged after commit, and different consistency guarantees can produce different capacity results. The worked example therefore defines completion semantics for each request class rather than relying on the phrase "representative work."

These targets, measurements, and architecture should be revised together as evidence arrives. The constraint is that a sizing recommendation should not be defended before its workload, acceptance criteria, correctness semantics, and fault assumptions are clear enough to evaluate.

From 10 million users to peak RPS

Candidates often restate the user count and stop there. But daily active users cannot directly tell us concurrency, machine count, or cost.

The useful chain is:

Daily active users → external eligible attempts per user → daily external eligible attempts → class-specific application work → average RPS → peak RPS → rounded design load

Traffic-volume and peak-factor assumptions produce the peak RPS. Little's Law then uses an application-tier arrival rate and mean residence time to estimate average in-flight occupancy.

Every link is an assumption the interviewer can challenge. Back-of-the-envelope estimates will be imprecise either way. The goal is not to create an accurate production forecast from fictional inputs. It is to turn vague architectural claims into assumptions that can be inspected, changed, and recalculated.

Production-validation assumptions and worked calculation

All values below are assumptions, not external benchmarks. In a real system, capacity would come from representative load tests or an existing service profile, and cost would come from the actual deployment.

Targets and acceptance rules

• Latency SLO: p99 at or below 800 milliseconds for successful eligible attempts over a rolling 30-day window. Measure from an external point immediately before service-controlled routing and edge infrastructure until the response is complete. Report the target both in aggregate and for each materially distinct request class.
• Mean assumption: 300 milliseconds of mean application-tier residence time over the Little's Law boundary defined below.
• Throughput: Sustain 4,000 external eligible attempts per second for one uninterrupted hour under the modeled healthy delivery path.
• Peak-test acceptance: Over the full hour, aggregate and per-class p99 remain at or below 800 milliseconds. Capacity useful-work goodput is at least 99.99% of offered load in aggregate and for each class. Completed throughput and useful-work goodput are reported separately.
• Availability: At least 99.99% of eligible attempts succeed over a rolling 30-day window.
• Service timeout: 2 seconds.
• Fault model: After traffic has been redistributed following the loss of one of three zones, retain nominal application-tier capacity for the rounded design load. This calculation sets no recovery-time or in-transition service guarantee.
• Application-compute budget: At most $0.001 per 1,000 offered external eligible attempts over a 30-day month, including zone-loss capacity but excluding databases and other supporting infrastructure.
• Fleet policy under evaluation: A fixed fleet with no autoscaling.

An eligible attempt is a syntactically valid request observed at the external measurement point. A contract-conforming 2xx or 3xx response, or an appropriate 4xx response for the request's authorization and resource state, counts as successful when completed within the timeout. Routing failures, 429 responses, inappropriate 4xx responses, 5xx responses, and timeouts count as service failures. The worked load contains no cancellations and expects no 4xx responses. In operational measurement, an attempt observed at the external boundary remains in the eligible denominator unless independent evidence establishes a genuinely client-originated cancellation or another explicitly defined exclusion. Cancellation before the service timeout is not sufficient evidence because service latency may have caused it. A cancellation that remains eligible and does not produce a qualifying response within the timeout counts as an availability failure and a useful-work-goodput miss, even though the latency SLO is reported for successful attempts.

For the capacity test, useful-work goodput counts an attempt only when it completes within 2 seconds, performs the class-specific work defined below, and returns its prescribed outcome. Late completions and fast rejections do not enter the numerator.

Traffic assumptions and arrival shape

• Daily active users: 10,000,000
• External eligible attempts per user per day: 17
• Peak-to-average factor: 2
• Rounded peak design load: 4,000 RPS, a conservative rounding of the exact 2× result
• Peak window: one uninterrupted hour
• Request mix: 90% lightweight reads, 8% writes, and 2% expensive operations
• Remaining daily traffic: the balance of the daily volume spread across the other 23 hours

Client retries that cross the external measurement point are already included in the 17 eligible attempts per user per day. They must not receive another retry multiplier. Only service-controlled retries, duplicate dispatches, or redispatches after that boundary can amplify application-tier arrivals. The model holds the client-retry policy and the retry share of those 17 attempts fixed. Each test must record that share; a change in the observed client-retry rate is a workload change that requires recalculation.

The load generator uses an open-loop schedule with aggregate arrivals evenly spaced at 250-microsecond intervals. Each one-second interval contains 3,600 reads, 320 writes, and 80 expensive operations, evenly interleaved. The test introduces no additional subsecond bursts, client retry amplification, or service-controlled redispatch. A different burst distribution would require a different test.

Class-specific completion semantics

Every offered test request is valid and authorized. For this example:

• A lightweight read completes only when it returns a contract-conforming value within the timeout.
• A write completes only after the mutation is durably committed to the modeled system of record and the service returns the prescribed response. Acknowledgement before durable commit does not count.
• An expensive operation completes only after all synchronous downstream work represented in the benchmark finishes and the prescribed response is returned.

Real services must replace these assumptions with their own durability, consistency, and completion rules. If those rules change the work performed before acknowledgement, the capacity test must change with them.

Map external attempts to application work by class and cause

A single external-to-application conversion factor is unsafe because different causes have different operational meanings. For each request class, account separately for:

• Legitimate edge completion: An attempt successfully completed at the edge may be removed from application-tier work only if the edge response satisfies that class's correctness semantics.
• Batching: Several external attempts may become one application work item, but the model must retain how many attempts and which classes that item represents.
• Service-controlled duplication or redispatch: Each additional application dispatch increases the corresponding class's application-tier arrival rate.
• Failed routing or delivery: The attempt remains a failed eligible attempt. A service-controlled failure must never be used to lower required design capacity.

This example assumes no edge completion, batching, failed delivery, or post-boundary amplification. Each external attempt produces one application-tier arrival of the same class. The modeled application-tier rates are therefore 3,600 read RPS, 320 write RPS, and 80 expensive-operation RPS, for 4,000 RPS in total. A real model must preserve this class composition rather than replacing it with one fleet-wide scalar.

Derive the design load

Daily external eligible attempts
  = 10,000,000 × 17
  = 170,000,000

Average external eligible-attempt RPS
  = 170,000,000 / 86,400 seconds
  ≈ 1,967.6 RPS

Exact 2× peak RPS
  = 1,967.6 × 2
  ≈ 3,935.2 RPS

Rounded peak design load
  = 4,000 RPS

Effective peak-to-average factor
  = 4,000 / 1,967.6
  ≈ 2.03

Under the modeled one-to-one, healthy delivery path, the rounded application-tier design load is also 4,000 RPS.

Define the Little's Law boundary

For this calculation:

• λ is all application-tier arrivals entering service instances.
• W is the mean time from instance entry until final termination of the corresponding application-owned work and release of the relevant application-tier resources.
• L is work from that same entrant population still resident within the application-tier boundary.

A terminal departure may be response handoff, completed rejection, cancellation cleanup, timeout cleanup, or another final release of application-owned work. If work continues after a client disconnect or timeout, its residence time continues until that work terminates. The mean covers the full entrant population, including failed and cancelled requests.

Average application-tier occupancy ≈ λ × W

At a 300 ms mean:
  4,000 × 0.30 seconds = 1,200 resident application work items

At a 200 ms mean over the same boundary:
  4,000 × 0.20 seconds = 800 resident application work items

Aggregate testing must measure the mean under the proposed fleet configuration and dependency load before either figure is treated as expected deployed-fleet occupancy.

Define the per-instance planning cap

Configuration A is one container limited to 4 vCPU and 8 GiB of memory. Its runtime build, process topology, worker settings, and concurrency limits are fixed for the model. Its effective price in the model is $0.50 per running instance-hour, without discounts or a separate idle rate. The price excludes supporting infrastructure, storage, and data transfer.

For this worked arithmetic, 350 RPS is the assumed planning input for Configuration A. After a 10-minute warm-up excluded from measurement, the test runs for one uninterrupted hour. Provisional resource and headroom acceptance thresholds for that hour are mean CPU utilization at or below 70% of the 4-vCPU limit; a maximum five-minute rolling mean resident memory at or below 6 GiB; a maximum one-minute rolling mean utilization at or below 80% for each bounded worker or connection pool; and, for each application-owned queue, a linear trend in one-minute mean depth no greater than one item per minute, with the final 10-minute mean no more than 5% or one item, whichever is greater, above the first 10-minute mean. At 350 RPS and the stated request mix, assume the configuration meets those thresholds and the aggregate and per-class p99 and useful-work-goodput criteria, with the same external boundary and open-loop pacing method scaled to an aggregate 350 RPS: 315 reads, 28 writes, and 7 expensive operations per second, with an average aggregate interval of about 2.857 milliseconds. Under this assumption, the run also produces the 300-millisecond mean application-tier residence time.

The 350-RPS input becomes a validated planning cap only after a representative per-instance test passes those conditions and aggregate fleet tests support the extrapolation. For each fleet size and zone-loss state used in the sizing claim, each healthy instance's mean offered rate in every non-overlapping five-minute interval must remain within 5% of the fleet-wide per-instance mean. The fleet must sustain N × 350 offered RPS while meeting the same latency, useful-work-goodput, correctness, and resource criteria, and a load sweep must show that the maximum accepted aggregate offered rate is no more than 5% below N times the maximum accepted single-instance rate. Shared dependencies must receive the full aggregate fan-out during these tests; dependency throttling or interference that causes either fleet criterion to fail invalidates the extrapolation. The tests also assume homogeneous instances. If any condition fails, the planning cap must be reduced or the fleet model revised before the 12- and 18-instance results are treated as validated capacity results.

Calculate the capacity and zone-loss candidate

Minimum application-tier count at peak
  = 4,000 / 350
  ≈ 11.4
  → 12 instances

For an even three-zone distribution:
  N × 2/3 ≥ 12
  N ≥ 18

Eighteen instances are therefore the minimum capacity-and-zone-loss candidate under these assumptions. This is a post-redistribution capacity result that assumes successful traffic redistribution, ready and active survivors, 350 RPS of retained capacity per survivor, and sufficient downstream capacity.

Evaluate application-compute cost

The budget measures application-compute cost per offered external eligible attempt under the stated attempt semantics, 90/8/2 class mix, and client-retry policy. It is not an inherent measure of cost per useful operation. Application-tier arrivals are not the denominator because service-controlled retries or duplicate dispatches could increase that denominator and make cost appear to improve without delivering more useful service. Comparisons using external eligible attempts are valid only while the attempt semantics and client-retry policy remain fixed, the observed client-retry rate matches the modeled baseline, and the workload retains the stated class mix. Report the retry rate and class mix with each result and treat a change in either as a workload change requiring recalculation. Measuring cost per useful operation would instead require a denominator based on deduplicated logical operations or successful operations. The arithmetic below preserves the stated offered-attempt budget.

Capacity-only baseline at the peak hour
  12 × $0.50/hour = $6.00/hour

External eligible attempts during the peak hour
  4,000 × 3,600 = 14,400,000

Application-compute contribution
  $6.00 / 14,400 thousand-attempt units
  ≈ $0.0004 per 1,000 external eligible attempts

Zone-loss-capacity candidate at the peak hour
  18 × $0.50/hour = $9.00/hour

Application-compute contribution
  $9.00 / 14,400 thousand-attempt units
  ≈ $0.0006 per 1,000 external eligible attempts

Monthly candidate instance-hours
  18 × 24 × 30 = 12,960 instance-hours

Monthly application-compute spend
  12,960 × $0.50 = $6,480

Monthly external eligible attempts
  170,000,000 × 30 = 5,100,000,000

Monthly application-compute contribution
  $6,480 / 5,100,000 thousand-attempt units
  ≈ $0.00127 per 1,000 external eligible attempts

Monthly application-compute budget
  = 5,100,000 thousand-attempt units × $0.001
  = $5,100

Monthly cost per always-on instance
  = 24 × 30 × $0.50
  = $360

Maximum whole instances under the application-compute budget
  = floor($5,100 / $360)
  = 14 instances

Affordable near-even three-zone placement
  = 5 / 5 / 4 instances

Survivors after loss of a five-instance zone
  = 14 - 5
  = 9 instances

Required planning cap per survivor
  = 4,000 / 9
  ≈ 444.4 RPS

Largest affordable even three-zone fleet
  = 12 instances

Survivors after one-zone loss
  = 12 × 2/3
  = 8 instances

Required planning cap per survivor
  = 4,000 / 8
  = 500 RPS

Maximum effective price for 18 always-on instances
  = $5,100 / (18 × 24 × 30)
  ≈ $0.3935 per instance-hour

Because the monthly result exceeds the $0.001 target, the fixed 18-instance candidate is not acceptable under the stipulated 350-RPS planning cap. Under a near-even 5/5/4 placement, the affordable 14-instance fleet would leave nine survivors after losing either five-instance zone, and each survivor would need a validated planning cap of approximately 444.4 RPS. If policy instead requires equal instance counts in every zone, the largest affordable fleet is 12 instances, and its eight survivors would each need a validated planning cap of at least 500 RPS. Alternatively, retaining 18 always-on instances requires an effective price no higher than about $0.3935 per instance-hour.

These thresholds identify the capacity or price changes that could make the capacity-and-cost arithmetic feasible.

What each result actually tells you

The design load defines the throughput test

The rounded 4,000-RPS load is tied to a specific one-hour traffic shape, class mix, and pacing rule. A fleet provisioned only for the daily average will run short during that hour if traffic follows the assumed shape.

The important discussion is whether 17 attempts per user, the 2× peak factor, the one-hour window, and the class-specific mappings are reasonable. A chat product, live event, enterprise batch workflow, or globally distributed service could produce a different pattern.

Four thousand cached reads and four thousand database-heavy writes are not equivalent workloads. Aggregate p99 is also insufficient when a minority class can be slow without moving the fleet-wide percentile. That is why the test applies latency and useful-work-goodput criteria to each material class as well as to the aggregate.

Little's Law estimates application-tier occupancy, not machine count

The occupancy estimates may help identify pressure on application memory, connection pools, worker slots, and queues associated with unfinished application-tier work. They are not acceptance tests against an application-tier resource ceiling.

They also do not establish concurrency at a downstream dependency. Fan-out can give each dependency a different arrival rate, and its occupancy requires its own arrival rate and residence time over that dependency's boundary.

A lower mean application-tier residence time may result from doing less work per request, adding machines, buying faster hardware, adding a cache, increasing database capacity, or reserving more idle headroom. Whether the fleet can shrink depends on the actual bottleneck.

Machine count depends on representative throughput

A throughput-based instance count is more defensible than saying that an instance can "hold 100 concurrent requests." An asynchronous service might hold thousands of waiting requests without consuming much CPU, while another service might saturate a core with only a handful of expensive operations.

The relevant question is:

How much representative traffic can this configuration sustain while meeting the latency, useful-work, correctness, and resource requirements?

Availability requires more than nominal redundancy

Zone loss is only one failure mode. Bad deployments, dependency outages, database failures, configuration mistakes, capacity exhaustion, slow recovery, and detection or routing delays can all consume the availability budget. The availability SLO describes the required outcome; the fault model describes one class of event the architecture must address.

Cost changes the fleet decision

The candidate might propose autoscaling to a smaller zone-loss-capable fleet during lower traffic, improving per-instance resource utilization, reducing always-on capacity, accepting a higher application-compute target, or revisiting the availability and fault-model requirements. Each option changes a stated target or assumption and must be evaluated on that basis.

Where the estimate stops

Little's Law is a steady-state identity about averages. During a sudden spike, requests may arrive faster than the system can complete them, causing queues and latency to grow. Burst handling may require additional headroom, queueing analysis, admission control, load shedding, or a higher application-tier arrival rate.

Representative production validation of the assumed 350-RPS cap would require a load sweep, repeated runs or an appropriate soak period, controlled warm-up, examination of run-to-run variance, checks for resource leaks and periodic work, and a stated headroom policy.

The zone-loss arithmetic covers capacity after redistribution. Evaluating rolling 30-day availability also requires evidence about expected failure frequency, detection, interruption, redistribution or replacement, and recovery behavior against the stated availability target.

The model shows which results depend on each input. If the peak factor changes, edge behavior alters the request-class mix, service-controlled retries amplify dispatches, writes prove more expensive than reads, or fleet testing reveals a shared bottleneck, only the dependent capacity and cost results need to be revised.

In an interview, this gives the candidate and interviewer a specific way to identify which assumption or measurement needs work. Here, the current inputs do not satisfy the capacity and cost constraints together. Changing a target, fleet policy, price, workload assumption, or measured capacity can make that arithmetic feasible.