Day 3: Vertical vs Horizontal Scaling

Vertical vs Horizontal Scaling: Bigger Server or More Servers?

By the end of Day 3, you will be able to compare scaling up with scaling out, estimate a starting capacity, and name the trade-off that would guide your choice.

This lesson is part of Korshub's System Design in 30 Days series.

The mental model

There are two basic ways to give a system more compute capacity:

• Vertical scaling, or scaling up: make one resource larger. For an application server, that could mean more CPU or memory.

• Horizontal scaling, or scaling out: add more resource instances and distribute work across them.

Microsoft's Azure Architecture Center uses the same distinction: vertical scaling changes the capacity of a resource, while horizontal scaling adds or removes instances. The exact mechanics vary by service, but the mental model stays useful.

Think of a restaurant kitchen. Scale up by replacing one small oven with a larger oven. Scale out by adding more ovens and coordinating which cook uses each one.

Neither approach is universally better. The right choice depends on the bottleneck, availability target, workload behavior, cost, and operational complexity.

Vertical scaling: make one machine bigger

Vertical scaling changes the capacity of one machine or managed resource. For a server, you might move from 4 vCPU and 16 GB of memory to a larger instance.

Why teams often start here:

• It can require fewer application changes.

• There are fewer instances to deploy, monitor, and coordinate.

• Local state and in-memory data are easier to reason about on one machine.

What limits it:

• Every machine type has a finite ceiling.

• One machine can remain a single failure point.

• A resize may require a restart or temporary unavailability, depending on the platform and service.

• A larger machine may not fix a bottleneck in the database, network, lock contention, or inefficient code.

Vertical scaling is often a sensible early move when the product is young, traffic is predictable, and simplicity matters more than independent failure handling.

Horizontal scaling: add more machines

Horizontal scaling changes the number of instances. A common application-tier pattern places several similar servers behind a load balancer. The load balancer routes requests to healthy instances.

Why teams use it:

• Capacity can grow in smaller increments.

• Multiple healthy instances can reduce dependence on one application server.

• Instances can be added or removed as demand changes.

• A failed instance can be replaced while other healthy instances continue serving traffic, if the architecture is designed for it.

What it adds:

• Requests must be distributed correctly.

• Health checks and replacement behavior must be reliable.

• Session or application state may need a shared store, replication, or partitioning.

• Logs, traces, deployments, and debugging now span several instances.

• Shared dependencies such as a database can become the next bottleneck.

Horizontal scaling does not automatically create high availability. The load balancer, database, network, and deployment design must also avoid critical single points of failure.

Worked example: from 400 to 1,200 requests per second

Start with explicit assumptions:

• One 4-vCPU application server handles an estimated 400 requests per second in a representative load test.

• The target peak is 1,200 requests per second.

• For this teaching estimate, throughput is assumed to grow approximately with added application-server compute.

• The database and other shared services do not bottleneck first.

The capacity multiple is:

1,200 req/s / 400 req/s = 3

That gives two simplified options:

The result is a starting estimate, not a production promise. Three servers do not always provide exactly three times the throughput. Load balancing, synchronization, network calls, lock contention, the database, and uneven traffic can reduce the gain.

You would also add headroom. If the peak target is 1,200 req/s, designing for exactly 1,200 req/s leaves no room for traffic spikes, instance loss, deployments, or measurement error.

How to choose in a system design interview

Use this decision sequence:

1. Name the bottleneck. CPU, memory, storage, database capacity, or network?

2. State the target. Peak traffic, latency, and availability requirements.

3. Choose the simplest move that meets the target. Scaling up may be enough for an early system.

4. Check the ceiling and failure model. A single large machine may not meet availability requirements.

5. Account for distributed complexity. Scaling out may require stateless application servers, shared session storage, coordination, and stronger observability.

6. Validate with testing. Measure the new design under representative load and failures.

A strong answer does not say, "Horizontal scaling is always better." It explains what the system needs and why the extra complexity is justified.

Try this today

Assume one tested application server handles an estimated 400 req/s. Your new peak target is 1,600 req/s.

Answer these questions:

1. How many similar servers are the arithmetic starting point?

2. What headroom would you add?

3. Which shared dependency might bottleneck next?

4. Does the design still serve traffic if one application server fails?

Completed example you can copy

Assumptions: One instance handles 400 req/s in a representative test. Target peak traffic is 1,600 req/s. The database has enough capacity for this exercise.

Calculation: 1,600 / 400 = 4 instances before headroom.

Decision: Start with four similar application instances behind a load balancer, then add capacity headroom based on the failure and traffic-spike target. Keep the application tier stateless or move session state to a shared store. Load-test the complete path because the database or cache may become the bottleneck.

Why: The capacity estimate points to four instances, while the multiple-instance design also supports the availability goal better than one application server. The trade-off is additional coordination and operational complexity.

Common mistakes

Treating scaling as a cure for inefficient code

More capacity can hide an inefficient query or algorithm temporarily. Identify the bottleneck before spending more.

Assuming scale out is perfectly linear

Three instances do not guarantee 3x throughput. Shared databases, locks, network overhead, and uneven requests matter.

Keeping local session state without a plan

If a user's next request reaches another server, local-only session state may be missing. Use an appropriate shared state strategy or deliberate request affinity, understanding its trade-offs.

Confusing more instances with full high availability

Availability depends on the whole request path, not only the application tier.

Ignoring scale-down

Capacity should also be reduced safely when demand falls. Scale-down rules need graceful connection draining and protection against rapid oscillation.

Knowledge check

1. What changes during vertical scaling?

2. What changes during horizontal scaling?

3. Why might three servers deliver less than three times one server's throughput?

4. Does horizontal scaling guarantee high availability?

Answers

1. The capacity of one resource changes, such as a larger server size.

2. The number of resource instances changes.

3. Shared bottlenecks, coordination, network overhead, locks, and uneven traffic can limit the gain.

4. No. The complete architecture, health checks, traffic distribution, dependencies, and failure handling determine availability.

Download the Day 3 notes

Keep learning with Korshub

Explore Rocking System Design for a structured path through the concepts that appear across this series. Course availability and details should be checked on the course page.

Series navigation

• Previous: Day 2 - Capacity Estimation

• Roadmap: System Design in 30 Days

• Next lesson: Day 4 - Caching