Load Balancing in System Design: How It Works

Learning outcome: By the end of Day 1, you will understand why a load balancer sits between users and application servers, how health checks influence routing, and why adding servers alone does not guarantee scalability.

This is Day 1 of 30 Days of System Design Explained, a beginner-friendly Korshub series that turns one important architecture concept into one practical mental model each day.

The surprising failure: three servers, but only one is working

Imagine your app runs on three identical servers: A, B and C.

Thousands of users arrive at once, but every request is sent directly to Server A. Server A becomes slow, queues grow and some requests fail. Meanwhile, Servers B and C sit almost idle.

The problem is not a lack of computing capacity. The problem is that traffic is not being distributed across that capacity.

That is the job of a load balancer.

What is a load balancer?

A load balancer is an entry point that accepts incoming traffic and routes each request or connection to an eligible backend.

In AWS terminology, a load balancer accepts traffic from clients, sends it to registered targets and monitors target health. When a target becomes unhealthy, the load balancer normally stops routing new traffic to it and resumes when it becomes healthy again. See How Elastic Load Balancing works and Application Load Balancer health checks.

The simplest mental model is:

Users → Load balancer → Healthy application servers

Users see one application address. Behind that address, the load balancer decides where each eligible request should go.

Follow one request through the system

Suppose a user opens the app:

1. The request reaches the load balancer.

2. The load balancer checks its current pool of eligible targets.

3. It selects a server using the configured routing strategy.

4. The selected server processes the request.

5. The response returns to the user.

The load balancer does not necessarily inspect every application-level detail, and different load-balancer types operate at different layers. For this lesson, focus on its two essential responsibilities: traffic distribution and target health awareness.

Practical example: nine requests and three servers

Without useful distribution, the traffic might look like this:

With a simple balanced setup, the same traffic could be distributed like this:

Real systems are not always perfectly even. Long-running requests, server capacity, connection state and the selected algorithm can produce different distributions.

Common routing strategies

NGINX documents round robin, least connected, weighted distribution and IP hash as distinct approaches. It also notes that round-robin or least-connected routing does not guarantee that the same client always reaches the same server. See the official NGINX HTTP load-balancing guide.

The best strategy depends on the workload. “Even request counts” and “even work” are not always the same thing.

What happens when Server B fails?

A health check is a probe used to determine whether a backend should remain eligible for new traffic.

If Server B fails enough health checks to cross the configured unhealthy threshold, the load balancer removes it from normal service and routes eligible new requests to Servers A and C. AWS also documents an important edge case: if every registered target is unhealthy, an Application Load Balancer can fail open and route to all targets. That is why “only healthy servers” is a useful beginner model, but not an absolute rule for every failure state. See AWS target-group health checks.

When Server B recovers and passes the required health checks, it can become eligible again.

What load balancing improves

• Availability: one failed backend does not automatically make the whole application unreachable.

• Fault tolerance: traffic can move away from a failed target.

• Scalability: new servers can join the eligible pool as demand grows.

• Maintenance: a server can be drained or removed while work is performed.

• Operational visibility: request, error, latency and target-health metrics become easier to observe at the traffic layer.

What a load balancer cannot fix

A load balancer is not a universal performance solution. It does not automatically repair:

• slow application code;

• inefficient database queries;

• memory leaks;

• a shared database bottleneck;

• a broken downstream API;

• insufficient total capacity; or

• a poorly designed health endpoint.

It distributes traffic. The services receiving that traffic must still be efficient, observable and resilient.

Try this today

Draw three boxes labelled A, B and C, then complete this exercise:

1. Send 12 requests directly to Server A.

2. Add a load balancer and distribute the next 12 requests across A, B and C.

3. Mark Server B unhealthy.

4. Route the next six requests only to eligible servers.

5. Write down one bottleneck the load balancer still cannot solve.

Optional hands-on extension: reproduce the three states in the Korshub interactive load-balancing diagram—uneven traffic, balanced traffic and Server B failed.

Completed example

Before: A receives 12 requests; B and C receive none.

Balanced: A, B and C receive approximately four requests each.

After B fails: new traffic is routed to A and C while B is excluded from normal routing.

Remaining bottleneck: if all servers depend on the same overloaded database, distributing application requests does not remove the database bottleneck.

Common mistakes

Mistake 1: “More servers automatically means more capacity”

Capacity only helps when requests can reach it. A routing layer must know which backends exist and which are eligible.

Mistake 2: “A load balancer always splits traffic equally”

The distribution depends on the routing algorithm, weights, active connections, session behaviour and target health.

Mistake 3: “Health checks prove the whole application is healthy”

A shallow endpoint might respond successfully even while a critical dependency is failing. Health checks must reflect the service behaviour you actually care about.

Mistake 4: “Load balancing and auto scaling are the same”

Load balancing distributes traffic across available capacity. Auto scaling changes how much capacity is available. They are often used together, but they solve different problems.

Mistake 5: “The load balancer removes every single point of failure”

The load-balancing layer, DNS, databases, dependencies and deployment topology must all be designed for availability.

Knowledge check

1. Why can Server A overload while Servers B and C remain idle?

2. What should happen to new requests after Server B fails its health check?

3. Why might round robin still produce uneven work?

4. Does a load balancer fix a slow shared database?

Answers

1. Traffic is being sent unevenly or directly to one server.

2. Normal routing should exclude B and send eligible new requests to healthy targets such as A and C.

3. Requests can require different amounts of time or computing work.

4. No. The database bottleneck must be addressed separately.

Continue learning with Korshub

Choose the path that matches your goal:

• Rocking System Design for broad system-design building blocks, including scaling, caching, sharding and load balancing.

• [NEW] Amazon EC2 Masterclass: Auto Scaling & Load Balancer for hands-on AWS EC2, Application Load Balancer and Auto Scaling practice.

• Mastering the System Design Interview for interview-focused scaling, resiliency and architecture trade-offs.

Course prices, coupons and availability can change. Check the live Korshub course page before enrolling.

Series navigation

• Previous: Start of series

• Roadmap: 30 Days of System Design Explained