System Design Day 08: Database Replication

Database Replication: Scale Reads Without Losing the Source of Truth

Learning outcome: By the end of Day 08, you will be able to explain how a primary database and read replicas distribute read traffic, how a replica can support recovery, and why replication lag matters.

This lesson is part of Korshub's System Design in 30 Days.

Watch the Day 08 lesson

The simplest mental model

Think of database replication as:

One source of truth writes. Multiple copies serve reads.

In a common primary-replica topology, the application sends writes to one primary database. The primary records each committed change and sends that change to one or more replicas. Those replicas can answer read-only queries, which reduces the read workload on the primary.

This is valuable when an application has far more reads than writes. Product catalogs, dashboards, reports, content feeds, and search-supporting workflows often contain reads that do not need to run on the primary.

Primary and read-replica flow

The flow is:

1. The application sends a write to the primary.

2. The primary commits the change.

3. Replication transfers that change to Replica A and Replica B.

4. Each replica applies the change.

5. Read traffic can be sent to the replicas according to the application's consistency needs.

The word "can" matters. Your application, driver, proxy, or data-access layer still needs a routing strategy. Adding replicas alone does not automatically distribute every query.

Worked example: 900 reads per second

Assume a product dashboard receives 900 read queries per second and has two read replicas. If the read router divides traffic evenly:

reads per replica = total reads / number of replicas
                  = 900 reads/second / 2
                  = 450 reads/second per replica

The conservation check is:

450 + 450 = 900 reads/second

Under this simplified assumption, each replica handles about 450 reads per second instead of one database handling all 900 reads.

This is an estimate, not a capacity guarantee. Real traffic is rarely distributed perfectly. Query cost, cache hit rate, replica hardware, connection pools, indexes, long-running reports, and hot keys can make one replica busier than another. Measure per-replica latency, throughput, CPU, connections, and replication lag before deciding that the topology is healthy.

The most important trade-off: replication lag

Many read-replica systems use asynchronous replication. The primary can commit a transaction before a replica has received and applied the corresponding change. The delay is replication lag.

Imagine this sequence:

1. A customer updates a delivery address.

2. The write commits on the primary.

3. The application immediately reads from a replica.

4. That replica has not applied the update yet.

5. The customer briefly sees the old address.

The database is not necessarily broken. The read was served from a copy that was slightly behind.

For workflows that require read-after-write consistency, a practical rule is to route the immediate read to the primary. Other reads that tolerate brief staleness may continue to use replicas. The correct choice depends on the product requirement, not on a universal rule.

Replication and availability

Replication keeps additional copies of data. If the primary becomes unavailable, an eligible replica may be promoted to become the new primary. Promotion is only part of recovery. A production design must also consider:

• how failure is detected;

• which replica is current enough to promote;

• whether promotion is automatic or operator-controlled;

• how clients discover and connect to the new primary;

• how split-brain writes are prevented;

• what recovery point and recovery time the business accepts.

Do not say "replicas guarantee zero downtime." They provide building blocks for availability, but failover behavior depends on the database technology, deployment mode, orchestration, and application routing.

What replication does not solve

It does not automatically scale writes

If every write still goes to one primary, that primary remains the write bottleneck. Read replicas scale read capacity; write scaling may require a different data model, sharding, partitioning, multi-primary architecture, or workload redesign.

It is not a replacement for backups

Replication copies changes, including accidental deletes or bad updates. Backups and point-in-time recovery protect against a different class of failures and should be designed separately.

It does not make all reads immediately current

Asynchronous replicas may return older data for a short period. The application must classify reads by consistency requirement.

Try this today

Classify each read as must be fresh or can be slightly stale:

1. A user updates a bank balance and refreshes immediately.

2. A public analytics dashboard refreshes every five minutes.

3. A customer changes a profile photo and opens another page.

4. A nightly business report scans millions of rows.

A reasonable first pass is:

• Bank balance after an update: primary, because correctness requires the committed value.

• Five-minute analytics dashboard: replica, if brief staleness is acceptable.

• Profile photo immediately after an update: primary or session-aware routing, depending on the product promise.

• Nightly report: replica, so the expensive scan does not compete with primary writes.

Common mistakes

1. Sending writes to a read replica. In this topology, application writes belong on the primary.

2. Ignoring lag. Monitor it and decide which reads may tolerate it.

3. Assuming equal traffic means equal work. One expensive query can cost more than many small ones.

4. Treating promotion as the whole failover plan. Client routing, fencing, health checks, and operational testing matter too.

5. Calling replication a backup. Replicas and backups protect against different failures.

6. Claiming write scalability. A single-primary design still has one write authority.

Knowledge check

1. Why can a read replica return old data?

Because an asynchronous replica may not yet have received and applied the primary's latest committed change.

2. If 900 reads per second are split evenly across two replicas, how many does each receive?

About 450 reads per second.

3. Where should an immediate read-after-write request go when the latest value must be visible?

To the primary in this design.

4. Does adding read replicas remove the primary write bottleneck?

No. All writes still go through the primary unless the architecture changes.

Download the handwritten notes

Continue learning with Korshub

Database replication becomes easier to reason about when you connect it to caching, indexing, consistency, sharding, and failover. Explore the Rocking System Design course on Korshub for a broader system design learning path.

Series navigation

• Previous: Day 07 - Database Indexing

• Roadmap: System Design in 30 Days

• Next planned lesson: Day 09 - Database Sharding