System Design Day 4: Caching Explained with Hits, Misses, and TTLs

System Design Day 4: Caching

By the end of Day 4, you will be able to explain a cache hit and cache miss, estimate how a hit rate changes database read load, and choose a TTL based on freshness requirements.

This lesson is part of Korshub's System Design in 30 Days.

The simple mental model

Check the fast nearby copy first.

A cache is a faster storage layer that keeps reusable copies of data. When the application needs a value, it asks the cache before repeating slower or more expensive work such as a database query, API call, or computation.

The cache is not automatically the source of truth. In our example, the product database remains authoritative. The cache contains a temporary copy of a product price so frequent readers do not force the database to answer the same question every time.

Example: a frequently read product price

Assume an application displays the price of product 42 as INR 1,999. The price is requested frequently but changes much less often than it is read.

A basic cache-aside, or lazy-loading, read path works like this:

1. The application asks the cache for product:42:price.

2. If the unexpired value exists, the cache returns INR 1,999. This is a cache hit.

3. If the value is absent or expired, the cache returns no value. This is a cache miss.

4. On a miss, the application reads the price from the database.

5. The application stores the returned price in the cache with a time-to-live, or TTL.

6. The application returns the price to the user.

AWS documents this same hit/miss sequence for lazy loading: a hit returns from cache, while a miss reads the database and then updates the cache. A miss costs more work because the application checks the cache, queries the database, and writes the result back to the cache.

What does hit rate change?

The cache hit rate is the fraction of lookups served from cache.

For this worked example, assume:

• Total price reads: 1,000 reads/minute

• Cache hit rate: 95% = 0.95

• Cache miss rate: 1 - 0.95 = 0.05

The estimated database read load is:

database reads/minute = total reads/minute x (1 - hit rate)

1,000 x (1 - 0.95) = 50 database reads/minute

The cache serves:

1,000 x 0.95 = 950 reads/minute

Cross-check:

950 cache reads/minute + 50 database reads/minute = 1,000 total reads/minute

Under these assumptions, about 50 reads per minute reach the database instead of 1,000. This result is an estimate, not a universal claim. Real results depend on traffic distribution, cache capacity, key popularity, failures, invalidation, and the latency of each layer.

TTL is a freshness decision

A TTL specifies how long a cached value may remain before it expires. After expiration, the next lookup becomes a miss and refreshes the value from the database.

Consider three possible TTLs for the product price:

There is no universally correct TTL. Start from the business requirement: how old may this price be before the user experience or transaction becomes incorrect?

TTL limits how long a forgotten entry can survive, but TTL alone does not guarantee freshness. If a price update must appear immediately, update or invalidate the cache when the database changes. One common approach is to write the database first and then delete the affected cache key so the next reader fetches the new value.

What should you cache?

Caching tends to help when data is:

• read frequently;

• more expensive to fetch or compute than to read from cache;

• stable enough to reuse for a meaningful period; and

• safe to serve within a defined freshness window.

Avoid caching everything by default. Data that changes continuously may be stale almost immediately. Data requested very rarely may occupy memory without producing meaningful hits. Redis guidance similarly recommends focusing on keys that are requested often and change at a reasonable rate.

Cache memory is finite

A cache cannot grow forever. When memory reaches its configured limit, the system needs an eviction policy to decide which keys to remove. Redis supports policies based on expiration, recency, frequency, or other rules.

Eviction and TTL solve different problems:

• TTL removes a value after a time-based freshness window.

• Eviction removes values when the cache needs memory.

• Invalidation removes or updates a value because the authoritative data changed.

Good cache design considers all three.

Try this today

Choose a TTL for the price of product 42: 30 seconds, 5 minutes, or 1 hour.

Write down:

1. How quickly must a price change become visible?

2. What happens if an old price is shown?

3. Is there an update event that can invalidate the cache immediately?

4. How many database reads are acceptable during a cache miss burst?

Your answer should name the requirement, not just the number.

Completed example you can copy

I choose a 5-minute TTL for browsing pages because a short display delay is acceptable there. When an administrator changes a price, the service deletes product:42:price immediately. The next request misses, reads the new database value, and repopulates the cache. Checkout reads the authoritative price again before payment so the final charged amount does not depend on a stale browse-page cache.

This is one reasonable design under those assumptions. A different business requirement can justify a different TTL and invalidation flow.

Common mistakes

Treating the cache as automatically correct

A cached value can be old. Define the source of truth and the acceptable freshness window.

Choosing TTL without a requirement

“Five minutes sounds fine” is not a design reason. Link the TTL to how quickly changes must appear and what stale data could cause.

Ignoring miss behavior

An empty or failed cache sends more work to the database. Ensure the database can tolerate miss bursts and avoid stampeding it with many simultaneous refreshes.

Caching cold data

Rarely requested data consumes memory but generates few hits. Measure key popularity and hit rate.

Forgetting memory limits and eviction

Specify what happens when the cache fills. Otherwise, new writes may fail or useful entries may be removed unpredictably.

Knowledge check

1. What is a cache hit?
   The requested unexpired value is found in cache and returned without reading the database.

2. What is a cache miss?
   The value is absent or expired, so the application must fetch it from the source and may repopulate the cache.

3. At 1,000 reads/minute and a 95% hit rate, how many reads reach the database?
   About 50 reads/minute: 1,000 x (1 - 0.95).

4. Does a TTL guarantee a value is never stale?
   No. It bounds expiration time, but the value can be stale before it expires. Use update or invalidation logic when stricter freshness is required.

Continue learning with Korshub

Explore Rocking System Design for a broader path through the components and trade-offs used in scalable systems. Course details may change, so use the course page as the current source.

Series navigation

• Previous: Day 3 - Vertical vs Horizontal Scaling

• Roadmap: System Design in 30 Days

• Next after publication: Day 5 - Content Delivery Networks at https://korshub.com/blog/system-design-day-05-content-delivery-networks