AI Agents Day 13: Chunking and Retrieval

Chunking and Retrieval: Give Your Agent the Right Context

Learning outcome: Split a document into useful chunks and understand how retrieval selects the pieces an AI model should read.

This is Day 13 of AI Agents in 30 Days. In Day 12, we learned how embeddings help search by meaning. Today we focus on what gets embedded and searched: the chunks.

The mental model

Imagine asking a question about a hundred-page textbook. Passing the entire book to a model for every question can consume unnecessary context and bury the most useful evidence.

A retrieval system handles this in two stages:

1. Chunking: split the source into smaller, searchable units.

2. Retrieval: compare a question with those units and return the most relevant ones.

The model then receives the question plus the retrieved chunks. The original RAG research describes this combination of a generator with access to retrieved external knowledge. Read the RAG paper.

Modern retrieval systems commonly use semantic similarity so relevant text can be found even when it shares few exact keywords with the query. OpenAI's Retrieval guide explains this vector-store search model.

Why chunk boundaries matter

Chunking is not just cutting text every fixed number of characters. Each chunk should preserve enough meaning to be useful after it has been separated from the original document.

Suppose your notes contain this section:

Binary Search

Repeatedly compare the target with the middle element of a sorted range.
Discard the half that cannot contain the target and continue.

A poor split might place Binary Search at the end of one chunk and its explanation at the beginning of the next. A search for "How does binary search work?" could then retrieve a heading without the explanation, or an explanation without its subject.

A better split keeps the heading, definition, and essential steps together.

Microsoft's RAG guidance describes chunking as creating appropriately sized units that contain semantically relevant content, and its document guidance shows how headings and layout can support structure-aware splitting. Review the chunking guidance and semantic document chunking example.

Small chunks, large chunks, and overlap

There is no universal best chunk size. The right choice depends on the document structure, the questions users ask, the embedding model, and the amount of context the application can pass to the model.

Treat chunk size and overlap as settings to evaluate, not magic numbers to copy.

Retrieval after chunking

Once the document is chunked:

Document
   ↓
Meaningful chunks
   ↓
Embeddings and search index
   ↓
User question
   ↓
Most relevant chunks
   ↓
Question + retrieved context → model

Retrieval does not automatically prove that a chunk is correct. It ranks chunks by relevance. Your application still needs clear instructions, source metadata, sensible score thresholds, and evaluations using real questions.

Try this today

Take one page of study notes containing at least two headings.

1. Split it at meaningful boundaries such as headings or complete paragraphs.

2. Read every chunk without looking at the original page.

3. Ask: Does this chunk make sense on its own?

4. If an explanation depends on the previous chunk, move the boundary or add a small overlap.

5. Write one question that each chunk should be able to answer.

Completed example

Source section

Binary Search
Works on a sorted search space. Compare the target with the middle value.
If the target is smaller, continue in the left half. Otherwise, use the right half.
Stop when the target is found or the range becomes empty.

Useful chunk

{
  "heading": "Binary Search",
  "content": "Works on a sorted search space. Compare the target with the middle value. If the target is smaller, continue in the left half. Otherwise, use the right half. Stop when the target is found or the range becomes empty.",
  "source": "algorithms-notes.md",
  "section": "searching"
}

Question it should answer: "What conditions and steps define binary search?"

The heading and explanation stay together, while the metadata helps the application cite and filter the source.

Common mistakes

1. Splitting only by character count

Fixed-size splitting is simple, but it can cut through a sentence, list, code block, or table. Respect structure where the document provides it.

2. Making every chunk tiny

Smaller is not always better. A focused fragment can still be useless if it lacks the context needed to interpret it.

3. Using excessive overlap

Overlap can protect boundary context, but too much creates near-duplicate chunks and may crowd retrieval results with repeated text.

4. Testing with only one easy question

Evaluate questions that target definitions, comparisons, steps, exceptions, and information near chunk boundaries.

5. Treating retrieval as a guarantee

Retrieval returns likely relevant evidence. The system should still cite sources, handle weak matches, and admit when the available context is insufficient.

Knowledge check

1. Why not send an entire document for every question?
It may waste context, increase processing cost, and make the relevant evidence harder to isolate.

2. What is the main purpose of overlap?
To preserve useful context that crosses a chunk boundary.

3. Are smaller chunks always more accurate?
No. They can improve focus but lose the surrounding meaning needed to answer correctly.

4. What is the best way to choose a chunking strategy?
Test it with representative documents and real user questions, then measure retrieval quality.

Download the handwritten notes

Use the one-page notes as a quick reference while testing your first retrieval pipeline.

Keep learning with Korshub

Korshub helps learners discover course deals across official learning platforms. You can explore relevant courses, review available price history where tracked, and continue to the official course platform to enroll.

Browse courses on Korshub

Continue the series

• Previous: Day 12 — Embeddings and Semantic Search

• Roadmap: AI Agents in 30 Days

• Next: Day 14 — Building a RAG Pipeline will be linked after publication.