Building a RAG Pipeline: Retrieve Before You Generate

Learning outcome: By the end of Day 14, you will be able to explain the four stages of a basic RAG pipeline and map them to a question-answering system built on trusted documents.

This lesson is part of the AI Agents in 30 Days roadmap. Day 12 introduced embeddings and semantic search, and Day 13 covered chunking and retrieval. Today, we connect those pieces into one complete pipeline.

The mental model: retrieve first, generate second

Large language models generate answers from the context available to them and from patterns learned during training. But your application may need answers from information that is private, recent, or specific to your domain - such as study notes, company documentation, policies, or product manuals.

Retrieval-augmented generation, usually shortened to RAG, adds a retrieval step before generation:

1. Search a trusted knowledge source for information relevant to the question.

2. Place the retrieved information in the model's context.

3. Ask the model to answer using that evidence.

The original RAG research combined a language model's internal knowledge with an external searchable memory. Modern implementations vary, but the central idea remains the same: retrieve useful evidence before asking the model to generate the final response. See the original paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.

The four stages of a basic RAG pipeline

Some managed retrieval systems automate much of the second stage. For example, OpenAI's current retrieval documentation explains that files added to a vector store can be automatically chunked, embedded, and indexed for search. The exact implementation is platform-specific, but the pipeline boundaries remain useful when designing or debugging the system. See the OpenAI Retrieval guide.

Practical example: answering from operating-systems notes

Imagine that a student uploads their operating-systems notes and asks:

What causes a deadlock?

Here is what the RAG pipeline does.

1. Ingest

The application accepts the student's notes and extracts the usable text. The notes become part of the system's trusted knowledge collection.

2. Chunk and index

The notes are split into coherent chunks. One chunk might define mutual exclusion, another might explain hold and wait, and another might cover no preemption and circular wait.

Each chunk is placed in a searchable index. Depending on the system, retrieval may use semantic similarity, keyword matching, or a combination of both.

3. Retrieve

The student's question is used to search the index. The system returns the chunks that appear most relevant to deadlocks instead of sending the entire document to the model.

4. Generate

The application sends the question, retrieved chunks, and instructions to the model. A useful instruction might be:

Answer only from the supplied notes. If the notes do not contain enough evidence, say that you do not have enough information. Cite the chunks used.

The generated answer can now explain the four necessary conditions for a deadlock and point the learner back to the relevant notes.

RAG compared with a model-only answer

RAG is not automatically the better choice for every task. If an answer does not require external knowledge, ordinary generation or deterministic code may be simpler.

Grounded does not mean guaranteed

RAG can reduce unsupported answers, but it does not guarantee correctness.

The final response can still be wrong when:

• the source documents are inaccurate or outdated;

• useful context was damaged during chunking;

• retrieval returned irrelevant or incomplete chunks;

• the model ignored or misinterpreted the evidence; or

• the application displayed a citation that did not actually support the claim.

This gives us an important engineering lesson: evaluate the retrieval and the generated answer separately. A fluent answer cannot repair missing evidence.

Try this today

Choose one question from your own notes and draw four boxes:

1. Ingest: Which documents will the system trust?

2. Chunk and index: How will those documents become searchable?

3. Retrieve: What evidence should the question return?

4. Generate: What instructions will constrain the answer?

Then add one fallback rule: what should the system do when retrieval finds no strong evidence?

Completed example you can copy

Question:
What causes a deadlock?

Trusted source:
My operating-systems notes

Ingest:
Extract the text and preserve each section heading.

Chunk and index:
Create one chunk per concept section and retain the page or section number.

Retrieve:
Return the most relevant chunks about deadlock conditions.

Generate:
Answer only from the retrieved chunks. Cite the section used.

Fallback:
If the retrieved notes do not contain enough evidence, say so instead of guessing.

Common mistakes

Retrieving too much

More context is not always better. Irrelevant chunks can distract the model and consume the context window.

Treating retrieval confidence as truth

A high similarity score means a chunk appears relevant to the query. It does not prove that the chunk is correct.

Losing source metadata

If page numbers, document names, or section identifiers disappear during ingestion, useful citations become much harder to produce.

Forcing an answer

A well-designed RAG system needs permission to say, "The available sources do not contain enough information."

Knowledge check

1. Why does retrieval happen before generation?
So the model receives relevant evidence in its context before producing the answer.

2. Does RAG guarantee a correct answer?
No. Source quality, chunking, retrieval, prompting, and generation can all fail.

3. Why should chunks retain source metadata?
So the application can trace retrieved evidence back to the correct document, page, or section.

4. What should happen when retrieval finds no useful evidence?
The application should use an explicit fallback, such as asking for clarification or stating that the sources are insufficient.

Download the Day 14 notes

Continue learning with Korshub

If you want a deeper course on RAG, vector search, or AI agents, use Korshub to discover learning options and continue to the official course platform when you find a suitable course. Listings and availability can change, so verify the current details on the destination platform before enrolling.

Navigation

• Previous: Day 13 - Chunking and Retrieval

• Roadmap: AI Agents in 30 Days