Building your first RAG application — a beginner's roadmap

A language model on its own knows what it was trained on. A useful product needs it to know what is in your documents, your knowledge base, or your customers' data. Retrieval-Augmented Generation — RAG — is how you bridge that gap.

This is a beginner's roadmap. By the end you will have a working RAG application that answers questions about a corpus of documents you provide. Then you will know what to learn next.

What RAG actually is

RAG is three steps:

When the user asks a question, find the parts of your corpus most relevant to it (retrieval)
Hand those parts to a language model along with the question (augmentation)
Let the model write the answer using the retrieved context (generation)

The model still does the writing. The retrieval system supplies the facts. Done well, the answer is grounded in your specific data. Done badly, you have a chatbot that hallucinates with extra steps.

Step 1: pick a problem and a corpus

Start small. Three good first projects:

A Q&A system over the documentation of one open-source project you use
A search assistant over your university's past exam questions
A customer support helper that reads through your product FAQs

Pick something where you actually know the right answers, so you can tell when the system is wrong. Build for a corpus you can reasonably load in a single session — under 10,000 documents.

Step 2: chunk the corpus

Models have a maximum amount of text they can take in at once. Each document needs to be broken into smaller "chunks" — usually 300 to 800 words — that can be retrieved individually.

Naive chunking splits at character boundaries. Better chunking respects structure: split on paragraph or section boundaries, never inside a code block, preserve headings as metadata.

A starting recipe: chunks of 500 tokens with 50 tokens of overlap, split at paragraph boundaries when possible.

Step 3: embed the chunks

Each chunk becomes a vector — a list of numbers that captures its meaning. Similar chunks have similar vectors. This is what makes "semantic search" possible.

Embedding options:

OpenAI text-embedding-3-small — cheap, fast, good enough for most projects
Cohere or Voyage embeddings — competitive quality, often cheaper at scale
Open-source models on Hugging Face (BGE, GTE, Nomic) — free if you have a machine to run them on

Embed every chunk. Store the resulting vectors alongside the chunk text. Repeat whenever your corpus changes.

Step 4: pick a vector store

For a first project, your laptop and one of these is enough:

SQLite + sqlite-vec — zero setup, fastest path to working code
Postgres + pgvector — if you already use Postgres, this is the natural choice
Qdrant or Chroma (local) — purpose-built vector databases that run in a Docker container

Avoid hosted services (Pinecone, Weaviate Cloud) for your first project. They add cost and complexity you do not need yet.

Step 5: retrieval

At query time:

Embed the question using the same embedding model you used for the corpus
Find the top-K most similar chunks — usually K is between 3 and 10
Return the chunks (with their metadata) for the next step

Pure vector search is your starting point. Once you have that working, try hybrid search: combine vector similarity with traditional keyword search (BM25), then merge the rankings. Hybrid usually beats either alone.

Step 6: prompt the model

Your prompt has three parts:

A system message that tells the model what to do — "You are an assistant that answers questions using only the context provided."
The retrieved chunks, clearly delimited
The user's question

Demand the model say "I don't know" when the context does not contain the answer. Most hallucination problems come from prompts that do not give the model permission to refuse.

Step 7: evaluate

You cannot tell if your system is working without measurement. Build a small evaluation set:

30 to 50 real questions about your corpus
For each, the right answer (or a list of acceptable answers)
For each, which chunks should have been retrieved

Run your system against the set every time you change something. Track three metrics: retrieval recall (did the right chunk get retrieved?), answer correctness (was the answer right?), faithfulness (did the answer stay grounded in the retrieved context?).

Step 8: ship the v1

Wire it to a small web frontend. Streamlit, Gradio, or a few-hundred-line React app — does not matter. Get real users in front of it. The first contact with reality will teach you more than a week of theoretical work.

What to learn next

Re-ranking — taking the top 20 chunks and using a more expensive model to score and re-order them
Query rewriting — using a model to clean up vague user questions before retrieval
Multi-hop retrieval — handling questions where the answer requires combining information from multiple chunks
Citations — making the model produce answers with footnotes pointing to specific chunks