Build a simple RAG pipeline with Claude

A simple RAGRetrieval-augmented generation: fetch relevant text from your own data and feed it to the model as context, so it answers from that instead of only its training. pipeline has four moving parts wired in order: ingest your documents, embed each piece into a vector, retrieve the closest pieces to a question, then answer with Claude using only those pieces. This guide builds that flow step by step and drives every step through the Claude Code CLI, so you describe the work and Claude writes the code.

If you want the theory behind each stage first, read RAG Fundamentals. This guide is the hands-on build.

The shape of the flow

Picture two halves. One half runs once when your data changes: it reads documents and stores their meaning. The other half runs on every question: it finds the relevant stored pieces and hands them to Claude.

The top row is the ingest path. The bottom row is the query path. Build them in that order, because you cannot retrieve from an empty store.

Stage 1: ingest

Ingest reads raw files and splits them into passages small enough to retrieve. A whole document is too coarse: a question about one paragraph should not pull in twenty pages. So you split each document into chunksA passage of text small enough to retrieve on its own, usually a few hundred words..

Start by pointing Claude Code at the folder you want to index. Open the CLI in your project and describe the job.

Claude writes the splitting script, runs it, and shows you the output. Inspect those first entries. If a chunk ends mid-sentence or merges two unrelated topics, the retrieval step later will suffer, so fix the boundaries now.

A few rules that keep ingest clean:

• Split on structure, not raw character counts. Break on paragraphs or headings first, then fall back to sentences only when a section is too long.
• Keep a small overlap. Carry the last sentence or two of one chunk into the next, so a fact sitting on a boundary survives in at least one chunk.
• Record the source. Store the file path and position with each chunk. You need this later so Claude can cite where an answer came from.

How do I pick a chunk size?

There is no single correct chunk size, but the size you pick shapes every later stage, so choose it on purpose. Small chunks of a paragraph or two retrieve precisely: a match points at one tight passage, and the answer stays focused. The cost is that a fact split across two paragraphs can land in neither chunk. Large chunks of a full section rarely split a fact, but they retrieve loosely, dragging in surrounding text that dilutes the relevant part.Common gotchaEvery embedding model has a token limit, and a chunk past it is usually truncated without error rather than rejected, so the tail of an oversized chunk never gets embedded and can never be retrieved.Cap chunk length below the model's limit and have Claude flag any chunk it had to cut.

Start around 500 words and adjust once you can read retrieval results. For the full set of sizing, overlap, and boundary choices, see chunk your documents. If the right passage keeps arriving buried inside a wall of unrelated text, the chunks are too big. If a question pulls a passage that answers half the question and the other half lives in a neighbouring chunk that did not get retrieved, the chunks are too small or the overlap is too thin.

The shape of your documents matters here too. Reference material with short, self-contained sections chunks cleanly on its headings. Prose that argues across several paragraphs needs more overlap so the argument is not severed at a boundary.

Stage 2: embed

An embeddingA list of numbers that captures the meaning of a piece of text, so two passages about the same thing land close together in that number space. turns a chunk of text into a list of numbers that captures its meaning. Two chunks about the same topic produce vectors that sit close together. That closeness is what makes retrieval work: you compare numbers instead of matching exact words.

You do not call an embedding model by hand. Pick an embedding service, then connect it to Claude Code as a tool through MCP so Claude can embed chunks for you on request. The Model Context Protocol is the open standard for giving a CLI access to outside tools and data.

Add a remote embedding server with one command:

claude mcp add --transport http embeddings https://embeddings.example.com/mcp

That registers the server under the default local scope, so it is available to you in this project. To share the server with everyone working in the repository, use the project scope, which writes the config to a checked-in .mcp.json file at the project root:

claude mcp add --transport http embeddings --scope project https://embeddings.example.com/mcp

The resulting .mcp.json holds the server under an mcpServers key:

{
  "mcpServers": {
    "embeddings": {
      "type": "http",
      "url": "https://embeddings.example.com/mcp"
    }
  }
}

The three scopes control who sees the server:

With the tool connected, ask Claude to embed your chunks:

Two things to watch:

• Use one embedding model for everything. A question and the chunks must be embedded by the same model, or their vectors live in different number spaces and the distances mean nothing.Common gotchaThe trap is silently changing the embedding model after you have already embedded your chunks, then embedding new questions with the new model.Nothing errors: cosine similarity still returns numbers, so retrieval looks healthy while quietly ranking by noise. Pin the exact model and dimension count, and re-embed every chunk whenever you change it.
• Embedding quality sets the ceiling. If similar passages do not land near each other, no later step can recover. Spot check by asking Claude to find the two closest chunks and reading them.

Stage 3: store and retrieve

You now have a vector per chunk. The store is where those vectors live so you can search them, and retrieval is the search itself: embed the question, then find the chunks whose vectors sit closest.

For a small project, a single JSON file scanned in memory is enough to learn the flow. When the collection grows past a few thousand chunks, move to a real vector store. If you already run Postgres, the pgvector extension covers a large share of cases. You can connect that database to Claude Code the same way, through MCP; see Connect Claude to your database.

The standard distance measure is cosine similarityA way to measure how similar two vectors are by the angle between them, so passages pointing the same direction count as close in meaning.. Ask Claude to wire retrieval:

Read the returned chunks before going further. This is the single highest-value check in the whole pipeline: if the right passage is not in the top results, Claude cannot use it, and the final answer will be wrong no matter how good Claude is.

How many chunks to retrieve is a tradeoff. Too few and you miss the relevant passage. Too many and you bury the good chunk among filler, which models attend to poorly. Starting around five and adjusting based on what you read is a sound default.

When does pure vector search fall short?

Vector search matches on meaning, which is exactly what you want for questions phrased differently from the source text. It has one blind spot: exact strings. An error code, an order number, or a function name like processRefund carries little semantic meaning, so a vector match can sail right past it.

The fix is hybrid retrieval: run a keyword search alongside the vector search and merge the results. Keyword search catches the literal processRefund; vector search catches the question that never names it. Ask Claude to add a keyword pass and combine the two rankings when your data is full of identifiers:

You do not need hybrid retrieval on day one. Add it once you see vector-only search miss questions that hinge on an exact term.

Stage 4: answer with Claude

The last stage hands the retrieved chunks to Claude and asks for an answer grounded in them. GroundingConstraining the model to answer only from the text you provide, rather than from its training. means the answer must come from the supplied passages, and the model says so plainly when the passages do not contain the answer. This is your main defence against confident wrong answers.

Tie the whole flow together in one CLI step:

The instruction to refuse when the answer is absent matters as much as the retrieval. A grounded prompt that still permits guessing gives you the same confident fabrication you were trying to avoid. For more on holding the model to its sources, see Stop Claude making things up.

Here is the full query path, end to end:

How do I test each RAG stage on its own?

The reason to build RAG as four separate steps is that you can test each one in isolation. When the final answer is wrong, you want to know which stage failed, and a single end-to-end script hides that. Walk the stages in order and stop at the first one that looks wrong.

• Ingest. Read a sample of chunks. Boundaries should fall between ideas, not inside a sentence. A chunk should make sense read on its own.
• Embed. Ask Claude to find the nearest neighbour of a chunk you understand. The neighbour should be about the same topic. If unrelated chunks come back close, the embedding step is the problem.
• Retrieve. Run a handful of real questions and read the top chunks. The passage that answers each question should be in the list. If it is not, no prompt fixes it.
• Answer. Only once retrieval is solid should you judge the answer. A wrong answer over correct chunks is a prompt problem; a wrong answer over wrong chunks is a retrieval problem.

This order saves time because a problem early in the pipeline shows up as a vague failure late in it. A bad chunk boundary looks like a bad answer. Testing forward, stage by stage, points you straight at the real cause.

How do I update a RAG pipeline when the data changes?

The query path runs on every question. The ingest and embed path runs only when the source documents change. Keep them separate so a content update does not force you to rebuild query logic.

When a document changes, re-chunk and re-embed that document and replace its entries in the store. Do not leave stale chunks behind, because a question can match an old passage and produce an answer that no longer matches the live source. Ask Claude to re-run ingest and embed for the changed files only, then confirm the store entry count moved as expected.

Why not only put everything in one prompt?

A reasonable first instinct is to skip retrieval and paste every document into the prompt. For a handful of short files that works, but it stops scaling for three reasons.

• Size. A document collection grows past what any context window holds. Retrieval selects the few passages a question needs, so the size of the collection stops mattering.
• Cost. Sending the whole collection on every question pays for text the question does not touch. Retrieving five chunks sends a fraction of the tokens for the same answer.
• Focus. A model reads a short, relevant context more accurately than a long one where the answer hides among unrelated text. Fewer, better chunks produce a sharper answer.

Retrieval is the step that turns a pile of documents into something a model can answer from at any scale.

Where to go next

You now have a working RAG pipeline: ingest splits documents, embed turns chunks into vectors, retrieve finds the closest, and Claude answers from those alone. Each stage is small, and each is checkable on its own.

To make it dependable rather than just working, read RAG Fundamentals for evaluation and the common failure modes, then RAG for Coding Agents for a domain where the standard recipe needs adjusting. To go deeper on what to put in the prompt alongside the chunks, see Context engineering.