How to Build a RAG Pipeline with LangChain and Pinecone

Retrieval-Augmented Generation is the most common way production teams give a language model access to knowledge it was never trained on — internal documentation, a product catalogue, last quarter's support tickets. The concept is simple: before you ask the model a question, you go and fetch the most relevant passages from your own data, and you paste them into the prompt. The model answers from what you gave it rather than from memory.

The concept is simple. Getting it to work reliably is not. Most RAG systems that fail in production don't fail at the model — they fail at retrieval, and the model faithfully answers using the wrong context it was handed. This walkthrough builds a working pipeline with LangChain and Pinecone, then spends real time on the parts that break.

The four stages of a RAG pipeline

Every RAG system, no matter how elaborate, is these four stages. Understanding them as distinct problems is what lets you debug the thing later.

1. Ingestion — load your source documents and split them into chunks small enough to be individually meaningful.
2. Embedding — convert each chunk into a vector, a list of numbers that encodes its meaning.
3. Storage & retrieval — put those vectors in a database that can find the nearest neighbours to a query vector, fast.
4. Generation — hand the retrieved chunks to the model as context, along with the user's actual question.

Stage 1: Chunking is where most RAG systems are lost

It is tempting to treat chunking as a formality — split every 1,000 characters and move on. This is the single most common reason a RAG system gives vague, unhelpful answers. If you split mid-sentence, or separate a table from its heading, or break a procedure away from the paragraph explaining when to use it, you have produced chunks that are individually meaningless. No amount of retrieval sophistication recovers from that.

Chunk on semantic boundaries wherever the format allows. Markdown headers, document sections, and paragraph breaks are all better split points than a raw character count. Use overlap so that a thought spanning a boundary survives in at least one chunk.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    # Try to split on the most meaningful boundary available,
    # falling back to weaker ones only when necessary.
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "],
)

chunks = splitter.split_documents(documents)

Stage 2: Embeddings

An embedding model turns text into a vector positioned so that semantically similar text lands nearby. This is what lets a query like "how do I cancel" retrieve a passage titled "Terminating your subscription" — the words don't overlap at all, but the meanings do.

The critical, non-negotiable rule: you must embed your queries with the same model you used to embed your documents. Mixing embedding models produces vectors in incompatible spaces, and your retrieval quietly becomes random. If you upgrade your embedding model, you must re-embed your entire corpus.

Stage 3: Storing and retrieving with Pinecone

Pinecone is a managed vector database — it handles the approximate-nearest-neighbour indexing so you don't build it yourself. Create an index whose dimension matches your embedding model's output, then upsert your chunks.

from langchain_pinecone import PineconeVectorStore
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vector_store = PineconeVectorStore.from_documents(
    documents=chunks,
    embedding=embeddings,
    index_name="company-docs",
)

# Retrieve more than you need, then narrow it down.
retriever = vector_store.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 8},
)

Note the k value. Retrieving a single best chunk feels efficient and is usually a mistake — the right answer is often spread across two or three passages. Retrieve generously, then use a reranker or let the model sift. Retrieval is cheap; being wrong is expensive.

Stage 4: Generation, and the prompt that prevents lies

Now you assemble the prompt. The most important line in the entire pipeline is the instruction that tells the model what to do when the retrieved context does not contain the answer. Without it, the model will cheerfully fall back on its training data and invent something plausible — and your users will have no way to tell the difference.

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_template("""
Answer the question using ONLY the context below.
If the context does not contain the answer, say
"I don't have that information" — do not guess.
Cite the source of each claim you make.

Context:
{context}

Question: {question}
""")

The failure modes nobody warns you about

Retrieval looks fine, answers are still wrong

Log the retrieved chunks for every query in development. Almost always, the problem is visible immediately: the model was handed the wrong passages and answered them correctly. You have a retrieval bug, not a model bug. Teams waste weeks tuning prompts to fix what is actually a chunking problem.

Semantic search misses exact terms

Embeddings capture meaning, which means they are surprisingly bad at exact identifiers — error codes, SKUs, function names, part numbers. A user searching for "ERR_4021" may get passages that are semantically about errors but never mention that code. The fix is hybrid search: combine dense vector similarity with old-fashioned keyword matching, and merge the results.

Stale data

Your vector store is a copy. When the source document changes, the embedding does not — and your system confidently serves last month's refund policy. Plan for re-indexing from the beginning, and store a document ID and version on every chunk's metadata so you can update in place rather than rebuilding the whole index.

Evaluate, or you're guessing

The hardest discipline in RAG is refusing to judge quality by vibes. Build a small evaluation set — thirty to fifty real questions with known correct answers — before you start tuning. Then measure two things separately:

• Retrieval quality: did the correct passage appear in the retrieved chunks at all? If not, no prompt can save you.
• Answer quality: given the correct passage, did the model actually use it faithfully and without embellishment?

Separating these two numbers tells you where to spend your effort. Almost every team that skips this step ends up optimising the wrong half of the system.

Where to go from here

A working RAG pipeline is roughly a hundred lines of code. A reliable one is an ongoing engineering discipline — chunking strategy, hybrid retrieval, reranking, evaluation harnesses, and re-indexing pipelines. That gap between "works in a notebook" and "survives real users" is exactly the gap that separates a tutorial from a production AI engineer.