Retrieval Augmented Generation with Vector Databasescopy link
Introduction to Retrieval Augmented Generation (RAG) using popular Vector databases like Chroma and Pinecone
overviewcopy link
This tutorial will walk you through creating a search index for a collection of documents by using a vector database, and then using Retrieval Augmented Generation (RAG) to build a chat bot that uses this database to answer questions.
While vector databases are useful on their own for indexing a collection of documents, combining this machine learning technique with generative AI provides agents with long-term memory, expanding their capabilities.
This hands-on tutorial is designed to help you integrate Retrieval Augmented Generation into your application as quickly as possible. For a deeper introduction checkout Vector Databases for Generative AI by Leandro Calado.
why vector database?copy link
Anyone implementing full-text search has to think about:
- Tokenizing, or breaking the search criteria into separate search terms, which is typically done by splitting the text a user typed into a search box by white-space and punctuation characters.
- Sub-string matching, so that a query term like “real” matches “realize”.
- Stemming, or transforming words into their most basic versions so that a query like “running” also matches documents containing “runs” or “run”.
- Stop words, or common words like “the”, “and”, “for” that must be filtered from the query to avoid matching every document.
- Similarity, or matching by meaning rather than exact words (“run” - “sprint”), which is usually done by loading a thesaurus for the target language.
- Ranking, or evaluating each maching document against the entire search criteria, usually based on how many of the terms match and their frequency in the document.
While a home-made search engine that uses tokenizing and sub-string matching is often sufficient for a personal blog or a hobby app, deployment at scale requires the intuitiveness of stemming, similarity matching, and ranking to meet varied needs of many users.
These advanced features were only supported by enterprise search engines like Elasticsearch until recently, requiring costly solutions like always-running virtual machines.
The newer open-source vector databases like Qdrant and Chroma were built on the Transformer architecture by using Embedding layers for converting text and images into numbers that neural networks can process, re-using the same mechanism for similarity search, and adding common database operations like inserts and queries.
This technology spawned many affordable serverless solutions with free tiers and pricing based on usage, bringing advanced full-text search within the grasp of hobbyists and lean startups.
selecting a databasecopy link
A survey of popular vector databases reveals the following:
| Solution | Deployment | Storage | Usage Limits |
|---|---|---|---|
| Chroma | Local Serverless | 1 GB | Low Cost: 1T transfer $5 / month |
| Pinecone | Serverless | 2 GB | Free: 1M read + 2M writes / month |
| Qdrant | Local VM | 4 GB | Free: 0.5 vCPU, unlimited requests |
| Upstash | Serverless | 2 GB | Free: 10K queries + 10K updates / day |
Qdrant and Chroma databases are open-source and can be operated locally or in the cloud while Pinecone and Upstash only run in the cloud and therefore require an API key.
Pinecone is a better choice for unattended deployments, such as a personal blog or an app. Unlike Qdrant which shuts down inactive virtual machine clusters, Pinecone is serverless and spins up virtual machines on request.
using a vector databasecopy link
Indexing with a vector database involves:
- Chunking and encoding a collection of documents
- Creating a database and inserting the encoded vectors
The next few sections will demonstrate how to setup a Chroma database locally, use it for indexing documents, retrieve documents based on a search criteria, and ochestrate Retrieval Augmented Generation with a locally running model in LM Studio.
The same steps are then repeated with a hosted Pinecone database to prepare you for deploying your solution to production.
/**/ The Python code snippets in this tutorial are from the vector companion repository included with this tutorial. The
mainbranch contains the code for using Chroma database locally, andpineconebranch contains the code for using the serverless Pinecone database with an API key.
chunking documentscopy link
Indexing a document requires feeding it through an encoder-only neural network implemented with the Transformer architecture.
A very small model is typically used for this purpose, such as the all-MiniLM-L6-v2. Unlike a generative LLM, this model produces a 384-dimensional embedding vector instead of text.
Since Transformer-based neural networks have a limited context window and cannot process arbitrarily large documents, the documents are split into bite-sized passages called chunks.
Chunking also aids retrieval, allowing us to return the matching passage instead of the entire document, which is a lot friendlier for the users.
The simplest chunking method is splitting each document into equal size chunks with some overlap. The overlap helps when the split happens mid-sentence:
# Split a document into chunksdef chunk_text(text, size=500, overlap=100): step = size - overlap
for start in range(0, len(text), step): chunk = text[start:start + size]
if chunk.strip(): yield chunkembeddingcopy link
The chunks are then sent to the embedding model to transform text into embedding vectors. In this tutorial we use all-MiniLM-L6-v2 from Hugging Face:
# Load embedding modelfrom sentence_transformers import SentenceTransformermodel = SentenceTransformer("all-MiniLM-L6-v2")The model is automatically loaded by the SentenceTransformer Python package:
/**/ Sentence Transformers (a.k.a. SBERT) is the go-to Python module for using and training state-of-the-art embedding and reranker models and unlocks a wide range of applications, including semantic search, semantic textual similarity, and paraphrase mining.
The indexing script included in the companion repository loads all markdown files in the /documents folder, splits them into chunks, and encodes each chunk into an array of vectors:
# Chunk all markdown documentsfrom pathlib import Path
for path in sorted(Path("documents").rglob("*.md")): text = path.read_text(encoding="utf-8")
chunks = list(chunk_text(text))
if not chunks: continue
# Convert chunks to vectors vectors = model.encode(chunks).tolist()indexingcopy link
All that remains is to insert a list of vectors for each chunk into the database, and record some metadata about each chunk, such as the name of the document where its content originated.
The Chroma database is created in the same folder as the script by using the chromadb Python package:
# Create index database and collectionimport chromadb
db = chromadb.PersistentClient(path="./database")collection = db.get_or_create_collection("documents")The encoded array of vectors for each chunk is inserted into the database:
# Store vectors and source textcollection.upsert( ids=[f"{path}:{i}" for i in range(len(chunks))], embeddings=vectors, documents=chunks, metadatas=[{"source": str(path)} for _ in chunks],)After following the instructions in readme for the companion repository, you will see the Chroma database created in the /database folder. This includes a .sqlite database that can be opened in a tool like DBeaver as well as some additional files.
retrievalcopy link
The retrieval script included in the companion repository queries the same Chroma database created in the previous step:
# Text to search forquery = "psychology"
# Load vector databaseimport chromadbdb = chromadb.PersistentClient(path="./database")collection = db.get_or_create_collection("documents")
# Load embedding modelfrom sentence_transformers import SentenceTransformermodel = SentenceTransformer("all-MiniLM-L6-v2")
# Encode query text into vectorsvectors = model.encode(query).tolist()
# Find matching vectorsresults = collection.query( query_embeddings=[vectors], n_results=5, include=["documents", "metadatas"],)The results are then printed to the output. This script can be piped into a file to conveniently view the results as markdown:
python ./search.py > results.mdserverless databasecopy link
Using a local database like Chroma lets you experience vector databases first-hand, without hiding too many implementation details behind a cloud service.
Indexing with a cloud service like Pinecone works much the same way. Cloud providers often integrate other related operations like embedding directly into their service, which lets us skip this step.
The indexing script included on pinecone branch of the companion repository looks very much like the Chroma script, but uses the Pinecone SDK, with API key set as an environment variable:
export PINECONE_API_KEY="your-api-key"The chunks are then uploaded to Pinecone in the same indexing loop:
# Index documents with Pineconefrom hashlib import sha256from pathlib import Path
# For each documentfor path in sorted(Path("documents").rglob("*.md")): # Split into chunks chunks = list(chunk_text(path.read_text(encoding="utf-8")))
if not chunks: continue
# Calculate checksum path_id = sha256(str(path).encode("utf-8")).hexdigest()
# Upload to Pinecone result = index.documents.batch_upsert( namespace="documents", documents=[ {"_id": f"{path_id}:{i}", "text": chunk, "source": str(path)} for i, chunk in enumerate(chunks) ] )ragcopy link
Once the documents are indexed, all that remains is orchestrating the user’s interaction with the chat LLM. This involves transforming their question into embedding vectors, querying the database, and injecting the search results (chunk contents and source file names) into the prompt.
The Retrieval Augmented Generation script in the companion repository then talks to a local LLM running in LM Studio, providing a System Prompt and injecting the retrieved chunks into the User Prompt.
# Connect to Chroma databaseimport chromadb
db = chromadb.PersistentClient(path="./database")collection = db.get_collection("documents")
# Load embedding modelfrom sentence_transformers import SentenceTransformerembedding_model = SentenceTransformer("all-MiniLM-L6-v2")
# Load chat modelfrom openai import OpenAIchat_model = OpenAI( base_url="http://127.0.0.1:1234/v1", api_key="lm-studio")
# Inputquestion = input("Question or topic: ").strip()
# Embedvector = embedding_model.encode(question).tolist()
# Retrieveresults = collection.query( query_embeddings=[vector], n_results=3, include=["documents", "metadatas"])
# Assemblecontext = "\n\n".join( f"Source: {metadata['source']}\n{text}"
for text, metadata in zip( results["documents"][0], results["metadatas"][0], ))
# Load system promptfrom pathlib import Pathsystem_prompt = Path(__file__).with_name("SYSTEM_PROMPT.md").read_text(encoding="utf-8").strip()
# Outputresponse = chat_model.chat.completions.create( model="google/gemma-3-12b", messages=[ { "role": "system", "content": system_prompt, }, { "role": "user", "content": ( f"Question or topic: {question}\n\n" f"Document excerpts:\n{context}" ), }, ],)
print(response.choices[0].message.content)Not surprisingly, the same script on the pinecone branch of the companion repository only differs in the simplicity resulting from using an integrated online service:
# Load chat modelfrom openai import OpenAIchat_model = OpenAI( base_url="http://127.0.0.1:1234/v1", api_key="lm-studio")
# Inputquestion = input("Question or topic: ").strip()
# Retrievefrom pinecone import EmbedQuery, Pinecone
results = Pinecone().index("vector").documents.search( namespace="documents", top_k=3, score_by=[EmbedQuery(field="text", query=question)], include_fields=["text", "source"],)
# Assemblecontext = "\n\n".join( f"Source: {match.source}\n{match.text}" for match in results.matches)
# Load system promptfrom pathlib import Pathsystem_prompt = Path(__file__).with_name("SYSTEM_PROMPT.md").read_text(encoding="utf-8").strip()
# Outputresponse = chat_model.chat.completions.create( model="google/gemma-3-12b", messages=[ { "role": "system", "content": system_prompt, }, { "role": "user", "content": ( f"Question or topic: {question}\n\n" f"Document excerpts:\n{context}" ), }, ],)
print(response.choices[0].message.content)closingcopy link
In this tutorial we’ve looked at using modern, open-source vector databases to easily replicate what used to be hours of work in setting up a complex deployment like Elasticsearch, or using advanced algorithms like BM25 to roll your own full-text search.
This sample code in this tutorial can be run on most Mac and Windows computers using a local vector database called Chroma.
For practical deployment in production, the code can be easily adopted to Pinecone, an online serverless vector database with a generous free tier that doesn’t ask for a credit card, making it a good choice for prototypes and personal projects.