Skip to content
NLEN
Illustration: Setting up a local vector database: Qdrant and Chroma

Setting up a local vector database: Qdrant and Chroma

By Ivo Donker — compiled with AI support (Claude & Gemini) · Last updated: 6 August 2026

When do you need a dedicated vector database?

When you start with Retrieval-Augmented Generation (RAG) or semantic search, the first instinct is often to install a vector database right away. In many cases, however, this is an example of premature optimization. Before you introduce the complexity of a separate database system and the associated network architecture, you should determine whether the size of your specific dataset and use case actually justify it. Setting up an additional service always brings administrative overhead, extra points of failure, and resource usage.

If you have, for example, fewer than 1,000 documents or text fragments (such as a personal collection of manuals, a small archive of blog posts, or a handful of internal PDF files), a dedicated vector database is often unnecessary. In such cases, you can store the vectors directly in your Python script's working memory using basic libraries like Numpy or Faiss, or simply write them to a flat JSON file on disk. Another excellent and often overlooked option is to use a SQLite extension, such as sqlite-vss or the newer sqlite-vec. This lets you run semantic searches within a familiar, file-based relational database without extra network overhead or container management. This is ideal for desktop applications or small-scale scripts that search documents locally with AI, as described in detail in the guide on searching documents locally. SQLite solutions are easy to back up and don't require background processes that continuously consume RAM.

When is a dedicated vector database worthwhile, then? The need arises when you're dealing with at least tens of thousands of document chunks, when your database needs to constantly receive live updates (think real-time additions, changes, or deletions by multiple users), or when you need to perform complex filtering operations based on metadata. Dedicated vector databases are specifically optimized for quickly searching high-dimensional spaces, efficiently indexing new vectors via algorithms like HNSW (Hierarchical Navigable Small World), and combining vector searches with traditional filters (such as date ranges or categories). If multiple applications or scripts need simultaneous access to the same index via a standardized API (HTTP or gRPC), a separate database setup is the most robust choice.

The practical differences between Chroma and Qdrant

If you decide you do need a database, Chroma and Qdrant are the most obvious choices for local networks and home setups (such as on a Mac mini, an Intel NUC, or a consumer NAS). Although they serve the same purpose — storing and searching vectors — they differ considerably in architecture, resource usage, and ease of deployment. A general overview of these and other systems can be found in the comparison of vector databases in the directory.

Chroma was designed with simplicity and quick integration as its starting point. It originated as a Python library that can run embedded in the same process as your application. This means you don't need to configure a separate server; the data is stored in a local folder in a SQLite database and a few flat files. However, Chroma can also be run as a standalone server via Docker. For a home setup, Chroma is extremely low-threshold and requires almost no configuration. Performance can decline, however, once the scale grows and complex filters or concurrent write operations come into play, because SQLite can limit concurrency under the hood.

Qdrant, on the other hand, is written in Rust. It is designed as a standalone, high-performance service that runs exclusively as a separate database (usually via Docker). Qdrant is extremely efficient with memory and processing power. The engine is capable of managing millions of vectors with minimal resources and offers advanced filtering capabilities directly in the index engine. For a home setup on a NAS or a server that runs 24/7, Qdrant offers more stability, predictability under load, and better integration options with programming languages other than Python, although the initial configuration requires a bit more networking knowledge.

Property Chroma (Embedded) Qdrant (Docker)
Language Python / C++ Rust
Architecture In-process or Server Client-Server
Data storage SQLite + Parquet/files Proprietary on-disk storage format
Resource usage Low at start, rises with large datasets Very low and predictable (Rust)
Network protocols HTTP / REST HTTP / REST and gRPC

Setting up Chroma: embedded versus server

Chroma offers the flexibility to run directly from a Python script. This makes it ideal for prototyping and small-scale projects. The simplest way to use Chroma embedded is via the PersistentClient. Here you specify a folder where Chroma should write the data.

import chromadb

# Initialiseer de embedded client met persistente opslag
client = chromadb.PersistentClient(path="/absolute/pad/naar/chroma_data")

# Maak een collectie aan of haal een bestaande op
collection = client.get_or_create_collection(name="lokale_documenten")

The persistent folder (in this example /absolute/pad/naar/chroma_data) contains a SQLite database (chroma.sqlite3) and folders for the vector indexes (usually HNSW files). When backing up an embedded Chroma installation, it's crucial that there are no active write operations. Because SQLite uses file locking, a backup taken during a write operation can result in a corrupted database file. The safest method is to fully stop the Python script and then copy or archive the entire folder.

If you want to run Chroma as a standalone server, for example to allow access from multiple scripts on your network or from different machines, you can start the database via Docker:

docker run -d \
  -p 8000:8000 \
  -v /absolute/pad/naar/chroma_data:/chroma/chroma \
  -e IS_PERSISTENT=TRUE \
  --name chromadb \
  chromadb/chroma:latest

In this server mode, your Python script connects via chromadb.HttpClient(host="127.0.0.1", port=8000). Keep in mind that Chroma's server mode is essentially an API shell around the embedded database. The underlying concurrency limitations of SQLite remain in place to some extent, making this setup less suitable for heavy parallel write workloads.

Setting up Qdrant via Docker

Qdrant is specifically designed to run as a network service. For a home setup, Docker is the most stable and configurable method. Running databases in containers requires specific attention to volume mapping to prevent data loss when updating the container. For broader context on running AI tools in containers, see also the article on running an LLM in Docker.

A basic docker-compose configuration for Qdrant looks like this:

version: '3.8'

services:
  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant_local
    ports:
      - "6333:6333"
      - "6334:6334"
    volumes:
      - /opt/qdrant/storage:/qdrant/storage
      - /opt/qdrant/config.yaml:/qdrant/config.yaml
    restart: unless-stopped

By default, Qdrant uses two network ports, each with its own purpose:

It's essential to map the storage folder (/qdrant/storage inside the container) to a folder on the host, such as /opt/qdrant/storage. Qdrant writes its collections, vectors, metadata, and index files here. If you don't configure this volume, the data is stored in the container's temporary write layer. This means all data disappears as soon as you stop, update, or remove the container. By keeping the storage folder outside the container, the data remains safely preserved and you can easily update the Qdrant image to a newer version by destroying and restarting the container.

Creating and configuring collections

Within a vector database, vectors aren't stored in arbitrary folders but in structured units called "collections" (or indexes). When creating a collection, you need to define two crucial parameters: the vector dimension and the distance metric.

The vector dimension is the length of the number sequence generated by your embedding model. Each model has a fixed dimension. For example, the popular model all-MiniLM-L6-v2 generates vectors with a dimension of 384, while larger models like text-embedding-3-large can go up to 1,536 or even 3,072 dimensions. It's technically impossible to store vectors of different dimensions in the same collection. You must therefore configure the collection exactly to match the model you're using. For a detailed overview of available models, see the overview of embedding models compared.

The distance metric determines how the database calculates the similarity between two vectors. The most commonly used metrics are:

The choice of distance metric belongs with the specifications of the embedding model. If a model was trained with cosine similarity, using L2 distance will lead to suboptimal search results. Always check the documentation of the chosen model before creating the collection, therefore.

Here's an example of how to create a collection in Qdrant with the Python client, configured for a model with 384 dimensions and Cosine similarity:

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams

client = QdrantClient(url="http://localhost:6333")

client.create_collection(
    collection_name="lokale_kennisbank",
    vectors_config=VectorParams(
        size=384, 
        distance=Distance.COSINE
    )
)

Metadata fields and filtering

A common mistake when setting up a local RAG environment is storing only the vectors and the associated raw text. While this suffices for simple searches, you'll quickly run into limits as the database grows and you want more control over the search results. It is therefore highly recommended to store relevant metadata fields right from the start of indexing.

The most important metadata fields you should include from the start are:

Storing this metadata allows you to apply pre-filtering. With pre-filtering, the database first selects all documents that meet the metadata criteria (for example category == 'handleidingen'), and only then performs the vector search on that specific subset. This is many times faster and more accurate than post-filtering, where you first retrieve the top 100 vectors and only afterward filter out the unwanted documents (which can result in too few usable results if the top 100 happens to contain no documents from the desired category).

In Qdrant you can define complex filters using filter objects:

from qdrant_client.models import Filter, FieldCondition, MatchValue

results = client.search(
    collection_name="lokale_kennisbank",
    query_vector=[0.1, 0.2, -0.3, 0.4], # Je query-embedding (voorbeeld)
    query_filter=Filter(
        must=[
            FieldCondition(
                key="category",
                match=MatchValue(value="handleidingen")
            )
        ]
    ),
    limit=5
)

Reindexing and model switches

The world of embedding models is evolving quickly. It's likely you'll want to switch to a newer, more accurate model in the future. This is where a fundamental characteristic of vector databases comes into play: vectors are not interchangeable between different models.

If you switch from, for example, an older MiniLM model to a larger model from BGE or Nomic, you need to rebuild the entire database. The new model's vector space is structured completely differently; a vector from model A has no semantic relationship whatsoever to a vector from model B, even if the dimensions happen to be identical. The database cannot convert the old vectors to the new space.

To avoid chaos, it's wise to record the model name and its associated parameters directly in the collection name (for example documenten_nomic_v1.5 instead of simply documenten). This prevents you from accidentally mixing embeddings from different models into this index, which would result in error messages or completely random search results.

In addition, a robust reindexing pipeline is essential. Since you can't simply convert the database to a new model, you need to re-read the source documents, generate chunk segments, run them through the new embedding model, and write them to a new collection. More information about the lifecycle and management of such indexes can be found in the guide on vector index maintenance.

System load and resource management

Running a vector database locally on consumer hardware such as a Mac mini, an older Intel NUC, or a Synology NAS with an ARM processor requires realistic expectations regarding system load and resource management. For Apple Silicon users, the specific optimization for macOS may be relevant, as discussed in the guide on local RAG on the Mac.

The memory usage (RAM) of a vector database is primarily determined by the indexing method. If you use an HNSW index (the default in both Qdrant and Chroma for fast searches), the vectors are loaded into RAM to enable fast graph searches. You can estimate the memory required for the raw vectors based on the number of vectors, the dimension, and the precision used. For float32 vectors, you use 4 bytes per dimension.

Memory estimate (raw vectors):
Number of vectors × Dimensions × 4 bytes

For 100,000 vectors with 768 dimensions, that amounts to about 307 MB of pure vector memory. The HNSW index itself, however, introduces significant overhead, often ranging from 50% to 100% extra memory usage. Qdrant offers options to reduce this footprint by storing vectors on disk (on-disk payload) or by applying quantization (for example Scalar Quantization), which can reduce memory usage by as much as 75% at the cost of a minor deviation in search accuracy.

A common problem in home setups is the database crashing due to import batches that are too large. When you try to index tens of thousands of documents at once, memory can fill up quickly. This leads to Out-Of-Memory (OOM) errors, where the operating system forcibly shuts down the database container or the Python process. It is therefore recommended to offer the data in batches of at most 100 to 500 vectors at a time and to give the database time to process the indexing and free up memory after each batch.

Backup, recovery, and network security

A good backup strategy for a local vector database consists of two parts: securing the physical database files and preserving the original data sources.

While it's tempting to back up only the storage folder of Chroma or Qdrant, in practice this is often insufficient. Vector indexes can become corrupted due to power outages or unexpected container crashes. Moreover, if you change the embedding model, you're forced to reprocess all documents anyway. The most important backup is therefore the export of the original source documents including their metadata. Store this in a standardized format (such as JSON-lines or an SQL dump) in a separate, secure backup folder. Should the vector database become corrupted or should you need to switch to a different system, you can always rebuild the index from scratch using your reindexing pipeline.

For the physical backup of Qdrant, you can use the built-in snapshot functionality. This creates a consistent backup of the database without you needing to stop the service:

# Maak een snapshot van een specifieke collectie via de REST-API
curl -X POST "http://localhost:6333/collections/lokale_kennisbank/snapshots"

The resulting .snapshot file can be safely copied to an external backup location.

Network security in the home environment

When you start Docker containers, it's tempting to open up ports to the entire network. By default, many configurations listen on 0.0.0.0, which means the database is reachable from every device on your local network (and potentially from the internet if your router is misconfigured or the host has a public IP address).

Vector databases like Chroma and Qdrant don't have authentication enabled by default. This means anyone on the network can view, modify, or delete your data. Restrict access as follows, therefore:

  1. Bind to localhost: If the database is only used by scripts on the same machine, bind the ports explicitly to 127.0.0.1 instead of 0.0.0.0: 
    ports:
  - "127.0.0.1:6333:6333"
  - "127.0.0.1:6334:6334"
  2. Enable API keys: If you're accessing Qdrant over the network, enable the built-in API key protection by setting the environment variable QDRANT__SERVICE__API_KEY in your Docker configuration. For Chroma, you can enable authentication via the configuration files using static API keys.

By combining these measures with a well-structured reindexing and backup setup, you create a stable and securely managed storage layer for all your local AI projects.

Further reading