How to Choose a Vector Database for Production RAG

Introduction

Learning how to choose a vector database for production RAG is now one of the most consequential infrastructure decisions an AI team makes. Retrieval-augmented generation was introduced in a 2020 research paper as a way to pair a language model with a dense vector index of external documents. Since then the retrieval layer has grown from a research component into the part of the stack that decides whether answers are accurate, fast, and affordable. Scale makes the stakes concrete, because Notion reported scaling its vector search roughly tenfold while cutting costs by about ninety percent over two years. A wrong choice shows up later as slow queries, recall that quietly decays, and monthly bills that surprise finance. This guide walks through index types, filtering, hybrid search, cost, and operations so that your decision rests on evidence rather than marketing. It closes with a seven-step selection process, real deployments, and answers to the questions teams ask most often.

Quick Answers on Choosing a Vector Database

How do you choose a vector database for production RAG?

Knowing how to choose a vector database for production RAG starts with testing recall, latency, filtering, and cost on your own data. Start with Postgres if your scale is modest.

Is pgvector a good vector database choice for production RAG?

pgvector suits production RAG when vectors number in the low millions and you already run Postgres, because HNSW indexes, SQL filtering, and one operational surface keep the vector database simple.

What matters most when you choose a vector database?

Recall at your latency target matters most, followed by filtered search quality, hybrid retrieval support, and total cost. Benchmark candidate vector databases with your own embeddings before committing to production RAG.

Key Takeaways

• Knowing how to choose a vector database for production RAG means measuring recall, latency, filtered search, and cost on your own documents, because public benchmarks rarely match your workload.
• HNSW graphs dominate production search, while quantization can shrink memory by four to thirty-two times if you verify recall with rescoring.
• Metadata filtering and hybrid keyword plus vector search separate engines more sharply than raw speed does.
• Postgres with pgvector is a legitimate starting point for modest scale, and purpose-built engines earn their place when scale, filtering, or operations demand it.

What Is a Vector Database for Production RAG?

Learning how to choose a vector database for production RAG means matching a retrieval engine’s index type, filtering, scale, and operating cost to your workload so answers stay accurate, fast, and affordable.

Why Vector Search Sits at the Center of Production RAG

Every retrieval-augmented system answers a question in two stages, and the first stage decides the quality ceiling of the second. The retriever finds passages that look relevant, and the language model can only reason over what the retriever hands it. If the right passage never reaches the prompt, no amount of prompt engineering or model size will recover the missing fact. This is why teams that debug disappointing answers usually find the root cause in indexing and search rather than in generation. Debugging therefore starts with the retrieved passages, because they reveal whether the model was ever given a chance to answer correctly.

A vector database is the component that makes the first stage fast enough to run on every user request. It stores embeddings, which are lists of numbers that place each chunk of text in a geometric space where similar meanings sit close together. At query time the system embeds the question, finds the nearest stored vectors, and returns the matching chunks. Brute force comparison works for a few thousand chunks, but production corpora reach millions or billions, and that scale forces the use of specialized indexes. Specialized indexes avoid comparing the query against every stored vector, which is what keeps latency in the tens of milliseconds at large scale.

Choosing the retrieval engine therefore shapes accuracy, latency, and cost at the same time, which is why the decision deserves a structured process. Our guide on retrieval-augmented generation versus fine-tuning explains why most knowledge-heavy applications start with retrieval instead of retraining. Once that choice is made, the vector store becomes the system of record for everything the assistant is allowed to know. Replacing it later means re-indexing the corpus, rewriting query code, and re-validating quality, so early care pays for itself. Anyone studying how to choose a vector database for production RAG should treat the retriever as a first-class component rather than a plumbing detail. Treat the database as long-lived infrastructure rather than a library you can swap in an afternoon, and plan the evaluation accordingly.

Embeddings, Dimensions, and What a Vector Store Actually Stores

Building on that foundation, it helps to be precise about what the database holds. Each record usually contains a vector, an identifier, the original text or a pointer to it, and a set of metadata fields such as source, author, date, and access group. The vector comes from an embedding model, and its length is fixed by that model, commonly a few hundred to a few thousand numbers. Our primer on what word embeddings are covers how those numbers capture meaning in the first place. Metadata is just as important as the vector, because filtering and access control depend on it. Plan the schema before loading data, since adding fields later can force a re-index.

Dimension count is the single biggest driver of storage and memory cost, so it belongs in the selection conversation from day one. A single float32 vector at 1,536 dimensions occupies about six kilobytes before any index overhead, and a hundred million of them need roughly six hundred gigabytes. The pgvector documentation states that its standard vector type uses four bytes per dimension plus eight bytes, while the half-precision type uses two bytes per dimension. Halving precision therefore halves raw storage, and the recall cost is often small enough to accept after testing. Keep this arithmetic in a spreadsheet during planning so that every proposed architecture is checked against real memory numbers.

Approximate Nearest Neighbor Indexes: HNSW, IVF, and Disk-Based Graphs

Turning to the search structures themselves, nearly every production system uses approximate nearest neighbor search, which trades a little accuracy for a very large speedup. The most widely deployed index is the hierarchical navigable small world graph, usually shortened to HNSW. The original HNSW paper describes a multi-layer stack of proximity graphs and reports that search cost scales logarithmically with the size of the dataset. Searches start at a sparse top layer, move greedily toward the query, and descend into denser layers until the closest neighbors are found. Their appeal is that recall stays high while the number of distance calculations stays small, even for very large collections.

Inverted file indexes, called IVF, take a different route by clustering vectors and searching only the clusters nearest to the query. They build faster and use less memory than graphs, but they need a training step and usually give lower recall at the same speed. Product quantization is often layered on top of IVF to compress vectors further. Teams with strict memory budgets and rarely changing data still find IVF variants attractive. The practical rule is that graph indexes favor quality and speed, while clustered indexes favor frugality and fast builds.

Disk-based graph indexes such as DiskANN keep most of the structure on solid state drives and only a compressed copy in memory. They let a single machine serve datasets that would otherwise need a large memory cluster. The cost is higher and less predictable latency, because some reads touch storage instead of RAM. If your corpus is huge and your latency budget is measured in hundreds of milliseconds, a disk-backed index can cut hardware cost sharply. Cost and latency should be modeled together, because a cheaper index that misses the latency target is not actually cheaper.

Every index exposes build and search parameters that move the same three dials: recall, speed, and memory. For HNSW the key knobs are the number of links per node, the effort spent during construction, and the effort spent during each search. Higher values raise recall and cost, and the right setting depends on your data rather than on any universal default. That is why the later benchmarking section insists on measuring your own workload. Document the chosen values and the reasons for them, so that future engineers can retune the index when the data or the traffic changes.

Recall, Latency, and the Trade-Offs You Cannot Escape

Stepping back from individual indexes, the core tension in vector search is that exact answers are expensive and fast answers are approximate. Recall measures how many of the true nearest neighbors your index actually returns, and it is the metric that connects infrastructure to answer quality. A recall of ninety-five percent at ten results means that on average nine and a half of the ten best chunks are found. The missing chunk may or may not matter, which depends on how redundant your content is. Setting a recall floor in advance prevents endless debate, because any configuration that falls below the floor is rejected without discussion.

Latency is usually reported as a percentile, because averages hide the slow tail that users actually feel. A p50 value tells you the typical query, while p95 and p99 tell you how bad the worst experiences are. Production RAG adds an embedding call and a generation call around retrieval, so the vector search budget is often a small slice of a multi-second request. Set the retrieval latency target from the end-to-end experience you promise, not from a vendor benchmark chart. Tail latency matters most when retrieval runs several times per request, as it does in agent loops that search repeatedly before answering.

The three dials interact in ways that surprise newcomers, because pushing recall higher at a fixed hardware size slows queries and pushing throughput higher at fixed recall requires more machines. Vendors choose whichever operating point flatters their product, so a published chart rarely transfers to your workload. The BEIR benchmark paper evaluated ten retrieval systems across eighteen datasets and found that results vary widely by domain. The lesson is that retrieval quality is empirical, and the only trustworthy number is the one measured on your own documents and questions. Insist on seeing the recall level attached to every speed claim, since a number without its recall is not comparable to anything.

Metadata Filtering and Why It Breaks Naive Setups

Moving on to a topic that catches many teams late, real applications almost never search the whole corpus. Users should see only documents they are permitted to read, from the right tenant, in the right language, within a date range. These conditions are expressed as metadata filters applied alongside the vector query. A system that cannot filter efficiently will either leak data or return too few results, and both failures are serious. Treat security filters as correctness requirements and test them with adversarial queries that try to cross tenant boundaries.

Filtering is where approximate indexes are most likely to disappoint, because the filter and the graph traversal do not naturally cooperate. The pgvector documentation explains that with an approximate index the filter runs after the scan. A condition matching ten percent of rows can therefore return only about four results with default settings. Recent pgvector releases added iterative index scans that keep searching until enough matches are found. Purpose-built engines handle this by integrating filters into the traversal itself, and the quality of that integration is a real differentiator. Ask each vendor to show filtered recall on a query that matches only one percent of the data, because that is where weak designs fail.

Hybrid Search: Combining Keyword and Vector Retrieval

Beyond pure semantic matching, many production systems blend vector search with classic keyword scoring. Embeddings are excellent at paraphrase and intent but can blur exact identifiers such as product codes, error messages, and legal citations. Keyword scoring using BM25 catches those exact terms reliably, which is why search engines have relied on it for decades. The BEIR study cited earlier called BM25 a robust baseline that dense retrievers often fail to beat outside their training domain. Hybrid retrieval therefore covers both meaning and exact wording, which is how people actually phrase questions at work.

The Weaviate hybrid search documentation describes running both searches in parallel and fusing the scores into one ranking. It offers a rank-based fusion that uses only list positions and a relative-score fusion that normalizes and sums the original scores. An alpha setting from zero to one controls the balance, with zero meaning keyword only and one meaning vector only. The documentation advises setting alpha explicitly whenever the weighting matters, because client libraries differ in their defaults. Reciprocal rank style fusion is simple and robust, while score-based fusion preserves how far ahead a strong keyword match really is.

Hybrid retrieval support should be a first-class selection criterion rather than an afterthought. Some engines provide it natively in one query, while others require you to run two systems and merge results in application code. Merging in your own code works but adds latency, complexity, and a second index to keep fresh. A reranking model after retrieval can then sort the combined candidates by true relevance before they reach the prompt. Check whether the engine tokenizes your language well, since keyword quality varies across languages and specialized vocabularies.

Managed Services Versus Self-Hosted Engines

Shifting from algorithms to operations, the first structural decision is whether someone else runs the database. Managed services remove capacity planning, upgrades, backups, and on-call duty from your team. They charge for that convenience through usage-based pricing, and costs can climb quickly with data volume and query rate. Self-hosted engines give you control and often lower unit cost, but every operational task becomes yours. Hybrid models also exist, such as a managed control plane running inside your own cloud account, and they deserve a place on the shortlist.

The honest comparison is total cost of ownership, which includes engineer time as well as the monthly invoice. A small team without database expertise often spends more on a self-managed cluster than on a managed subscription once incident response is counted. A large team with strong infrastructure skills may save substantially by running its own. Data residency rules and private networking requirements can also decide the question before cost enters the picture. Ask finance and security for their constraints early, so that the shortlist already reflects what the organization can approve.

Postgres With pgvector: When the Database You Have Is Enough

Among the open options, the most pragmatic starting point for many teams is the Postgres database they already operate. The pgvector extension adds a vector column type, distance operators, and both HNSW and IVFFlat indexes. Your embeddings then live next to your relational data, so filtering, joins, transactions, and backups come for free. For corpora in the low millions of vectors this single-system approach is hard to beat on simplicity. Teams also value that vector data inherits existing monitoring, access control, and disaster recovery without any new tooling.

The pgvector project documents default HNSW settings of sixteen links per node, a construction effort of sixty-four, and a search effort of forty. It also lists indexing limits of two thousand dimensions for the standard type and four thousand for the half-precision type. Those limits matter if you plan to use very large embedding models. Storing half-precision vectors or applying binary quantization with expression indexes are the documented ways to stay within them. Plan for dimension limits early, because discovering them after choosing an embedding model forces an unwelcome redesign.

The risk with the Postgres route is scale rather than correctness, because index builds and memory pressure grow with the dataset. HNSW indexes need to fit mostly in memory to perform well, so the database server must be sized for the vectors rather than for the relational data alone. Heavy vector workloads can also compete with transactional traffic on the same instance. Many teams solve this with a dedicated read replica for retrieval, which keeps the operational model familiar. Load test with the full expected corpus rather than a small sample, since behavior at ten million vectors can differ sharply from behavior at one million.

Extensions built on top of pgvector push the ceiling higher by adding disk-friendly indexes and compression. The next sections examine measured results, but the strategic point is that Postgres is a legitimate production option rather than a toy. Starting there also preserves a clean exit, because embeddings can be exported and loaded into a specialized engine later. A decision that keeps options open is usually worth taking when the future load is uncertain. That flexibility is the main reason experienced engineers rarely regret starting with the database they already trust.

Purpose-Built Engines: Pinecone, Qdrant, Weaviate, and Milvus

On the dedicated side, a handful of engines dominate production conversations, and each has a distinct personality. Pinecone is a fully managed service whose serverless design separates storage from compute and bills by usage. Qdrant is an open-source engine written in Rust that emphasizes filtering quality, quantization, and efficient memory use. Weaviate combines vector search with built-in hybrid retrieval and an object model, while Milvus targets very large distributed deployments with many index types. Pricing models, licensing terms, and cloud availability differ as well, so the differences go well beyond the index algorithm.

Choosing among these engines is less about a universal winner than about matching strengths to requirements. A team that wants zero operations and accepts vendor pricing leans toward a managed service. A team with heavy filtering needs or tight memory budgets examines quantization and payload indexing closely. A team planning billions of vectors looks at distributed architectures, storage tiering, and multi-tenancy controls. Each of these choices narrows the field before any benchmark is run, which saves weeks of effort.

Evaluate each engine against your own non-negotiables first, then use benchmarks only to break ties. Write down your maximum acceptable latency, minimum recall, expected corpus size, filter patterns, and compliance constraints before opening any vendor page. Engines that fail a hard requirement drop out immediately, which shortens the shortlist to two or three candidates. Those survivors then go through the hands-on benchmark described later in this guide. Share the requirements sheet with every vendor so that demonstrations address your real workload instead of a generic showcase.

Scaling Strategies: Sharding, Replication, and Multi-Tenancy

Looking at growth, every vector workload eventually outgrows a single machine in memory, throughput, or availability. Sharding splits the data across nodes so each holds a slice of the index, while replication copies slices so queries can be served in parallel and failures are survivable. Both add coordination cost, and both change how filters and ranking behave across nodes. Ask any candidate how it merges partial results and how it behaves when one shard is slow. Test failure scenarios deliberately by removing a node during a benchmark and watching how recall and latency respond.

Multi-tenancy deserves special attention because many RAG products serve thousands of customers whose data must never mix. One pattern gives each tenant its own namespace or collection, which isolates data cleanly but can waste resources when most tenants are tiny. Another pattern stores all tenants together with a tenant identifier filter, which is efficient but depends entirely on filter correctness. Engines that keep cold tenants in cheap storage and load them on demand can change the economics dramatically. Whichever pattern you choose, write automated tests that prove one tenant can never retrieve another tenant’s documents.

Quantization and the Economics of Memory

Building on the memory discussion, quantization is the most powerful lever for reducing vector database cost. It stores each number with fewer bits, which shrinks the index and speeds distance calculations. The Qdrant quantization documentation reports about a four times memory reduction for scalar quantization and a thirty-two times reduction for binary quantization. It adds that binary quantization can reach up to a forty times speedup, with accuracy that depends on the embedding model. These figures describe one vendor’s implementation, so confirm them on your own embeddings and hardware before planning around them.

Accuracy loss is manageable when you use oversampling and rescoring, which fetch extra candidates from the compressed index and re-rank them with the original vectors. The same documentation reports recall of 0.9966 for a 3,072-dimension model with three times oversampling and rescoring. It also warns that one-bit compression loses significant precision for vectors under roughly a thousand dimensions. Treat quantization as a tested configuration rather than a default, and measure recall before and after on your own queries. Product quantization reaches higher compression still, with documentation quoting up to sixty-four times, but it slows distance calculations and costs more accuracy.

The economic effect is large because memory is the most expensive resource in a vector cluster. A hundred million vectors at 1,536 dimensions need about six hundred gigabytes in float32 but only about 150 gigabytes with scalar quantization. That difference can mean the gap between a single large node and a multi-node cluster. Teams often discover that quantization pays for the whole evaluation effort. Hardware cost scales roughly with memory, so every halving of the footprint tends to cut the monthly bill by a similar share.

Compression also interacts with the embedding model you choose, since some models tolerate low precision better than others. Models trained with this in mind produce vectors whose quality survives aggressive quantization. When you compare embedding providers, test them at the precision you plan to deploy. Our article on generative AI cost per token offers a wider budgeting frame for those decisions. A small recall difference between embedding models can outweigh a large difference in database cost, so evaluate them together.

How Much Does a Vector Database Cost to Run

Turning to money, a credible cost model separates four components that vendors often blur together. Storage covers the bytes held at rest, memory covers the portion that must stay hot, compute covers query and indexing work, and egress covers data moved across networks. Each component scales differently with corpus size, query rate, and update rate. A model that tracks all four will reveal which pricing structure fits your workload. Query rate, which is often forgotten, drives compute spending in usage-priced services and sets the size of the cluster in self-hosted ones.

Usage-based managed pricing is cheap at low volume and can become expensive at high query rates, while fixed-capacity clusters are the reverse. A workload with steady heavy traffic often favors reserved capacity, and a spiky workload with long idle periods favors serverless pricing. Build the model from your projected queries per second and corpus growth rate rather than from list prices alone. Run three scenarios, a conservative one, an expected one, and an aggressive one, to see where each option breaks. Review the model every quarter, because traffic growth and new features can shift the cheapest option over time.

Do not forget the cost outside the database, because embedding generation can dominate. Every document you index and every query you serve calls an embedding model, and changing models later forces a full re-embedding of the corpus. Notion’s engineering team reported that avoiding needless re-embedding after small edits cut its data volume by seventy percent. Our guide to enterprise AI cost optimization strategies places these line items in the broader budget. Caching embeddings for repeated queries and for unchanged documents is one of the cheapest optimizations available.

Finally, you should price the engineering effort honestly in every scenario. Migration projects, index tuning, and on-call burden rarely appear on vendor calculators, yet they often exceed the infrastructure bill for smaller teams. A cheaper engine that needs weekly attention can cost more than a pricier service that needs none. Include a realistic estimate of staff hours per month in every scenario you compare. Revisit the estimate after the first month in production and compare it with actual invoices and incident logs.

Putting Candidates to the Test on Your Own Data

Moving on to implementation, the most reliable way to choose is to measure candidates with your documents, your embeddings, and your questions. Public leaderboards use generic datasets and idealized hardware, and they rarely reflect your filter patterns or update rates. A one-week bake-off with a representative sample gives far better evidence than weeks of reading comparisons. The how-to section later in this guide turns this idea into a concrete procedure. Document the dataset, queries, and settings so that results can be reproduced when a vendor ships a new release. Teams that learn how to choose a vector database for production RAG by measuring, rather than by reading brochures, make fewer expensive reversals.

A good benchmark set pairs real user questions with human-labeled relevant passages, so you can compute recall and ranking quality directly. If labels do not exist yet, a small team can label a few hundred questions in a day or two and generate more with a language model for coverage. Include hard cases such as rare terms, near-duplicate documents, and questions that require a filter. Our walkthrough on evaluating retrieval pipelines with RAGAS shows one way to score the end-to-end result. Keep the judgment criteria fixed before running tests, so that surprising results are not rationalized after the fact.

Measure latency under realistic concurrency instead of a single sequential client. Run warm and cold caches, because the first queries after a restart can be many times slower. Record index build time and memory use as well, since they drive operating cost and recovery time. Report percentiles, not averages, and keep the hardware identical across candidates. Finally, test failure and recovery, because a database that restores in minutes is worth more than one that is slightly faster but takes hours to rebuild.

Operational Concerns: Freshness, Re-Embedding, and Upgrades

In practice, a vector database is never finished, because documents change and so does the embedding technology. Freshness measures how quickly an edit becomes searchable, and the requirement ranges from seconds for chat history to hours for a document archive. Engines differ in how they handle continuous writes while serving reads. Some rebuild segments in the background, and others pause or degrade during heavy ingestion. Ask vendors how deletes are handled, since some engines mark records and reclaim space later, which affects both cost and compliance.

Plan for an embedding model change from the beginning, because a better model will arrive within a year and you will want it. A change means re-embedding every document and often rebuilding the index, so the migration path needs a blue-green strategy with two indexes serving in parallel. Store the model name and version alongside every vector so mixed generations can be detected. Our explainer on context rot in large language models shows why retrieving less but better context keeps quality high. Rehearse the migration on a copy of production data and measure how long a full rebuild actually takes before you need it.

Security, Privacy, and Compliance Risks

Despite the focus on performance, the largest production incidents in retrieval systems come from security failures. Embeddings are not anonymous, and research has shown that original text can sometimes be partially reconstructed from them. Treat the vector store with the same classification as the source documents. Encrypt data at rest and in transit, restrict network access, and log administrative actions. Run periodic penetration tests that include the retrieval layer, since attackers increasingly probe the data that feeds language models.

Access control must be enforced at query time on every request, not assumed from the application layer. The safest pattern attaches permission metadata to each chunk and applies it as a mandatory filter in the database. Retrieved text also feeds the model, which creates an injection path when documents contain hostile instructions. Our guide to prompt injection attacks and defenses describes how poisoned content reaches the prompt. Defense in depth means filtering retrieved text, limiting what the model can do with it, and monitoring outputs for signs of manipulation.

Compliance requirements such as data residency and deletion rights should be checked against each candidate before benchmarking begins. A managed service that cannot keep data in a required region is disqualified regardless of speed. A deletion request must reach the vectors, the backups, and any caches, which is easier in systems with clear deletion semantics. Our overview of AI data privacy best practices for businesses lists the governance steps that surround the database. Involve legal and security reviewers before the shortlist is final, because late objections are expensive to resolve.

Ethics: Bias, Consent, and the Right to Erasure

Beyond the technical risks, a retrieval system embodies decisions about whose knowledge is searchable and who benefits. The corpus you index determines which perspectives the assistant can cite. If it over-represents one source type or viewpoint, answers will lean the same way no matter how fair the model is. Audit the corpus for coverage gaps before blaming the model. Include domain experts in the review, since they notice missing viewpoints that engineers and metrics overlook.

Consent and provenance matter because indexing a document copies its meaning into a new system that people may not know exists. Employees, customers, and authors may have expectations about how their writing is reused. Record the origin and permitted use of every source, and give people a path to request removal. Our article on AI bias and fairness metrics offers ways to test whether retrieval results treat groups consistently. Retention policies should specify how long vectors and source text are kept, and who can order their deletion.

Common Failure Modes in Production Retrieval

Looking at what goes wrong in practice, most failures fall into a small set of repeating patterns. Poor chunking splits an answer across two passages so neither is retrieved on its own. Stale indexes serve outdated policies with complete confidence, and users have no way to tell. Embedding drift appears when a provider silently updates a model and new vectors no longer align with old ones. Naming these patterns up front gives the team a checklist to review whenever answer quality drops unexpectedly.

Another frequent failure is silent recall decay as the corpus grows while index settings stay fixed. A configuration that gave ninety-eight percent recall at one million vectors can fall noticeably at fifty million. Schedule a recurring recall check against a labeled query set so degradation is caught by a dashboard instead of by an angry customer. Alert when the score drops by more than a few points. Logging the retrieved chunk identifiers for every answer makes these investigations far faster.

Over-retrieval also hurts, because stuffing the prompt with many marginal passages dilutes the signal. The Lost in the Middle study found that model performance is often highest when relevant information sits at the beginning or end of the context. Return fewer, better chunks and rerank them rather than relying on a large top-k. Our guide on reducing LLM hallucinations in production shows how grounding quality connects to answer reliability. Passing only high-scoring candidates to the model also lowers token cost and shortens response time.

The Future of Vector Databases and Retrieval Infrastructure

Looking ahead, several trends are reshaping the market, and they affect how long a choice will stay valid. Object-storage-first designs separate compute from storage so that idle data costs little and only active data occupies memory. Vector search is also becoming a standard feature of general-purpose databases, which narrows the gap between dedicated engines and incumbent systems. The practical effect is that the decision is shifting from which product has vectors to which product fits the rest of your architecture. Teams that design for these trends today will find later migrations smaller and less risky.

Retrieval itself is also evolving beyond single-vector similarity toward richer, structured approaches. Graph-based approaches combine embeddings with entity relationships, as discussed in our comparison of GraphRAG and traditional RAG. Agents that plan multi-step searches place new demands on latency and on memory architecture, which our piece on AI agent memory architecture explores. Choose infrastructure that supports hybrid search, rich filtering, and cheap re-indexing, because those capabilities will matter more than raw speed. Interfaces are standardizing too, which should make switching engines easier than it was in the early years of the market.

How to Run a Vector Database Selection in Seven Steps

Step 1 – Write down requirements and hard limits

Start with a one-page requirements sheet that every stakeholder signs before any tool is installed. List the expected number of chunks today and in two years, the embedding dimension, and the peak queries per second. State the p95 latency you can tolerate, such as 150 milliseconds. Add the filter patterns you need, including tenant, language, date range, and access group. Record compliance limits like data region, encryption, and deletion deadlines. A candidate that violates any hard limit is removed immediately, which keeps the evaluation short and honest. Share the sheet with finance and security so that nobody raises a new constraint after the benchmark is finished.

Step 2 – Assemble a representative corpus and query set

Sample your real documents so the test data has the same length, formatting noise, and duplication as production. Take at least 50,000 chunks, or the full corpus if it is smaller, and keep the sample stable across every candidate. Gather real user questions from search logs, support tickets, or interviews instead of inventing tidy examples. Include rare-term lookups, ambiguous questions, and queries that need a metadata filter. Remove personal data from the sample unless the test environment is approved to hold it. Record how many of the 50,000 chunks come from each source system so coverage is visible.

Label the relevant chunks for at least 200 questions, using a subject expert for the hardest cases. Store the labels in a simple file so any engine can be scored the same way. If labeling is slow, have a language model propose matches and let a person approve or reject them. A labeled set of this size is enough to detect recall differences of a few points between engines. Keep a separate holdout of 50 questions that nobody tunes against, so the final comparison stays honest.

Step 3 – Fix the embedding model and chunking strategy

Hold the embedding model and chunking rules constant while you compare databases, because changing them mixes two experiments into one. Pick the model you expect to deploy, and record its name, version, and dimension, such as 1,536 for many API models. Chunk by structure, such as headings and paragraphs, with a small overlap so answers are not cut in half. Embed the whole sample once and cache the vectors so every candidate loads identical data. Store the chunk text, identifiers, and metadata in the same cache file for later scoring. Revisit chunking as a separate experiment after the database is chosen.

Step 4 – Stand up each candidate with realistic settings

Install each shortlisted engine on identical hardware, or use managed tiers that match your expected size within 10 percent. Use the vendor’s recommended index type and start from default parameters before tuning. Pin every software version so the results can be reproduced later. The first snippet creates a Postgres table with an HNSW index, using the documented default link and construction settings. Record the build time of the index, since it becomes your recovery time after a failure. Capture the exact hardware, such as 8 vCPUs and 64 GB of memory, in the experiment log.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id bigserial PRIMARY KEY,
  tenant_id integer NOT NULL,
  body text NOT NULL,
  embedding vector(1536) NOT NULL
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

The second snippet creates a Qdrant collection with scalar quantization held in memory, which lets you compare compressed and uncompressed recall in the same engine. Keep the same distance metric across candidates, because cosine and dot product give different rankings for unnormalized vectors. Load the cached vectors in batches and record the total ingest time for each engine. Note the memory and disk usage after the index finishes building. Repeat the load with 2 or 3 batch sizes to see whether ingestion speed affects index quality.

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")
client.create_collection(
    collection_name="chunks",
    vectors_config=models.VectorParams(size=1536, distance=models.Distance.COSINE),
    quantization_config=models.ScalarQuantization(
        scalar=models.ScalarQuantizationConfig(
            type=models.ScalarType.INT8,
            always_ram=True,
        )
    ),
)

Step 5 – Measure recall and latency under load

Compute exact nearest neighbors for your labeled queries by brute force, then score each engine against that ground truth. The helper below calculates recall at k for a single query, and you can average it over the whole set. Run the queries from several concurrent clients at the load you expect in production, and capture p50, p95, and p99 latency. Repeat the run after a restart to see how each engine behaves with a cold cache. Discard the first 100 queries of each run as warm-up unless cold starts matter to your users.

def recall_at_k(approx_ids, exact_ids, k=10):
    """Share of the true top-k neighbors that the index returned."""
    hits = len(set(approx_ids[:k]) & set(exact_ids[:k]))
    return hits / k


scores = [recall_at_k(a, e) for a, e in zip(approx_results, exact_results)]
print(f"mean recall@10: {sum(scores) / len(scores):.3f}")

Sweep the search effort setting to draw a recall versus latency curve for each engine rather than a single point. An engine that wins at one operating point may lose at another, and the curve reveals where each one shines. Track memory use and CPU during the run, because those numbers drive your monthly bill. Record every setting so the experiment can be repeated by a colleague. Compare the curves at your recall floor, for example 95 percent, and choose the engine with the lowest latency or cost there.

Step 6 – Test filters, hybrid queries, and updates

Repeat the measurements with the filters you wrote down in the first step, since filtered recall is where candidates diverge most. The snippet below enables iterative scanning in Postgres so that a tenant filter still returns a full set of results from the HNSW index. Compare that behavior with engines that apply filters during graph traversal, using a filter that matches only 1 percent of rows. Then add hybrid queries that combine keyword and vector scores, and test how each system fuses them. Finish by writing 1,000 new documents while queries run, to measure freshness and the effect of ingestion on latency.

SET hnsw.iterative_scan = relaxed_order;

SELECT id, body
FROM chunks
WHERE tenant_id = 42
ORDER BY embedding <=> $1
LIMIT 10;

Step 7 – Model total cost and make a documented decision

Combine the measured memory, CPU, and storage numbers with vendor pricing to project the monthly cost at your expected scale and at 3 times that scale. Add estimated engineering hours for setup, tuning, and on-call duty, and include the cost of a future migration if you choose a closed platform. Score each candidate on recall, latency, filtering, hybrid support, operations, and cost using weights agreed in the first step. Write a short decision record that lists the winner, the runner-up, and the conditions that would trigger a re-evaluation. Teams that document how to choose a vector database for production RAG this way rarely need to relitigate the decision a year later.

Key Insights

• Binary quantization compresses vectors thirty-two times and can speed search up to forty times, according to the Qdrant quantization documentation, though accuracy depends on the embedding model.
• A Tiger Data benchmark on fifty million vectors reported 28 times lower p95 latency and 16 times higher throughput than Pinecone's storage-optimized index at ninety-nine percent recall.
• On one million 1,536-dimension vectors, Supabase measured HNSW at more than six times the performance of IVFFlat with equal accuracy, while admitting that HNSW builds slower and uses more memory.
• Notion's two-year retrospective links migrations and change detection to a tenfold scale increase alongside a ninety percent cost reduction, although every figure is self-reported.
• The BEIR benchmark tested ten retrieval systems on eighteen datasets and still called BM25 a robust baseline, which is why hybrid retrieval belongs on every shortlist.
• According to the pgvector documentation, a filter matching ten percent of rows can return only about four results from an approximate index with default settings.
• The Lost in the Middle paper found accuracy is often highest when relevant facts sit at the start or end of the context, so fewer, better chunks win.
• Ada's retrieval study found int8 vectors were four times smaller while top-10 recall moved only from 90.7 to 90.6 percent on its production queries.

Taken together, these numbers show that the best vector database is the one that holds recall steady at your latency target for the lowest total cost. Benchmarks from vendors can differ by an order of magnitude depending on the recall level and hardware they choose to report. Compression and index tuning often matter more than the brand on the invoice, which makes hands-on testing the highest-value activity in any selection. Filtering and hybrid search quietly decide user-visible quality because real queries are rarely pure semantic lookups. Operational questions such as re-embedding, migration, and multi-tenancy then determine whether the choice still looks wise two years from now.

Dimension	pgvector on Postgres	Pinecone	Qdrant	Weaviate	Milvus
Deployment model	Extension inside your own Postgres	Fully managed service	Open source or managed cloud	Open source or managed cloud	Open source or managed cloud
Best-fit scale	Low millions of vectors per server	Small to very large, usage priced	Millions to billions with sharding	Millions to large clusters	Large distributed deployments
Index options	HNSW and IVFFlat	Managed, tuned by the vendor	HNSW with payload indexes	HNSW with compression options	HNSW, IVF, and disk-based types
Metadata filtering	SQL filters, with iterative scans for recall	Metadata filters on queries	Filters integrated into graph search	Filters with inverted indexes	Scalar filters with partitions
Hybrid keyword and vector search	Combine with Postgres full-text search in SQL	Sparse plus dense vectors	Sparse vectors with fusion	Native BM25 plus vector fusion	Sparse plus dense with reranking
Compression and quantization	Half-precision and binary expression indexes	Handled internally	Scalar, binary, and product quantization	Several compression options	Several quantization options
Operational burden	Low if you already run Postgres	Lowest, vendor operated	Moderate when self-hosted	Moderate when self-hosted	Highest when self-hosted
Typical trade-off	Memory pressure at large scale	Usage cost and vendor lock-in	Tuning effort for quantization	Resource use of richer features	Complexity of a distributed system

Vector Databases in Practice: Three Measured Setups

Postgres With pgvectorscale Against Pinecone

In practice, Tiger Data ran a benchmark on fifty million 768-dimension Cohere embeddings, described in its published benchmark write-up. It compared PostgreSQL with pgvector and pgvectorscale against Pinecone's storage-optimized and performance-optimized pod indexes. At ninety-nine percent recall the Postgres setup reported p95 latency twenty-eight times lower and throughput sixteen times higher than the storage-optimized index. Against the performance-optimized index at ninety percent recall, the gap narrowed to 1.4 times lower latency and 1.5 times higher throughput. The company priced self-hosting at about $835 per month on AWS against $3,241 and $3,889 for the two Pinecone options. The result is a useful critique of the assumption that dedicated engines always win. The vendor of the Postgres offering ran the test, and the performance-optimized index was not tested at ninety-nine percent recall, so it remains a directional signal.

HNSW Versus IVFFlat on One Million Vectors

Moving to index choice, Supabase used pgvector version 0.5.0 to compare the new HNSW index with the older IVFFlat index, and it published its pgvector HNSW results. On 224,482 vectors of 1,536 dimensions it measured about three times the performance of IVFFlat with better accuracy. On one million vectors of the same size it reported more than six times the performance at equal accuracy. Reaching accuracy of 0.98 required raising the search effort from the default of forty to one hundred, and 0.99 required 250, a 6 times increase over the default. The post notes that HNSW is almost always slower to build and uses more memory than IVFFlat. Hardware differed between the two tests and only 1,536-dimension vectors were used, which is a limit on how far the numbers generalize.

Binary Quantization With Rescoring in Qdrant

Turning to compression, Qdrant tested binary quantization on OpenAI embeddings and documented the results in its quantization guide. For the 3,072-dimension text-embedding-3-large model, accuracy rose from about 76 percent without rescoring to between 97 and 99 percent with it. Recall reached 0.9966 with three times oversampling, while storing one bit per component cut memory thirty-two times compared with float32 vectors. The 512-dimension small model improved from about 53 percent to between 71 and 91 percent, a weaker result that shows the technique favors high-dimensional vectors. The documentation states that one-bit compression loses significant precision below roughly a thousand dimensions, so compact embeddings still need the gentler scalar option. Rescoring also requires the original vectors to stay available, which limits the memory saving unless they sit on disk.

Lessons From Real Retrieval Deployments

Case Study: Notion Scales Workspace Search Ten Times

Looking at hyperscale operations, Notion launched its question-answering product in November 2023 with millions of workspaces on a waitlist. The problem arrived quickly, because its dedicated index clusters neared capacity within a month. Capacity grew through successive index generations, which the team later called operationally complex and expensive. In May 2024 the team moved to a serverless design that cut costs by about 50 percent from peak and removed storage limits. A later migration to turbopuffer reduced search engine spend by 60 percent and embedding pipeline compute by 35 percent. It also lowered p50 query latency from 70 to 100 milliseconds down to 50 to 70 milliseconds.

Notion's two-year retrospective credits a change detection system for a further 70 percent cut in data volume. Hashing page spans meant that a one-character edit no longer triggered a full re-embedding. The headline claim is a tenfold scale increase with a ninety percent cost reduction over two years. Every figure is self-reported, and the projected savings from a self-hosted Ray embedding pipeline were still rolling out at publication. The team also named the risk of adopting a young vendor, which required a full re-index and a gradual cutover. Treat the numbers as one company's experience rather than a promise for your workload.

Case Study: Ada Rebuilds Retrieval for Support Knowledge Bases

Considering multi-tenant support software, Ada faced a problem familiar to many SaaS teams. Its previous vector database demanded constant tuning of shard counts and node sizes, while rarely queried knowledge bases occupied expensive live shards. The incumbent OpenAI embedding model also fell outside a sub-200 millisecond p95 latency target in its setup. The solution combined a move to turbopuffer, where cold namespaces sit in object storage, with a bake-off of embedding providers on 15,000 production queries. Cohere embed-v4 reached 90.7 percent top-10 recall in float32 and 90.6 percent in int8, against 88.0 percent for the OpenAI model. Int8 vectors were four times smaller, at 1 kilobyte instead of 4 kilobytes for 1,024 dimensions.

The team chose self-hosted models on dedicated SageMaker instances for the online path, reaching about 77 milliseconds p50. For offline indexing it chose a dedicated vendor deployment that delivered roughly 2.7 times the throughput per dollar. Ada's vector database migration report is candid about its limits. The latency figures are first-pass measurements in one multi-tenant configuration, and no absolute dollar savings are published. The results also come from a single company's workload, so they may not generalize.

Case Study: predori Moves Patent Search to Weaviate Cloud

Stepping into a smaller team's story, predori built a patent research platform in 2020 whose first search layer ran on a Lucene-based AWS service. That service was not designed for approximate nearest neighbor search, and the problem was cost. The team kept everything in memory and over-provisioned hardware to keep queries fast, which the founders described as extremely expensive. In 2021 they evaluated self-hosted Faiss and the managed Milvus and Pinecone offerings before adopting Weaviate Cloud. They cited predictable pricing, offloaded operations, and a simple migration that swapped one client connection. The predori case study reports an operational cost reduction of more than 80 percent, while its results section describes savings by a factor of ten.

Those two figures do not match, and the page is a vendor-published story with quotes from the company's own executives. It omits data volume, latency, recall, baseline costs, and the measurement period, so the savings cannot be verified independently. The engineering benefits, such as faster time to market and expansion into agentic retrieval, are described only qualitatively. Multi-tenancy and vector compression work were still in progress, so their effects were not reported. Ask any reference customer for the baseline behind a percentage before you trust it.

Common Questions About Choosing a Vector Database