You May Not Need a Vector Database

The first architecture diagram for a retrieval system usually contains a dedicated vector database. For a large share of systems, especially early ones, that box adds a service to run, a second source of truth to keep in sync, and a bill, in exchange for a capability nobody needs yet.

This is not an argument against vector databases. It is an argument for adding one when a measured limit says so, not because the reference architecture had one.

What the job actually is

Retrieval needs one operation: given a query vector, find the stored vectors most similar to it. There are two ways to do that.

Exact search compares the query with every stored vector. It always returns the true nearest neighbours. Its cost grows linearly with the number of vectors.

Approximate search builds an index, such as an HNSW graph or an inverted-file index, that lets the search skip most comparisons. It is much faster at scale, and it sometimes misses the true nearest neighbours.

Everything else, sharding, filtering, replication, a managed service, is engineering around that single operation.

How far exact search goes

The following is illustrative arithmetic, not a benchmark.

One million vectors of 768 dimensions, stored as 32-bit floats, take about 3 GB of memory. An exact search over them is roughly 770 million multiply-adds per query. That is real work, but it is the kind of dense arithmetic that CPUs and especially GPUs do well, and for many applications it is fast enough, with perfect recall and nothing to tune.

Whether it is fast enough for you depends on your hardware, your query rate and your latency budget. That is a measurement you can make in an afternoon, and it should come before the architecture decision rather than after it.

The step most teams skip

Between a matrix in memory and a dedicated service sits an option that many systems already have: the database they run anyway.

pgvector adds vector search to Postgres. Its documentation says it performs exact nearest neighbour search by default, with perfect recall, and supports HNSW and IVFFlat indexes when you want approximate search, which trades some recall for speed. Your vectors then live next to the rows they describe, filtered by the same permissions and updated in the same transactions.

That last point is underrated. Keeping embeddings in a separate system means keeping two stores consistent, and a stale vector pointing at a deleted or changed document is precisely the kind of fault that never shows up as an error.

When a dedicated database earns its place

Move up the ladder when a measured limit appears, for example:

• the vectors no longer fit comfortably on one machine
• the write rate is high enough that index maintenance interferes with queries
• many tenants need isolation and their own indexes
• filtered search at scale is too slow in your general-purpose database

Those are real and common at scale. They are rarely true in a system's first months.

HNSW or IVFFlat

If you do add an approximate index, the two common choices behave differently. pgvector's documentation summarises the trade:

	HNSW	IVFFlat
How it works	A multilayer graph of neighbours	Vectors divided into lists; only the lists nearest the query are searched
Query speed for a given recall	Better	Lower
Build time	Slower	Faster
Memory	More	Less

Both expose a dial between speed and recall. For HNSW in pgvector it is hnsw.ef_search: per the documentation, a higher value provides better recall at the cost of speed, and it can be set per query. That makes recall a setting you choose, and therefore one you should measure.

Filtering is where it gets hard

Real queries rarely ask only "what is most similar". They ask what is most similar among this customer's documents, or in this category, or from the last year.

Approximate indexes and filters interact badly. The index finds candidates by similarity first, and if the filter then discards most of them, you can get back fewer results than you asked for, or worse ones. pgvector's documentation suggests starting with an ordinary index on the filter column, offers iterative index scans that automatically scan more of the index when filtering removes too many candidates, and recommends partial indexes when filtering by only a few distinct values.

This is often the real reason teams move to a dedicated vector database: not raw vector count, but filtered search at scale. It is worth testing your actual filters before concluding that you need one.

Measuring recall in an afternoon

Recall is easy to measure, which makes not measuring it hard to excuse.

1. Take a sample of real queries, a couple of hundred is plenty to start.
2. Run each with exact search to get the true top results. That is your ground truth.
3. Run the same queries through the approximate index.
4. For each query, count how many of the true top results the index returned. The average is your recall.
5. Adjust the index settings and repeat, recording latency alongside recall.

You end with a small table of recall against latency for your own data, which is the only version of that trade-off that applies to you.

Similarity is not the only signal

Embeddings are good at meaning and weak at exact strings: product codes, error messages, names, identifiers. A query for a specific part number can retrieve passages about similar parts and miss the one that matters.

Combining vector similarity with keyword search, usually called hybrid search, covers both. pgvector's documentation points to using it together with Postgres full-text search for this. If your users search for exact terms, test hybrid retrieval before tuning embeddings, since no amount of vector tuning makes a model reliably match strings it treats as meaning.

What a dedicated database actually costs

The price of a dedicated vector database is rarely the licence. It is the second system.

It has to be deployed, monitored, backed up and secured, and it holds copies of data that already lives somewhere else. Every update and deletion in the source has to reach it, and every permission change has to be reflected in what it will return. A document removed from your main store but still present as a vector can keep surfacing in answers long after it should have disappeared.

There is also a cost that no choice of store avoids: changing your embedding model means re-embedding the entire corpus, because vectors from different models are not comparable. Wherever the vectors live, plan for that migration before you need it.

Recall is the hidden cost

Approximate indexes are tuned with parameters that trade speed for recall, and the Faiss authors describe vector search as a trade-off space rather than a solved problem. When recall drops, nothing reports it. The system still returns the requested number of results; some of them are simply not the best ones available.

That matters because retrieval quality, not the store, usually decides answer quality. A high similarity score does not guarantee a relevant passage, and adding more passages to compensate runs into the fact that models use the middle of a long prompt poorly. Measure recall on your own queries whenever you adopt an approximate index.

You may not need any of this

If your corpus is a few thousand documents, keep the embeddings in memory or in your existing database and spend the saved effort on chunking and evaluation. If you are already at the scale where a dedicated vector database is clearly justified, you knew that before reading this.

What we do not know

There is no universal threshold at which a dedicated database becomes worth it, and we do not offer one. The crossover depends on hardware, query rate, write rate, filtering and team capacity, and the honest answer is to measure exact search on your data first and let that number make the decision.

The figure in this note may be reused with attribution and a link to this page.