The Case Against Vector Databases

Vector databases became the default RAG infrastructure choice in 2023. Pinecone. Weaviate. pgvector. Embeddings became free. Retrieval became a default phrase in every AI product spec.

Three years in, most production RAG systems would have been faster, cheaper, and more accurate if they'd been built with keyword search and a JSON file. The vector database was added because it was the expected pattern, not because it was the right tool.

Vector search has two advantages over keyword search.

First, it handles synonyms and paraphrases. A user asks 'how do I cancel my subscription' and the documentation says 'how to end your plan.' Keyword search misses this. Vector search finds it via semantic similarity.

Second, it handles long, conversational queries. The user types a paragraph; the system finds the relevant chunk. Keyword search struggles when there are too many query terms — the noise overwhelms the signal.

If your use case has both of those properties consistently, vector search earns its keep. If only one, or neither, you're paying overhead for a feature you don't use.

Vector search wins on synonyms and long queries. Most RAG use cases have neither.

Keyword search wins on three dimensions.

Speed: it's an order of magnitude faster than vector search at most scales. Querying an in-memory inverted index runs in microseconds. Vector search runs in tens to hundreds of milliseconds depending on index size.

Cost: no embedding model. No vector storage. No re-indexing infrastructure. Your 'vector DB' is a JSON file or a SQLite database with full-text search.

Debuggability: when a query returns wrong results, you can read the index and understand why. When a vector search returns wrong results, you stare at 768-dimensional vectors and shrug. Keyword search lets you see what's happening.

For a knowledge base under 1,000 items, keyword search is almost always the right starting point. You can layer vector search on top later if and when you observe queries that keyword search fails on.

If you do need vector search, the right architecture is hybrid: keyword search runs first as the primary retrieval method, vector search runs in parallel and contributes additional candidates, and a final reranking step picks the top N.

This sounds complicated but is actually simple to implement. Keyword search is cheap, vector search is your backup for the queries keyword misses, reranking is one extra LLM call that picks from a small candidate set.

The pure-vector RAG pipeline you see in tutorials — embed query, retrieve top-K, stuff into prompt — is the simplest possible RAG and also the lowest-quality. Hybrid retrieval consistently outperforms pure vector at any meaningful scale. The teams that ship good RAG run hybrid. The teams that ship demo-quality RAG run pure vector.

Real signals that you've outgrown keyword + JSON:

- Your knowledge base has > 10,000 items and is growing. - Real users (not demo cases) regularly type long, conversational queries. - You've measured keyword search recall on a sample of real queries and it's below 70%. - Your team has at least one engineer comfortable maintaining vector infrastructure long-term.

None of these are present? Skip the vector DB. Build the keyword-search RAG. Spend your engineering time elsewhere. The vector DB will still be there when you actually need it.