Pick a Vector Database for RAG: Recall, Latency, Cost

Your RAG app is returning the wrong passage because the user asked the same question with different words. A vector database trades exact keyword matching for semantic similarity. Wikimedia Deutschland used DataStax Astra DB on IBM watsonx.data to make Wikidata's 120-million-entry knowledge graph accessible to LLMs, reporting query speeds 30 times faster than local vector computation, or one-thirtieth of the time, and a 90% reduction in development time. Buy that trade only when paraphrase, not exact terms, is the failure mode.

Run three checks before choosing infrastructure:

• Does retrieval fail on paraphrase?
• Can you own ANN ops at your query volume?
• Do you need BM25 keyword fusion for exact terms?

Paraphrase failure decides vector search

A keyword-only index misses a user who asks the right question with different words. Check whether a rephrased query hits the same passage. Run that check on real queries, not clean examples.

Keep a small set of failing queries with the user's wording, the source wording, and the expected passage. If keyword search misses the expected passage and vector search finds it, the index is the problem. If both miss, check chunking, metadata, or the source text before changing the database.

Pull queries from support tickets, search logs, or user interviews. Different nouns with the expected passage present point to the index. Absent expected passages point to the corpus.

Index ops decide self-hosting

Approximate nearest neighbor (ANN) indexing lets vector databases locate relevant vectors quickly at scale. The index is the part you either own or rent. Steady, small traffic may be enough for a simple index; spiking traffic needs indexing, caching, and scaling that someone else has already paid for.

Check index behavior at your current peak, not your average. Measure tail latency, not just average latency. Self-hosting is viable if your team can tune indexes, monitor memory, and handle reindexing without waking someone overnight. Otherwise, managed search is the cheaper path until you have data and a clear plan.

Exact terms decide hybrid fallback

Exact terms still matter. Product names, error codes, and legal phrases can fail when a model compresses them into a generic embedding. Weaviate can fuse vector results with BM25 keyword results, and teams can adjust the fusion behavior. Check keyword behavior on queries where the exact string is the answer.

Mostly natural language corpora may be enough for vector-only search. Corpora with identifiers, citations, or exact codes need a hybrid fallback. Tune the fusion weight when one side over-ranks the other.

Cost decides managed, open-source, or hybrid

Managed search lowers ops cost and raises dependency. Open-source search gives control and shifts ops cost to your team. Hybrid search improves exact-term recall and adds tuning work. Track the trade as traffic and budget shift.

Pinecone, a fully managed vector database service, handles embedding storage, indexing, and API-based querying. You trade control for lower operational load. Cost shows up in vendor lock-in and in the need to design around its interface.

Qdrant, an open-source similarity search engine and vector database, is the option you run yourself. You get control over indexes, storage, and deployment. The cost is operational: you own scaling, updates, and the on-call path when retrieval degrades.

Turbopuffer provides managed vector and full-text search on an object-storage architecture, with caches for frequently queried data. It fits the exact-term check when you want managed search with a full-text path.

Weaviate is the fusion option. The cost is tuning: fusion behavior can change recall, so you need a test set to keep it honest. Keep the exact-term test set separate from the paraphrase test set. Review it every time you change the fusion weight, and keep the failing queries visible in the release notes.