What Is a Vector Database? How It Works?

Most of the data an organization holds is unstructured: documents, tickets, chat logs, images, audio. Machine learning models made that data searchable by converting it into embeddings, numeric vectors whose geometry captures meaning, so that two similar documents land near each other even when they share no keywords. That conversion created a search problem existing databases were not built for: finding the nearest neighbors of a point among millions of high-dimensional vectors, quickly and repeatedly. Vector databases emerged to solve it, and the rise of retrieval-augmented generation has moved them from a niche similarity-search tool to a standard component of AI infrastructure.

This post explains what a vector database is, why the category exists, how one works internally, and where it fits: embeddings, similarity search, the comparison with traditional databases, benefits, use cases, implementation steps, and the limitations to plan around.

What is a vector database?

A vector database is a database designed to store high-dimensional vectors and retrieve them by similarity. Its core operation is “find the k vectors closest to this query vector,” rather than the exact lookups relational indexes serve. That operation is called nearest-neighbor search, and everything distinctive about a vector database, its indexes, its query interface, its performance characteristics, exists to make nearest-neighbor search fast at scale.

Each record typically holds three things: the vector itself, an identifier, and a metadata payload (the document’s source, timestamp, access tags, or any other attributes worth filtering on). A query supplies a vector and gets back the most similar records, ranked by a distance score, optionally restricted by metadata filters such as “only documents from this tenant.”

The category spans two implementation shapes. Purpose-built engines such as Pinecone, Milvus, Weaviate, and Qdrant are standalone systems built around vector search. Vector-capable extensions add the same primitive to an existing database: pgvector brings vector columns and indexes to PostgreSQL, and Elasticsearch and OpenSearch expose approximate nearest-neighbor search alongside their text indexes. Both implement the same core ideas; the choice between them is operational more than functional.

Why vector databases matter

The reason the category exists is that similarity search does not decompose into the operations traditional databases optimize. A B-tree index accelerates exact matches and range scans on ordered values. An inverted index accelerates keyword lookup. Neither helps with “which of these ten million 1,536-dimensional points is closest to this one,” because proximity in high-dimensional space is not a predicate on any single column; a naive answer requires comparing the query against every stored vector.

For small collections, a brute-force scan is fine, and a NumPy array or a SQL table with a distance function will do. The problem arrives with scale: embedding a document corpus, a product catalog, or a user-behavior history produces millions to billions of vectors, and applications query them interactively, often on every request. At that point nearest-neighbor search needs an index of its own, and maintaining that index, keeping it consistent with inserts and deletes, filtering it by metadata, and distributing it across nodes, is a database workload rather than a library call.

Generative AI turned this from a specialized need into a common one. Language models answer from their training data unless an application supplies fresher or private context, and the standard way to find that context is embedding similarity. A retrieval-augmented generation system runs a similarity search on every user question before the model answers, which puts a vector store in the critical path of a large class of AI applications.

How a vector database works

The lifecycle of a vector database has two paths: an ingest path that builds the searchable structure, and a query path that uses it.

 Ingest. Raw data is converted to vectors by an embedding model, which runs outside the database; the database stores what the model produces. Each vector arrives with its identifier and metadata payload and is written into an index structure designed for proximity search rather than ordered lookup. Index building is the expensive part of ingestion, and the reason inserts into a vector database cost more than appends to a log: the new vector has to be placed relative to its neighbors.

Query. An incoming query, a user’s question, an image, a product a customer viewed, is first embedded by the same model that embedded the corpus, so that query and data live in the same space. The database searches its index for the nearest stored vectors, applies any metadata filters, and returns the top k results with distance scores. The application decides what those results become: search hits, recommended items, or context passages handed to a language model.

The approximation trade-off. At scale, exact nearest-neighbor search is too slow to run on every request, so production systems use approximate nearest neighbor (ANN) indexes. An ANN index finds the nearest neighbors almost certainly in milliseconds, rather than provably in seconds. The quality measure is recall: the fraction of true nearest neighbors the index actually returns. Vector databases expose tuning parameters that trade recall against latency and memory, and choosing that operating point is a core part of running one.

What are vector embeddings?

An embedding is a learned numeric representation of an object: a list of floating-point numbers, typically hundreds to a few thousand dimensions long, produced by a machine learning model trained so that semantically similar objects map to nearby points. The distance between two embeddings approximates the similarity of the things they represent. “Cardiac arrest” and “heart attack” share no words, but a text embedding model places them close together; an image model does the same for two photos of the same landmark from different angles.

Embedding models exist for most modalities: text, images, audio, code, user behavior, products. Multimodal models embed text and images into a shared space, which is what lets a text query retrieve matching images. In every case the model defines the space, and that has a practical consequence that shapes vector database operations: vectors from different models, or different versions of the same model, are not comparable. A corpus embedded with one model must be queried with that model, and switching models means re-embedding everything.

Embeddings are the input contract of a vector database: everything the database sees is points in a space and the distances between them. Search quality is bounded by how well the embedding model captures the similarity the application actually cares about.

Similarity search in vector databases

Similarity search makes “similar” precise with a distance metric, and three are standard. Cosine similarity compares the angle between two vectors and ignores their length, which suits text embeddings, where direction carries the meaning. Euclidean distance measures straight-line distance and is the natural choice when magnitude matters. Dot product combines angle and magnitude and is common in recommendation models, where a vector’s length can encode popularity or confidence. The embedding model’s training determines the appropriate metric; the database is configured to match.

The harder problem is running that metric against millions of vectors per query, and ANN indexes are the answer. Two families dominate.

Graph-based indexes. HNSW (Hierarchical Navigable Small World) builds a layered proximity graph over the vectors: sparse upper layers for coarse navigation, denser lower layers for precision. A search enters at the top, greedily walks toward the query, and descends. HNSW delivers high recall at low latency and is the default index in much of the category; its cost is memory, since the graph lives alongside the vectors.

Clustering-based indexes. IVF (inverted file) partitions the vectors into clusters and searches only the clusters nearest the query, scanning a small fraction of the data. It builds faster and holds less memory than a graph index, at some cost in recall for the same latency. Quantization techniques such as product quantization compress the vectors themselves into compact codes, shrinking memory and speeding comparisons at some cost in precision, and are often layered onto an IVF index for very large collections.

Real queries are rarely pure similarity; they carry conditions such as “nearest neighbors among in-stock products only.” Combining ANN search with metadata filtering is its own engineering problem, because filtering after the search can leave too few results while filtering before it can gut the index’s assumptions, and vector databases differ meaningfully in how well they handle it.

Vector databases vs. traditional databases

The differences run deeper than the data type. A relational database and a vector database disagree about what a query is and what an answer is.

The correctness row is the one that surprises teams coming from relational systems. A SQL query that misses a matching row is broken; an ANN query that misses a true neighbor is operating as designed, within a recall target someone chose. That shift moves quality from a property of the engine to a property of the configuration, which is why vector search needs measured evaluation in a way relational queries do not.

In practice the two are complements, not competitors: the vector database holds embeddings and answers similarity queries, while the system of record keeps the transactions, the relationships, and the authoritative copy of the data the embeddings were derived from. The boundary is also blurring, with relational and document databases adding vector columns and indexes, pgvector being the clearest example. For many workloads, vector search inside the existing database is enough, and a dedicated engine earns its place only at a scale or latency the extension cannot meet.

Benefits of using a vector database

Retrieval by meaning rather than keywords. Similarity search finds relevant results that share no vocabulary with the query, which is the failure mode that caps keyword search quality. Synonyms, paraphrases, and cross-language matches come along without rule-writing, because the embedding model learned them.

Scale with sublinear search cost. ANN indexes answer queries against millions to billions of vectors without scanning the collection, and purpose-built engines shard indexes across nodes to grow horizontally.

One primitive across modalities. Text, images, audio, and product embeddings are all points in a space, so one retrieval system serves search, recommendations, and multimodal lookup, rather than a separate engine per data type.

Filtered similarity in one query. Production retrieval is similarity plus constraints: tenant boundaries, freshness windows, access tags. Vector databases combine metadata filtering with ANN search in a single query, which is where they most clearly outgrow a bare index library such as FAISS.

A managed home for the AI retrieval path. Putting embeddings in a database rather than an in-process index buys the operational properties applications need from any datastore: persistence, replication, incremental updates, and concurrent access.

The common thread is that these are database virtues applied to a new query shape. The embedding model supplies the intelligence; the vector database makes it operable at production scale.

Common use cases of vector databases

Semantic search. The query “laptop won’t turn on” retrieves the support article titled “troubleshooting power issues,” because the embeddings match even though the words do not. This is the direct replacement for, or re-ranking layer on top of, keyword search in support portals, documentation, and enterprise search.

Retrieval-augmented generation. RAG grounds a language model by embedding a user’s question, retrieving the most similar passages from an embedded corpus, and placing them in the prompt. The vector database is the retrieval half of that loop, and much of the category’s recent adoption has been driven by RAG; retrieval quality bounds answer quality, which makes the retrieval techniques around the database matter as much as the database itself.

Recommendation systems. Users and items embedded in the same space turn “what should this user see next” into a nearest-neighbor query: items close to the user’s vector, or to the items the user just engaged with. Similarity search serves the candidate-generation stage, producing a shortlist a ranking model then orders.

Image and multimodal search. Reverse image search, find-similar-products, and text-to-image lookup all run on embeddings from vision or multimodal models. The same mechanics support media deduplication, finding near-identical images at scale.

Anomaly and fraud detection. Similarity has a mirror image: a transaction whose vector is far from every cluster of normal behavior is an outlier worth flagging, and a new case close to a known fraud pattern is a lead worth pursuing. Vector search supplies the primitive for finding more things like a known-bad example in investigation workflows.

Across these cases the pattern repeats: an embedding model turns a domain’s objects into points in a shared space, and the vector database turns proximity in that space into a production query. The use cases differ in modality and in what similarity means; the retrieval machinery underneath is the same.

How to implement a vector database

Start from the retrieval requirement, not the database. Corpus size, query rate, latency budget, filtering needs, and update frequency determine everything downstream; a million vectors behind an internal tool and a billion inside a consumer request path are different projects.

Choose an embedding model before choosing an engine. The model determines vector dimensionality, the appropriate distance metric, and ultimately search quality. For document workloads, chunking strategy (how source text is split before embedding) belongs to this step and affects results more than most index tuning.

Pick the deployment shape. If the data already lives in PostgreSQL and the scale is moderate, an extension like pgvector adds vector search without a new system to run. Dedicated engines earn their place at larger scale, tighter latency, or heavier filtering. Managed services trade cost for not operating index infrastructure yourself.

A minimal pgvector example shows how compact the core mechanics are; the same three ideas, a vector column, an ANN index, an order-by-distance query, appear in every engine’s API:

CREATE EXTENSION vector;

CREATE TABLE documents (
  id bigserial PRIMARY KEY,
  content text,
  embedding vector(1536)
);

CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);

SELECT id, content
FROM documents
ORDER BY embedding <=> '[0.011, -0.032, ...]'
LIMIT 5;

Tune the index against a recall target. Build a ground-truth set with exact search over a sample, then adjust index parameters until measured recall meets the application’s bar within the latency budget. Skipping this step is how retrieval quality degrades silently.

Plan the update and re-embedding path. Decide how new and changed documents flow into the index, and budget for full re-embedding when the embedding model changes, since old and new vectors cannot share a space. Treat the embedding model version as part of the schema.

The order of these steps carries most of their value: the retrieval requirement constrains the embedding model, the model constrains the engine and index, and the update path keeps the result valid over time. Working through them in sequence produces an engine fitted to the workload, with the operating point (recall, latency, cost) chosen deliberately rather than inherited from defaults.

Challenges and limitations

Memory and cost. High-recall ANN indexes, HNSW in particular, want to live in RAM, and at hundreds of millions of vectors the memory bill becomes the dominant cost. Quantization and disk-based indexes reduce it, at a recall or latency price that has to be measured rather than assumed.

Approximation is hard to debug. When a relational query is wrong, the row is provably missing. When retrieval quality drifts, the index still returns plausible results, and only a maintained ground-truth evaluation reveals that recall fell. Teams that skip evaluation find out from end users.

Coupling to the embedding model. The database’s contents are derived data, valid only for one model version. Model upgrades, which arrive frequently, mean re-embedding the corpus and rebuilding indexes, a pipeline that has to exist before it is needed.

Filtered search is uneven terrain. Metadata filtering interacts with ANN structures differently across engines, and aggressive filters can collapse recall or latency in ways unfiltered benchmarks never show. This is the capability to test hardest during selection.

Similarity is one retrieval shape, not all of them. A vector database answers “what is most like this?” It does not answer “what is connected to this, and through what path?”: which accounts does this customer control, which services depend on this component, which entities does this alert touch. Those are traversal questions over explicit relationships, and embedding similarity cannot substitute for following edges. Retrieval stacks increasingly pair the two, similarity search over content and graph retrieval over relationships, so that a system can fetch passages that resemble the question and the entities connected to it. PuppyGraph covers the relationship half without adding another storage system: it defines a graph schema over the tables already in SQL databases, data warehouses, and data lakes or lakehouses (including direct reads of open table formats like Iceberg and Delta Lake) and executes openCypher and Gremlin traversals against them directly, with no ETL and no second copy of the data. The full comparison of when each retrieval shape fits is in our vector database vs. graph database post.

These limitations argue for adopting the category deliberately, with an evaluation harness, a re-embedding plan, and a clear view of which questions similarity search is the right tool for.

Conclusion

A vector database stores embeddings and retrieves them by similarity, using ANN indexes to keep nearest-neighbor search fast at scales where exact search cannot keep up. It exists because meaning-based retrieval became a production workload: semantic search, recommendations, and above all RAG put similarity queries in the request path of everyday applications. Getting value from one is less about the engine choice than the discipline around it: an embedding model matched to the task, an index tuned against a measured recall target, and clarity about which questions are similarity questions and which are relationship questions that belong to a graph.

Try the forever-free PuppyGraph Developer Edition and book a demo with the team to see how openCypher and Gremlin queries run over warehouse and lakehouse tables, with no graph-specific ETL, alongside the vector search layer of a retrieval stack.