From Text Search to Vector Search: Generalizing Unstructured Data Retrieval

When people think of search engines they first imagine ElasticSearch, which excels at textual search, but the data foundation of search has expanded far beyond plain text to include video, audio, images, social graphs, and spatio‑temporal data.

Traditional text search relies on explicit semantics: each term becomes a dimension in a high‑dimensional TF‑IDF vector, and inverted indexes prune documents that lack required keywords. This model works well for pure text but cannot directly handle the implicit semantics of multimodal data.

Vector databases store these implicit semantic representations as high‑dimensional vectors produced by neural‑network embeddings. Queries are also mapped to the same vector space, and similarity is measured by distance (e.g., cosine similarity). The article illustrates this with a simple example where three Chinese sentences are converted to TF‑IDF vectors and a query "偷袭" AND "不讲武德" is evaluated.

$ f(Q,x)=\left\{ \begin{aligned}&cos(Q, x),\ \text{if "偷袭" in x and "不讲武德" in x \ (1)\\&0,\ \text{if "偷袭" not in x or "不讲武德" not in x \ (2)\end{aligned} \right.$

The model shows that explicit keyword matching yields a similarity of zero for documents missing any required term, which mirrors the pruning behavior of inverted indexes.

However, many unstructured data types lack explicit tokenizable units, and real‑world search often needs to combine multiple modalities (e.g., video recommendation using visual features, duration, language, user behavior). To achieve high accuracy, modern systems favor implicit semantic embeddings over interpretable token‑based representations.

A generalized unstructured‑data search system therefore moves the "mapping to vector space" step outside the search engine, using encoders built with deep learning frameworks (Spark, PyTorch, TensorFlow, etc.). The engine then only needs to store vectors, compute distances, and organize data via clustering or graph‑based indexes.

This decoupling simplifies engine design, allows diverse data types to be handled uniformly, and aligns the search engine with big‑data and AI ecosystems. Open‑source projects such as JINA are beginning to fill the missing pieces.

In summary, the proposed generalized model features (1) a vector space whose dimensions correspond to implicit semantics, (2) external, possibly domain‑specific encoders that map raw data to vectors, and (3) relevance search powered by clustering or graph indexes rather than traditional term‑based inverted indexes.