Full Text Search vs Vector (Semantic) Search? And Beyond.
Search is the one of the most important feature in any application. In this article we will discuss about Full Text search vs Vector search and the possibilities beyond that.
In full-text search, the emphasis is primarily on keyword matching. It treats each word independently, without considering the relationships between them or the overall context. It retrieves documents or web pages that contain the exact keywords specified in the query. On the other hand, semantic search aims to understand the user’s intent behind queries by analyzing its semantics, context, and relationships between words or concepts.
Here’s the flow for semantic search:
Storing record in DB: doc -> transformer model -> embedding -> write doc + embedding
Search: question → transformer model → embedding → vector search → relevant docs
Here’s what it looks like without semantic search :
Search: question → tokenizer or model → extracted terms/ property filters → full-text search → relevant docs
Huggingface has a number of transformers trained on various datasets.You can use appropritate transformers for your use case.
Embeddings are dense vector matrices that are stored as metadata to perform KNN search in vector databases. There are a number of vector databases; I personally prefer Elasticsearch as it allows me to work with full-text as well as semantic search use cases.
Is Semantic Search better than Full-Text Search?
It would be wrong to say that one is better than the other; there is no silver bullet. Depending on the use cases, you can opt to implement the type of search.
What’s beyond semantic search?
Natural Language (English) to native DB search (SQL or query in the underlying DB) is the next big thing. English or those daily conversational languages are powerful and most widely used. If you can develop a system that accurately interprets the user’s search query and returns results accordingly, then it would be a very intelligent search system. Retrieval Augmented Generation (RAG) provides the tools to build such intelligent systems.
