Why you should pause before adopting a vector database
As the popularity of large language models continues to surge, there is also a growing interest in vector databases. Many tutorials show how vector databases can enable you to customize the behavior of LLMs to their proprietary data without the need for extensive model fine-tuning.
But does your enterprise need to invest in vector databases? A new study by researchers at Waterloo University and Roma Tre University posits that enterprises can reap the benefits of sophisticated "embedding search" capabilities offered by LLMs without using a vector database.
They suggest using Lucene, an open-source platform that is the backbone of search engines used in many enterprises, including Elasticsearch. Accordingly, their paper is titled “Vector Search with OpenAI Embeddings: Lucene Is All You Need.”
Key findings:
Embeddings enable you to represent the meaning of text as a vector of numbers
With embeddings, you can measure the similarity of prompts and documents and retrieve relevant documents for a user’s query
Vector databases are designed for storing, comparing and retrieving embeddings with high efficiency
Popular thought is that every enterprise needs to invest in vector databases to create LLM applications that leverage their proprietary documents
However, the new research paper shows that if you already have Elasticsearch or another search engine based on Lucene, you can use it for vector search
Recent versions of Lucene support HNSW, the main algorithm in vector search
The researchers implemented a prototype of vector search with Lucene and obtained results that were comparable to state-of-the-art vector database results
Lucene still has its challenges, including speed and difficulty of implementation, but these challenges will likely be solved soon
The benefit of Lucene is that many enterprises already use some form of it, including Elasticsearch
Lucene can enable enterprises to prototype and experiment with embedding search without making heavy investments in a new technology
The researchers write, “If you’ve built search applications already, chances are you’re already invested in the Lucene ecosystem. In this case, Lucene is all you need. Of course, time will tell who’s right.”
Read the full article on TechTalks.
Recommendations:
To learn more about Lucene, read Getting Started with Elastic Stack 8.0
Transformers for Natural Language Processing is an excellent introduction to the technology underlying LLMs. It provides a very accessible explanation of how transformers work and how you can use different transformer architectures (BERT, T5, GPT, etc.)
For more on LLMs: