Combining Vector and Keyword Search

Duration: 5 min

This module delves into the integration of vector and keyword search techniques to enhance the performance and accuracy of retrieval-augmented generation (RAG) systems. By combining these methods, we aim to leverage the strengths of both approaches to deliver more relevant and contextually appropriate results. This is crucial for applications requiring nuanced understanding and precise information retrieval.

Understanding Vector Search

Vector search involves converting text into numerical vectors using embeddings, which capture semantic meaning. These vectors are stored in a vector database, allowing for efficient similarity searches. This method excels in understanding context and relationships between different pieces of text, making it ideal for tasks requiring semantic understanding.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

# Sample documents
documents = ["The cat sat on the mat.", "The dog played in the park.", "The cat chased the mouse."]

# Create TF-IDF vectorizer
vectorizer = TfidfVectorizer()

# Fit and transform documents
tfidf_matrix = vectorizer.fit_transform(documents)

# Convert to dense array for simplicity
tfidf_matrix_dense = tfidf_matrix.toarray()

# Calculate cosine similarity
similarity_matrix = cosine_similarity(tfidf_matrix_dense)

print(similarity_matrix)

Try it in Google Colab:

[[1.         0.24847395 0.69922657]
 [0.24847395 1.         0.24847395]
 [0.69922657 0.24847395 1.        ]]

Understanding Keyword Search

Keyword search relies on exact matches or partial matches of words within a text corpus. This method is straightforward and efficient for retrieving documents containing specific terms. However, it may lack the contextual understanding provided by vector search, often resulting in less relevant results when dealing with complex queries.

from collections import defaultdict

# Sample documents
documents = ["The cat sat on the mat.", "The dog played in the park.", "The cat chased the mouse."]

# Create an inverted index
index = defaultdict(list)

for doc_id, doc in enumerate(documents):
    words = doc.lower().split()
    for word in words:
        index[word].append(doc_id)

# Search for a keyword
keyword = "cat"
results = index[keyword.lower()]

print(f"Documents containing '{keyword}': {results}")

Documents containing 'cat': [0, 2]

💡 Tip: When combining vector and keyword search, ensure that the weighting of each method is balanced according to the specific requirements of your application. Over-reliance on one method can lead to suboptimal results.

Key Concepts

Check Your Understanding