Product Recommendation System Using Elasticsearch and ChatGPT: Revolutionizing Personal Shopping Experiences

Today, businesses try making shopping not just a transaction but an engaging and highly personalized journey as well. At ExpertGuru, we achieved this by integrating Elasticsearch with advanced language models like ChatGPT to create a Personal Shopper chatbot. Combining these technologies can revolutionize product recommendation systems, offering a seamless, efficient, and tailored shopping experience.

But how? And why?

ChatGPT now needs no introduction. Given textual instructions and proper data, it generates output for the given task. Hmmm… seems powerful enough. But here is the catch: it has no real-time world information. In fact, it has no information post its training data day. Why is this important? Then how can we get properly updated product recommendations?

This limitation is precisely where Elasticsearch comes into play, bridging the gap between ChatGPT's extensive language understanding capabilities and the need for up-to-date product data. While ChatGPT excels at parsing user queries and generating human-like interactions, it lacks access to real-time updates on products, prices, and trends. By integrating Elasticsearch, we ensure that our Personal Shopper chatbot has instant access to the latest product information, allowing it to provide recommendations that are not only contextually accurate but also reflect current availability and customer preferences. This synergy between Elasticsearch's dynamic search and analytics engine and ChatGPT's conversational AI creates a powerful tool that keeps recommendations relevant and personalized, catering to the evolving needs of today's shoppers.

How Have We Designed the System?

Using Elasticsearch and ChatGPT to create a Personal Shopper chatbot that not only understands the nuances of human language but also delivers instant, relevant product recommendations. Here's how the system works:

1. Understanding user queries. When a user interacts with the Personal Shopper chatbot, we use ChatGPT to process the input text to understand the user's request. This could range from specific product inquiries to more abstract shopping goals, such as finding a gift for an occasion. The output is a concise and contextual search text which tells what the user is looking for.

2. Querying the product database. Once the intent is understood and we get this contextual search text, our system formulates a query that is sent to Elasticsearch, which then searches through the product database to find items matching the criteria set by the user's request. To do this, we have used a hybrid search approach. This approach ensures highly accurate and relevant product recommendations, even for complex user queries that specify product features. Here's a breakdown of how it works:

i. Formulating the search query: Based on the understood intent, our system crafts a query that incorporates both textual and contextual elements aimed at finding the best match within our product database.

ii. Employing text-embedding for contextual search: To address the contextual aspect of the search, we use the text-embedding-ada-002 model. Using this model, we processed the product descriptions and any relevant textual content to create embeddings. These embeddings are already indexed in our ElasticSearch database. These allow the system to perform a contextual search, effectively matching the user's query with products based on the semantic meaning and nuances contained in their descriptions.

iii. Keyword search for detailed product attributes: Alongside contextual search, our system also performs a keyword-based search targeting other meta fields of the product database, such as specifications, categories, and tags. This dual approach ensures that specific product details, which may be directly mentioned in the user's query, are accurately captured and matched.

Hybrid search advantage: By combining contextual search with keyword search, the system can handle complex queries that require understanding both the broader context and the fine details of product features. This approach significantly enhances the accuracy and relevance of the product recommendations, catering to users' specific needs and preferences.

3. Personalization and re-ranking of products: To enhance the personalization and re-ranking of products in our system, we've adopted a sophisticated method involving the integration of Large Language Models (LLMs) and custom algorithms. We leveraged the advanced capabilities of ChatGPT-4 to understand and process user queries. This allows for a deeper comprehension of what the user is looking for, beyond just basic keyword matching. To do this, we integrated a custom Retrieval-Augmented Generation (RAG) prompt using ChatGPT-4. This combination helps in interpreting the user's current query in the context of their historical interactions and preferences. Using this approach, we were able to achieve the following capabilities: