RAG and its limitations

Day 11 / 366

Yesterday I used vector database and embeddings to provide an additional knowledge base for an LLM to use. This is a popular technique known as Retrieval-Augmented Generation or RAG.

The idea is simple. An LLM only knows about the data that they were trained on. Through RAG, we develop a pipeline to retrieve relevant facts and provide it to an LLM to help it answer a question.

I used this to build a bot that can answer any question about a pdf file that you give it. It worked amazingly well at first. But as I kept playing around with it, I soon realized that it had a big limitation.

Where did my tool fail?

To test my tool I gave it a PDF that had 100 short stories in it. And then I would pick any story, create a question and ask it. The RAG pipeline would then take the question, get the pages that were relevant to it, and pass it to the LLM. The LLM would then use the text from those pages to answer my question.

So far so good.

But then I thought of the following question -

Out of all these stories, how many had a happy ending?

And my tool failed. The exact response I got was

The reference document doesn’t have enough information to answer your question

Why did my tool fail?

The response was wrong. The PDF has all the stories, and so it has enough information to count how many have happy endings. When you think about it, it's clear why my RAG pipeline would fail. It tries to look for pages related to my question, and there won’t be any.

If at the end of the pdf, there was a conclusion chapter where the author would have listed down all stories with happy endings, then the RAG would have picked up the page and the LLM would have read it and answered my question.

Looking for other solutions

My first thought was that maybe I was a noob at this, and as I said, there were a lot of chats with PDF applications made last year. One such app was pdf.ai, an app that did more than 100k dollars in revenue in less than a year. Surely this app will be able to solve my question.

So I uploaded my document and asked it the same question, here is the response I got

And this is an app that people are paying money for. Surely something is wrong here.

Conclusion

It’s clear to me now that RAG has a lot of limitations when it comes to complex questions. Even the highly successful paid app said that it couldn't find any mentions of the story having a happy ending.

Of course, there are solutions, like adding metadata about the stories in the vector database. But then it just becomes an app to chat with stories. What if the PDF was a maths textbook? And if I have to handle every case separately, what’s the point of even using an LLM?

It’s times like these that reassure me that LLMs and AI have a long long way to go before they become smarter than humans.