Is ChatGPT getting Lazier?

Day 14 / 366

Today I was doing some research for my new GPT-powered learning app. For one of the features, I need to use AI to answer complex questions based on a document. As I had explained earlier, the normal RAG-based approach does not work here, as the AI needs to come up with a strategy to get to the solution.

ChatGPT Plus also supports uploading documents and then asking questions about it, so I thought I should give it a try, just to see if what I was trying to achieve was even feasible or not with GPT-4.

I uploaded a PDF with 100 short stories in it and then asked ChatGPT to list all stories that had a happy ending.

ChatGPT first said that this process would take time, and then gave me names of 3 stories. Now I know it's a wrong answer. That PDF had way more stories with happy endings. But let’s give ChatGPT the benefit of the doubt here, maybe the definition of a “Happy Ending” is ambiguous.

So I decided to give it a more clear task — listing down all the character names that appear in the pdf

You can see how the laziness started to show here. It ended with saying that it would “provide me updates”, but nothing happened. I finally asked it to proceed, and it again gave me the same complaint of this being a tough task which will take a long time.

But it finally gave characters for 2 more stories.

And again it ended mysteriously saying that it would provide me with updates. But nothing happened.

I had read on some tweets that to get an answer from ChatGPT these days, sometimes people had to be rude to it and shout at it. So I thought I would give that a try as well.

And this was the worst answer so far. The same complaining about it being a tough task and then the promise at the end of updates that never came.

So either ChatGPT is not as good as people are making it out to be, or OpenAI has purposely dumbed it down. I think the latter might be true. The same thing happened with GPT-3 right before GPT-4 was announced. So this might mean that GPT-5 is coming soon.