Getting JSON responses from LLMs

Day 60 / 366

A lot of people want to use LLM’s to add AI to their applications. LLMs are text-based models, you provide them text as input, and they give you back a text output. This is good enough in chat-based applications. But what if you want to use it differently? Is it possible to get the LLM to respond in a different more structured response so that our code can read it reliably and use it?

The most popular structure when coding applications is JSON. It’s a key value-based structure that has support for nestng and lists as well. When I was playing around with GPT APIs trying to build a book suggestion app, I wanted GPT to give me 4 suggestions, that I could then display in my web app in the form of a list. It seems trivial, but it wasn’t. Let’s see why that is.

Let’s move over to the OpenAI playground. We will try the GPT-4 model with a system prompt specifying that we need the output to be JSON. And then we will ask the model for some book suggestions

This worked well on the first try, just like you would expect. But will it always be the case?

I try it again, and this is what I get now —

You see the ```json at the top? That got added because the model is trained to display code in markdown, and so it tried to output the JSON with markdown as well.

This seems like an easy thing to fix, we could just add to our system prompt that we don't want markdown in our output.

And sure enough, it works now. But can you be absolutely sure it will always work? The answer is no. While I could not get it to fail right now to get an example for this blog post, in practice I have seen the LLM fail to provide a proper JSON. It would add some text in the beginning like “Here are your suggestions..” or sometimes just change the keys and provide a different JSON.

And no matter how well you write your prompt, the moment you change your model, or if OpenAI forces you to change your model, your prompt may no longer work.

OpenAI has tried to fix this by adding a JSON mode in some of their models. My suggestion is to fine-tune the model on your own with the required response structure for best results.