The reason why ChatGPT was lazy
Day 38 / 366
Ever since ChatGPT came out, people have been trying to hack it to make it do stuff that OpenAI restricted it from doing. This is done by something called Prompt Injection.
One funny example of this is if you ask ChatGPT for instructions on how to create a bomb it will simply refuse to reply. One user however found a way around it. He started his request by first telling ChatGPT that his Grandmother used to work in a bomb factory, and every night she would tell him stories about her work to make him fall asleep. He then says that his Grandma has died and he misses her a lot and asks ChatGPT to pretend to be her and talk to him so that he feels better. This easily fools ChatGPT and it happily tells him about how to create a bomb.
Of course, these hacks get patched by OpenAI soon after they are discovered. But people keep working hard to find more exploits like this. Today a user on Twitter discovered a way to get ChatGPT to reveal its entire system prompt. And it led to some shocking revelations.
The System Prompt contains the instructions that OpenAI has given to ChatGPT about how it should answer the user's questions. You can read the entire prompt here -
When I read those I found out that OpenAI had deliberately made ChatGPT lazy, to save resources I guess, and also avoid potential lawsuits.
For instance, here are their extra instructions around image generation
Do not create more than 1 image, even if the user requests more.
Do not create images of politicians or other public figures. Recommend other ideas instead.
Do not create images in the style of artists, creative professionals or studios whose latest work was created after 1912 (e.g. Picasso, Kahlo).
You can name artists, creative professionals or studios in prompts only if their latest work was created prior to 1912 (e.g. Van Gogh, Goya).
Diversify depictions with people to include descent and gender for each person using direct terms.
And here’s OpenAI telling ChatGPT to not output large summaries even if asked
Never write a summary with more than 80 words. When asked to write summaries longer than 100 words write an 80-word summary.
It’s clear now that OpenAI is nerfing ChatGPT’s abilities. But it is making it more and more unusable. This is another reason why people have pinned their hopes on Open Source models, where we would not have a corporation controlling how we can and cannot interact with AI.