I tried using an Open Source LLM

Day 12 / 366

Whenever someone comes up to me with an idea for a product using Generative AI, I ask them the same question

How do you deal with the platform risk that comes with using OpenAI APIs?

GPT-4 is miles ahead of any other LLM out there, and to build a good product you would probably want to use it. But GPT-4 is a closed model, which means that to use it, you need to access it through APIs that OpenAI provides you.

And that’s where the platform risk comes in. If tomorrow OpenAI decides to disable the API, nerf it, or add usage limits, your product is gone. And if you do manage to create an amazing product, OpenAI can just copy and ship it directly with ChatGPT, thus removing the need for your product. This happened a lot last year, with every update that ChatGPT got, several small startups became worthless overnight.

Open Source LLMs

All such startups have their hopes on Open Source LLMs. These are LLM models that you can download and run on your own, and therefore you won't be dependent on any other company for your Generative AI needs.

So far, I have not used any Open Source LLMs, so today I decided to try some out and write a blog comparing them with GPT-4. Apart from being not as good as GPT-4, the biggest pain with Open Source LLM is how tough it is to use them. And I was not able to run even a single one today.

Why are Open Source LLMs hard to use

The first plan I had was to just try to run an open-source LLM locally on my computer. But I soon realized that to run any good enough LLM, I would require a really expensive graphic card.

I could run some smaller or trimmed-down versions of LLMs, but with my old graphic card, they would either run too slowly to be usable, or not run at all.

Assuming that I even have one running, my goal is to create tools that I can share with everyone. So ideally I would want it hosted on a server so that I can access it through an API. But from what I found, building a server even for a small 7B model for 10 users will require a large amount of GPUs.

In short, open-source LLMs might be cool to experiment with, but they are nowhere near GPT, in terms or quality or accessibility.