Stable Diffusion APIs — Control Net

Day 18 / 366

A few days ago I started playing around with Image generation AI models. But I lost interest pretty quickly because of how difficult it was to get them running and coding with them. I could not find any reliable GPU hosts online, and setting them up locally would have meant buying a 1 Lakh plus Graphic Card.

Today I found https://stablediffusionapi.com/ a hosted version of Stable Diffusion that can be accessed through an API, just like I access the GPT models through OpenAI APIs.

While it is paid, there are some free things to try out as well. The best part is that like OpenAI, this provides you with a playground as well, that you can use to try out some of the features.

We had already seen txt2img, i.e. providing the model with a text prompt and getting an image in return. Today I wanted to try out img2img, or control net. Control Net is supposed to help you control how the final image will look more strictly and easily than just a text prompt could.

Let me show you how this works.

Suppose I wanted to use Stable Diffusion to create a picture of a plant. If I use normal txt2img, I can give it the following prompt

A red flower in a pot beside a window

And this is the image it gives me

Looks good, but maybe I had a simpler plant and a simpler pot in mind, and rather than trying to explain what I want using the text, I could just draw a rough sketch of what I want and provide that to the model as well.

So I drew this quick sketch and provided this to Control Net with the same prompt. And this is what it came back with

While I agree that this is not as good as the original image, part of the reason is that I am a total noob when it comes to Stable Diffusion, and I know that using the correct parameters, settings, and prompts I could have gotten a better result. But it's still a good proof of concept, as it shows that the model, through ControlNet, was able to take my crappy drawing and generate a realistic image using it.