In this article, we will develop a custom Sketch-to-Image API for converting hand-drawn or digital sketches into photorealistic images using stable diffusion models powered by a ControlNet model. We will extend the Automatic 1111’s txt2img API to develop this custom workflow.

Prerequisites

Stable Diffusion Web UI (Automatic 1111) running on your local machine. Follow the instructions here if you are starting from scratch.
SD APIs Enabled. Follow the instructions on this page (scroll down to the Enabling APIs section) to enable the APIs if you haven’t already done so.
ControlNet extension installed:

Click on the Extensions tab on Stable Diffusion Web UI.
Navigate to the Install from URL tab.
Paste the following link in URL for extension’s git repository input field and click Install.
After the successful installation, restart the application by closing and reopening the run.bat file if you’re a PC user; Mac users may need to run ./webui.sh instead.
After restarting the application, the ControlNet dropdown will become visible under the Generation tab in the txt2img screen.

Download and add the following models to Automatic 1111:

RealVisXL_V4.0_Lightning: HuggingFace: SG161222/RealVisXL_V4.0_Lightning. Copy this model to the Stable Diffusion models folder which is under the project root directory:/models/Stable-diffusion
diffusers_xl_canny_full – HuggingFace: lllyasviel/sd_control_collection
Copy the downloaded model to /extensions/sd-webui-controlnet

Payload

Now that we have all our prerequisites in place, let’s build the payload for the/sdapi/v1/txt2img API.