Skip to content
learn
Global menu

ComfyUI, from beginner to expert — Mission

Why I’m learning this

Section titled “Why I’m learning this”

The site’s other machine learning courses load models from code: Candle in Rust, and ONNX Runtime in GA’s AI course. ComfyUI is the other way round. It is an application where you wire models together in a graph, in a browser, and its template library covers images, video, audio and 3D models.

I want to know how it works under the graph, what makes an image reproducible, and how a C# or Java service can use it as a render engine: for instance to generate textures for this site and for GuitarAlchemist, which is the project of the last lesson.

Who this course is for

Section titled “Who this course is for”

You write C# or Java. You know HTTP, JSON and asynchronous code. You don’t need to know Python, PyTorch, or how diffusion models work: lesson 2 explains what you need, and links to the papers. The lessons’ images were rendered on a 16 GB NVIDIA GPU, and SDXL used about 7 GB of it; how the later models fit on smaller cards is lesson 8’s subject. Without a GPU, you can still follow the JSON and API lessons with the CPU workflow that CI uses.

By the end of this course, I will be able to

Section titled “By the end of this course, I will be able to”
  • install ComfyUI on Windows, Linux or macOS, and run a text-to-image graph;
  • explain what the checkpoint’s networks, the seed, steps, CFG, sampler and scheduler do, and what makes a render reproducible;
  • read, convert, validate and diff workflows in both JSON formats;
  • queue workflows and follow them from C# and Java through the HTTP and WebSocket API;
  • edit images with img2img, inpainting and outpainting, and control them with ControlNet and LoRA;
  • choose a recent model for a machine and a use, read its license, and fit it in VRAM with quantized weights;
  • upscale images and make seamless textures, and generate short videos;
  • judge a custom node’s code before installing it;
  • run ComfyUI as a service behind an API of my own.

Outline

Section titled “Outline”
# Lesson In C# or Java terms
1 The node graph, installing, and a first image a dataflow graph, like TPL Dataflow blocks
2 Diffusion, and what makes an image reproducible new Random(seed), and floating-point determinism
3 Workflows as JSON: the UI format, the API format, and diffs System.Text.Json, Jackson, a schema
4 The HTTP and WebSocket API from C# and Java HttpClient, ClientWebSocket, java.net.http
5 Img2img, inpainting and outpainting —
6 ControlNet: edges and depth —
7 LoRA: loading, stacking, and what training one involves a plugin that patches weights
8 Recent models and their licenses, quantization and VRAM choosing a dependency and its license
9 Upscaling, seamless textures and HDR —
10 Video —
11 Custom nodes, and their security NuGet or Maven packages that run code at install
12 ComfyUI in production: a service, a queue, several GPUs a worker behind a job queue
13 Project: textures for this site and for GuitarAlchemist —
14 Guitar Alchemist lab: measured experiments a benchmark suite with recorded results
15 Audio: music, sound effects and speech a sample buffer and a sample rate, as in NAudio or javax.sound
— Gallery every image, with the workflow that made it
— Journal

Lessons 5 to 13 are the plan; they will change as the first ones teach me what matters.

Models and images

Section titled “Models and images”

The course starts with Stable Diffusion XL base 1.0, under the CreativeML Open RAIL++-M License, and adds models lesson by lesson. Each lesson gives each model’s license and download size. No model file is in the repository. Every image in the course has a caption with its model, seed and workflow, and no image shows a real person or a brand.

Resources

Section titled “Resources”