How to Use ControlNet for Game Development: Fix Random Poses and Get Consistent Character Art

Here is the problem with Stable Diffusion after you get it running: every image is a dice roll. You write a prompt for a warrior in a combat stance and get a warrior standing with their arms at their sides. You try again and get someone leaning casually against a wall. On the fifth attempt you get something close, but the sword arm is anatomically impossible and one hand has eight fingers.

This is not a prompt engineering problem. Text prompts are probability nudges, not instructions. When you type "perfect hands, five fingers, correct anatomy" into Stable Diffusion, you are nudging a statistical weighting system — you are not issuing a command. The model has no spatial awareness. It does not know where the hands are or how joints connect.

ControlNet is the fix. It takes a structural guide — an edge map, a body pose, a depth map — and forces the diffusion model to follow that structure. Same prompt, but now the output matches the layout you specified. That is a genuinely different category of control. This guide covers everything from installation to the specific workflows that actually matter for game concept art: character sheets, consistent poses across multiple images, and turning rough sketches into finished art.

 

What ControlNet Actually Does

Standard Stable Diffusion uses text prompts to guide image generation. ControlNet adds a second layer of guidance: input images. But it does not copy the input image like standard img2img. Instead, it extracts specific information — edges, depth maps, or human skeletons — and forces the AI to follow that structure while generating entirely new textures, lighting, and styles based on your prompt.

The workflow is: give ControlNet a reference image, it processes that image through a preprocessor to extract a control map (a skeleton wireframe, an edge outline, a depth gradient), and that map constrains what the diffusion model generates. Your warrior prompt now produces a warrior in the exact pose from your reference photo, not whatever pose the model felt like.

For game development specifically this matters because you need repeatable results. A character design sheet needs the same character from multiple angles. Your enemy AI needs five different attack poses that all look like the same design. Your environment concepts need consistent lighting across a zone. None of that is achievable with prompts alone.

Installing ControlNet

Assumes you have Forge or AUTOMATIC1111 set up from the Stable Diffusion guide. If you are using ComfyUI, the process is slightly different and covered at the end of this section.

In Forge or AUTOMATIC1111

1. Open the webui and go to the Extensions tab.
2. Click Install from URL.
3. Paste: https://github.com/Mikubill/sd-webui-controlnet
4. Click Install, then go to the Installed tab and click Apply and Restart UI.

After restart, a ControlNet section appears below the prompt fields in txt2img and img2img. It starts collapsed — click the arrow to expand it.

Downloading Models

The extension is just the interface. The actual ControlNet models are separate downloads. For SD 1.5, there are 14 official ControlNet models. For SDXL, the most practical option is xinsir's ControlNet Union — one file that covers 10+ control types. For Flux, Shakker Labs Union Pro 2.0 is the current best option.

You do not need all of them. For game development, download these three first and add others only when you have a specific use case for them:

• control_openpose — for character poses
• control_canny — for edge-based composition from sketches or photos
• control_depth — for 3D spatial layout from reference images

Place downloaded .safetensors files in: stable-diffusion-webui-forge/models/ControlNet/

Restart the webui. The models now appear in the ControlNet model dropdown.

In ComfyUI

Install the comfyui_controlnet_aux node pack through ComfyUI Manager, then drop ControlNet models into models/controlnet/. The workflow is node-based — add a Load ControlNet Model node, connect it through an Apply ControlNet node between your positive conditioning and KSampler. It takes fifteen minutes to build once you understand the node graph from the Stable Diffusion guide.

The Five Models Worth Knowing

There are over a dozen ControlNet models available, but you will spend 90% of your time using about five of them. Here is what each does and when to use it for game art specifically.

Model	What It Controls	Game Dev Use Case
OpenPose / DWPose	Extracts a skeletal stick figure from a reference photo, transfers that exact pose to the generated character.	Character design sheets, attack animations, idle poses, crowd scenes. The most-used model for character work.
Canny	Extracts edge lines from an image or sketch and uses them as a structural guide.	Turning rough concept sketches into finished AI art, preserving composition while changing style, reference-based environment blocking.
Depth	Creates a grayscale depth map from a reference image, preserving the 3D spatial relationships between objects.	Environment concept art where you want to lock the spatial layout, multi-character scenes where depth separation matters.
Lineart	Similar to Canny but preserves line weight variation better for hand-drawn inputs.	Converting your own sketches or existing lineart into fully rendered AI illustrations while keeping your original composition.
IP-Adapter	Uses an image as a style/appearance reference rather than a structural guide. Injects the visual features of a reference into the generation.	Keeping the same character face or design consistent across multiple images. Works best combined with OpenPose.

The preprocessor and model have to match. If you use a Canny preprocessor, you must select the control_canny model. If you use OpenPose, select control_openpose. Mismatching them is the most common beginner mistake and produces garbage output with no useful error message.

Real Workflow 1: Character Poses With OpenPose

This is where most game developers start, and for good reason. You want to generate a knight swinging a sword. Upload a photo of yourself swinging a broom, and OpenPose will transfer that exact motion to the fantasy character.

Here is the actual step-by-step:

1. Take a reference photo of the pose you want — yourself, a friend, or a stock photo. The character's appearance does not matter at all, only the body position.
2. In the ControlNet panel, enable it and upload the reference photo to the image canvas.
3. Set Preprocessor to openpose (or dw_openpose for better hand detection).
4. Set Model to control_openpose (or xinsir Union for SDXL).
5. Click the preview icon — you should see a skeleton wireframe overlay on your reference image. This is what ControlNet is extracting.
6. Write your normal character prompt and generate.

The generated character will match the extracted skeleton's pose. Body proportions, clothing, face, and style all come from your prompt and checkpoint. Only the pose transfers from the reference.

Pro tip: always photograph your own hand holding an object against a blank wall to use as the ControlNet reference. It is 100x faster than trying to articulate individual finger joint positions through text prompt iteration. I spent twenty minutes trying to prompt "hand gripping sword with correct finger wrap" and got nonsense every time. Took a photo of my own hand gripping a pen, ran it through OpenPose, generated in ten seconds. Different category of result.

Real Workflow 2: Sketch-to-Art With Canny

You sketch a rough environment layout — a ruined building on the left, open ground in the center, trees on the right. Canny extracts the edges from that sketch and forces the generation to follow your composition. The actual style, lighting, and detail all come from your prompt. Your terrible Quick sketch turns into a polished game environment concept.

1. Upload your sketch or rough drawing to the ControlNet image canvas.
2. Set Preprocessor to canny.
3. Set Model to control_canny.
4. Adjust the Canny Low Threshold and High Threshold sliders. Lower values pick up more edges including noise; higher values keep only the strongest structural lines. For a clean sketch, start at 100/200 and adjust from there.
5. Generate with your environment prompt.

Canny works on any image, not just hand-drawn sketches. Taking a screenshot of a Unity greybox level and running it through Canny gives you a quick concept visualization of what that geometry could look like as finished art. This is genuinely useful for showing non-developers what a level will look like before any real assets exist.

Real Workflow 3: Stacking for Character Sheets

The most powerful ControlNet technique for game developers is stacking multiple units simultaneously. The professional approach is to decouple the elements: one slot handles identity, one handles structure, one handles environment.

For a character design sheet with the same character in multiple poses:

• ControlNet Unit 1 (IP-Adapter) — upload your character's face/design reference. This locks who the character is.
• ControlNet Unit 2 (OpenPose) — upload a reference photo for each pose. This locks how they are standing.

VRAM requirements stack. Running multiple ControlNet models simultaneously requires a minimum of 12GB in 2026. On a 6-8GB card, run one unit at a time and composite the results. On 12GB+, stacking gives you dramatically better consistency across a character sheet than either approach alone.

The Control Weight slider matters here. A weight of 1.0 means the structure is rigidly followed. Lower values (0.6-0.8) give the model more creative freedom while still respecting the general structure. For IP-Adapter identity control, 0.7-0.8 usually produces better-looking faces than forcing it to 1.0, which can make faces look plasticky.

ControlNet vs IP-Adapter: What is the Difference

This confuses almost everyone. Where ControlNet controls spatial structure — where things are — IP-Adapter controls appearance — what things look like. They solve different problems and work better together than either does alone.

If you need a character to be in a specific pose: ControlNet OpenPose.

If you need a character to look like a specific reference design: IP-Adapter.

If you need both at once: stack them.

T2I-Adapter is worth knowing about too. It is a lightweight alternative to ControlNet at only 158MB per model versus ControlNet's roughly 1.4GB. It runs conditioning once for the entire generation instead of every step, uses 93% less storage, and adds minimal extra VRAM. The control is less precise, but on a 6-8GB card where ControlNet is straining your VRAM budget, T2I-Adapter lets you get some structural control without the full cost.

What Does Not Work Well

Being straight about this is useful: ControlNet is not magic.

OpenPose struggles with extreme camera angles, particularly top-down or very low-angle shots where limb occlusion confuses the skeleton extraction. If the preprocessor preview looks wrong (missing limbs, incorrect joint positions), the generated image will also be wrong regardless of how good your prompt is.

Hands are still hard even with DWPose, which is specifically designed for better hand tracking than the original OpenPose. ControlNet inpainting is the workaround — generate the image, then inpaint only the hand area with a high denoising strength while keeping the rest intact. More steps, but the results are significantly better than hoping the full generation gets hands right.

On SDXL, ControlNet support is less mature than on SD 1.5. Xinsir's Union model covers most use cases, but some preprocessor types that work perfectly on SD 1.5 behave inconsistently on SDXL. SD 1.5 still offers the most control types, the smallest model files, the most community resources, and the lowest VRAM requirements if ControlNet is central to your workflow.

Common Errors and Fixes

The generated image ignores the ControlNet input completely.

Either the Enable checkbox in the ControlNet panel is not checked (it defaults to unchecked), or the preprocessor and model do not match. Check both. Also confirm the model file is in the correct ControlNet folder and appears in the dropdown — if you can not select it, the file is missing.

The skeleton preview looks correct but the generated character ignores the pose.

Control Weight is too low. Raise it toward 1.0. If it is already at 1.0, the base checkpoint you are using may not respond well to ControlNet — try a different SD 1.5 or SDXL checkpoint that is known to work with pose conditioning.

CUDA out of memory when using multiple ControlNet units.

Each additional ControlNet stacks VRAM usage — three ControlNets can add 4-6GB on SD 1.5. Solutions in order: reduce the number of active ControlNet units, lower the image resolution, enable the Low VRAM checkbox in the ControlNet settings panel, or switch to T2I-Adapter for units where you do not need full ControlNet precision.

IP-Adapter makes all characters look the same regardless of prompt.

IP-Adapter weight is too high. Drop it to 0.5-0.7. At high weights it overwrites the text prompt's influence — the reference image appearance dominates so strongly that changing the prompt makes little difference.

Practical Tips Before You Start

• Always check the preprocessor preview before generating. If the skeleton, edge map, or depth map looks wrong, the output will be wrong. Fix the reference image or adjust thresholds before wasting generation time.
• Start with a single ControlNet unit and get it working before adding a second. Stacking doubles the variables when something goes wrong.
• Save working ControlNet configurations as presets using the save button in the ControlNet panel. Good OpenPose settings for character work are worth preserving across sessions.
• For character sheets specifically: generate all poses at the same resolution and seed (or similar seeds) to reduce variation between outputs. The ControlNet handles pose consistency; seed consistency handles background and lighting.
• Blender is genuinely useful here. You can pose a Blender character rig, screenshot it from Forge's ControlNet input, and use that as an OpenPose reference — Blender's 3D modeling capabilities make it useful for creating posing skeletons for Stable Diffusion workflows. This connects directly to the Blender guide from this series.

FAQ

Do I need ControlNet or can I just use better prompts?

For specific poses, no amount of prompt engineering reliably replaces ControlNet. Prompts nudge probabilities. ControlNet enforces structure. They are not substitutes for each other.

Which ControlNet model should I start with?

OpenPose for character work, Canny for environment sketches. Download those two first and learn them before anything else.

Can I use ControlNet with Midjourney or DALL-E?

No. ControlNet is a Stable Diffusion extension. Midjourney has its own --cref and --sref reference parameters (covered in the Midjourney guide), which offer some of the same character consistency features without the same level of pose precision. DALL-E has no equivalent system.

How much VRAM do I actually need?

Single ControlNet on SD 1.5: 8GB is comfortable. Single ControlNet on SDXL: 12GB recommended. Stacking two or more units: 12GB minimum, 16GB+ for reliability. T2I-Adapter is the workaround for lower VRAM budgets.

Does ControlNet work on Flux models?

Yes, with limitations. Shakker Labs Union Pro 2.0 is the current best ControlNet option for Flux, but Flux ControlNet support is less mature than SD 1.5. You also need 24GB VRAM total for Flux plus a ControlNet model running simultaneously.

What is the difference between OpenPose and DWPose?

DWPose is the newer, more accurate variant. It tracks hand and face keypoints more reliably than the original OpenPose, which makes a meaningful difference for character close-ups and poses where hand position matters. Use DWPose if your Forge or ComfyUI install has it — it is the better choice for most character work in 2026.

What to Try First

Install the extension, download control_openpose, take a photo of yourself in any pose, and generate a character concept using it as the reference. That alone will immediately show you what ControlNet actually does more clearly than any description can. Then try Canny on a rough sketch. Then look at IP-Adapter for the face consistency problem.

The stacking workflow for professional character sheets takes longer to set up but is worth learning if you are generating large amounts of concept art for a project. Once you have a working pose reference library and a character IP-Adapter reference image, generating a full character sheet becomes fast and repeatable in a way that pure prompt work simply is not.

Next Topics To Learn

• Stable Diffusion for Game Developers — the foundation this guide builds on, covering installation, checkpoints, LoRAs, and basic workflows.
• Midjourney for Game Developers — for comparison, Midjourney's --cref and --sref parameters offer character and style consistency through a simpler interface without ControlNet's precision.
• Blender Basics for Unity Developers — Blender's pose rigging connects directly to ControlNet workflows for generating consistent character reference images.