Skip to content
learn
Global menu

1. Why Candle

Candle is a machine learning framework written in Rust by Hugging Face. It gives you tensors, automatic differentiation, neural network layers and implementations of well-known models, and it compiles into your program like any other crate: no Python, no C++ runtime to install next to it. This lesson places it among the tools a C# or Java developer may already know, takes the workspace apart, and runs a first program.

Where Candle sits

Section titled “Where Candle sits”
Language you write What computes Training What you ship
PyTorch Python libtorch, in C++ and CUDA yes a Python environment, or a model exported for another runtime
tch-rs Rust libtorch, through a C API yes your binary plus the libtorch libraries
TorchSharp C#, F# libtorch, through native packages yes your .NET application plus a libtorch-* NuGet package
ONNX Runtime C#, Java, Python, C++, Rust through ort its own C++ engine inference first your application plus the native runtime and an .onnx file
ML.NET C#, F# .NET trainers, and ONNX Runtime or TorchSharp for deep learning yes a .NET application
DJL Java an engine you choose: PyTorch, ONNX Runtime and others depends on the engine a JVM application plus the engine’s native libraries
Burn Rust backends behind a Backend trait, from CUDA to a pure Rust CPU backend yes a Rust binary
Candle Rust Rust code, with optional CUDA, Metal, MKL or Accelerate yes a Rust binary, or a WebAssembly module

Sources: the READMEs of tch-rs, TorchSharp, ort, DJL, Burn and Candle.

The row that matters for this course is the last column. With tch-rs or TorchSharp, the tensor math happens in libtorch, a native library of several hundred megabytes that you deploy next to your program. With ONNX Runtime, you deploy its native library and a model exported from Python. Candle’s own README states its goals in the same terms: make serverless inference possible with small binaries that start fast, and remove Python from production workloads.

What you give up is breadth. PyTorch has thousands of operations and a large ecosystem; Candle has the operations its models need. Lesson 2 meets some of the edges, such as operations that panic on integer tensors instead of returning an error.

The workspace

Section titled “The workspace”

Candle is published as several crates. The version studied here is 0.11.0, the latest on crates.io (June 26, 2026), whose sources are the tag 0.11.0, commit 31f35b1. Every link to Candle’s code in this course points to that commit.

Crate What it holds Lessons
candle-core Tensor, DType, Device, the CPU, CUDA and Metal backends, autodiff, quantized tensors, the safetensors and GGUF formats 2 to 4, 8
candle-nn layers (Linear, Embedding, normalizations), activations, losses, optimizers, VarBuilder 5, 6
candle-transformers implementations of published models: 125 entries under models/, from BERT to quantized Qwen3 7, 8
candle-datasets loaders for common data sets such as MNIST 5
candle-examples 111 command-line programs, one per model or task 7 to 9
candle-onnx an evaluator for ONNX graphs 11
candle-pyo3 Python bindings, the reverse of this course’s direction —
candle-kernels, candle-metal-kernels, candle-flash-attn GPU kernels in CUDA and Metal 9
candle-wasm-examples the same models compiled to WebAssembly for the browser 10

candle-core switches its backends on with Cargo features, declared in candle-core/Cargo.toml, lines 48-62. None is on by default, so a plain dependency gives you the CPU backend only.

Feature What it adds Where
cuda the CUDA backend, through the cudarc crate, and the CUDA kernels NVIDIA GPUs, needs the CUDA toolkit
cudnn cuda plus cuDNN for convolutions same
nccl cuda plus NCCL, to split a model over several GPUs same
metal the Metal backend Apple GPUs, macOS
accelerate Apple’s Accelerate framework for matrix products on the CPU macOS
mkl Intel’s Math Kernel Library for matrix products on the CPU x86 CPUs

candle-nn and candle-transformers forward the same features to candle-core. Lesson 9 covers the GPU features; this course’s code uses none of them.

Running the code

Section titled “Running the code”

The course code is a Cargo workspace in code/candle, with one example per topic and one per set of exercise solutions. You need Rust and a C compiler, for a reason explained below: on Windows, the Visual Studio Build Tools that rustup asks for are enough.

Terminal window
cd code\candle
cargo run --release --example l01_hello
bash check.sh

check.sh runs cargo fmt --check, clippy, the tests (including the compile_fail doctests of the snippets this course shows being rejected), then every example, and compares each output with the file in expected/. On Windows, run it from Git Bash.

The first program

Section titled “The first program”

The Candle book adds candle-core with cargo add --git, which follows the main branch. This course pins the published version instead, so that the outputs below stay true: Cargo.toml asks for exactly 0.11.0 with =, as the Cargo reference describes, and Cargo.lock is committed.

[workspace.dependencies]
candle-core = "=0.11.0"
candle-nn = "=0.11.0"

The program is the one of Candle’s README, a matrix product, with one change. The README fills the matrices with Tensor::randn, random numbers, and Candle’s CPU random generator can’t be seeded (lesson 2 shows the error), so its output changes on every run. examples/l01_hello.rs uses arange instead:

use candle_core::{Device, Tensor};


fn main() -> Result<(), Box<dyn std::error::Error>> {
    let device = Device::Cpu;


    // arange instead of the README's randn: the same numbers on every run and every OS
    let a = Tensor::arange(0f32, 6., &device)?.reshape((2, 3))?;
    let b = Tensor::arange(0f32, 12., &device)?.reshape((3, 4))?;


    let c = a.matmul(&b)?;
    println!("{c}");
    Ok(())
}
[[20., 23., 26., 29.],
 [56., 68., 80., 92.]]
Tensor[[2, 4], f32]

Four things to notice, coming from C# or Java:

  • Every operation returns a Result. arange, reshape and matmul can all fail, on a shape that doesn’t fit for instance, and they report it as a candle_core::Error value instead of throwing. The ? after each call returns the error to the caller, as the Rust course explains. Where TorchSharp throws an exception, Candle hands you a value.
  • The device is explicit. Every tensor is created on a Device: Device::Cpu here, a CUDA or Metal device in lesson 9.
  • The shape and the element type are run-time values. Tensor is one type for every shape and every element type; the last line of the output prints both, [2, 4] and f32. A shape mistake is found when the program runs, not when it compiles.
  • 0f32, 6. gives the first number a type suffix; Rust infers f32 for the second from it.

A (2, 3) matrix times a (3, 4) matrix gives a (2, 4) matrix: the first element is 0·0 + 1·4 + 2·8 = 20.

Two errors every beginner meets

Section titled “Two errors every beginner meets”

Printing the result of matmul without ? prints a Result, not a tensor, and Result has no Display implementation:

let c = a.matmul(&b);
println!("{c}");
error[E0277]: `Result<candle_core::Tensor, candle_core::Error>` doesn't implement `std::fmt::Display`
 --> course\examples\x_result.rs:8:15
  |
8 |     println!("{c}");
  |               ^^^ `Result<candle_core::Tensor, candle_core::Error>` cannot be formatted with the default formatter

The second one comes from the cheat sheet of Candle’s own README, line 278, which converts a tensor with tensor.to_dtype(&DType::F16)?. to_dtype takes the DType by value:

error[E0308]: mismatched types
    --> course\examples\x_readme.rs:5:32
     |
   5 |     let half = tensor.to_dtype(&DType::F16)?;
     |                       -------- ^^^^^^^^^^^ expected `DType`, found `&DType`
     |                       |
     |                       arguments to this method are incorrect

DType is a small Copy enum, so there is no reason to pass a reference; the compiler’s suggestion, removing the &, is the fix. Both snippets are compile_fail doctests in src/lib.rs, so the course notices if a later version of Candle or Rust accepts them.

What it costs to compile

Section titled “What it costs to compile”

The first cargo build --release of this workspace took 73 seconds and compiled 128 crates, on an Intel Core Ultra 9 285K under Windows 11 with Rust 1.94.0, limited to 4 parallel jobs (-j 4) because the machine was shared. The l01_hello executable is 1.4 MB. Later builds recompile only the course crate, in about a second.

Most of those crates are what you’d expect: gemm for matrix products, rayon for threads, half for 16-bit floats, safetensors for model files. One is not. cargo tree shows where the C compiler comes in:

> cargo tree -p candle-course -e normal -i onig_sys
onig_sys v69.9.3
└── onig v6.5.3
    └── tokenizers v0.22.2
        └── candle-core v0.11.0

candle-core 0.11.0 depends on Hugging Face’s tokenizers crate on every target except WebAssembly (Cargo.toml, lines 37-38), and with its onig feature, which builds the Oniguruma regular expression library from C sources. Only one file uses it, quantized/tokenizer.rs, which rebuilds a tokenizer from the metadata of a GGUF file (lesson 8). Of the 119 crates behind candle-core, 72 are in the dependency tree of tokenizers. On main, after 0.11.0, the dependency is still there, but with the pure Rust fancy-regex feature instead of onig (Cargo.toml at ddf1b87), so the C compiler requirement should go away with the next release (to verify when it is published).

Asking for a GPU

Section titled “Asking for a GPU”

examples/l01_devices.rs asks the build which backends it has, then asks for a CUDA and a Metal device:

println!("cuda feature:       {}", utils::cuda_is_available());
match Device::new_cuda(0) {
    Ok(d) => println!("Device::new_cuda(0): {d:?}"),
    Err(e) => println!("Device::new_cuda(0): error: {e}"),
}
// Falls back to the CPU instead of failing
let device = Device::cuda_if_available(0)?;
cuda feature:       false
metal feature:      false
mkl feature:        false
accelerate feature: false
Device::new_cuda(0): error: the candle crate has not been built with cuda support
Device::new_metal(0): error: the candle crate has not been built with metal support
Device::cuda_if_available(0): Cpu

The output is the same on a machine with an NVIDIA GPU: utils::cuda_is_available tells whether the cuda feature was compiled in, not whether a GPU is present. Without the feature, Candle compiles a stand-in backend whose constructors return this error. cuda_if_available is the usual way to write a program that uses a GPU when it was built for one, and the CPU otherwise.

The CPU backend has one more switch that isn’t a feature. examples/l01_machine.rs prints what the build knows about the processor; check.sh runs it without comparing, since it depends on the machine:

os / arch:        windows / x86_64
compiled for avx2: false
compiled for neon: false
compiled for f16c: false
threads (rayon):   24

The processor has AVX2, but the program wasn’t compiled for it: Rust targets a baseline x86-64 CPU unless you pass -C target-cpu=native. Candle’s own repository does, in .cargo/config.toml, so its examples are built for the machine they run on, and a crate that depends on Candle isn’t. Lesson 3 measures what that changes.

In GA, IX and TARS

Section titled “In GA, IX and TARS”

None of the three repositories this site studies depends on Candle today.

  • GA runs its embedding models in C# with ONNX Runtime: GA.Business.ML.csproj references Microsoft.ML.OnnxRuntime 1.23.2 and Microsoft.ML.Tokenizers 2.0.0, and OnnxEmbeddingService.cs opens an InferenceSession. That is the “ONNX Runtime” row of the table above; lesson 7 computes embeddings with Candle, and lesson 11 calls a Candle model from C#.
  • TARS does the same from F#: TarsEngine.FSharp.Cli.fsproj references ONNX Runtime 1.19.2 and Microsoft.ML.Tokenizers 0.22.0.
  • IX is written in Rust and chose not to build on Candle. Its automatic differentiation crate, ix-autograd, is written from scratch over ndarray; its documentation records the decision (lib.rs, line 27) and keeps a place for “a future Candle or GPU backend” (tensor.rs, lines 9-19). Lesson 4 runs the same gradient through both.

IX’s guide to Rust libraries describes Candle as “optimised for LLM inference on NVIDIA GPUs via cuTENSOR and cuDNN” (code-analysis-tools.md, line 129). The CUDA features of Candle’s workspace ask cudarc for cuBLAS, cuBLASLt, cuRAND and NVRTC, with cuDNN as an option (Cargo.toml, lines 50-61); cuTENSOR isn’t among them, and the CPU, Metal and WebAssembly backends aren’t mentioned. The journal lists it.

Key takeaways

Section titled “Key takeaways”
  • Candle is a Rust framework for tensors, autodiff and models, compiled into your binary. Its selling point next to tch-rs, TorchSharp and ONNX Runtime is that there is no native runtime to ship.
  • candle-core holds tensors and backends, candle-nn layers and optimizers, candle-transformers models. GPU backends and MKL or Accelerate are opt-in Cargo features.
  • Every tensor operation returns a Result; shapes and element types are checked when the program runs.
  • Pin the crates.io version with = for reproducible results. Version 0.11.0 needs a C compiler because candle-core pulls in tokenizers with Oniguruma.
  • cuda_is_available reports a compiled feature, not hardware. Without -C target-cpu=native, the CPU-specific kernels behind with_avx aren’t compiled in.

Exercises

Section titled “Exercises”

The solutions are in examples/l01_exercises.rs, and their output in expected/l01_exercises.txt.

  1. In l01_hello, compute b.matmul(&a) instead of a.matmul(&b). Read the error, then find a product of transposed matrices (t()) that is defined and gives the numbers of the lesson, transposed.
Solution

Lines 11-18:

outcome("b.matmul(&a)", b.matmul(&a));
// (3, 4) x (4, 3) is defined if we transpose b instead: b.t() is (4, 3), a.t() is (3, 2)
let c = b.t()?.matmul(&a.t()?)?;
println!("{c}");
// (AB)^T = B^T A^T: the same numbers as a.matmul(&b) of the lesson, transposed
let same = c.eq(&a.matmul(&b)?.t()?)?.min_all()?.to_scalar::<u8>()? == 1;
== exercise 1: b x a, then b x a transposed
b.matmul(&a): error: shape mismatch in matmul, lhs: [3, 4], rhs: [2, 3]
[[20., 56.],
 [23., 68.],
 [26., 80.],
 [29., 92.]]
Tensor[[4, 2], f32]
b.t() x a.t() equals (a x b).t(): true

A (3, 4) matrix can only multiply a matrix with 4 rows. The transpose of a product is the product of the transposes in the other order, so b.t() times a.t(), a (4, 3) times a (3, 2), gives the (4, 2) transpose of the lesson’s result. eq compares element by element and returns a tensor of u8, whose minimum is 1 only if every element matched.

  1. Convert a and b to F64, F16, BF16, U32, I64 and U8 with to_dtype, and try matmul on each. Which element types does the CPU backend multiply?
Solution

Lines 20-37:

for dtype in [DType::F64, DType::F16, DType::BF16, DType::U32, DType::I64, DType::U8] {
    let result = a.to_dtype(dtype)?.matmul(&b.to_dtype(dtype)?);
    match result {
        Ok(c) => println!("{dtype:?}: ok, first row {:?}", c.to_dtype(DType::F64)?.to_vec2::<f64>()?[0]),
        Err(e) => println!("{dtype:?}: error: {e}"),
    }
}
== exercise 2: matmul on each dtype
F64: ok, first row [20.0, 23.0, 26.0, 29.0]
F16: ok, first row [20.0, 23.0, 26.0, 29.0]
BF16: error: unsupported dtype BF16 for op matmul
U32: error: unsupported dtype U32 for op matmul
I64: error: unsupported dtype I64 for op matmul
U8: error: unsupported dtype U8 for op matmul

Only F16, F32 and F64: the CPU matrix product calls gemm, and cpu_backend/mod.rs, lines 1382-1385 rejects every other type. BF16, the “brain float” format many models are stored in, has to be converted to F32 first on the CPU. Integer tensors hold indexes and masks, not weights.

Sources

Section titled “Sources”