# Colibri runs 744B-parameter AI models on a 25 GB PC by streaming experts from disk. The real speeds

> Colibri, a dependency-free C engine that keeps mixture-of-experts weights on NVMe and loads them on demand, is back on GitHub's daily trending list at about 47K stars. It really does run frontier-size open models on ordinary hardware, but independent tests show giant models crawl well below one token per second.

Published Nov 3, 2026 by TechMeasured Staff · Repos · Canonical: https://techmeasured.com/repos/colibri-run-744b-moe-models-consumer-hardware-real-speeds

**Colibri**, an open-source inference engine written in plain C with no dependencies, lets you run very large mixture-of-experts (MoE) language models on hardware you already own by keeping most of the model on disk and loading only the parts it needs. It was on GitHub's daily trending list on November 3 with about 800 new stars that day, and the repo showed 46.7K stars, 5.4K forks and an Apache-2.0 license when we checked.

Why it matters: the biggest open-weight models now run to hundreds of billions of parameters, far beyond the memory of a normal PC. Colibri's bet is that MoE models only use a few "experts" per token, so you can stream those from a fast SSD instead of buying hundreds of gigabytes of RAM or VRAM. That works, but whether it is usable depends almost entirely on the model size and your drive.

## How it works

According to the project README, the dense part of a model (attention layers, shared experts, embeddings) stays in RAM, while routed experts are read from disk whenever the router asks for them. A cache learns which experts your workload uses most. A GPU is optional: it can hold the hottest experts and dense layers via Vulkan, CUDA or Metal. The project says placement only affects speed, not precision or routing.

Its first tagged release, v1.0.0 in July, claimed GLM-5.2 (744B parameters) running in about 25 GB of RAM, output checked token-for-token against a Hugging Face `transformers` reference, and a compressed KV cache it says is 57x smaller. The latest release on the releases page, v2.0.0 from October 6, lets every MoE engine use any Vulkan GPU and serve up to 16 conversations at once, and adds MiMo-V2.6, Qwen3-Coder and image-model support.

Supported models listed on the repo include GLM-5.2/5.3, Kimi K3, DeepSeek V4 Flash and V4.1 Flash, MiMo-V2.6, Qwen3.6 and Qwen3.8 variants, and OLMoE. It can also read existing GGUF files.

## The speeds, from the project and from testers

The project's own benchmark table shows a wide gap between mid-size and giant models:

| Model | Hardware | Decode speed |
|---|---|---|
| Qwen3.6-35B-A3B | RTX PRO 4000 (24 GB), CUDA | 104.8 tok/s (project) |
| Qwen3.6-35B-A3B | Same machine, CPU only | 13.9 tok/s (project) |
| GLM-5.2 (744B) | Ryzen AI Max+ 395, 128 GB, CPU | 1.83 tok/s warm (project) |
| GLM-5.2 (744B) | 25 GB laptop | 0.05–0.1 tok/s cold (project) |
| GLM-5.2 (744B) | Strix Halo, 128 GB, PCIe 4.0 NVMe | 0.42 tok/s best warm (Sleeping Robots) |
| GLM-5.2 (744B) | RTX 4090 desktop, 64 GB, internal NVMe | ~0.8 tok/s (Better Stack) |
| GLM-5.2 (744B) | M2 Max MacBook Pro, 32 GB, external SSD | ~0.1 tok/s, 2+ min to first token (Better Stack) |

Two independent write-ups agree on the bottleneck. Sleeping Robots measured roughly 2.4 seconds per token on a 128 GB Strix Halo machine, with about 40% of decode time spent waiting on expert reads from disk; the author notes that 27–35B models run 80 to 130 times faster. Better Stack found 145 of 158 seconds went to I/O wait on a MacBook with an external SSD, and called an internal NVMe drive the minimum meaningful setup. Both tests were short, with one or two runs per configuration, so treat them as indications rather than benchmarks.

## Try it

The README lists 8 GB of RAM as the minimum (16 GB or more recommended) and 22 GB of free disk for the smallest model. The big ones are huge: GLM-5.2 is a 429 GB download and Kimi K3 is 1.56 TB.

```bash
# Linux (Ubuntu/Debian)
sudo apt install git python3 build-essential
git clone https://github.com/JustVugg/colibri
cd colibri
./start-here.sh
```

On Windows, the README says to download the ZIP and run `START-HERE.bat`. Once a model is in place, `./coli chat`, `./coli web` and `./coli serve` (an OpenAI-compatible API) are the main entry points.

## Who should care

If you want a capable local model at interactive speeds, Colibri's sweet spot is mid-size MoE models like Qwen3.6-35B-A3B on a machine with a decent GPU. Running a 744B model on a laptop is a real demonstration of what disk streaming can do, but at under one token per second it suits batch jobs or experiments, not chat. Put models on a fast internal NVMe, not a USB drive, and keep folders off `/mnt/c` if you use WSL, which the README warns is many times slower.

*TechMeasured did not run Colibri. Speeds come from the project's README and the two independent write-ups linked above.*


## Sources

1. [GitHub — JustVugg/colibri](https://github.com/JustVugg/colibri)
2. [GitHub — colibri releases](https://github.com/JustVugg/colibri/releases)
3. [GitHub — Trending repositories today](https://github.com/trending)
4. [Sleeping Robots — Running a 744B model on Strix Halo with Colibri](https://sleepingrobots.com/dreams/colibri-744b-moe-strix-halo)
5. [Better Stack — Colibri: how to run a 744-billion-parameter model on consumer hardware](https://betterstack.com/community/guides/ai/colibri-glm/)