FreeToken: run huge MoE models on the GPU you already own

FreeToken is not free tokens. It is an open-source Ollama/vLLM alternative built specifically for Mixture-of-Experts models.

The Ollama problem

You have a model bigger than your VRAM - say a 36GB model on a 24GB RTX 5090. Ollama spills the rest into system RAM. Every layer then shuffles between GPU and CPU, and the CPU grinds through it. Slow.

The FreeToken trick

In an MoE model, only a fraction of parameters activate per query - maybe 13B of a 300B model. Instead of moving the whole model in and out of VRAM, FreeToken streams only the experts each query needs into the GPU.

The GPU becomes a cache of RAM: hot, frequently-used parameters stay warm.

The verdict

• Low VRAM + MoE -> use FreeToken.
• Everything else -> Ollama is still better.

Details

• Apache-2.0, free, open source.
• Model is 36GB on 24GB GPU? FreeToken shuffles far less.

Running MoE on a low-VRAM GPU, FreeToken beats Ollama. For everything else, Ollama remains the better call. Give it a go.