Skip to content
Weights Not Walls
Menu

inference

by xorbitsai

In its repository's words: “Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.”

Repository on GitHubinference.readthedocs.io

Facts from GitHub

Licence
Apache-2.0 (Apache License 2.0), permissive, our reading
Activity
Last push 19 Nov 2026: pushed to in the last 90 days
Latest release
not fetched yet
Stars
9.7k
Main language
Python
On GitHub since
14 Jun 2023
Topics
artificial-intelligence, diffusers, gemma, glm, glm-5-3, inference, kimi, kimi-k3, llama-cpp, llamacpp, llm, machine-learning, mlx, openai-api, pytorch, qwen, sglang, transformers, vllm, whisper

What it says it supports

Loads
safetensors, MLX
Runs on
Linux, macOS, Windows
Hardware
NVIDIA, Apple silicon, CPU only

Each item is on our list for this tool and named in its README. A mention is not a test: check the project's documentation for your setup.

Reviewed models it can load

Matched by file format only, from our snapshot of the Hub: GGUF where we measured GGUF files, other formats where a conversion exists, safetensors for the original weights. Support for a new architecture can lag behind its release.

safetensors 428

openai-community/gpt2, microsoft/Florence-2-base, Qwen/Qwen3-0.6B, Qwen/Qwen3-14B, deepseek-ai/DeepSeek-V3.2, Qwen/Qwen3.8-27B, Qwen/Qwen3.5-9B, Qwen/Qwen2.5-7B-Instruct, zai-org/GLM-5.3-Flash, Qwen/Qwen3-VL-8B-Instruct, google/gemma-4-31B-it, Qwen/Qwen3-8B, and 416 more, most downloaded first

MLX 284

openai-community/gpt2, Qwen/Qwen3-0.6B, Qwen/Qwen3-14B, deepseek-ai/DeepSeek-V3.2, Qwen/Qwen3.8-27B, Qwen/Qwen3.5-9B, Qwen/Qwen2.5-7B-Instruct, zai-org/GLM-5.3-Flash, Qwen/Qwen3-VL-8B-Instruct, google/gemma-4-31B-it, Qwen/Qwen3-8B, baidu/Unlimited-OCR, and 272 more, most downloaded first

Other tools in Serve

All of Serve

Tool data from the GitHub API, fetched 19 Nov 2026, 15:36 UTC. Listing is not endorsement and nobody can pay to be listed. Something wrong here? Write to hello@weightsnotwalls.com.