exllamav2
by turboderp-orgIn its repository's words: “A fast inference library for running LLMs locally on modern consumer-class GPUs”
Facts from GitHub
- Licence
- MIT (MIT License), permissive, our reading
- Activity
- Last push 4 Mar 2026: no push for 90 to 365 days
- Latest release
- v0.3.213 Jul 2025
- Stars
- 4.7k
- Main language
- Python
- On GitHub since
- 30 Aug 2023
What it says it supports
- Loads
- EXL2, GPTQ
- Runs on
- Linux, Windows
- Hardware
- NVIDIA
Each item is on our list for this tool and named in its README. A mention is not a test: check the project's documentation for your setup.
Reviewed models it can load
Matched by file format only, from our snapshot of the Hub: GGUF where we measured GGUF files, other formats where a conversion exists, safetensors for the original weights. Support for a new architecture can lag behind its release.
EXL2 23
openai-community/gpt2, Qwen/Qwen2.5-Coder-14B-Instruct, Qwen/Qwen2-VL-2B-Instruct, Qwen/Qwen2.5-32B-Instruct, Qwen/Qwen2-VL-7B-Instruct, Qwen/Qwen2.5-Coder-32B-Instruct, meta-llama/Llama-3.3-70B-Instruct, Qwen/Qwen2.5-14B-Instruct-1M, Qwen/Qwen2.5-72B-Instruct, ibm-granite/granite-3.3-8b-instruct, deepseek-ai/DeepSeek-R1-Distill-Qwen-14B, mistralai/Mistral-Small-24B-Instruct-2501, and 11 more, most downloaded first
GPTQ 142
openai-community/gpt2, Qwen/Qwen3-0.6B, Qwen/Qwen3-14B, Qwen/Qwen3.8-27B, Qwen/Qwen2.5-7B-Instruct, Qwen/Qwen3-VL-8B-Instruct, google/gemma-4-31B-it, Qwen/Qwen3-8B, Qwen/Qwen2.5-VL-7B-Instruct, google/gemma-4-26B-A4B-it, Qwen/Qwen2.5-1.5B-Instruct, google/gemma-4-E4B-it, and 130 more, most downloaded first
Other tools in Run locally
Tool data from the GitHub API, fetched 19 Nov 2026, 15:36 UTC. Listing is not endorsement and nobody can pay to be listed. Something wrong here? Write to hello@weightsnotwalls.com.