Skip to content
Weights Not Walls
Menu

Open-source tools

135 tools for running, serving, fine-tuning, quantising and evaluating open-weight models. Each one is checked against GitHub: an open-source licence, when it was last pushed to, its latest release, and which model formats its own documentation says it loads.

Open-source tools
Tool, and what its repo says it isCategoryLicenceStarsLatest releaseLast pushLoads
opencodeanomalycoThe open source coding agent.Agents and codingLicenceMITStars228kReleasenot fetched yetPush19 Nov 2026
ollamaollamaGet up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.Run locally, ServeLicenceMITStars188kReleasev0.40.169 Nov 2026Push18 Nov 2026LoadsGGUF, safetensors
transformershuggingfaceTransformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.LibrariesLicenceApache-2.0Stars169kReleasenot fetched yetPush19 Nov 2026Loadssafetensors, GGUF, AWQ, GPTQ, bitsandbytes
codexopenaiLightweight coding agent that runs in your terminalAgents and codingLicenceApache-2.0Stars136kReleasenot fetched yetPush19 Nov 2026
llama.cppggml-orgLLM inference in C/C++Run locally, Serve, Quantise and convertLicenceMITStars136kReleasev1.0.013 Nov 2026Push19 Nov 2026LoadsGGUF
vllmvllm-projectA high-throughput and memory-efficient inference and serving engine for LLMsServeLicenceApache-2.0Stars97kReleasenot fetched yetPush19 Nov 2026Loadssafetensors, AWQ, GPTQ, FP8, NVFP4, GGUF, bitsandbytes
OpenHandsOpenHandsOpenHands: AI-Driven DevelopmentAgents and codingLicenceMITStars94kReleasenot fetched yetPush19 Nov 2026
NextChatChatGPTNextWebZero-config AI chat assistant. No API key needed - sign up and instantly chat with GPT-5, Claude 4, Gemini 2.5, DeepSeek & 100+ top models. Pay-as-you-go saves you more. Available on Web, iOS, macOS, Android, Linux, Windows.Chat interfacesLicenceMITStars89kReleasenot fetched yetPush20 Oct 2026
unslothunslothaiLocal UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.Fine-tune, Quantise and convertLicenceApache-2.0Stars80kReleasenot fetched yetPush19 Nov 2026
gpt4allnomic-aistaleGPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.Run locally, Chat interfacesLicenceMITStars77kReleasev3.10.025 Feb 2025Push27 May 2025LoadsGGUF
LlamaFactoryhiyougaUnified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)Fine-tuneLicenceApache-2.0Stars76kReleasenot fetched yetPush19 Nov 2026
clineclineAutonomous coding agent as an SDK, IDE extension, or CLI assistant.Agents and codingLicenceApache-2.0Stars72kReleasenot fetched yetPush19 Nov 2026
openinterpreteropeninterpreterA coding agent for open models like Kimi K3.1 and GLM 5.3Agents and codingLicenceApache-2.0Stars69kReleasenot fetched yetPush15 Nov 2026
anything-llmMintplex-LabsStop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experienceChat interfacesLicenceMITStars68kReleasenot fetched yetPush19 Nov 2026
gooseaaif-goosean open source, extensible AI agent that goes beyond code suggestions - install, execute, edit, and test with any LLMAgents and codingLicenceApache-2.0Stars57kReleasenot fetched yetPush19 Nov 2026
cherry-studioCherryHQYour AI workspace for models, agents, and productivity.Chat interfacesLicenceAGPL-3.0Stars54kReleasenot fetched yetPush19 Nov 2026
LocalAImudlerLocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.ServeLicenceMITStars50kReleasenot fetched yetPush19 Nov 2026LoadsGGUF, safetensors
aiderAider-AIaider is AI pair programming in your terminalAgents and codingLicenceApache-2.0Stars50kReleasenot fetched yetPush22 May 2026
exoexo-exploreRun frontier AI locally.Run locallyLicenceApache-2.0Stars48kReleasenot fetched yetPush6 Nov 2026LoadsMLX
textgenoobaboogaOpen-source desktop app for local LLMs. Text, vision, tool-calling, OpenAI/Anthropic-compatible API. 100% private.Run locally, Chat interfacesLicenceAGPL-3.0Stars48kReleasev4.920 May 2026Push17 Aug 2026LoadsGGUF, safetensors, EXL3
LibreChatLibreChat-AIEnhanced ChatGPT Clone: Features Agents, MCP, Skills, DeepSeek, Anthropic, AWS, OpenAI, Responses API, Azure, Groq, o1, GPT-5, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, Code Interpreter, langchain, DALL-E-3, OpenAPI Actions, Functions, Secure Multi-User Auth, Presets, open-source for self-hosting. ActiveChat interfacesLicenceMITStars47kReleasenot fetched yetPush19 Nov 2026
janjanhqJan is an open source alternative to ChatGPT that runs 100% offline on your computer.Run locally, Chat interfacesLicenceApache-2.0*Stars46kReleasev0.8.828 Oct 2026Push18 Nov 2026LoadsGGUF
chatboxchatboxaiPowerful AI ClientChat interfacesLicenceGPL-3.0Stars42kReleasenot fetched yetPush18 Nov 2026
BitNetmicrosoftOfficial inference framework for 1-bit LLMsRun locallyLicenceMITStars42kReleasenot fetched yetPush27 Jul 2026LoadsGGUF
Codewhalecodewhale-hqAn open-source coding agent for your terminal, written in Rust. It works with any model, can split one task across several agents, and every line of it is yours to read.Agents and codingLicenceMITStars41kReleasenot fetched yetPush19 Nov 2026
sglangsgl-projectSGLang is a high-performance serving framework for large language models and multimodal models.ServeLicenceApache-2.0Stars39kReleasenot fetched yetPush19 Nov 2026Loadssafetensors, AWQ, GPTQ, FP8, NVFP4
VaneItzCrazyKnsVane is an AI-powered answering engine.Chat interfacesLicenceMITStars38kReleasenot fetched yetPush1 Sep 2026
khojkhoj-aiYour AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (gpt, claude, gemini, llama, qwen, mistral). Get started - free.Chat interfacesLicenceAGPL-3.0Stars38kReleasenot fetched yetPush2 Aug 2026
continuecontinuedevopen-source coding agentAgents and codingLicenceApache-2.0Stars37kReleasenot fetched yetPush18 Nov 2026
SillyTavernSillyTavernLLM Frontend for Power Users.Chat interfacesLicenceAGPL-3.0Stars36kReleasenot fetched yetPush15 Nov 2026
qwen-codeQwenLMAn open-source AI coding agent that lives in your terminal.Agents and codingLicenceApache-2.0Stars31kReleasenot fetched yetPush19 Nov 2026
smolagentshuggingfacesmolagents: a barebones library for agents that think in code.Agents and coding, LibrariesLicenceApache-2.0Stars30kReleasenot fetched yetPush18 Nov 2026
mlxml-exploreMLX: An array framework for Apple siliconLibrariesLicenceMITStars29kReleasenot fetched yetPush18 Nov 2026LoadsMLX
kilocodeKilo-OrgKilo is the all-in-one agentic engineering platform. Build, ship, and iterate faster with the most popular open source coding agent.Agents and codingLicenceMITStars28kReleasenot fetched yetPush19 Nov 2026
promptfoopromptfooTest your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.EvaluateLicenceMITStars27kReleasenot fetched yetPush19 Nov 2026
llamafilemozilla-aiDistribute and run LLMs with a single file.Run locally, ServeLicenceApache-2.0*Stars27kRelease0.10.615 Sep 2026Push9 Nov 2026LoadsGGUF
verlverl-projectverl/HybridFlow: A Flexible and Efficient RL Post-Training FrameworkFine-tuneLicenceApache-2.0Stars24kReleasenot fetched yetPush19 Nov 2026
Roo-CodeRooCodeIncarchivedRoo Code gives you a whole dev team of AI agents in your code editor.Agents and codingLicenceApache-2.0Stars24kReleasenot fetched yetPush15 May 2026
omlxjundotLLM inference server with continuous batching & SSD caching for Apple Silicon - managed from the macOS menu barServeLicenceApache-2.0Stars24kReleasenot fetched yetPush19 Nov 2026LoadsMLX
mlc-llmmlc-aiUniversal LLM Deployment Engine with ML CompilationRun locally, ServeLicenceApache-2.0Stars23kReleasenot fetched yetPush18 Nov 2026
pefthuggingfacePEFT: State-of-the-art Parameter-Efficient Fine-Tuning.Fine-tune, LibrariesLicenceApache-2.0Stars22kReleasenot fetched yetPush19 Nov 2026
guidanceguidance-aiA guidance language for controlling large language models.LibrariesLicenceMITStars22kReleasenot fetched yetPush21 May 2026
candlehuggingfaceMinimalist ML framework for RustLibrariesLicenceApache-2.0Stars21kReleasenot fetched yetPush18 Nov 2026LoadsGGUF, safetensors
trlhuggingfaceTrain transformer language models with reinforcement learning.Fine-tuneLicenceApache-2.0Stars20kReleasenot fetched yetPush19 Nov 2026
ktransformerskvcache-aiA Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune OptimizationsRun locally, ServeLicenceApache-2.0Stars20kReleasenot fetched yetPush13 Nov 2026
deepevalconfident-aiThe LLM Evaluation FrameworkEvaluateLicenceApache-2.0Stars20kReleasenot fetched yetPush19 Nov 2026
web-llmmlc-aiHigh-performance In-browser LLM Inference EngineRun locallyLicenceApache-2.0Stars19kReleasenot fetched yetPush18 Nov 2026
MNNalibabaMNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.Run locally, LibrariesLicenceApache-2.0Stars17kReleasenot fetched yetPush19 Nov 2026
transformers.jshuggingfaceState-of-the-art Machine Learning for the web. Run Transformers directly in your browser, with no need for a server!LibrariesLicenceApache-2.0Stars16kReleasenot fetched yetPush18 Nov 2026LoadsONNX
ms-swiftmodelscopeUse PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).Fine-tuneLicenceApache-2.0Stars16kReleasenot fetched yetPush19 Nov 2026
ggmlggml-orgTensor library for machine learningLibrariesLicenceMITStars16kReleasenot fetched yetPush16 Nov 2026
outlinesdottxt-aiStructured OutputsLibrariesLicenceApache-2.0Stars16kReleasenot fetched yetPush21 Sep 2026
TensorRT-LLMNVIDIATensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.ServeLicenceApache-2.0*Stars15kReleasenot fetched yetPush19 Nov 2026Loadssafetensors, FP8, NVFP4, AWQ
cactuscactus-computeSub-50mb agentic, voice and vision intelligence for consumer devices.Run locallyLicenceApache-2.0Stars15kReleasenot fetched yetPush18 Nov 2026
lm-evaluation-harnessEleutherAIA framework for few-shot evaluation of language models.EvaluateLicenceMITStars15kReleasenot fetched yetPush4 Nov 2026
litgptLightning-AI20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.Fine-tune, ServeLicenceApache-2.0Stars14kReleasenot fetched yetPush10 Nov 2026
axolotlaxolotl-ai-cloudGo ahead and axolotl questionsFine-tuneLicenceApache-2.0Stars13kReleasenot fetched yetPush19 Nov 2026
OpenLLMbentomlRun any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.ServeLicenceApache-2.0Stars13kReleasenot fetched yetPush16 Nov 2026
koboldcppLostRuinsKoboldCpp - Run GGUF AI models locally with a KoboldAI UI. One File. Zero Install.Run locally, Chat interfacesLicenceAGPL-3.0Stars12kReleasev1.124.231 Oct 2026Push19 Nov 2026LoadsGGUF
openvinoopenvinotoolkitOpenVINO is an open source toolkit for optimizing and deploying AI inferenceLibrariesLicenceApache-2.0Stars11kReleasenot fetched yetPush19 Nov 2026
chat-uihuggingfaceThe open source codebase powering HuggingChatChat interfacesLicenceApache-2.0Stars11kReleasenot fetched yetPush19 Nov 2026
text-generation-inferencehuggingfacearchivedLarge Language Model Text Generation InferenceServeLicenceApache-2.0Stars11kReleasenot fetched yetPush21 Mar 2026Loadssafetensors, AWQ, GPTQ, FP8, bitsandbytes
petalsbigscience-workshopstaleRun LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloadingServeLicenceMITStars11kReleasenot fetched yetPush7 Sep 2024
llama-cpp-pythonabetlenPython bindings for llama.cppLibrariesLicenceMITStars11kReleasenot fetched yetPush15 Nov 2026LoadsGGUF
garakNVIDIAthe LLM vulnerability scannerEvaluateLicenceApache-2.0Stars10kReleasenot fetched yetPush18 Nov 2026
PowerInferTiiny-AIHigh-speed Large Language Model Serving for Local DeploymentRun locallyLicenceMITStars9.9kReleasenot fetched yetPush11 May 2026LoadsGGUF
inferencexorbitsaiOpen-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.ServeLicenceApache-2.0Stars9.7kReleasenot fetched yetPush19 Nov 2026Loadssafetensors, MLX
pocketpal-aia-ghorbaniAn app that brings language models directly to your phone.Run locally, Chat interfacesLicenceMITStars9.6kReleasenot fetched yetPush19 Nov 2026LoadsGGUF
oumioumi-aiEasily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!Fine-tune, EvaluateLicenceApache-2.0Stars9.4kReleasenot fetched yetPush18 Nov 2026
bitsandbytesbitsandbytes-foundationAccessible large language models via k-bit quantization for PyTorch.Quantise and convert, LibrariesLicenceMITStars8.5kReleasenot fetched yetPush2 Nov 2026
GenieXqualcommRun frontier LLMs and VLMs locally on Qualcomm devices across NPU, GPU, and CPU with a few lines of codeRun locallyLicenceBSD-3-ClauseStars8.5kReleasenot fetched yetPush19 Nov 2026LoadsGGUF
dynamoai-dynamoA Datacenter Scale Distributed Inference Serving FrameworkServeLicenceApache-2.0*Stars8.5kReleasenot fetched yetPush19 Nov 2026
page-assistn4ze3mUse your locally running AI models to assist you in your web browsingChat interfacesLicenceMITStars8.3kReleasenot fetched yetPush15 Nov 2026
osaurusosaurus-aiOwn your AI. The open-source agent harness for macOS. Run local or cloud models with persistent memory, skills, tools, and autonomous execution. Built in Swift.Run locally, Chat interfacesLicenceMITStars8.2kReleasenot fetched yetPush19 Nov 2026LoadsMLX
lmdeployInternLMLMDeploy is a toolkit for compressing, deploying, and serving LLMs.Serve, Quantise and convertLicenceApache-2.0Stars8.2kReleasenot fetched yetPush19 Nov 2026Loadssafetensors, AWQ
mistral.rsEricLBuehlerFast, flexible LLM inferenceRun locally, ServeLicenceMITStars7.8kReleasev0.9.519 Oct 2026Push9 Nov 2026LoadsGGUF, safetensors
mlx-lmml-exploreRun LLMs with MLXRun locally, Fine-tune, Quantise and convert, ServeLicenceMITStars7.7kReleasev0.32.216 Oct 2026Push18 Nov 2026LoadsMLX, safetensors
opencompassopen-compassOpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.EvaluateLicenceApache-2.0Stars7.6kReleasenot fetched yetPush18 Nov 2026
mergekitarcee-aiTools for merging pretrained large language models.Quantise and convertLicenceLGPL-3.0Stars7.5kReleasenot fetched yetPush31 Oct 2026
big-AGIenricorosAI suite powered by state-of-the-art models and providing advanced AI/AGI functions. Includes AI personas, AGI functions, world-class Beam multi-model chats, text-to-image, voice, response streaming, code highlighting and execution, PDF import, presets for developers, much more. Deploy on-prem or in the cloud.Chat interfacesLicenceMITStars7.2kReleasenot fetched yetPush19 Nov 2026
gemma.cppgooglelightweight, standalone C++ inference engine for Google's Gemma models.Run locallyLicenceApache-2.0Stars7.1kReleasenot fetched yetPush19 Nov 2026
codecompanion.nvimolimorrisAI Coding, Vim StyleAgents and codingLicenceApache-2.0Stars6.9kReleasenot fetched yetPush18 Nov 2026
LiteRT-LMgoogle-ai-edgeLiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.Run locallyLicenceApache-2.0Stars6.9kReleasenot fetched yetPush19 Nov 2026
Liger-KernellinkedinEfficient Triton Kernels for LLM TrainingFine-tune, LibrariesLicenceBSD-2-ClauseStars6.7kReleasenot fetched yetPush19 Nov 2026
lemonadelemonade-sdkLemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8ZkRun locally, ServeLicenceApache-2.0Stars6.2kReleasenot fetched yetPush19 Nov 2026LoadsGGUF, ONNX
SWE-benchSWE-benchSWE-bench: Can Language Models Resolve Real-world Github Issues?EvaluateLicenceMITStars6.2kReleasenot fetched yetPush18 Sep 2026
gpustackgpustackA GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.ServeLicenceApache-2.0Stars6.1kReleasenot fetched yetPush19 Nov 2026
shimmyMichael-A-KuykendallPure-Rust WebGPU inference engine - OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.Run locally, ServeLicenceApache-2.0Stars6.0kReleasenot fetched yetPush12 Nov 2026LoadsGGUF
mlx-vlmNeywa-LabsMLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.Run locally, Fine-tuneLicenceMITStars5.9kReleasev0.8.016 Nov 2026Push18 Nov 2026LoadsMLX, safetensors
torchtunemeta-pytorchPyTorch native post-training libraryFine-tuneLicenceBSD-3-ClauseStars5.8kReleasenot fetched yetPush9 Nov 2026
Model-OptimizerNVIDIAA unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.Quantise and convertLicenceApache-2.0Stars5.6kReleasenot fetched yetPush19 Nov 2026
transformerlab-apptransformerlabThe open source research environment for AI researchers to seamlessly train, evaluate, and scale models from local hardware to GPU clusters.Fine-tuneLicenceAGPL-3.0Stars5.2kReleasenot fetched yetPush5 Nov 2026
h2o-llmstudioh2oaiH2O LLM Studio - a framework and no-code GUI for fine-tuning LLMs. Documentation: https://docs.h2o.ai/h2o-llmstudio/Fine-tuneLicenceApache-2.0Stars5.2kReleasenot fetched yetPush19 Nov 2026
executorchpytorchOn-device AI across mobile, embedded and edge for PyTorchLibrariesLicenceBSD-3-Clause*Stars5.2kReleasenot fetched yetPush19 Nov 2026
AutoGPTQAutoGPTQarchivedAn easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.Quantise and convertLicenceMITStars5.1kReleasenot fetched yetPush11 Apr 2025
CTranslate2OpenNMTFast inference engine for Transformer modelsLibrariesLicenceMITStars4.8kReleasenot fetched yetPush18 Nov 2026
simple-evalsopenaiNo description on GitHub.EvaluateLicenceMITStars4.7kReleasenot fetched yetPush22 Apr 2026
exllamav2turboderp-orgA fast inference library for running LLMs locally on modern consumer-class GPUsRun locally, Quantise and convertLicenceMITStars4.7kReleasev0.3.213 Jul 2025Push4 Mar 2026LoadsEXL2, GPTQ
lmms-evalEvolvingLMMs-LabOne-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio TasksEvaluateLicenceMIT AND Apache-2.0*Stars4.5kReleasenot fetched yetPush18 Nov 2026
VLMEvalKitopen-compassOpen-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarksEvaluateLicenceApache-2.0Stars4.5kReleasenot fetched yetPush19 Nov 2026
Rapid-MLXraullenchaiRapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.Run locally, ServeLicenceApache-2.0Stars4.2kReleasenot fetched yetPush19 Nov 2026LoadsMLX
huggingface_hubhuggingfaceThe official CLI and Python client for the Hugging Face Hub.LibrariesLicenceApache-2.0Stars4.1kReleasenot fetched yetPush19 Nov 2026
llm-compressorvllm-projectState-of-the-art LLM compression, built for production inference with vLLMQuantise and convertLicenceApache-2.0Stars4.0kReleasenot fetched yetPush19 Nov 2026
safetensorssafetensorsSimple, safe way to store and distribute tensorsLibrariesLicenceApache-2.0Stars4.0kReleasenot fetched yetPush19 Nov 2026
loraxpredibasearchivedMulti-LoRA inference server that scales to 1000s of fine-tuned LLMsServeLicenceApache-2.0Stars3.8kReleasenot fetched yetPush28 May 2026Loadssafetensors
LLamaSharpSciSharpA C#/.NET library to run LLM (LLaMA/LLaVA) on your local device efficiently.LibrariesLicenceMITStars3.8kReleasenot fetched yetPush17 Nov 2026LoadsGGUF
llm-awqmit-han-labstale[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and AccelerationQuantise and convertLicenceMITStars3.7kReleasenot fetched yetPush17 Jul 2025
optimumhuggingfaceAccelerate inference and training of Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization toolsQuantise and convert, LibrariesLicenceApache-2.0Stars3.5kReleasenot fetched yetPush16 Nov 2026
ik_llama.cppikawrakowllama.cpp fork with additional SOTA quants and improved performanceRun locally, ServeLicenceMITStars3.4kReleasenone on GitHubPush19 Nov 2026LoadsGGUF
inspect_aiUKGovernmentBEISInspect: A framework for large language model evaluationsEvaluateLicenceMITStars3.3kReleasenot fetched yetPush19 Nov 2026
distributed-llamab4rtazDistributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.Run locallyLicenceMITStars3.1kReleasenot fetched yetPush17 Nov 2026
ramalamacontainersRamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.Run locally, ServeLicenceMITStars3.1kReleasev0.26.04 Nov 2026Push14 Nov 2026LoadsGGUF, safetensors
aopytorchPyTorch native quantization for training and inferenceQuantise and convert, LibrariesLicenceBSD-3-Clause*Stars3.0kReleasenot fetched yetPush19 Nov 2026
helmstanford-crfmHolistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.EvaluateLicenceApache-2.0Stars3.0kReleasenot fetched yetPush1 Sep 2026
ChatterUIVali-98A frontend for running models on mobile or connecting to your preferred API providers.Run locally, Chat interfacesLicenceAGPL-3.0Stars2.9kReleasenot fetched yetPush13 Nov 2026LoadsGGUF
lightevalhuggingfaceLighteval is your all-in-one toolkit for evaluating LLMs across multiple backendsEvaluateLicenceMITStars2.6kReleasenot fetched yetPush18 Nov 2026
AutoAWQcasper-hansenarchivedAutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:Quantise and convertLicenceMITStars2.3kReleasenot fetched yetPush11 May 2025
node-llama-cppwithcataiRun AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation levelLibrariesLicenceMITStars2.2kReleasenot fetched yetPush18 Nov 2026LoadsGGUF
alpaca_evaltatsu-labstaleAn automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.EvaluateLicenceApache-2.0Stars2.0kReleasenot fetched yetPush9 Aug 2025
xgrammarmlc-aiFast, Flexible and Portable Structured GenerationLibrariesLicenceApache-2.0Stars2.0kReleasenot fetched yetPush18 Nov 2026
sonardphnAILarge-scale LLM inference engineServeLicenceAGPL-3.0Stars1.9kReleasenot fetched yetPush17 Nov 2026Loadssafetensors, FP8
exllamav3turboderp-orgAn optimized quantization and inference library for running LLMs locally on modern consumer-class GPUsRun locally, Quantise and convertLicenceMITStars1.8kReleasev1.6.713 Nov 2026Push16 Nov 2026LoadsEXL3
auto-roundintelPTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度Quantise and convertLicenceApache-2.0Stars1.7kReleasenot fetched yetPush19 Nov 2026
AlpacaJeffserLocal and online AI hubChat interfacesLicenceGPL-3.0Stars1.7kReleasenot fetched yetPush2 Oct 2026
llama.vscodeggml-orgVS Code extension for LLM-assisted code/text completionAgents and codingLicenceMITStars1.6kReleasenot fetched yetPush13 Nov 2026
tabbyAPItheroyallabThe official API server for Exllama. OAI compatible, lightweight, and fast.ServeLicenceAGPL-3.0Stars1.5kReleasenot fetched yetPush14 Nov 2026LoadsEXL3
GPTQModelModelCloudLLM model quantization (compression) toolkit with HW acceleration support for Nvidia CUDA, AMD ROCm, Intel XPU, Tenstorrent, Huawei Ascent NPU, and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.Quantise and convertLicenceApache-2.0*Stars1.3kReleasenot fetched yetPush19 Nov 2026
hollamafmaclenA minimal LLM chat app that runs entirely in your browserChat interfacesLicenceMITStars1.2kReleasenot fetched yetPush19 Nov 2026
llmperfray-projectarchivedLLMPerf is a library for validating and benchmarking LLMsEvaluateLicenceApache-2.0Stars1.1kReleasenot fetched yetPush9 Dec 2024
arena-hard-autoArena-OSSstaleArena-Hard-Auto: An automatic LLM benchmark.EvaluateLicenceApache-2.0Stars1.1kReleasenot fetched yetPush21 Jun 2025
optimum-quantohuggingfaceA pytorch quantization backend for optimumQuantise and convertLicenceApache-2.0Stars1.1kReleasenot fetched yetPush17 Nov 2026
keras-hubkeras-teamPretrained model hub for Keras 3.LibrariesLicenceApache-2.0Stars999Releasenot fetched yetPush18 Nov 2026
LiveCodeBenchLiveCodeBenchstaleOfficial repository for the paper "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code"EvaluateLicenceMITStars979Releasenot fetched yetPush16 Jul 2025
chatllm.cppfoldlPure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)Run locallyLicenceMITStars943Releasenot fetched yetPush15 Nov 2026
inference-benchmarkerhuggingfaceInference server benchmarking toolEvaluateLicenceApache-2.0Stars177Releasenot fetched yetPush23 Sep 2026

How tools are chosen

  • The list is kept by hand. It started from tools we knew of and from GitHub's most starred repositories on topics such as local-llm, gguf, llm-inference, fine-tuning and llm-evaluation, and every entry is checked against the GitHub API each time the data is refreshed.
  • A tool is listed only with an open-source licence: one that the SPDX licence list marks as OSI approved, as GitHub's licence detection reports it. Where GitHub cannot identify the licence file, we read the file; those licences are marked * and the reading is on the tool's page. A tool whose application is not under an open-source licence is not listed here, however well known; the ones we checked are below with the reason.
  • "Loads" shows a model format only when the tool's README or documentation names it. It says the format is supported, not that every model in it works.
  • Archived means the owners made the repository read only. Stale means no push for over 365 days; "pushed to in the last 90 days" counts any push to the repository, as of the fetch time below.
  • Descriptions are each repository's own words, with emoji removed. Stars are GitHub's count and say how many people bookmarked a repository, not how good it is.
  • Listing is not endorsement. Nobody can pay to be listed or to be placed higher, and the order is the sort you choose.
  • To suggest a tool, or to tell us something here is wrong, write to hello@weightsnotwalls.com with a link to its repository.

Checked, not listed

These repositories were on our list, but GitHub reports no open-source licence for them, or the licence file is not one.

  • charmbracelet/crush GitHub reports: NOASSERTIONOur reading of the licence file: The Functional Source License 1.1 (FSL-1.1-MIT), a source-available licence that is not OSI approved.
  • BerriAI/litellm GitHub reports: NOASSERTIONOur reading of the licence file: Open core: MIT outside the litellm/enterprise/ directory, which has its own licence.
  • lobehub/lobehub GitHub reports: NOASSERTIONOur reading of the licence file: The LobeHub Community License: Apache License 2.0 with added commercial conditions, so not an OSI-approved licence.
  • open-webui/open-webui GitHub reports: NOASSERTIONOur reading of the licence file: The Open WebUI License: BSD-style terms plus conditions on keeping its branding, so not an OSI-approved licence.
  • airockchip/rknn-llm GitHub reports: NOASSERTIONOur reading of the licence file: A BSD-style licence with an added fourth clause, not a standard OSI-approved text.
  • TabbyML/tabby GitHub reports: NOASSERTIONOur reading of the licence file: Open core: Apache License 2.0 outside the ee/ directory, which has its own licence.

Tool data from the GitHub API, fetched 19 Nov 2026, 15:44 UTC. Licence approval from the SPDX licence list (version ddf7382). Formats from each repository's README and the documentation pages named in our list.