Open-source tools
135 tools for running, serving, fine-tuning, quantising and evaluating open-weight models. Each one is checked against GitHub: an open-source licence, when it was last pushed to, its latest release, and which model formats its own documentation says it loads.
| Tool, and what its repo says it is | Category | Licence | Stars | Latest release | Last push | Loads |
|---|---|---|---|---|---|---|
| opencodeanomalycoThe open source coding agent. | Agents and coding | LicenceMIT | Stars228k | Releasenot fetched yet | Push19 Nov 2026 | |
| ollamaollamaGet up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. | Run locally, Serve | LicenceMIT | Stars188k | Releasev0.40.169 Nov 2026 | Push18 Nov 2026 | LoadsGGUF, safetensors |
| transformershuggingfaceTransformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training. | Libraries | LicenceApache-2.0 | Stars169k | Releasenot fetched yet | Push19 Nov 2026 | Loadssafetensors, GGUF, AWQ, GPTQ, bitsandbytes |
| codexopenaiLightweight coding agent that runs in your terminal | Agents and coding | LicenceApache-2.0 | Stars136k | Releasenot fetched yet | Push19 Nov 2026 | |
| llama.cppggml-orgLLM inference in C/C++ | Run locally, Serve, Quantise and convert | LicenceMIT | Stars136k | Releasev1.0.013 Nov 2026 | Push19 Nov 2026 | LoadsGGUF |
| vllmvllm-projectA high-throughput and memory-efficient inference and serving engine for LLMs | Serve | LicenceApache-2.0 | Stars97k | Releasenot fetched yet | Push19 Nov 2026 | Loadssafetensors, AWQ, GPTQ, FP8, NVFP4, GGUF, bitsandbytes |
| OpenHandsOpenHandsOpenHands: AI-Driven Development | Agents and coding | LicenceMIT | Stars94k | Releasenot fetched yet | Push19 Nov 2026 | |
| NextChatChatGPTNextWebZero-config AI chat assistant. No API key needed - sign up and instantly chat with GPT-5, Claude 4, Gemini 2.5, DeepSeek & 100+ top models. Pay-as-you-go saves you more. Available on Web, iOS, macOS, Android, Linux, Windows. | Chat interfaces | LicenceMIT | Stars89k | Releasenot fetched yet | Push20 Oct 2026 | |
| unslothunslothaiLocal UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more. | Fine-tune, Quantise and convert | LicenceApache-2.0 | Stars80k | Releasenot fetched yet | Push19 Nov 2026 | |
| gpt4allnomic-aistaleGPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use. | Run locally, Chat interfaces | LicenceMIT | Stars77k | Releasev3.10.025 Feb 2025 | Push27 May 2025 | LoadsGGUF |
| LlamaFactoryhiyougaUnified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024) | Fine-tune | LicenceApache-2.0 | Stars76k | Releasenot fetched yet | Push19 Nov 2026 | |
| clineclineAutonomous coding agent as an SDK, IDE extension, or CLI assistant. | Agents and coding | LicenceApache-2.0 | Stars72k | Releasenot fetched yet | Push19 Nov 2026 | |
| openinterpreteropeninterpreterA coding agent for open models like Kimi K3.1 and GLM 5.3 | Agents and coding | LicenceApache-2.0 | Stars69k | Releasenot fetched yet | Push15 Nov 2026 | |
| anything-llmMintplex-LabsStop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience | Chat interfaces | LicenceMIT | Stars68k | Releasenot fetched yet | Push19 Nov 2026 | |
| gooseaaif-goosean open source, extensible AI agent that goes beyond code suggestions - install, execute, edit, and test with any LLM | Agents and coding | LicenceApache-2.0 | Stars57k | Releasenot fetched yet | Push19 Nov 2026 | |
| cherry-studioCherryHQYour AI workspace for models, agents, and productivity. | Chat interfaces | LicenceAGPL-3.0 | Stars54k | Releasenot fetched yet | Push19 Nov 2026 | |
| LocalAImudlerLocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required. | Serve | LicenceMIT | Stars50k | Releasenot fetched yet | Push19 Nov 2026 | LoadsGGUF, safetensors |
| aiderAider-AIaider is AI pair programming in your terminal | Agents and coding | LicenceApache-2.0 | Stars50k | Releasenot fetched yet | Push22 May 2026 | |
| exoexo-exploreRun frontier AI locally. | Run locally | LicenceApache-2.0 | Stars48k | Releasenot fetched yet | Push6 Nov 2026 | LoadsMLX |
| textgenoobaboogaOpen-source desktop app for local LLMs. Text, vision, tool-calling, OpenAI/Anthropic-compatible API. 100% private. | Run locally, Chat interfaces | LicenceAGPL-3.0 | Stars48k | Releasev4.920 May 2026 | Push17 Aug 2026 | LoadsGGUF, safetensors, EXL3 |
| LibreChatLibreChat-AIEnhanced ChatGPT Clone: Features Agents, MCP, Skills, DeepSeek, Anthropic, AWS, OpenAI, Responses API, Azure, Groq, o1, GPT-5, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, Code Interpreter, langchain, DALL-E-3, OpenAPI Actions, Functions, Secure Multi-User Auth, Presets, open-source for self-hosting. Active | Chat interfaces | LicenceMIT | Stars47k | Releasenot fetched yet | Push19 Nov 2026 | |
| janjanhqJan is an open source alternative to ChatGPT that runs 100% offline on your computer. | Run locally, Chat interfaces | LicenceApache-2.0* | Stars46k | Releasev0.8.828 Oct 2026 | Push18 Nov 2026 | LoadsGGUF |
| chatboxchatboxaiPowerful AI Client | Chat interfaces | LicenceGPL-3.0 | Stars42k | Releasenot fetched yet | Push18 Nov 2026 | |
| BitNetmicrosoftOfficial inference framework for 1-bit LLMs | Run locally | LicenceMIT | Stars42k | Releasenot fetched yet | Push27 Jul 2026 | LoadsGGUF |
| Codewhalecodewhale-hqAn open-source coding agent for your terminal, written in Rust. It works with any model, can split one task across several agents, and every line of it is yours to read. | Agents and coding | LicenceMIT | Stars41k | Releasenot fetched yet | Push19 Nov 2026 | |
| sglangsgl-projectSGLang is a high-performance serving framework for large language models and multimodal models. | Serve | LicenceApache-2.0 | Stars39k | Releasenot fetched yet | Push19 Nov 2026 | Loadssafetensors, AWQ, GPTQ, FP8, NVFP4 |
| VaneItzCrazyKnsVane is an AI-powered answering engine. | Chat interfaces | LicenceMIT | Stars38k | Releasenot fetched yet | Push1 Sep 2026 | |
| khojkhoj-aiYour AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (gpt, claude, gemini, llama, qwen, mistral). Get started - free. | Chat interfaces | LicenceAGPL-3.0 | Stars38k | Releasenot fetched yet | Push2 Aug 2026 | |
| continuecontinuedevopen-source coding agent | Agents and coding | LicenceApache-2.0 | Stars37k | Releasenot fetched yet | Push18 Nov 2026 | |
| SillyTavernSillyTavernLLM Frontend for Power Users. | Chat interfaces | LicenceAGPL-3.0 | Stars36k | Releasenot fetched yet | Push15 Nov 2026 | |
| qwen-codeQwenLMAn open-source AI coding agent that lives in your terminal. | Agents and coding | LicenceApache-2.0 | Stars31k | Releasenot fetched yet | Push19 Nov 2026 | |
| smolagentshuggingfacesmolagents: a barebones library for agents that think in code. | Agents and coding, Libraries | LicenceApache-2.0 | Stars30k | Releasenot fetched yet | Push18 Nov 2026 | |
| mlxml-exploreMLX: An array framework for Apple silicon | Libraries | LicenceMIT | Stars29k | Releasenot fetched yet | Push18 Nov 2026 | LoadsMLX |
| kilocodeKilo-OrgKilo is the all-in-one agentic engineering platform. Build, ship, and iterate faster with the most popular open source coding agent. | Agents and coding | LicenceMIT | Stars28k | Releasenot fetched yet | Push19 Nov 2026 | |
| promptfoopromptfooTest your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic. | Evaluate | LicenceMIT | Stars27k | Releasenot fetched yet | Push19 Nov 2026 | |
| llamafilemozilla-aiDistribute and run LLMs with a single file. | Run locally, Serve | LicenceApache-2.0* | Stars27k | Release0.10.615 Sep 2026 | Push9 Nov 2026 | LoadsGGUF |
| verlverl-projectverl/HybridFlow: A Flexible and Efficient RL Post-Training Framework | Fine-tune | LicenceApache-2.0 | Stars24k | Releasenot fetched yet | Push19 Nov 2026 | |
| Roo-CodeRooCodeIncarchivedRoo Code gives you a whole dev team of AI agents in your code editor. | Agents and coding | LicenceApache-2.0 | Stars24k | Releasenot fetched yet | Push15 May 2026 | |
| omlxjundotLLM inference server with continuous batching & SSD caching for Apple Silicon - managed from the macOS menu bar | Serve | LicenceApache-2.0 | Stars24k | Releasenot fetched yet | Push19 Nov 2026 | LoadsMLX |
| mlc-llmmlc-aiUniversal LLM Deployment Engine with ML Compilation | Run locally, Serve | LicenceApache-2.0 | Stars23k | Releasenot fetched yet | Push18 Nov 2026 | |
| pefthuggingfacePEFT: State-of-the-art Parameter-Efficient Fine-Tuning. | Fine-tune, Libraries | LicenceApache-2.0 | Stars22k | Releasenot fetched yet | Push19 Nov 2026 | |
| guidanceguidance-aiA guidance language for controlling large language models. | Libraries | LicenceMIT | Stars22k | Releasenot fetched yet | Push21 May 2026 | |
| candlehuggingfaceMinimalist ML framework for Rust | Libraries | LicenceApache-2.0 | Stars21k | Releasenot fetched yet | Push18 Nov 2026 | LoadsGGUF, safetensors |
| trlhuggingfaceTrain transformer language models with reinforcement learning. | Fine-tune | LicenceApache-2.0 | Stars20k | Releasenot fetched yet | Push19 Nov 2026 | |
| ktransformerskvcache-aiA Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations | Run locally, Serve | LicenceApache-2.0 | Stars20k | Releasenot fetched yet | Push13 Nov 2026 | |
| deepevalconfident-aiThe LLM Evaluation Framework | Evaluate | LicenceApache-2.0 | Stars20k | Releasenot fetched yet | Push19 Nov 2026 | |
| web-llmmlc-aiHigh-performance In-browser LLM Inference Engine | Run locally | LicenceApache-2.0 | Stars19k | Releasenot fetched yet | Push18 Nov 2026 | |
| MNNalibabaMNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI. | Run locally, Libraries | LicenceApache-2.0 | Stars17k | Releasenot fetched yet | Push19 Nov 2026 | |
| transformers.jshuggingfaceState-of-the-art Machine Learning for the web. Run Transformers directly in your browser, with no need for a server! | Libraries | LicenceApache-2.0 | Stars16k | Releasenot fetched yet | Push18 Nov 2026 | LoadsONNX |
| ms-swiftmodelscopeUse PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025). | Fine-tune | LicenceApache-2.0 | Stars16k | Releasenot fetched yet | Push19 Nov 2026 | |
| ggmlggml-orgTensor library for machine learning | Libraries | LicenceMIT | Stars16k | Releasenot fetched yet | Push16 Nov 2026 | |
| outlinesdottxt-aiStructured Outputs | Libraries | LicenceApache-2.0 | Stars16k | Releasenot fetched yet | Push21 Sep 2026 | |
| TensorRT-LLMNVIDIATensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way. | Serve | LicenceApache-2.0* | Stars15k | Releasenot fetched yet | Push19 Nov 2026 | Loadssafetensors, FP8, NVFP4, AWQ |
| cactuscactus-computeSub-50mb agentic, voice and vision intelligence for consumer devices. | Run locally | LicenceApache-2.0 | Stars15k | Releasenot fetched yet | Push18 Nov 2026 | |
| lm-evaluation-harnessEleutherAIA framework for few-shot evaluation of language models. | Evaluate | LicenceMIT | Stars15k | Releasenot fetched yet | Push4 Nov 2026 | |
| litgptLightning-AI20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale. | Fine-tune, Serve | LicenceApache-2.0 | Stars14k | Releasenot fetched yet | Push10 Nov 2026 | |
| axolotlaxolotl-ai-cloudGo ahead and axolotl questions | Fine-tune | LicenceApache-2.0 | Stars13k | Releasenot fetched yet | Push19 Nov 2026 | |
| OpenLLMbentomlRun any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud. | Serve | LicenceApache-2.0 | Stars13k | Releasenot fetched yet | Push16 Nov 2026 | |
| koboldcppLostRuinsKoboldCpp - Run GGUF AI models locally with a KoboldAI UI. One File. Zero Install. | Run locally, Chat interfaces | LicenceAGPL-3.0 | Stars12k | Releasev1.124.231 Oct 2026 | Push19 Nov 2026 | LoadsGGUF |
| openvinoopenvinotoolkitOpenVINO is an open source toolkit for optimizing and deploying AI inference | Libraries | LicenceApache-2.0 | Stars11k | Releasenot fetched yet | Push19 Nov 2026 | |
| chat-uihuggingfaceThe open source codebase powering HuggingChat | Chat interfaces | LicenceApache-2.0 | Stars11k | Releasenot fetched yet | Push19 Nov 2026 | |
| text-generation-inferencehuggingfacearchivedLarge Language Model Text Generation Inference | Serve | LicenceApache-2.0 | Stars11k | Releasenot fetched yet | Push21 Mar 2026 | Loadssafetensors, AWQ, GPTQ, FP8, bitsandbytes |
| petalsbigscience-workshopstaleRun LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading | Serve | LicenceMIT | Stars11k | Releasenot fetched yet | Push7 Sep 2024 | |
| llama-cpp-pythonabetlenPython bindings for llama.cpp | Libraries | LicenceMIT | Stars11k | Releasenot fetched yet | Push15 Nov 2026 | LoadsGGUF |
| garakNVIDIAthe LLM vulnerability scanner | Evaluate | LicenceApache-2.0 | Stars10k | Releasenot fetched yet | Push18 Nov 2026 | |
| PowerInferTiiny-AIHigh-speed Large Language Model Serving for Local Deployment | Run locally | LicenceMIT | Stars9.9k | Releasenot fetched yet | Push11 May 2026 | LoadsGGUF |
| inferencexorbitsaiOpen-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs. | Serve | LicenceApache-2.0 | Stars9.7k | Releasenot fetched yet | Push19 Nov 2026 | Loadssafetensors, MLX |
| pocketpal-aia-ghorbaniAn app that brings language models directly to your phone. | Run locally, Chat interfaces | LicenceMIT | Stars9.6k | Releasenot fetched yet | Push19 Nov 2026 | LoadsGGUF |
| oumioumi-aiEasily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM! | Fine-tune, Evaluate | LicenceApache-2.0 | Stars9.4k | Releasenot fetched yet | Push18 Nov 2026 | |
| bitsandbytesbitsandbytes-foundationAccessible large language models via k-bit quantization for PyTorch. | Quantise and convert, Libraries | LicenceMIT | Stars8.5k | Releasenot fetched yet | Push2 Nov 2026 | |
| GenieXqualcommRun frontier LLMs and VLMs locally on Qualcomm devices across NPU, GPU, and CPU with a few lines of code | Run locally | LicenceBSD-3-Clause | Stars8.5k | Releasenot fetched yet | Push19 Nov 2026 | LoadsGGUF |
| dynamoai-dynamoA Datacenter Scale Distributed Inference Serving Framework | Serve | LicenceApache-2.0* | Stars8.5k | Releasenot fetched yet | Push19 Nov 2026 | |
| page-assistn4ze3mUse your locally running AI models to assist you in your web browsing | Chat interfaces | LicenceMIT | Stars8.3k | Releasenot fetched yet | Push15 Nov 2026 | |
| osaurusosaurus-aiOwn your AI. The open-source agent harness for macOS. Run local or cloud models with persistent memory, skills, tools, and autonomous execution. Built in Swift. | Run locally, Chat interfaces | LicenceMIT | Stars8.2k | Releasenot fetched yet | Push19 Nov 2026 | LoadsMLX |
| lmdeployInternLMLMDeploy is a toolkit for compressing, deploying, and serving LLMs. | Serve, Quantise and convert | LicenceApache-2.0 | Stars8.2k | Releasenot fetched yet | Push19 Nov 2026 | Loadssafetensors, AWQ |
| mistral.rsEricLBuehlerFast, flexible LLM inference | Run locally, Serve | LicenceMIT | Stars7.8k | Releasev0.9.519 Oct 2026 | Push9 Nov 2026 | LoadsGGUF, safetensors |
| mlx-lmml-exploreRun LLMs with MLX | Run locally, Fine-tune, Quantise and convert, Serve | LicenceMIT | Stars7.7k | Releasev0.32.216 Oct 2026 | Push18 Nov 2026 | LoadsMLX, safetensors |
| opencompassopen-compassOpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety. | Evaluate | LicenceApache-2.0 | Stars7.6k | Releasenot fetched yet | Push18 Nov 2026 | |
| mergekitarcee-aiTools for merging pretrained large language models. | Quantise and convert | LicenceLGPL-3.0 | Stars7.5k | Releasenot fetched yet | Push31 Oct 2026 | |
| big-AGIenricorosAI suite powered by state-of-the-art models and providing advanced AI/AGI functions. Includes AI personas, AGI functions, world-class Beam multi-model chats, text-to-image, voice, response streaming, code highlighting and execution, PDF import, presets for developers, much more. Deploy on-prem or in the cloud. | Chat interfaces | LicenceMIT | Stars7.2k | Releasenot fetched yet | Push19 Nov 2026 | |
| gemma.cppgooglelightweight, standalone C++ inference engine for Google's Gemma models. | Run locally | LicenceApache-2.0 | Stars7.1k | Releasenot fetched yet | Push19 Nov 2026 | |
| codecompanion.nvimolimorrisAI Coding, Vim Style | Agents and coding | LicenceApache-2.0 | Stars6.9k | Releasenot fetched yet | Push18 Nov 2026 | |
| LiteRT-LMgoogle-ai-edgeLiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices. | Run locally | LicenceApache-2.0 | Stars6.9k | Releasenot fetched yet | Push19 Nov 2026 | |
| Liger-KernellinkedinEfficient Triton Kernels for LLM Training | Fine-tune, Libraries | LicenceBSD-2-Clause | Stars6.7k | Releasenot fetched yet | Push19 Nov 2026 | |
| lemonadelemonade-sdkLemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk | Run locally, Serve | LicenceApache-2.0 | Stars6.2k | Releasenot fetched yet | Push19 Nov 2026 | LoadsGGUF, ONNX |
| SWE-benchSWE-benchSWE-bench: Can Language Models Resolve Real-world Github Issues? | Evaluate | LicenceMIT | Stars6.2k | Releasenot fetched yet | Push18 Sep 2026 | |
| gpustackgpustackA GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances. | Serve | LicenceApache-2.0 | Stars6.1k | Releasenot fetched yet | Push19 Nov 2026 | |
| shimmyMichael-A-KuykendallPure-Rust WebGPU inference engine - OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary. | Run locally, Serve | LicenceApache-2.0 | Stars6.0k | Releasenot fetched yet | Push12 Nov 2026 | LoadsGGUF |
| mlx-vlmNeywa-LabsMLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX. | Run locally, Fine-tune | LicenceMIT | Stars5.9k | Releasev0.8.016 Nov 2026 | Push18 Nov 2026 | LoadsMLX, safetensors |
| torchtunemeta-pytorchPyTorch native post-training library | Fine-tune | LicenceBSD-3-Clause | Stars5.8k | Releasenot fetched yet | Push9 Nov 2026 | |
| Model-OptimizerNVIDIAA unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed. | Quantise and convert | LicenceApache-2.0 | Stars5.6k | Releasenot fetched yet | Push19 Nov 2026 | |
| transformerlab-apptransformerlabThe open source research environment for AI researchers to seamlessly train, evaluate, and scale models from local hardware to GPU clusters. | Fine-tune | LicenceAGPL-3.0 | Stars5.2k | Releasenot fetched yet | Push5 Nov 2026 | |
| h2o-llmstudioh2oaiH2O LLM Studio - a framework and no-code GUI for fine-tuning LLMs. Documentation: https://docs.h2o.ai/h2o-llmstudio/ | Fine-tune | LicenceApache-2.0 | Stars5.2k | Releasenot fetched yet | Push19 Nov 2026 | |
| executorchpytorchOn-device AI across mobile, embedded and edge for PyTorch | Libraries | LicenceBSD-3-Clause* | Stars5.2k | Releasenot fetched yet | Push19 Nov 2026 | |
| AutoGPTQAutoGPTQarchivedAn easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm. | Quantise and convert | LicenceMIT | Stars5.1k | Releasenot fetched yet | Push11 Apr 2025 | |
| CTranslate2OpenNMTFast inference engine for Transformer models | Libraries | LicenceMIT | Stars4.8k | Releasenot fetched yet | Push18 Nov 2026 | |
| simple-evalsopenaiNo description on GitHub. | Evaluate | LicenceMIT | Stars4.7k | Releasenot fetched yet | Push22 Apr 2026 | |
| exllamav2turboderp-orgA fast inference library for running LLMs locally on modern consumer-class GPUs | Run locally, Quantise and convert | LicenceMIT | Stars4.7k | Releasev0.3.213 Jul 2025 | Push4 Mar 2026 | LoadsEXL2, GPTQ |
| lmms-evalEvolvingLMMs-LabOne-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks | Evaluate | LicenceMIT AND Apache-2.0* | Stars4.5k | Releasenot fetched yet | Push18 Nov 2026 | |
| VLMEvalKitopen-compassOpen-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks | Evaluate | LicenceApache-2.0 | Stars4.5k | Releasenot fetched yet | Push19 Nov 2026 | |
| Rapid-MLXraullenchaiRapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights. | Run locally, Serve | LicenceApache-2.0 | Stars4.2k | Releasenot fetched yet | Push19 Nov 2026 | LoadsMLX |
| huggingface_hubhuggingfaceThe official CLI and Python client for the Hugging Face Hub. | Libraries | LicenceApache-2.0 | Stars4.1k | Releasenot fetched yet | Push19 Nov 2026 | |
| llm-compressorvllm-projectState-of-the-art LLM compression, built for production inference with vLLM | Quantise and convert | LicenceApache-2.0 | Stars4.0k | Releasenot fetched yet | Push19 Nov 2026 | |
| safetensorssafetensorsSimple, safe way to store and distribute tensors | Libraries | LicenceApache-2.0 | Stars4.0k | Releasenot fetched yet | Push19 Nov 2026 | |
| loraxpredibasearchivedMulti-LoRA inference server that scales to 1000s of fine-tuned LLMs | Serve | LicenceApache-2.0 | Stars3.8k | Releasenot fetched yet | Push28 May 2026 | Loadssafetensors |
| LLamaSharpSciSharpA C#/.NET library to run LLM (LLaMA/LLaVA) on your local device efficiently. | Libraries | LicenceMIT | Stars3.8k | Releasenot fetched yet | Push17 Nov 2026 | LoadsGGUF |
| llm-awqmit-han-labstale[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration | Quantise and convert | LicenceMIT | Stars3.7k | Releasenot fetched yet | Push17 Jul 2025 | |
| optimumhuggingfaceAccelerate inference and training of Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools | Quantise and convert, Libraries | LicenceApache-2.0 | Stars3.5k | Releasenot fetched yet | Push16 Nov 2026 | |
| ik_llama.cppikawrakowllama.cpp fork with additional SOTA quants and improved performance | Run locally, Serve | LicenceMIT | Stars3.4k | Releasenone on GitHub | Push19 Nov 2026 | LoadsGGUF |
| inspect_aiUKGovernmentBEISInspect: A framework for large language model evaluations | Evaluate | LicenceMIT | Stars3.3k | Releasenot fetched yet | Push19 Nov 2026 | |
| distributed-llamab4rtazDistributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference. | Run locally | LicenceMIT | Stars3.1k | Releasenot fetched yet | Push17 Nov 2026 | |
| ramalamacontainersRamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers. | Run locally, Serve | LicenceMIT | Stars3.1k | Releasev0.26.04 Nov 2026 | Push14 Nov 2026 | LoadsGGUF, safetensors |
| aopytorchPyTorch native quantization for training and inference | Quantise and convert, Libraries | LicenceBSD-3-Clause* | Stars3.0k | Releasenot fetched yet | Push19 Nov 2026 | |
| helmstanford-crfmHolistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models. | Evaluate | LicenceApache-2.0 | Stars3.0k | Releasenot fetched yet | Push1 Sep 2026 | |
| ChatterUIVali-98A frontend for running models on mobile or connecting to your preferred API providers. | Run locally, Chat interfaces | LicenceAGPL-3.0 | Stars2.9k | Releasenot fetched yet | Push13 Nov 2026 | LoadsGGUF |
| lightevalhuggingfaceLighteval is your all-in-one toolkit for evaluating LLMs across multiple backends | Evaluate | LicenceMIT | Stars2.6k | Releasenot fetched yet | Push18 Nov 2026 | |
| AutoAWQcasper-hansenarchivedAutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation: | Quantise and convert | LicenceMIT | Stars2.3k | Releasenot fetched yet | Push11 May 2025 | |
| node-llama-cppwithcataiRun AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level | Libraries | LicenceMIT | Stars2.2k | Releasenot fetched yet | Push18 Nov 2026 | LoadsGGUF |
| alpaca_evaltatsu-labstaleAn automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast. | Evaluate | LicenceApache-2.0 | Stars2.0k | Releasenot fetched yet | Push9 Aug 2025 | |
| xgrammarmlc-aiFast, Flexible and Portable Structured Generation | Libraries | LicenceApache-2.0 | Stars2.0k | Releasenot fetched yet | Push18 Nov 2026 | |
| sonardphnAILarge-scale LLM inference engine | Serve | LicenceAGPL-3.0 | Stars1.9k | Releasenot fetched yet | Push17 Nov 2026 | Loadssafetensors, FP8 |
| exllamav3turboderp-orgAn optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs | Run locally, Quantise and convert | LicenceMIT | Stars1.8k | Releasev1.6.713 Nov 2026 | Push16 Nov 2026 | LoadsEXL3 |
| auto-roundintelPTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度 | Quantise and convert | LicenceApache-2.0 | Stars1.7k | Releasenot fetched yet | Push19 Nov 2026 | |
| AlpacaJeffserLocal and online AI hub | Chat interfaces | LicenceGPL-3.0 | Stars1.7k | Releasenot fetched yet | Push2 Oct 2026 | |
| llama.vscodeggml-orgVS Code extension for LLM-assisted code/text completion | Agents and coding | LicenceMIT | Stars1.6k | Releasenot fetched yet | Push13 Nov 2026 | |
| tabbyAPItheroyallabThe official API server for Exllama. OAI compatible, lightweight, and fast. | Serve | LicenceAGPL-3.0 | Stars1.5k | Releasenot fetched yet | Push14 Nov 2026 | LoadsEXL3 |
| GPTQModelModelCloudLLM model quantization (compression) toolkit with HW acceleration support for Nvidia CUDA, AMD ROCm, Intel XPU, Tenstorrent, Huawei Ascent NPU, and Intel/AMD/Apple CPU via HF, vLLM, and SGLang. | Quantise and convert | LicenceApache-2.0* | Stars1.3k | Releasenot fetched yet | Push19 Nov 2026 | |
| hollamafmaclenA minimal LLM chat app that runs entirely in your browser | Chat interfaces | LicenceMIT | Stars1.2k | Releasenot fetched yet | Push19 Nov 2026 | |
| llmperfray-projectarchivedLLMPerf is a library for validating and benchmarking LLMs | Evaluate | LicenceApache-2.0 | Stars1.1k | Releasenot fetched yet | Push9 Dec 2024 | |
| arena-hard-autoArena-OSSstaleArena-Hard-Auto: An automatic LLM benchmark. | Evaluate | LicenceApache-2.0 | Stars1.1k | Releasenot fetched yet | Push21 Jun 2025 | |
| optimum-quantohuggingfaceA pytorch quantization backend for optimum | Quantise and convert | LicenceApache-2.0 | Stars1.1k | Releasenot fetched yet | Push17 Nov 2026 | |
| keras-hubkeras-teamPretrained model hub for Keras 3. | Libraries | LicenceApache-2.0 | Stars999 | Releasenot fetched yet | Push18 Nov 2026 | |
| LiveCodeBenchLiveCodeBenchstaleOfficial repository for the paper "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code" | Evaluate | LicenceMIT | Stars979 | Releasenot fetched yet | Push16 Jul 2025 | |
| chatllm.cppfoldlPure C++ implementation of several models for real-time chatting on your computer (CPU & GPU) | Run locally | LicenceMIT | Stars943 | Releasenot fetched yet | Push15 Nov 2026 | |
| inference-benchmarkerhuggingfaceInference server benchmarking tool | Evaluate | LicenceApache-2.0 | Stars177 | Releasenot fetched yet | Push23 Sep 2026 |
How tools are chosen
- The list is kept by hand. It started from tools we knew of and from GitHub's most starred repositories on topics such as local-llm, gguf, llm-inference, fine-tuning and llm-evaluation, and every entry is checked against the GitHub API each time the data is refreshed.
- A tool is listed only with an open-source licence: one that the SPDX licence list marks as OSI approved, as GitHub's licence detection reports it. Where GitHub cannot identify the licence file, we read the file; those licences are marked * and the reading is on the tool's page. A tool whose application is not under an open-source licence is not listed here, however well known; the ones we checked are below with the reason.
- "Loads" shows a model format only when the tool's README or documentation names it. It says the format is supported, not that every model in it works.
- Archived means the owners made the repository read only. Stale means no push for over 365 days; "pushed to in the last 90 days" counts any push to the repository, as of the fetch time below.
- Descriptions are each repository's own words, with emoji removed. Stars are GitHub's count and say how many people bookmarked a repository, not how good it is.
- Listing is not endorsement. Nobody can pay to be listed or to be placed higher, and the order is the sort you choose.
- To suggest a tool, or to tell us something here is wrong, write to hello@weightsnotwalls.com with a link to its repository.
Checked, not listed
These repositories were on our list, but GitHub reports no open-source licence for them, or the licence file is not one.
- charmbracelet/crush GitHub reports: NOASSERTIONOur reading of the licence file: The Functional Source License 1.1 (FSL-1.1-MIT), a source-available licence that is not OSI approved.
- BerriAI/litellm GitHub reports: NOASSERTIONOur reading of the licence file: Open core: MIT outside the litellm/enterprise/ directory, which has its own licence.
- lobehub/lobehub GitHub reports: NOASSERTIONOur reading of the licence file: The LobeHub Community License: Apache License 2.0 with added commercial conditions, so not an OSI-approved licence.
- open-webui/open-webui GitHub reports: NOASSERTIONOur reading of the licence file: The Open WebUI License: BSD-style terms plus conditions on keeping its branding, so not an OSI-approved licence.
- airockchip/rknn-llm GitHub reports: NOASSERTIONOur reading of the licence file: A BSD-style licence with an added fourth clause, not a standard OSI-approved text.
- TabbyML/tabby GitHub reports: NOASSERTIONOur reading of the licence file: Open core: Apache License 2.0 outside the ee/ directory, which has its own licence.
Tool data from the GitHub API, fetched 19 Nov 2026, 15:44 UTC. Licence approval from the SPDX licence list (version ddf7382). Formats from each repository's README and the documentation pages named in our list.