Skip to content
Weights Not Walls
Menu

TensorRT-LLM

by NVIDIA

In its repository's words: “TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.”

Repository on GitHubnvidia.github.io/TensorRT-LLM

Facts from GitHub

Licence
Apache-2.0 (Apache License 2.0), permissive, our readingGitHub's licence detection could not identify the licence file, so we read the file on 19 Nov 2026: The project is under the Apache License 2.0; the file then lists third-party code under its own licences.
Activity
Last push 19 Nov 2026: pushed to in the last 90 days
Latest release
not fetched yet
Stars
15k
Main language
Python
On GitHub since
16 Aug 2023
Topics
blackwell, cuda, llm-serving, moe, pytorch

What it says it supports

Loads
safetensors, FP8, NVFP4, AWQ
Runs on
Windows
Hardware
NVIDIA

Each item is on our list for this tool and named in its README. A mention is not a test: check the project's documentation for your setup.

Reviewed models it can load

Matched by file format only, from our snapshot of the Hub: GGUF where we measured GGUF files, other formats where a conversion exists, safetensors for the original weights. Support for a new architecture can lag behind its release.

safetensors 428

openai-community/gpt2, microsoft/Florence-2-base, Qwen/Qwen3-0.6B, Qwen/Qwen3-14B, deepseek-ai/DeepSeek-V3.2, Qwen/Qwen3.8-27B, Qwen/Qwen3.5-9B, Qwen/Qwen2.5-7B-Instruct, zai-org/GLM-5.3-Flash, Qwen/Qwen3-VL-8B-Instruct, google/gemma-4-31B-it, Qwen/Qwen3-8B, and 416 more, most downloaded first

FP8 235

openai-community/gpt2, Qwen/Qwen3-0.6B, Qwen/Qwen3-14B, Qwen/Qwen3.8-27B, Qwen/Qwen3.5-9B, Qwen/Qwen2.5-7B-Instruct, zai-org/GLM-5.3-Flash, Qwen/Qwen3-VL-8B-Instruct, google/gemma-4-31B-it, Qwen/Qwen3-8B, Qwen/Qwen3.5-2B, Qwen/Qwen2.5-VL-7B-Instruct, and 223 more, most downloaded first

NVFP4 164

Qwen/Qwen3-14B, deepseek-ai/DeepSeek-V3.2, Qwen/Qwen3.8-27B, Qwen/Qwen3.5-9B, zai-org/GLM-5.3-Flash, Qwen/Qwen3-VL-8B-Instruct, google/gemma-4-31B-it, Qwen/Qwen3-8B, baidu/Unlimited-OCR, Qwen/Qwen3.5-2B, Qwen/Qwen2.5-VL-7B-Instruct, google/gemma-4-26B-A4B-it, and 152 more, most downloaded first

AWQ 213

Qwen/Qwen3-0.6B, Qwen/Qwen3-14B, deepseek-ai/DeepSeek-V3.2, Qwen/Qwen3.8-27B, Qwen/Qwen3.5-9B, Qwen/Qwen2.5-7B-Instruct, zai-org/GLM-5.3-Flash, Qwen/Qwen3-VL-8B-Instruct, google/gemma-4-31B-it, Qwen/Qwen3-8B, baidu/Unlimited-OCR, Qwen/Qwen3.5-2B, and 201 more, most downloaded first

Other tools in Serve

All of Serve

Tool data from the GitHub API, fetched 19 Nov 2026, 15:35 UTC. Listing is not endorsement and nobody can pay to be listed. Something wrong here? Write to hello@weightsnotwalls.com.