▲ 85 today1k starsPythonNew · 12d old
Open-source LLM inference engine runs on AWS Trainium
What is trnrt?
TrnRT lets you serve large language models on a single AWS Trainium2 instance through an OpenAI-compatible API. It uses fused NKI megakernels and a 1-million-token context cache to keep latency low.
Who it is for
If you need high-throughput LLM serving without GPUs, you can spin up a Trainium box and use TrnRT to deploy models like GLM-5.3-Flash.
Try it in 60 seconds
You need: Docker, an AWS trn2.48xlarge Trainium2 instance, and an x86_64 host with Neuron SDK 2.32 and Python 3.13
git clone https://github.com/RunanywhereAI/trnrt.git && cd trnrtdocker build -f docker/Dockerfile \ --build-arg TRNRT_VERSION=$(python3 -c 'import trnrt; print(trnrt.__version__)') \ --build-arg TRNRT_REVISION=$(git rev-parse HEAD) -t trnrt .
Apache-2.0 licenseUpdated yesterdayPython10 open issues
New projects are not vetted. Skim the code, and run it in a fresh folder or a container before giving it your files or keys.
The story so far
Early Signal first spotted RunanywhereAI/trnrt on November 10, 2026, when it had 804 GitHub stars. It has since grown to 1k. On its busiest day it gained 85 stars. The project is 12 days old and written mainly in Python. It is published under the Apache-2.0 license and has 9 forks. Early Signal files it under Inference, LLMs, Agents.
Questions about trnrt
- Is trnrt free and open source?
- Yes. RunanywhereAI/trnrt is open source under the Apache-2.0 license, and the code is on GitHub at https://github.com/RunanywhereAI/trnrt.
- How do I install trnrt?
- Use one of these, depending on your setup: git clone https://github.com/RunanywhereAI/trnrt.git && cd trnrt · docker build -f docker/Dockerfile \ --build-arg TRNRT_VERSION=$(python3 -c 'import trnrt; print(trnrt.__version__)') \ --build-arg TRNRT_REVISION=$(git rev-parse HEAD) -t trnrt .
- What do I need to run trnrt?
- Docker, an AWS trn2.48xlarge Trainium2 instance, and an x86_64 host with Neuron SDK 2.32 and Python 3.13