134 likesNew · 21d old
NVIDIA releases tiny 4-bit LLM for on-device voice assistants
What is NVIDIA-Nemotron-3.5-Bolt-E4B-4GB?
The Nemotron-3.5-Bolt-E4B-4GB is an 8-billion-parameter language model quantized to 4-bit, tuned for fast, low-memory inference. It can handle up to 64 K tokens and is aimed at local voice assistants, interactive personas, and edge applications.
Who it is for
Developers building on-device or embedded AI can run a responsive LLM locally without a high-end GPU.
How to try it
Download the GGUF file from Hugging Face and run it with llama.cpp (e.g. `llama-cli -m NVIDIA-Nemotron-3.5-Bolt-E4B-4GB.Q4_K_M.gguf`).