r/JetsonNano

My Jetson Orin–optimized engine, little-gemma V1.0, substantially outperforms llama.cpp. Even after exhausting every practical GGUF option, however, it still falls short of the ideal performance level for Gemma E4B on the Jetson Orin Nano Super 8GB.

I forked exllamav3 and ported key code from little-gemma V1.0. This delivered the prefill improvement I needed. Although the decoding rate took a hit, overall voice-chat latency is lower. EXL3 on the Jetson Orin Nano isn’t for everyone, but if you need the superior prefill rate, this is it.

Fork: https://github.com/cortexist/exllamav3
Weights: https://huggingface.co/cortexist/gemma-4-E4B-it-qat-exl3-4.0bpw

Ran out of projects for your Jetson Orin Nano? Don't give it away. Here's a better idea.

For the last two years, we have been developing Mino, a hackable companion robot for makers. Mino can carry your NVIDIA Jetson Orin Nano Super Developer Kit and give it a new life. It comes with building blocks for you to develop robot applications based on ROS 2: motion control, gestures, stereo vision, speech-to-text, LLM, emotion classifier, remote control from a mobile app... and more to come.

We just put Mino on Kickstarter. The campaign will launch soon, with a limited number of super-early-bird and early-bird discounts. To increase your chance of getting one, open the pre-campaign link below and click on "Notify me on launch". Happy making!

https://www.kickstarter.com/projects/minrobotics/mino-hackable-smart-robot