r/JetsonNano

What my AI Waifu can do right now:

✅Voice output to the browser connect remotely to my Waifu running in Jetson Orin Nano over local LAN. The stereo glitch from before seems to have gone away.

✅End to end gap between my input and her voice output is shortened to between 5 to 25sec for normal chat depends on prompt length, memory recall and so on

✅Web-search to find update info such as the 0-3 defeat of Canada by Morocco in the World Cup match.

✅Simple Agentic workflow like go online do research for recipes and consolidate all the info and write the ingredients and preparations in a file in her workspace.

What she cannot do yet for now:

❌Voice input still couldn't work via the local mic in remote workstation via the browser. Works on Jetson itself but ASR cannot get some niche case words.

❌Gap cannot shorten to 3sec even though I have refactored the whole memory vector memory, and reduce the input tokens (system prompt, memory context) to the minimum.

❌Cannot separate websearch snippet and web fetch deep search to save time.

❌Complicated Agentic workflow will exhaust the context window and halt halfway.

❌Can setup reminder to sound alarm at certain time, but the alarm can only sound after I quit the program and restart, not able to sound alarm in same session.

Been procrastinating buying one for CV experiments, kept putting it off while it was $250, finally decided to buy it and now it's 3 options starting from $375

Eight tiny LLMs on a $250 Jetson Orin Nano Super — what I learned about running inference at the edge

I spent the last week running 8 small language models, from 135M parameters all the way to 1.2B -- on a single Jetson Orin Nano Super 8GB.

The models I tested:

• SmolLM2-135M
• SmolLM2-360M
• Qwen2.5-0.5B
• LFM2.5-350M
• LFM2.5-1.2B
• Qwen3-0.6B
• Llama3.2-1B
• Gemma3-1B.

All running on both llama.cpp CUDA and Ollama, across all four Jetson power modes - 7W, 15W, 25W, and MAXN.

Why both backends? Because I wanted to know if theres any real, noticeable difference between llama.cpp and Ollama inference and it turns out llama.cpp beats Ollama at sub-1B and almost same 1 B models.

Here's what I found.

At SmolLM2-135M Q4_K_M under llama.cpp at 25W:

• up to 165 tok/s (Ollama: 121 tok/s), 29.6 output tok/J (Ollama: 21.3)
• 0.31 s TTFT at ctx=2048 (Ollama: 0.46 s) -- llama.cpp is 1.37× faster on throughput, 1.39× on tok/J
• 487 total tok/J at ctx=2048, gen=64: best in suite

At LFM2.5-350M Q4_K_M under llama.cpp at 25W:

• 115 tok/s -- nearly matching SmolLM2-360M (369 MB) in only 219 MB
• Ollama drops to 28 tok/s at the same mode -- 4.20× gap, purely a kernel issue
• 17.16 output tok/J (Ollama: 6.39)
• 0.39 s TTFT at ctx=2048 (Ollama: 0.50 s)

At LFM2.5-1.2B Q4_K_M under llama.cpp at 25W:

• 54.1 tok/s: leads the ~1B class (15 % over Llama3.2-1B at 47.1, 33 % over Gemma3-1B at 40.8)
• Ollama: 21.8 tok/s -- llama.cpp is 2.48× faster
• 6.37 output tok/J (Ollama: 3.94), 1.03 s TTFT (Ollama: 1.11 s)
• Only 698 MB -- smallest footprint in the 1B class

At Llama3.2-1B Q4_K_M under llama.cpp at 25W:

• 47.1 tok/s (Ollama: 44.8) -- just 5 % apart, nearly identical across backends
• 5.48 output tok/J (Ollama: 5.38), 0.98 s TTFT (Ollama: 1.23 s)

Benchmark Methodology

• For each model × prompt × gen combo, aiperf sends 20 single-concurrency requests with synthetic prompts at the exact target token count.

• Power is sampled from tegrastats VDD_CPU_GPU_CV (mW → W) at 500 ms intervals. Tegrastats samples are assigned to exact prefill/decode phase windows using per-request nanosecond timestamps from profile_export.jsonl (aiperf's stats).

• Clocks were locked with jetson_clocks at all modes. Each run's power and clock speed was capped through nvpmodel and monitored for thermal stability (no sustained throttling; junction temp ≤ 73 °C).

• Latency percentile used throughout: all TTFT, ITL, and request latency (RL) values reported use the p50 (median) over the 20 requests per combo.

Analysis here

It was 249 and I had it in my cart, yesterday it was pushed to 300 and now it's 299. I looked elsewhere to buy it but I can't find it anywhere.

I'm new to Jetson Nano stuff. Started a project 2 months back and I will be needing more. Just wondering if this is a normal problem for these? Are they hard to get or going out of stock often?