r/ChatGPTArduino

I trained a transformer from scratch on an ESP32-S3. Not inference, actual on-chip training with hand-written backprop.

Most ESP32 language model projects run inference: you train on a GPU, quantize, flash it to the board. I wanted the opposite. The chip starts with random weights and trains itself.

Everything happens on board: random init (and no, not seed 42), tokenising the corpus, forward pass, cross-entropy, backprop, SGD with momentum (not Adam, not AdamW), checkpoint to flash, and generation from the weights it learned. Nothing outside the chip.

No PyTorch, no autograd. Every derivative in the backward pass is written out by hand in C.

Setup:

* ESP32-S3 N16R8, about $8

* SH1106 OLED showing the live loss

* Single block transformer, single head causal attention, tied embeddings, ReLU FFN, LayerNorm

* ~319K params, char level, vocab 31, context 32

* 5,000 steps, roughly two days on a phone charger

The training loss moving average went from 2.137 to 1.871 over the stretch I photographed. With vocab 31 a randomly initialised model has to start somewhere around ln(31) ≈ 3.43, but I never photographed the first steps, so I can't prove that part from the OLED.

The interesting constraint isn't the parameter count, it's memory. To train you need weights, gradients, optimizer momentum, activations and scratch buffers all resident at the same time. Inference has it much easier: it still needs activations, but no gradients and no optimizer state.

Where it's weak:

* No validation split. The checkpoint I keep is just the one with the lowest moving average of training loss.

* The corpus is Klingon: small, regular, agglutinative, and published under Apache 2.0. The output shows plausible use of suffixes like `-wI'`, `-Daq` and `-taHvIS`, but it isn't reliably meaningful.

* With a corpus this small I can't cleanly separate generalisation from memorisation.

* No full serial log. It ran unattended, so what I have is the code, the checkpoint and photos of the OLED at three points.

This is not ChatGPT on a microcontroller. It's a small experiment showing that an $8 ESP32-S3 can run the whole training loop of a transformer starting from random weights.

Apache 2.0. The corpus is in the repo so you can reproduce a run, but the fun part is swapping it for your own text.

https://github.com/Carloscodix/qapla

Note: written by me, translated and adapted to Reddit with AI help.

u/flash_speed3412 — 1 day ago

▲ 124 r/ChatGPTArduino+1 crossposts

Turned a wall-switch-box ESP32-S3 module into a desk AI quota display.

I kept hitting my Claude Code quota limit mid-task, so I built a small always-on panel that just shows how much I have left. And I also added later codex and antigravity. Now sits on my desk.

The device is a Waveshare ESP32-S3 Smart 86 Box with a touch 480×480 touch display, 8 MB octal PSRAM. 86×86 mm.

The display was the easy part. Four things that were harder:

1. The panel desynced permanently under load — and only on production builds. Under a sustained full-screen scroll the image would slip into a fixed vertical offset and stay there until reboot. Framebuffers live in PSRAM with a 30-line bounce buffer (28.8 KB) in internal DMA SRAM. At a 12 MHz pixel clock, scan-out alone pulls ~19 MB/s off the octal PSRAM — that buffer is 1.37 ms of runway. Add LVGL in DIRECT mode redrawing the whole framebuffer plus XIP fetches and the refill misses its deadline. It's a runway, not a queue, so the slip never recovers. 8 MHz → 2.06 ms → fixed. Except not quite: flash and PSRAM share the MSPI bus and the cache is off while flash is being programmed, and production has flash encryption on, so each page is AES-256'd and takes longer than the plaintext dev path. That's the entire reason it looked production-specific.

2. Internal DRAM, not PSRAM, is the thing you actually run out of. I had 8 MB of PSRAM and 6.2 KB of free internal heap. Everything that matters — TLS handshake buffers, DMA descriptors, ISR stacks — has to be internal, and LVGL plus the RGB driver had quietly eaten it in .bss. Moving the cold buffers to PSRAM freed 41,764 bytes and took free internal heap from 6,239 to 48,043. The rule I ended up with isn't "avoid PSRAM", it's "no PSRAM traffic on the live-render path" — which is a much more useful rule.

3. Signed OTA on a secure-boot device needed Ed25519, which IDF doesn't give you. Updates are gated behind a signed manifest, but mbedTLS as shipped in IDF has no EdDSA arithmetic at all. On top of that: anti-rollback, per-version failure records so a bad build can't boot-loop forever, and a check that refuses to start an update on battery — a brownout halfway through a 2.1 MB write is exactly how you brick one of these.

4. The device holds no credentials, which is a design constraint, not a feature list. Putting API keys on a Wi-Fi gadget that sits on a desk seemed like a bad idea, so it doesn't have any. A small broker runs on my own machine, reads the token counts from the CLI logs already on disk, and the device polls it over the LAN — HMAC-SHA256 with a timestamp and nonce so a captured request can't be replayed. Worst case if someone owns the device is they learn how much quota I have left.

Happy to go deeper on any of these.

u/flash_speed3412 — 1 day ago

▲ 44 r/ChatGPTArduino+1 crossposts

I built an ESP32-S3 desk terminal (3.5in, mic array) — the memory and LVGL gotchas that nearly killed it

Short version: it's a small desk terminal built around an ESP32-S3 that mirrors what my coding agent is doing on my PC . It talks over WebSocket to a small bridge running on my machine. I've had it on my desk for a few months now.

Hardware:

- ESP32-S3
- 3.5" 480x320, ST7796 over SPI, driven through esp_lcd + LVGL 9.2
- ES7210 mic array, ES8311 codec, NS4168 amp
- Rotary encoder + 3 mechanical key switches

The parts that actually cost me time:

Internal RAM. This was the whole war. Display buffers, LVGL, WiFi, WebSocket and the audio pipeline all want internal SRAM and PSRAM doesn't help for most of it. At the worst point I had 4.7 KB free internal heap during a voice session. It sits around 27 KB in the same state now. What actually moved the needle: task stacks into PSRAM where allowed, WiFi TX buffers from 16 down to 8, killing BT in sdkconfig.defaults (it quietly claims ~30 KB even when you never call it), and tracking down a heap_caps alloc/free mismatch that leaked on every reconnect.
PSRAM stacks and flash I/O do not mix. Follow-on from the above: I moved a pile of task stacks into PSRAM, and one of them was the task that reads audio assets off SPIFFS. Instant crash, every time, very confusing. If a task touches flash or SPIFFS, its stack has to stay in internal RAM. That one cost me a day.
Half-drawn characters in LVGL. Text in my cards would occasionally render with a glyph sliced down the middle. Root cause was two render paths disagreeing: the full-frame path skips a glyph entirely when it doesn't fit, the partial-redraw path happily clips it mid-glyph. Same string, different result depending on which path ran. Fixed by truncating to whole characters myself before handing the string to LVGL instead of trusting it to clip.
Regenerating sdkconfig loses things you can't see. I deleted and regenerated sdkconfig once and silently lost two settings I had only ever touched in menuconfig (mbedtls external mem, sys_evt stack size). The result was a boot loop that took a while to trace back to that. Anything load-bearing lives in sdkconfig.defaults now.

Firmware is being open sourced in stages. Happy to go deeper on any of this.

Fair warning: I am turning this into a product eventually, so weigh my enthusiasm accordingly. Nothing to sell yet and no link.

Anyone else fighting the internal-RAM-versus-everything war on the S3? Curious what people cut first.

u/Cold_Arm3819 — 1 day ago

▲ 141 r/ChatGPTArduino+2 crossposts

I trained a 582k-parameter transformer from scratch that runs entirely on an ESP32-S3 and controls GPIO from natural language.

I trained a 582K-parameter transformer from scratch that runs entirely on an ESP32-S3 and controls GPIO from natural language

I wanted to see how small a language model could get if it only had to understand a very narrow domain instead of knowing about the world.

So I trained a \~582K-parameter transformer from scratch that runs entirely on an ESP32-S3 and translates natural-language commands into hardware actions.

No cloud, no API key, and no PC involved during inference.

For example:

\> turn on the led on pin 17

Pin 17 is HIGH.

\> set pin 5 to 75 percent

Pin 5 at 75% PWM.

\> move the servo 30 degrees more

Servo on pin 13 moved +30 degrees.

\> stop everything

Done.

It currently understands things like:

GPIO digital output

PWM / brightness

frequency generation

absolute and relative servo movement

ADC / digital reads

chained commands

English and Spanish

One design decision I thought was especially important: the neural network never decides whether touching a GPIO is safe.

The transformer only translates the sentence into a small DSL representing intent. A deterministic C layer then validates the requested pins and hardware operation before anything is executed.

So if the model hallucinates something like GPIO 30, which is part of the flash bus on my module, the firmware simply refuses to execute it.

The base model is only \~2.2 MB and lives in PSRAM. I also included the complete training pipeline, dataset generator, tokenizer, inference implementation and the trained English/Spanish models in the repository.

The interesting part for me isn't really controlling LEDs with an LLM (I know it's not a LLM maybe I can call this a SML stands for Small Language Model). It's the idea that a very small domain-specific transformer can provide a natural-language interface to embedded hardware without needing an internet connection.

I'm curious what other closed embedded domains this approach could be useful for.

Repository:

https://github.com/BareiroJorge/esp32-llm-commands

u/flash_speed3412 — 1 day ago

▲ 55 r/ChatGPTArduino+4 crossposts

We built a micromouse on perfboard with an ESP32-S3 and placed 3rd at AAMC 2026. Firmware & build log are open source.

Hey everyone,

My teammate and I competed at the All America Micromouse Contest (AAMC 2026) at UCLA IEEE a few months back and took 3rd place overall. We just cleaned up and open-sourced our entire codebase and build log: https://github.com/enkhbold470/neuromouse26

A few interesting engineering details from the build:

The "Ugly Protoboard" Pivot:
Our V1 was a custom-designed, clean PCB. But every time we had power rail noise or needed to tweak sensor positioning, we were stuck waiting a week for a board respin. We scrapped it and built V2 on raw perfboard with point-to-point soldering and a mechanical keyboard blue switch for mode select. It looked like a rat's nest, but being able to desolder and reposition an IR emitter in 15 minutes is what got us to the competition.
ESP32-S3 instead of STM32:
Almost every competitive micromouse runs on STM32. We went with an ESP32-S3 running PlatformIO. We used the ESP32 hardware PCNT (Pulse Counter) peripheral for 4x encoder decoding so the CPU didn't choke on interrupts, and cached explored maze walls into ESP32 NVS flash so the fast run could skip sensing entirely.
Motion Control & Algorithms:
- 16x16 flood-fill BFS solver.
- 200 Hz PID control loop timed purely with "micros()" 😂
- no RTOS tasks or "delay()" in the control path.
- Trapezoidal velocity profiling that fuses consecutive straight cells into a single acceleration corridor so the mouse doesn't brake every 180mm cell.
- 4x IR emitter/receiver pairs (SFH4545 + TEFT4300) with lookup tables for distance calibration + MPU-6500 gyro for yaw-hold.
6x3 Home Maze vs 16x16 Real Maze:

We tested at home on a tiny 6x3 grid made of homedepot whiteboard ~$10 board + 3D printed walls. Scaling to the official 16x16 (256 cells) UCLA maze was brutal because millimeter errors compound fast over long straightaways.

The 0.96" OLED display was the real MVP on competition day— like seeing live battery, IR readings, and flood-fill maps on-robot meant we could debug in the 5-minute prep window without opening a laptop.

> 🎬 Competition full run video is on YouTube: https://www.youtube.com/watch?v=2M4ZANPrZ4s

> ⭐️ Repo / Schematics / Firmware: https://github.com/enkhbold470/neuromouse26

Happy to answer any questions about the sensor tuning, flood-fill implementation, or motor control!

u/Puzzleheaded-Seat201 — 1 day ago

▲ 58 r/ChatGPTArduino+3 crossposts

ESPHome MCP server for managing ESPHome devices directly

I tried using a couple of existing ESPHome MCP add-ons recently, but ran into some startup errors with them. Instead of running extra container infrastructure, I created a direct Python MCP server that interfaces right with ESPHome and works seamlessly with AI coding tools like Claude Code, Antigravity, and Cursor.

The nice part about this setup is that it does not require installing any extra Home Assistant add-ons. It just connects over your local network to the ESPHome dashboard you already have running.

Once connected, it lets your AI assistant:

List your devices and see what is online
Read and edit your device YAML files
Check configs for errors
Send Over-The-Air (OTA) updates over Wi-Fi

A few examples of things you can ask it in chat:

"Check which of my ESPHome devices are currently online"
"Show me the YAML file for esp32-shelf.yaml"
"Update esp32-shelf.yaml so pressing the door button flashes my LED lights for half a second as feedback, check for syntax errors, and push the OTA update over Wi-Fi"

The project is completely open source (MIT license). If you'd like to try it out, open issues, or suggest features, check out the GitHub repo: https://github.com/saihgupr/esphome-mcp

u/DiggingForDinos — 1 day ago

▲ 13 r/ChatGPTArduino+1 crossposts

Problems with my new esp32c3

So I bought an Esp32C3 Super Mini from Allegro (the Polish equivalent of Amazon) for my new project and soldered the board to the goldpins. Immediately after connecting the board to my computer, unexpected problems began. For some unknown reason, the Arduino IDE claimed I had connected an Ozoobot kit, and every second I heard the sound of a device connecting. The sound calmed down after pressing the boot button and uploading some blank code. Unfortunately, the Arduino IDE still insisted it was the Ozobot kit, leaving me with no other options. I don't know what I did exactly, but I managed to select the correct board. The problems so far aren't the ones that prompted this post. I'm mentioning them because I don't know if they're related. Now the real headache begins. I tried to connect to Wi-Fi, but for some reason the board wouldn't connect. I downgraded the Esp32 firmware to a lower version, like 3.2. Nothing helped. I ran countless mindless tests with the gpt chat without any components connected, uploading a lot of his code to my board and following his advice. He told me it was most likely a faulty board or something wrong with the software. In any case, he couldn't get my board to connect to my Wi-Fi. I'd also previously connected my ESP32S3 to Wi-Fi without any problems. For some reason, the ESP32S3's MAC address was always zero-based in tests. The board detects nearby Wi-Fi networks without any problems. Tests show that the receiver works, but the transmitter doesn't work. Personally, I think it's a faulty board. I'm sending photos because I'm a bit of a soldering amateur and I'm not sure if I've just desoldered the antenna. Here's my latest test connecting to a Wi-Fi network without any security features. Even this is beyond my board's capabilities: Testing code for connecting esp32c3 super mini to wifi network without any security.

PS: sorry for the text in Polish instead of English. I'm too lazy for that.

#include <WiFi.h>

const char* ssid = "TESTESP";

const char* password = "";

void WiFiEvent(WiFiEvent_t event, WiFiEventInfo_t info) {

Serial.print("[EVENT] ");

switch (event) {

case ARDUINO_EVENT_WIFI_STA_START:

Serial.println("STA START");

break;

case ARDUINO_EVENT_WIFI_STA_CONNECTED:

Serial.println("POŁĄCZONO Z AP");

break;

case ARDUINO_EVENT_WIFI_STA_GOT_IP:

Serial.println("DOSTANO IP");

Serial.print("IP: ");

Serial.println(WiFi.localIP());

break;

case ARDUINO_EVENT_WIFI_STA_DISCONNECTED:

Serial.print("ROZŁĄCZONO! reason = ");

Serial.println(info.wifi_sta_disconnected.reason);

Serial.print("reason name = ");

Serial.println(WiFi.disconnectReasonName(

(wifi_err_reason_t)info.wifi_sta_disconnected.reason

));

break;

default:

Serial.print("inne zdarzenie: ");

Serial.println((int)event);

break;

}

void setup() {

Serial.begin(115200);

delay(1000);

Serial.println();

Serial.println("========== WIFI DIAGNOSTYKA ==========");

WiFi.onEvent(WiFiEvent);

WiFi.mode(WIFI_STA);

Serial.print("MAC: ");

Serial.println(WiFi.macAddress());

Serial.println();

Serial.print("Łączenie z: ");

Serial.println(ssid);

WiFi.begin(ssid, password);

for (int i = 0; i < 30; i++) {

delay(1000);

Serial.print(i + 1);

Serial.print("s | status=");

Serial.print(WiFi.status());

Serial.print(" | RSSI=");

Serial.println(WiFi.RSSI());

if (WiFi.status() == WL_CONNECTED) {

Serial.println();

Serial.println("***** SUKCES *****");

Serial.print("IP: ");

Serial.println(WiFi.localIP());

break;

}

void loop() {

}

Output test results:

========== WIFI DIAGNOSTYKA ==========

[EVENT] inne zdarzenie: 101

[EVENT] STA START

MAC: 9C:CC:01:67:A1:F0

Łączenie z: TESTESP

1s | status=6 | RSSI=0

2s | status=6 | RSSI=-71