u/gitzian

Wanna share something I think is pretty cool.

Recently I worked with a Raspberry Pi Zero 2 W and played some AAA games over the internet using my custom encoder/decoder pipeline. There is also some gameplay on YouTube if you are curious, with voiced explanation. The system streams from Windows or Linux with low latency to the device.

I was curious if I could lower the microcontroller requirements even more, slower wifi and less cpu, and do the same with an Arduino Nano ESP32. It works pretty well. This little ESP32 part fits the setup surprisingly nicely. I am basically close to the ceiling, afaik, of what it can do here: receiving the stream over wifi, calculating CRC, handling some control logic, and forwarding the encoded data to my decoder via SPI.

The ESP32 is not doing the heavy video decoding itself. The actual decoder is my small square board in the setup. It is optimized for this use case, decodes the stream, and then drives the parallel RGB display directly.

The codec is custom too. It is not just sending full frames all the time. It uses delta-frame logic, so if only something small changes, like a cursor, only a small part of the frame has to be sent. That is one of the reasons why this can work on such a small part.

The whole thing on the breadboard runs with very low power consumption, below 1.5 W during active streaming and full brightness. In this video I stream over the local network. I am pretty sure I can also get useful results over the internet, like with the Raspberry Pi, if I route correctly to my home PC.

The Arduino provides everything I need here: a clean enough clock for my decoder, I2C for the backlight driver and touch input, and SPI for data transmission. Touch input is converted and sent to the server as mouse input.

My home PC is just an older 2017-ish gaming PC with an Nvidia GTX 1050 Ti. The encoder is written with DirectX and OpenGL, which means it can run on many GPUs or iGPUs that support basic hardware acceleration. In turn, I can stream nearly every graphics program or game that also runs on my PC.

I have seen many cool display projects in this sub, so I am curious if anyone knows something comparable, especially in terms of resolution, frame rate, latency, or power consumption. If you have seen or built something similar, please let me know.

I am looking forward to building a small device around this, probably with a simple keyboard. I am not sure yet if I should aim more for gaming inputs, a QWERTY keyboard, or both.

Feedback is welcome. I would also be happy to answer questions if you have any.

I’m currently using a Pi to transmit a low-latency video stream for gaming over 30 FPS, (shown here on Youtube). Now I’m wondering how realistic something similar is with an ESP32.

Has anyone here managed to get over 15, 20, or even 30 Mbit/s with TCP, mostly receiving? And what about ping RTT? I guess the advertised Wi-Fi/link-rate numbers are not the same as TCP or UDP throughput. That’s why I’m mainly interested in real measured results from people who actually tested it, ideally under both good and not-so-good conditions.

Also, does anyone have experience with forwarding incoming TCP or UDP data to SPI and measuring the total throughput and latency? Maybe with lwIP tuning, DMA, or similar bottlenecks?

I also managed to get good results with ST microcontrollers + NRF7002, but for those results I had to write a ton of code myself and I want to experiment with alternatives. it would be useful to get an upfront reality check before spending too much time benchmarking the ESP32.

So basically I’m looking for a good balance of low power consumption, decent Wi-Fi throughput, and low latency. Low-latency has higher priority than throughput for me.

I’m working on this project here, and it has been a lot of fun. Some things were never really planned, but worked out surprisingly well.

In this video, of course, the Raspi doesn’t RUN the games shown on the 720x720 screen. But I still think it is kinda funny that it is possible to PLAY them with a Raspi Zero 2 W at over 30 FPS. The game is streamed low-latency over WiFi/internet, then forwarded via SPI over the GPIO pins to this little low-power hardware.

The video comes from the host PC, which has the GPU and renders the game. The Pi acts more like a network/input bridge here. It receives the stream, pushes the encoded video data to my decoder board, and sends input back to the host.

I wanna try to build an open-source, battery-powered handheld device like this. Maybe a simple gaming pad with PlayStation-controller-style input.

Originally, I was trying to build something more like a communicator, with the goal of a real Linux phone-ish device. But I was honestly surprised that I was able to make this work with Windows as well. Just to make it clear: Windows is not installed on the Pi. It is streamed to it. Under good circumstances, it feels native and very responsive, even over the internet.

I use the Pi’s capabilities wherever I can. GPCLK0 on GPIO 4 feeds my decoder with a clean enough clock. The encoded video stream goes over SPI at up to 64 MHz, but it can be much lower and still be sufficient. I also use I2C for the backlight driver, turning it on/off and adjusting brightness, plus touch input. Basically, my Python code handles all this on the Pi side.

For those who don’t know: with raw pixel data, this would not work at this resolution and frame rate over normal SPI. A 720x720 frame has 518,400 pixels. At 64 MHz SPI, raw RGB565 would only be around 7.7 FPS max, and raw RGB888 would only be around 5.1 FPS max, before any overhead. So the trick here is video compression.

A simple way to explain it is BMP vs PNG. Both can show the exact same image, but BMP stores raw pixel information, while PNG uses tricks to make the file smaller. My setup does something similar in spirit: it avoids sending full raw frames whenever possible.

I also bitbang some GPIO pins for JTAG to flash firmware to the decoder, which is the little square board.

Let me hear what you think. Ask me questions. I know there are plenty of ways to attach a display, be it HDMI, DSI, normal SPI displays, etc. But I like the idea of using a simple serial protocol and a small low-power decoder board.

Hi, I wanna find out if you like my project and if any of you have done similar projects.

I have implemented a video decoder algorithm in the FPGA, currently reaches up to 40 FPS at 720x720. I still have plenty of room to further optimize it.

The encoder part is also controlled by me and written in DirectX and OpenGL, so basically I am GPU-vendor agnostic. Also, I don’t depend on any special FPGA vendor IPs and can move without much friction from vendor to vendor, as long as the part itself has proper specs: speed, internal memory, and enough IO ports. In this weird chip market, this can be really helpful.

I am currently using a Lattice CrossLink device. Additionally, I can easily synthesize different display resolutions by changing just a few params, which is cool imo and uses the FPGA’s flexibility to flash different bitstreams at any time.

I try to focus on simple parallel RGB displays, which gives me the most flexibility. Parallel RGB eases PCB development due to lower frequencies, but of course it costs more space and IO ports.

The main bottleneck is the attached 8-bit HyperRAM bandwidth, especially if I wanna keep the decoder PCB at 21x17 mm size. The whole decoder consumes very little power.

I invite you to share your experience with me if you have done any similar stuff, or to ask me questions.

My main background is computer science. I would say I am more the algorithm guy rather than the electrical engineering guy, but low-level programming is something I really like to do. Here is a video with some gameplay and current state of the project.

Hey, I have this niche project I started working on months ago, and I’m looking for feedback. I invite you to ask me as many questions as you can throw at me.

My original plan was to build a low-power thin-client communicator with real Linux on it. Long story short: I gave up finding proper hardware for my needs. I don’t wanna depend on too many vendors, protocols, or software stacks. I want to be able to adapt and change devices smoothly.

I ended up moving the GPU workload to the host machine and remotely streaming a real desktop OS like Windows or Linux. I have written my own shader-based video encoder for streaming, which means I can create and read frames directly on the host GPU, bypassing any special Nvidia or AMD features. So it is fully GPU-vendor agnostic. I can alter the rendering/encoding pipeline however I want, using simple DirectX, Vulkan, or OpenGL. It is also possible to render entirely with a CPU, aka software renderers.

I use my own simple adaptive FPS algo to keep the pipeline as responsive as possible. And it is really fast imo. Encoding a 720x720 stream on the GPU, e.g. GTX 1050, takes around ~3 ms on average. The decoder (fpga) on the client side is fast enough to render 40 FPS, and the client still has some performance headroom left. The decoder receives frames simply by SPI or QSPI. Im a video maker noob, I don’t lie when I say the screen looks in real way better than on camera

Here I use a Raspi on the client side, which just acts as a network device: transmitting the encoded frames to the decoder and sending user inputs, like keyboard, touch, mouse, gamepad, etc., back to the server.

A Raspi 5 can also be used as a host. With its new video chip, it can currently render up to 25 FPS using my GPU code. Of course, this steals GPU resources from other GPU-heavy applications, but it is absolutely usable for watching simple YouTube videos or using the KDE Plasma GUI.

Most of the latency comes from the network and the OS itself. For example, the path from keyboard input on the client side to triggering a new frame update on the server side.

With a Raspi Zero 2 W, I am able to play GTA V via WiFi and over the internet, with about 70 ms pure network latency intentionally added on top, smoothly enough at 20 to 35 FPS. It depends on screen complexity, bandwidth drops, etc. Of course, a real hardcore gamer with a 120 FPS setup won’t have much fun, but I used this as a benchmark because it is absolutely usable. The streaming algo is specialized for low latency and sacrifices frame rate to keep latency low.

All software used here is free to use, and parts of it are open source. I also have hardware files on my website (gitzian.com), but please note that some docs are still missing. But if you are interested in building similar projects, contact me for details.

I do sell hardware related to this project, but I try to keep it customizable and hackable for people who want to adapt it to their own needs.

I know some things are not easy to read here, so I invite you to ask me questions. Please, there are no stupid questions. There is a high chance I left out crucial details, so just ask.

As you may notice, the project is evolving, and from time to time I add new features or improvements. I aim for a cool, artsy device that fits my needs. A cherry on top would be if my personal needs and experience resonate with some of you. Here is another video I made with other details and more gameplay. Playing with this tiny keyboard is not very fun and cramps my fingers, but it is not hard to add other input methods, which I’m looking forward to building in the future.