To get this out of the way first: should you do this yourself? No, absolutely not, the price of Pi's and computer parts in general right now is insane and this build is overkill in every way. But, I had fun with it and at least bought most of the parts before the latest Pi price hike :)
I've been working on this project for the past 4-5 months or so and am finally ready to unveil it! My Pi Lab, a cluster of 9 Raspberry Pi 5s acting as a NAS and server cluster. It consists of:
- 8GB Raspberry Pi 5 acting as a NAS
- ATX hat for supplying power
- PCIe x16 expansion board
- PCIe 3 x16 NVMe switch
- 8-bay U.2 enclosure connected to the NVMe switch
- As many U.2 drives as I could cheaply get my hands on in RAID1 array
- I only felt the need to get 2x 1.6TB drives for now given how expensive storage is now
- These are heavily bandwidth limited anyways given the x16 NVMe switch is connected to an x1 port on the Pi at PCIe 3 speeds
- DSI display for showing status information about the cluster
- M.2 usb enclosure for the boot drive
- 3x 4GB Raspberry Pi Compute Module 5, 5x 8GB Raspberry Pi Compute Module 5 w/ 128GB 2230 M.2 drive
- Custom designed backplane for supplying power and ethernet to each compute module
- Custom designed daughter board acting as a blade server and carrier board for the CM5, connects to the compute module and slots into a PCIe x1 connector on the backplane
- Custom designed case, drive sleds and fan bracket
- ATX PSU supplying power to:
- NAS via the ATX Hat, also connected to a power button which controls power to the PSU and can shut down the entire cluster
- U.2 drive enclosure via an 8 pin CPU header
- Compute module cluster via an 8 pin CPU header
- 2.5 GbE switch via a molex to 12V adapter
- 2.5 GbE switch with 9 ports
- Connected via SFP+ to my main router on port 9
- Connected to NAS via 2.5GbE USB adapter
- Connected to 7 compute modules via the backplane
- Final compute module is connected to the 1GbE port on the NAS Pi which is in bridge mode
- Backplane currently only supports 1GbE to each compute module. In theory each daughter board could take a 2.5GbE USB IC to do 2.5GbE, but added more cost and complexity than I was willing to accept
- 6U 10" mini-rack to house it all
- Some custom cut faceplates to hide the switch internals and PSU from the front and bracket the DSI display in place
- Probably some other smaller components that I'm forgetting about.
This build was originally inspired by the U.2 drive enclosure after I purchased 2 of them for another server build and thought it would be a really cool way to house compute modules and hot swap them as needed. Performance was meh, but I was more interested in the hardware side and designing the backplane and carrier boards than actually using it for real work (I have other servers for that already).
Getting everything to fit into a reasonably sized 6U case was a lot of fun, kind of incredibly the 8 Pi cluster acting as the centerpiece of the build only makes up a very small part of it. I had originally wanted just this enclosure and nothing else, but putting an ethernet switch on the backplane as well as figuring out an integrated power supply that could do up to 250W was challenging to say the least and adding the extra space opened up possibilities for the separate NAS device to be included anyways so I like the final result.
As for the software side, its pretty standard. I have some ansible scripts to set up the NAS and each of the compute modules, but in general it is just installing docker and setting it up in swarm mode. I'm not actually doing much with the server right now, I have OpenClaw installed on the NAS and ollama with gemma4:e2b running on each of the 8gb compute modules. Plus some other random monitoring and file browsing services. I'm not sure what I'll expand this to do in the future yet and built it without any real use case in mind, I was more excited about the hardware side and PCB design than the software side for this project.