Pitfalls I should avoid going from Windows only to Proxmox Superstation
Summary
Building a threadripper workstation as my daily. No experience with proxmox, zfs, and currently read, pretend to understand and then execute linux commands I find on the internet because windows has been my existence for the past 35 years. This is what I am hoping to change with this workstation. I want to use windows daily but make it super painless to switch to linux while also having dedicated VMs for my local AI and research projects. I am committed to putting everything together on one box so that I can use all the resources at once if necessary.
My primary question
Assuming I configure everything correctly, what are the pitfalls of using your proxmox box as your daily as well. How great can it be, and when it goes wrong, how bad is it? War stories of how you recovered from a disaster or failure with it (assuming single node) are welcome.
Config suggestions
Since I am still learning about ZFS, I am open to thoughts. Assume that all important files have an off-site encrypted backup that will run automatically. I prioritize reliability and quick recovery in non-catastrophic events so usually I run a local RAID0 dedicated backup drive and then sync that to an off-site. By incredible luck, I timed going on a spending spree in 2022 to build new workstations and got some really cheap enterprise hardware at a bankruptcy sale. Only the ps, cpu, and motherboard are new. Combining these with other items that will be repurposed gives me:
Here is the parts list (only the powersupplies, cpu and motherboard are new):
- Threadripper 9970x
- ASRock TRX50 WS
- MSI MPG Ai1600TS PS
- MSI MPG Ai1300TS PS
- 4x48GB Corsair Vengeance with acceptable timings
- 2x RTX 6000 Blackwell WS (power limit 500W for temperature reasons)
- 1x RTX 3090 (I may not have the airflow to support this one, but I am hoping to have it in)
- 2x Optane P5801X 1.6TB (into slimsas ports on MB)
- 1x Broadcom 9600-24i HBA (Ordering a 2nd to have a spare ready)
- 12x Micron 5300 Pro SATA 7.68TB (very healthy)
- 3x IronWolf SATA 12TB
- 4x Samsung 860evo SATA 4TB
- 1x Samsung 990pro NVME 4TB
- 1x Samsung 980pro NVME 2TB
- 1x Samsung 970pro NVME 1TB
- 1x Highpoint 7101A PCI 3.0x16 to 4 x M.2 Adapter
- 3x Samsung 970evo 2TB (I really don't like these drives and may just omit them along with the adapter)
- 1x Samsung 970pro NVME 1TB
- Off the MB chipset
- 1x 32GB Intel SLC Sata SSD (into chipset
My thoughts were to use the SLC drive as the boot drive for proxmox through a chipset sata port. After that, I am very torn. The way I usually approach storage is: high throughput drives for DBs, durable low-latency for high write scenarios, a monster pool with RAID capabilities for general purpose, and then a spinning disk RAID0 to mirror important directories that then backup offsite to backblaze. Part of me wants to experiment with using the Optane drives for KV Cache offloading and be ready for better MoE offloading in the future, but I know right now I need them for DB logs, and SQL Server temp DB. I also use them for intraday logging on important apps. I rarely come close to saturating them though, so based on the L1T vids, I am guessing they should be apart of the ZFS Journal. This is where my knowledge gap really shows so any help would be appreciated. After that, I am expecting to use the older and smaller nvme ssds to be boot drives for scratch vms and test projects.
