u/Any-Winter-4079

Hello everyone.

I've recently ran some experiments comparing DDR4/PCIe4 and DDR5/PCIe5 for AI workstations on a pre-training run, and would like to hear yours thoughts.

First of all, and as a summary of my results ( code here: https://github.com/Any-Winter-4079/DDR4-PCIe-4-vs-DDR5-PCIe-5-for-CUDA-training ), I rented two machines on Vast.ai, one with an H12SSL-i motherboard, an EPYC 7352, 192 GB of RAM and of course using PCIe4 (26.3 GB/s) and another with a WRX90E-SAGE SE motherboard, a 9975WX CPU, 256 GB of DDR5 RAM and PCIe5 (54.3 GB/s), and DDR5/PCIe5 is about 15-20% faster on pre-training (depending on whether you include or exclude validation and other costs) under the same number of GPUs.

With the current RAM prices, however, for the cost of 256 GB DDR5 RAM at 6400 MT/s you can get a full (extra) RTX PRO 6000 WS/Max-Q, at which point the comparison clearly favors DDR4/PCIe4 (with 2 GPUs), with about 50% extra throughput vs a single GPU at equal(ish) cost.

Now, there aren't a lot of downsides in my mind to choosing DDR4/PCIe4, but there can be a few:

at least some of these DDR4/PCIe4 motherboards are on the older side, and were one to need replacement, they are not so easy to get (for example, the H12SSL-i used, I can only find it for sale as refurbished now, so who knows in a few years if it will even be available for retail).
newer GPUs (as in, new NVIDIA generations) may stop working with older motherboards (meaning yes, PCIe is backwards compatible but the motherboard's BIOS/UEFI sometimes has issues during POST with newer GPUs (e.g., some older PCIe3 motherboards already have trouble recognizing Blackwell cards, and this may be the case for PCIe4 and newer cards in the future). Meaning if one were to buy a newer GPU in the future, the whole workstation may not be suitable.
If one has to go ahead and bite the bullet on RAM prices, the million question is when? To me prices now are **awful** but so were RTX PRO 6000 prices and here we are (i.e., even higher).
For pre-training, I would still choose DDR4/PCIe4, but I suspect for inference DDR5/PCIe5 might be a fair bit better than the 15-20% that it gives you on pre-training, plus we might be moving into some techniques soon such as dynamic expert/data loading into the model at runtime, which again would favor better DDR/PCIe speeds.

With all of this, I am curious if anyone has benchmarked this, and what are your thoughts on it. Would you hold out on DDR5 at the moment, and therefore go for PCIe4, or would you bite the DDR5 bullet early? Another issue with RAM is channels and DIMM count/channel, because if you want to go 'cheap' like let's only get 192 or 256 GB of DDR5 on 8 channels at 1 DIMM/channel (e.g., 8x32 to get 256), then upgrade to 512 later (when budget allows), that means you have to replace your full RAM (because all the slots are occupied, requiring new 8x64 to get 512 for instance)... And if you get fewer DIMMs like 4x64 to get 256 GB (leaving 4 DIMM slots unoccupied) then you get half the bandwidth because only 4 channels are populated. So maybe a machine that has dual DIMM support per channel is the answer to this (fully populating 8x32 to get the full bandwidth, and still allowing you to expand to another 8x32 to get 512), but in general it's a tricky point too.

So, what do you do/are you guys doing? Have you recently bought a workstation or upgraded to one, for pre-training, fine-tuning, RL, inference, whatever your use case may be, and come up with this dilemma? Are you choosing DDR4/PCIe4 as it would seem reasonable or are you going for DDR5/PCIe5 and if so, why? I am interested in all use cases and opinions!

Three things that I want to say about Codex these last 2 days:

First, in case you do not know, be smart about banked resets.

I don't remember them telling me it will push my reset date 7 days, and I thought they wouldn't be so cheap, but they are, meaning, if you have a reset in 2 days (July, 25th), the dumb way is: Use your reset now (July, 23rd), and get 100% with next reset date on July 30th. I say it is the dumb way because had you not used the reset, you would have waited 2 days, and had 100% until August, 1st. In other words, you still have 100% of usage, and they only let you use it 2 days earlier.

The smarter way is: Wait until July 25th, get your free reset, use the 100% of tokens in a day, then on July 26th use your banked reset. The result: 200% usage, next reset on August 2nd.

I know this is probably an obvious strategy if you have used resets before (which by now might be a good portion of you). However, I wanted to share it anyway because in my case I had 3 resets (had not used any yet), used my first, and I don't remember Codex telling me the banked reset will change my next reset date. They only show the expiration date, as far as I remember, although I could be wrong.

Second thing, usage is terrible. I don't want to get much into this because there are 1k other posts about it, but I've used 300% (3 resets) in 1.5 days. I had NEVER come across anything like this before. I normally fit what I do in the weekly limit, and I use Extra High usage always (now, Sol; before, 5.5).

Third thing, are we sure they are not using more tokens in part because of a serious degradation in performance? As in, the model is more stupid and needs to redo things more. I've caught Sol 5.6 Extra High doing INCREDIBLY stupid things, such as:

Asking it to create questions and answers, and writing correct answers 2-3x longer than the others, so they are all identifiable this way, without needing knowledge on the topic.

Asking it to fix it, and creating answers that don't even address the question.

Asking it to check all answers, and lazily creating rules in hidden files (leading .), based on my examples (e.g., I may tell it, please avoid using 'never', 'full', 'entire', etc., in wrong answers, because the right answers keep being identifiable via over-claim). In short, what did it do? A hidden file was being run checking for those words, instead of treating it as the example I said it was.

Asking it to create long and short summaries of papers, and while some are good, mid-run this is what the difference between a long and short summary becomes. Ridiculous:

https://preview.redd.it/d4ntzrsp10fh1.png?width=2658&format=png&auto=webp&s=1721e371bb6ae6c827e6027c33b759bf62c9cbe6

For comparison, this is one of the earlier summary pairs:

https://preview.redd.it/qp8neiuo20fh1.png?width=2724&format=png&auto=webp&s=5598850b77a8d36830412288b422589f6bbbbe18

Much more fitting.

Asking it to create 3 levels of answers, and mid-run defaulting to creating one (together with shorter summaries).

And like this, I could share like 20 other examples of stupid things a model of its intelligence was not making before.

Finally: Yes, the task is not "easy", as in, it is questions about research papers, etc., but my tasks are always somewhat technical, and this performance is unheard of to me (in a bad way).

DDR4/PCIe4 vs DDR5/PCIe5 for LLMs- I benchmarked them for pre-training. What are your thoughts?

Be smart about banked resets, and performance is hindered (with examples)