u/In_der_Tat

If it weren't for power consumption, I'd go for a 4x or even 8x setup, but as the adage goes,

>If my grandma had wheels, she'd be a streetcar.

Now, 2x V100 seemingly strikes a balance between, on one hand:

• a non-negligible amount of VRAM, namely 64 GB (capable of running such models as the Qwen 3.6 family at GGUF Q8, for instance);

• a fair 900 GB/s of bandwidth;

• a helpful 300 GB/s GPU interconnect thanks to NVLink;

• affordable upfront capital outlay;

• non-inordinate power consumption--admittedly a bold claim, especially in energy-starved EU;

and, on the other:

• lack of BF16 hardware support,

• planned obsolescence in the software ecosystem--mitigated thanks to the community.

Since this will be my first setup for local inference (plus training experiments, why not?) and I'm an utter novice, I'm looking for advice on the NVLinked V100-PCIe connection. It looks like a dual V100 baseboard may be connected to the motherboard by means of:

• a couple of PCIe adapter, and in such case it is called "direct-through connection" (what does it mean?); or

• a single PCIe adapter (listing is here).

What are the implications of each alternative, and which one is the most computationally effective or energetically efficient? Should I pursue this project? If not, why not, and what are alternatives with better trade-offs?

I understand the baseboard needs its own PSU, and I suppose one rated at 800 W should be enough, or 600 W if the Teslas are power-limited to 200 W. Moreover, each GPU is to be cooled with a voluminous heatsink, and this raises another question: how do I house this GPU duplet? Note I have a 3D printer.

For context, the core of the system will be as follows:

1x Xeon E5-2699 V4; 4x 16 GB of DDR4 2400 MHz RDIMM RAM (should I get 4x 32GB or even 8x 32GB instead and, if so, why?); HP Z440 motherboard with two PCIe 3.0 16x sockets; perhaps a GTX 1650 for video output--in this regard, an RTX 5050 would be better, but it's dual slot, and so would an RTX A1000, but for €300+ used it's too expensive, even though it's got an attractive 50 W TDP).

Thank you for your attention to this matter.