u/voyager256

As in the title. I came across a post here that referenced the blog post below :

https://blog.hellas.ai/blog/thunderbolt-ibverbs/

Are there any vLLM (or llama.cpp etc.) forks or at least early implementation attempts/ plans that make use of USB4STREAM?
Basically , it should allow to drastically reduce cross node latency(so improve inference) vs ordinary USB 4 (or non RDMA Ethernet network) as it removes the whole network stack overhead .

So e.g. with two PCs , equipped with USB 4 40Gbit ports you’d just need to get a cheap passive cable for pipeline parallelism to be viable solution.

llama.cpp RPC can be used for such , but standard Ethernet latency / overhead makes it far from ideal.

As in the title. I came across a post here that referenced the blog post below :

https://blog.hellas.ai/blog/thunderbolt-ibverbs/

Are there any vLLM (or llama.cpp etc.) forks or at least early implementation attempts/ plans that make use of USB4STREAM?
Basically , it should allow to drastically reduce cross node latency(so improve inference) vs ordinary USB 4 (or non RDMA Ethernet network) as it removes the whole network stack overhead .

So e.g. with two PCs , equipped with USB 4 40Gbit ports you’d just need to get a cheap passive cable for pipeline parallelism to be viable solution.

llama.cpp RPC can be used for such , but standard Ethernet latency / overhead makes it far from ideal.

As in the title. I came across a post here that referenced the blog post below :

https://blog.hellas.ai/blog/thunderbolt-ibverbs/

Are there any vLLM (or llama.cpp etc.) forks or at least early implementation attempts/ plans that make use of USB4STREAM?
Basically , it should allow to drastically reduce cross node latency(so improve inference) vs ordinary USB 4 (or non RDMA Ethernet network) as it removes the whole network stack overhead .

So e.g. with two PCs , equipped with USB 4 40Gbit ports you’d just need to get a cheap passive cable for pipeline parallelism to be viable solution.

llama.cpp RPC can be used for such , but standard Ethernet latency / overhead makes it far from ideal.

As in the title. I came across a post here that referenced the blog post below :

https://blog.hellas.ai/blog/thunderbolt-ibverbs/

Are there any vLLM (or llama.cpp etc.) forks or at least early implementation attempts/ plans that make use of USB4STREAM?
Basically , it should allow to drastically reduce cross node latency(so improve inference) vs ordinary USB 4 (or non RDMA Ethernet network) as it removes the whole network stack overhead .

So e.g. with two PCs , equipped with USB 4 40Gbit ports you’d just need to get a cheap passive cable for pipeline parallelism to be viable solution.

llama.cpp RPC can be used for such , but standard Ethernet latency / overhead makes it far from ideal.

Hi, I haven’t seen it posted here yet, but it’s worth watching, especially for folks not that familiar with Falcon BMS.

Does anyone know if the high CPU overhead for Nvidia GPUs in some games (e.g. Borderlands 4) is likely to be mitigated/resolved in future generation GPUs( Rubin ) ?

IIRC the issue was covered by Hardware Unboxed(most likely among few others) at least few years ago and affects some games' performance more than others, for example is less of an issue in Battlefield 6. It was even more pronounced for Intel GPUs, but they managed to significantly mitigate it via software updates.

On one of the Moore’s Law Is Dead episode (YouTube channel and podcast) a guest claimed it has something to do with Nvidia moving scheduler logic to graphics drivers , somewhere around 10xx series launch , but I don’t know which scheduler. I know Windows has one that is responsible for e.g. assigning CPU cores to threads. Is this the same one? Also is it feasible to fix / improve for Nvidia? Cause I also recall hearing it has something to do with ATI libraries (Mesa?) that were obviously created very long time ago and I guess it is virtually a standard , but Nvidia GPUs are just not that tailored for it.

Anyone has more insight on this and care to clarify?

Anyone know if the high CPU overhead for Nvidia GPUs in some games (e.g. Borderlands 4) is likely to be mitigated/resolved in future generation GPUs( Rubin ) ?

IIRC the issue was covered by Hardware Unboxed(most likely among few others) at least few years ago and affects some games' performance more than others, for example is less of an issue in Battlefield 6. It was even more pronounced for Intel GPUs, but they managed to significantly mitigate it via software updates.

On one of the Moore’s Law Is Dead episode (YouTube channel and podcast) a guest claimed it has something to do with Nvidia moving scheduler logic to graphics drivers , somewhere around 10xx series launch , but I don’t know which scheduler. I know Windows has one that is responsible for e.g. assigning CPU cores to threads. Is this the same one? Also is it feasible to fix / improve for Nvidia? Cause I also recall hearing it has something to do with ATI libraries (Mesa?) that were obviously created very long time ago and I guess it is virtually a standard , but Nvidia GPUs are just not that tailored for it.

Anyone has more insight on this and care to clarify?

Hi,

This is not Linux specific, as it certainly applies to Windows too, but perhaps someone here knows if the high CPU overhead for Nvidia GPUs in some games (e.g. Borderlands 4) is likely to be mitigated/resolved in future generation GPUs( Rubin ) ?

IIRC the issue was covered by Hardware Unboxed(most likely among few others) at least few years ago and affects some games' performance more than others, for example is less of an issue in Battlefield 6. It was even more pronounced for Intel GPUs, but they managed to significantly mitigate it via software updates. Perhaps the recent updates that were merged in DXVK 3.0 would somehow help with the CPU overhead on Linux for Nvidia too?

On one of the Moore’s Law Is Dead episode (YouTube channel and podcast) a guest claimed it has something to do with Nvidia moving scheduler logic to graphics drivers , somewhere around 10xx series launch , but I don’t know which scheduler. I know Windows has one that is responsible for e.g. assigning CPU cores to threads. Is this the same one? Also is it feasible to fix / improve for Nvidia? Cause I also recall hearing it has something to do with ATI libraries (Mesa?) that were obviously created very long time ago and I guess it is virtually a standard , but Nvidia GPUs are just not that tailored for it.

I recently saw there where some

Anyone has more insight on this and care to clarify?

Hi,

Does anyone know if the high CPU overhead for Nvidia GPUs in some games (e.g. Borderlands 4) is likely to be mitigated/resolved in future generation GPUs( Rubin ) ?

IIRC the issue was covered by Hardware Unboxed(most likely among few others) at least few years ago and affects some games' performance more than others, for example is less of an issue in Battlefield 6. It was even more pronounced for Intel GPUs, but they managed to significantly mitigate it via software updates.

On one of the Moore’s Law Is Dead episode (YouTube channel and podcast) a guest claimed it has something to do with Nvidia moving scheduler logic to graphics drivers , somewhere around 10xx series launch , but I don’t know which scheduler. I know Windows has one that is responsible for e.g. assigning CPU cores to threads. Is this the same one? Also is it feasible to fix / improve for Nvidia? Cause I also recall hearing it has something to do with ATI libraries (Mesa?) that were obviously created very long time ago and I guess it is virtually a standard , but Nvidia GPUs are just not that tailored for it.

Anyone has more insight on this and care to clarify?