u/schaka

I had been building nightly images for every ARCH via GitHub CI for quite a while now, because I found that official AMD images were quite a bit behind and lacking a few architectures as well.

However, until just now, gfx803, aka Polaris was still pinned to 6.4.4, because it got completely abandoned after 5.7 and it was generally accepted that 6.4.x were the last builds that would still allow you to compile against gfx803 at all.

Well, I decided to take a patch an AMD employee already posted and got 7.14 to compile - fully. I also found a bunch of kernel bugs and at least one bug with memory management that I patched, though for the latter it's more of a workaround as I couldn't fully identify the source - may be an AMD driver bug but I'm out of my depth here.

I've dedicated its own repository to gfx803 in hopes of being able to clean up some of the patches and submit them upstream and maybe bring Polaris support back. RX 470 8GB mining cards are about $20-30 on the used market and if you don't need to run the latest 25-35B LLM, that's still plenty. I'm currently using it for Audiomuse AI and I can absolutely see it being useful for Immich as well.

The docs and some of the comments are pretty much AI generated from my explanation. I am aware that I still have some cleaning up to do - especially for the patches themselves. But I've been in software development for over 10 years (albeit not C++) and am fairly confident I can clean up a bit better.

If you're completely against any use of AI (not sure why you'd be in the ROCm subreddit), just ignore this post.

I'm mosty interested in 128-192GB VRAM with 128-256GB RAM to spare, so SSD streaming is basically not even necessary. Seems only FP4 is supported, so older hardware will likely be slow - no Unsloth GGUF supported either.

I'd be curious what people are getting with V100s, R9700s, etc, just to have some comparison.

What's prefill like >200k context? Tg/s high enough to support agentic workloads?

It's probably wishful thinking, but when I saw the release, my immediate thought was Sonnet 5 level model being "affordable" to consumers.

For a while now I've wondered how I'm getting way fewer recommendations for this sub. It used to be how I got update notes for a lot of projects, especially those not directly on GitHub or if I missed the new release on my GitHub feed, this subreddit and the selfhosted newsletter were my backups.

I still see a few posts for new versions of existing software being released here and there, so I didn't think too much of it.

That is, until I tried posting an update to my own software (i.e. version 2.1.0 release) and the automod removed at least 5 or 6 attempts at getting past whatever horrendous word filter triggers the auto deletion.

Everything gets deleted for rule 6, which clearly only covers new projects. And since it's not done by a human, you can't ask to reinstate the post. Messaging the mods does nothing.

I'm not just posting this to air my grievances (though clearly part of it). A pattern I noticed in the past few months hast just gotten confirmed.

If the automod stays as aggressive as is, it just defeats the purpose of subeddit.