u/LeatherRub7248

Sometime back i made a proof of concept and over the last couple of weeks i had time to pick things up again and fleshed out more content and game mechanics. The engine is more complete now (full races / classes / spells), and the world content has expanded.

Check it out:
https://blackgate.manafox.dev
Key points:

1. browser-first, mobile friendly game play.
2. built from ground up with multi-character scripting / automation in mind (that was a large part of the fun for me back in the day)
3. its almost a complete re-imagining, but retaining some of the majormud UI elements + the race / class / spells database. Otherwise all new. (Fans of MAJORMUD might find it familiar!)

To the earlier testers, the old server just dropped abruptly... but hope to see you guys again.. Drop by and say hi :)

Simple extension to save you time getting PNGs from datacat into ST.

Install from source:
https://github.com/datacat-run/datacat-sillytavern-browser

EDIT: updated to support earlier versions of ST as well

(if your wondering what datacat is, https://datacat.run is a mega source for character cards)

These last few weeks have been godsend for 24GB (and below) gpu poor peeps.

1. Killer models released (Gemma 4 / Qwen 3.6)
2. Free intelligence via QAT
3. Bonus speed via MTP

We're at the tipping point where GPU poor (24gb and below) people are actually NOT poor any more.

I was already happy with Gemma 4 31b running at 40tok/s but now its 70-80tok/s

Its not a wonder 3090 prices are increasing.

For ref:
- limit=1, OSL=192, concurrency 1, temp=1.0/top_k=64/top_p=0.95, ctx=40960, q8_0 KV cache, parallel=1
- For the 12b, did test for both TEXT only as well as mmproj multimodal. Same speedup increase.
(Im TOTALLY Loving the fact that you can actually TALK to the model, and its a split second before it starts generating a response. No TTS yet though)

• Hardware
- CPU: Intel Core i9-13900H, 14 cores / 20 threads
- RAM: 62 GiB system RAM, 8 GiB swap
- GPU: NVIDIA GeForce RTX 3090, 24 GiB VRAM
- Driver/CUDA: NVIDIA driver 595.71.05, CUDA 13.2
- OS/kernel: Ubuntu 24.04-ish, Linux 6.17.0-35-generic

Startup config:

llama-server \
  -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
  --model-draft gemma-4-12B-it-qat-assistant-MTP-Q8_0.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --parallel 1 \
  --ctx-size 40960 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 \
    --spec-draft-ngl all \
    --spec-draft-type-k q8_0 \
    --spec-draft-type-v q8_0 \

UPDATE: 
for 26b, turns out best N-max is 1, which gives a 1.26x speedup:
 setting     tok/s    speedup    accept
  ━━━━━━━━━  ━━━━━━━━  ━━━━━━━━━  ━━━━━━━━
   no MTP     143.01      1.00x         -
  ─────────  ────────  ─────────  ────────
   n-max 1    180.01      1.26x     0.765
  ─────────  ────────  ─────────  ────────
   n-max 2    175.77      1.23x     0.654
  ─────────  ────────  ─────────  ────────
   n-max 3    170.37      1.19x     0.576
  ─────────  ────────  ─────────  ────────
   n-max 4    165.90      1.16x     0.492
  ─────────  ────────  ─────────  ────────
   n-max 5    155.51      1.09x     0.444

NOTE: These are Temp 1.0, so there is some stochastic voltatility to the numbers, but i think they are directionalyl correct.


Also what are the deets on this quick test?
11 requests, one each for coding, humanities, math, QA, RAG, reasoning, STEM, writing, multilingual, summarization, roleplay. Context allocated is 40960, but prompt lengths were only about 22 to 1578 tokens, average about 280. Output target is --osl 192 per turn; some samples are multi-turn, so max full-length total is 15 turns * 192 = 2880 generated tokens, but stop tokens can end samples early. 

This is meant to be a quick and dirty benchmark to get a rough idea of potential impact of QAT + MTP on Gemma4 (on a 3090 GPU) A full proper grid of context + depth will be done separately.

Any recommendations for this?

Initially, i was thinking that LLMs probably not the right thing for this (assuming your source data is all listening metrics), HOWEVER, if you combine
a) user listening data; AND
b) user comments / text data / reccs/ reviews / forum posts / social media mentions etc
and put taht ALL inside the LLM, it might work. Like your ultimate LLM DJ that is intune with not just data, but the zeitgeist as well.

anyway, I've did the obligatory search and seems like nothing really worthy comes up.

Apart from last.fm / various APIs which are heavily limited, there's also this
https://www.reddit.com/r/navidrome/comments/1eoc0cz/generating_weekly_recommendations_playlists_for/

but it seems pretty janky and not exacltly what I'm thinking of.

Is this obscure / rare because BULK user listening data is not really public (ie all hidden behind spotify / youtube / soundhound / shazam walled gardens?)

The ask:
Put in a song / list of songs, and it generates playlist based on that.

So far, spotify's reccs are best for me, i can do endless listening and enjoy most of their suggestions.

There's a dearth of information (in the english world) about these cards.

The good recent video is probably this one:
https://www.youtube.com/watch?v=TcRGBeOENLg

even in this subreddit, there's seems to be few reviews of these cards.

Last couple of decent threads:
https://www.reddit.com/r/LocalLLaMA/comments/1s62b23/bought_rtx4080_32gb_triple_fan_from_china/
https://www.reddit.com/r/LocalLLaMA/comments/1nifajh/i_bought_a_modded_4090_48gb_in_shenzhen_this_is/

Is there really NOONE else who has tried these?

In particular

1. Software / bios / quirks that make them NOT run as per unmodded card
2. Short term consistency, does it run fast for a test, but hang / die when stressed?
3. Long term reliability - does the whole thing fail within 2 months of regular usage?
4. Are the benchmarks good? Where are the results??
5. source and price?

chinese video site blibli has ton of videos, and taobao (and other ecomm) sites also lots of sellers.

If i can piece together enough research, i may also visit shenzhen to pick up a few.

If you're interested in this space, DM me . hope to form a group to split up research efforts.

Also any native chinese speakers who are familiar in this space also please join in.

EDIT:
Some downvotes going on. Unclear if its some larger suppression of this topic, or just angry people.