MTP quitting too early for long runs: for oMLX developers
Dear oMLX developers,
First, thanks for an amazing tool.
Here is an issue that might be worth investigating. I've been playing with oMLX running Qwen 3.8 27B - 4 bit MTP. I've set up the model with 96k context, 16k max_tokens, and 8k reasoning limit. Qwen is on xhigh thinking. I'm using it to generate a large web page that is a fluid dynamics visualizers. On my MacBook Pro M5 Max, the full run with many turns takes about 90 mins. It is well behaved throughout - no memory thrashing (took some tuning with the GDN settings).
In analyzing the log, it seems that on many long generation runs, MTP is quitting early - after several hundred tokens - on a many thousand token generation run. Claude suggested that performance might be better if MTP re-armed more often. Here is a table that lists context length (prompt length) (growing over about an hour or so), and then generation length for a turn, and MTP status and success for that turn. Stop means that MTP got through the whole run, and parked means that it stopped after x tokens in the run.
Full post-GDN table, every completion I can pair with an MTP outcome:
| Prompt | Gen | tok/s | MTP | Accept |
|---|---|---|---|---|
| 69 | 10,207 | 29.0 | — | — |
| 1,182 | 252 | 13.7 | stop | 75.8% |
| 8,729 | 8,599 | 23.9 | parked at 640 | 71.5% |
| 17,772 | 61 | 28.6 | stop | 70.5% |
| 17,915 | 4,964 | 33.7 | stop | 93.8% |
| 22,900 | 1,979 | 29.9 | stop | 87.3% |
| 24,899 | 363 | 27.2 | stop | 82.2% |
| 41,338 | 1,085 | 29.7 | — | — |
| 42,443 | 141 | 23.6 | stop | 83.9% |
| 44,123 | 1,100 | 28.8 | stop | 94.2% |
| 45,243 | 289 | 25.9 | stop | 93.1% |
| 45,552 | 102 | 23.3 | stop | 83.8% |
| 46,646 | 314 | 17.0 | parked at 134 | 74.3% |
| 46,980 | 570 | 22.2 | stop | 80.3% |
| 47,998 | 120 | 21.3 | stop | 78.6% |
| 48,201 | 7,559 | 15.9 | parked at 101 | 79.3% |
| 55,780 | 132 | 18.2 | parked at 79 | 81.6% |
| 56,230 | 2,243 | 24.0 | stop | 92.9% |
| 58,493 | 4,763 | 15.5 | parked at 499 | 85.6% |
| 63,276 | 5,865 | 15.2 | parked at 499 | 90.9% |
| 69,161 | 3,296 | 14.9 | parked at 155 | 81.6% |
| 72,477 | 280 | 19.0 | stop | 85.5% |
| 72,777 | 713 | 21.0 | stop | 87.3% |
| 73,510 | 79 | 22.3 | stop | 86.8% |
| 73,618 | 188 | 23.4 | stop | 92.3% |
| 73,835 | 7,814 | 14.5 | parked at 145 | 74.1% |
thanks for considering!
