Benchmark notes: Kimi K3 reaches 88/98; Qwen3.8-Max is stronger than its score looks; Gemini 3.6 Flash slips vs 3.5
I ran Kimi K3, Qwen3.8-Max and Gemini 3.6 Flash on the current 98-task MindTrial set with the same Python executor available to all models.
A few results stood out.
Kimi K3 was the biggest surprise: 88/98, up from 66/98 for Kimi K2.6, with hard errors dropping from 22 to 1. That ties Claude Opus 5 for the highest raw pass count among single models in this set. The downside is runtime: about 8h48m summed across the tasks.
Qwen3.8-Max finished at 74/98, but that number hides a much stronger solver. It went 39/39 on text and had 92.5% accuracy on completed tasks. Of its 18 hard errors, 15 were response-parsing failures; in a number of those cases the useful answer was there, but the outer response was malformed. So strict output discipline is currently a major weakness.
There was also an interesting API/agent interaction with both Kimi and Qwen. In the original runs using provider-enforced JSON object/schema output, neither model used the supplied Python tool across the benchmark. I reran using API response_format: text while keeping the same JSON schema in the prompt. Tool use came back immediately: 387 calls for Kimi and 423 for Qwen, with much better task performance. For Qwen, the constrained-output path also seemed to interfere with how reasoning translated into actions/final answers.
Gemini 3.6 Flash was less impressive: 74/98 versus 77/98 for Gemini 3.5 Flash. It was faster (~1h45m vs ~2h13m) and used fewer tokens overall, but made more Python calls (712 vs 597). In this workload the extra tool use often looked exploratory rather than efficient.
The strict scores are still the scores—I did not repair malformed model answers after the fact. I just found the differences in failure mode and tool behavior interesting.
Results/data: http://www.petmal.net/shared/mindtrial/results/2026-08-06/mindtrial-eval-all-models-03-2026_26.html