I created a small testing rig to evaluate new open source models as they drop, and with the much anticipated release of Qwen3.8-27B, I was eager to see how it cross-compares with frontier and strong local models.

The rig

My test rig is an M5 Max MacBook Pro, 128GB. Locally I ran Qwen3.8-27B on xhigh and medium thinking modes via LM Studio, MLX 8-bit, temp 1.0 / top_p 0.95 / top_k 20, context 131,072 and DeepSeek V4 Flash "0731" 2-bit-imatrix q2-q4, served by antirez's ds4-server at -ctx 400,000, thinking enabled. For cloud I included Opus 5, Sonnet 5, GPT-5.6-sol at xhigh reasoning, and even Haiku. Every model gets the same prompt. The algorithm tasks are executed against fixed-seed differential harnesses, and the repo tasks run against hidden test suites plus a cached baseline of the whole repo. If a fix inadvertently breaks something else it gets caught.

The methodology

The model assessment is broken into 4 batteries:

1. algorithms easy-hard

2. algorithms extremely hard

3. repo work easy

4. repo work hard

The models get run through the algorithm tests 3 times each to derive a mean score whereas the repo work is single pass/fail per task. If a task fails to produce a response, it's retried and time added to total wall clock time for task completion. The total test battery can take anywhere from 12-24 hours of wall time for slower local models. It's a long test.

For the repo batteries I had Fable build a small double-entry ledger CLI and plant bugs that pass the visible test suite while still reproducing a real symptom, then handed each model the repo and a bug report written in a theoretical 'user' voice to simulate how it might be reported in the real world. There's also a subjective code quality assessment that measures the model's ability to not just solve the problem but to conform to the repo's coding style, to fix the actual root cause rather than the symptom, and to keep the diff minimal instead of faffing about and rewriting a bunch of stuff.

I had Opus and GPT blind eval the results and compute a code quality score broken up by 'fixes' and 'features' as these appear to be separable skills for the models. The goal with all of this was to try and create a replicable and automated answer the question: How useful is this model in the real world?

Caveats

This is a home baked assessment and susceptible to bias or less than perfect methodology. It also includes subjective criteria like 'code quality'. I built this for myself as an adjacent tool to on the ground testing. I think the best way to evaluate any model is to test it against your own codebase to see how well it integrates into your workflow.

All that being said, let's move to the scorecard.

Results

Qwen3.8-27B is a very capable model that compares well against frontier models on code quality and correctness. The cost is wall time on Apple silicon. As many have observed, 3.8 has a tendency to over-think, burning up tokens. The time spent earns higher code quality for the most part, but what surprised me is that there are some instances in which less thinking is actually more accurate. On the repo battery, medium went 8/8 while xhigh went 7/8 — the most-thinking configuration failed a task there, and it took four times longer to do it (the wait time with Qwen was tiresome at times).

The caveat is that xhigh excels on extremely hard algorithms, where medium begins to fall apart. Medium didn't even finish the hard algo tasks. There may be some value in matching the thinking to the kind of work you're setting it upon. Lastly Qwen xhigh won outright on quality of surgical fixes and patches to existing code. Interestingly the global trend for locals is that they're competitive along fixes and less so along features where cloud still dominates. This fits anecdotally into my own experience with gravitating to frontier for planning and local for implementing.

GPT 5.6 Sol is the only cloud model with a perfect card on both repo tiers and near perfect algorithms. It's also among the fastest to completion. This all tracks with my own anecdotal experience with this model over the past several months. Highly competent and quick if not a bit stark.

DS4 0731 (a 2-bit quant running on my laptop) is the only local model to get 8/8 on both repo batteries, and one of only two models overall to do it, alongside GPT. It does this all at a respectable wall time. The expense is less elegant code: it sometimes mutates unrelated docstrings and writes dense inline solutions in a codebase that is overtly broken apart and stylistically explicit. Feature code quality is stronger and it's the only model that scored better/equal in the harder repo tier vs the easy one.

Opus 5 is the most reliable model in the set and best code quality of the cloud models. It stumbled only in the hard repo tier where it lost a task by being trying to outsmart the test. A doc string promised one behavior while the code did another, and Opus redesigned the function around what it looked like it should do instead of honoring the documented contract. This also falls inline with my anecdotal experience with Opus 5 where it occasionally ignores your directions completely and just does whatever it wants. The Alaskan Husky of frontier models.

Sonnet 5 is a steady pair of hands that performs reasonably well across tasks for a modest token budget. I think sonnet is kind of underrated as an implementer. Does the same quality of work as the locals cheaply and quickly.

Haiku 4.5 races to the end of the test but has a tendency to fall over and force retries. Worst code quality of all the cloud models.

Conclusion

Hopefully you find these comparisons interesting. For me personally, DS4 has been my goto local, but this test is making me consider trading it out for Qwen3.8-27B. I think they're on equal footing, which is crazy b/c DS4 needs like 90gb of ram. I'd like to try the MTPLX variant of Qwen3.8-27b that's meant to improve tok/s on apple silicon. Slowness to task completion is the real bottleneck for me right now when considering Qwen. Perhaps that'll be my next test.

Curious to know if these results track with your own real world experiences.

The goal was to create something minimal, stylized and fun.  I have more than 15 years experience as a dev, but this project allowed me to deep dive into a number of audio coding challenges that I had only touched on in other projects.

Here a few things I had to wrestle with.  Hopefully this pass along is useful to others.

^((Oh and the app is free to download, btw.  Link at bottom).) 

Sample-accurate playback under load takes work.

A sequencer has to fire hits at exact sample positions.  Seems simple enough, but everything about the platform conspires against you.  Audio playback renders in chunks (buffers) and timestamps live in two clock domains (sample time for the sequencer, host time for the player nodes).  Where this starts to get thorny is when the user is adding or editing the grid in real-time.  I found myself battling flams, jitter, and clipped transients.

The solution was setting the app's clock as an AVAudioSourceNode that outputs nothing but silence; its render callback just increments a monotonic frame counter every IO cycle, and that counter is the single "now" for the entire app.  A producer thread plans a 12 ms rolling window of events from a snapshot of the grid, so live edits never race the audio thread. 

On the render side, sample time is converted to host time for the player nodes and, most importantly, *both clock reads happen inside a single render cycle*, so they can't straddle an IO boundary and smear the timing.  Employing the above meant the sequencer stays sample-accurate even while the grid is being rapidly reshaped by the user mid-playback.

Touch screens lie about when you touch them! This totally kills 'feel' when playing live.

Tap the screen and hear a sound?  Piece of cake, just wire up a button action and you're good to go. If it were only this easy. 

The issue is that UITouch events do not arrive at constant, predictable times. Delivery is influenced by main thread congestion, processor speed, thermals, etc. One touch might reach the audio engine in 10ms while the next takes 100ms. This creates a real issue where live play feels wrong, because hits land randomly early or late against the timing of your actual taps.  

The fix: every hit is scheduled at a small fixed delay from the touch's hardware timestamp (which is rock steady) rather than played whenever the event finally arrives. Your hands adapt to a steady delay very quickly, the same way a pianist adapts to a piano.  This subjectively feels much better than trying to adapt to random jitter. 

The method: calibrate the delay to the user's actual device empirically. The app measures your specific iPad over your first couple hundred taps, picks the smallest delay that makes things steady, and locks it in as a constant. 

Every user's device ends up with the lowest real-time latency that also effectively removes the jitter.  You can play a rapid fire snare roll and hear every hit land in sequence rather than random flaming.

Your export path must match your live playback path. 

This might sound obvious, but during development it's easy to let these two paths quietly diverge until you realize pan controls or some FX knob didn't actually migrate to the export path, and now your WAV file sounds slightly different than what's happening in the app. From the user's perspective this is a massive fail and they'll never fully trust your product going forward. 

Solution: parity between live and export becomes a test target. Live playback and offline export build from the same FX list and execute the same render code (a new parameter physically can't be wired to one path and not the other), and most importantly, *a WAV-hash regression suite pins it.* Any structural refactor must produce byte-identical renders. This keeps you honest while building.

Download Vaporwave Beats for free:   https://apps.apple.com/us/app/vaporwave-beats-drum-machine/id6786917390

The website:  www.rarefyaudio.com