I tested a different approach to vocal-to-sheet-music transcription, and the rhythm was the biggest surprise
I've been experimenting with several tools for converting sung vocals into MIDI and sheet music, and I kept running into the same problem: the pitch transcription can be surprisingly good, while the resulting rhythm is almost unusable.
You get strange rests, unnecessary sixteenth notes, very short notes caused by vocal artifacts, overlapping notes, and all kinds of rhythmic clutter.
At first I assumed the problem was simply poor pitch detection. But after testing different approaches, I started to think the problem happens after the pitch information has already been extracted.
A singer doesn't perform a melody as a perfectly quantized MIDI sequence. There are tiny fluctuations in pitch, consonants, breaths, vibrato, slides, and other details that are musically irrelevant when you're trying to create a readable melodic line.
So I tried a different approach.
Instead of asking the transcription system to produce the final notation directly, I used a workflow with several stages:
1. Vocal isolation
The original recording is processed to isolate the main vocal and remove as much background material, reverb, speech, screams, etc. as possible.
2. Pitch extraction
A pitch-transcription engine is then used to generate the MIDI. In my tests, engines based on YourMT3 produced much better raw material than some of the simpler audio-to-MIDI solutions I had previously tried.
3. Rhythmic/musical cleanup
This was the interesting part.
Rather than simply quantizing the MIDI or deleting notes below a certain duration, I tested an HTML-based processing system that evaluates the extracted notes in context and tries to reconstruct what a human copyist would consider the intended melodic line.
The objective isn't to preserve every tiny fluctuation in the original performance.
It's to produce something that makes musical sense on paper.
The result was much more useful to me.
Instead of getting a technically detailed MIDI full of tiny rhythmic events, I could open the result in MuseScore and work on a relatively clean melodic draft. There can still be small things to correct manually — an occasional unwanted tie, a note that needs adjusting, etc. — but the basic rhythmic structure is already much closer to something a musician would actually write down.
The tool I used for this is called ScoreDraft. I originally came across the project while looking for a way to automate the tedious MIDI-cleaning stage of my own transcription workflow.
I'm not presenting this as a magical one-click transcription solution. It isn't. The final score still needs a human review.
What I found interesting is that better transcription may not necessarily mean capturing more information.
Sometimes the opposite is true: you need to deliberately throw away information that is musically irrelevant.
That made me wonder whether the future of practical audio-to-score transcription is less about making the AI hear every microscopic detail, and more about having a good musical logic layer decide which details should actually survive into the notation.
Has anyone else noticed that the rhythmic cleanup is often a bigger problem than the actual pitch detection when working with vocal transcriptions?