How exactly do you "mess up" the ghost note combo? I even tried just hitting random buttons in between notes on slower songs and it doesn't seem to change at all.
I took the note speed from the level I've been playing at forever, note speed 9 or about hyperspeed 2 and upped it to 11 yesterday. Suddenly I could play songs that I couldn't even come close to before. Then after a day of that I decided to try note speed 13 and bam another level up. Suddenly I can play Dragonforce songs and songs that I used to barely beat I'm easily getting 5 stars on. I just spent the day beating song after song that I basically never thought I'd be able to touch, let alone get 5 stars on.
It's clear that it's because I haven't been able to parse the notes at the lower speeds when they're all clumped together in a big block of color vs the faster speeds where you can actually see the individual notes.
So give it a try, you may find you're much better than you thought at the game but your eyes/brain haven't been able to read what you're playing properly.
Edit: As someone else mentioned, you probably want to increase the Highway length as well, it's in the same options as note speed. I have it at 120%