How close are LLMs now to copying Scott's style?
Back in 2023, a prediction market was set up asking whether AI would be able to mimic Scott Alexander's style by January 2026. When the market was posted to this sub, I'd just gotten early API access to GPT-4, so I posted a few examples of that model attempting to write a Scott essay to try and get an idea of how close the market was to resolving "yes".
The market hovered around 65% for a while, then pretty unambiguously resolved to "no" at the start of this year. This evening, however, I happened across a tweet from someone who was able to get Fable 5 to produce some pretty decent abstract idea brainstorming, so I thought I'd check to see if the market might have just set the date a few months early.
At the "Max" effort level, Fable produced The Therapeutic Index of Ideas, an essay arguing that problems of modern culture are often the result of people "overdosing" on true ideas like someone overdosing on medicine.
I'm not entirely sure what to think. The model gives a convincing impression of depth- tying the idea into broad trends, addressing objections, giving convincing examples- but it often feels more like someone doing a good job of selling a slightly clever idea than someone struggling to express a deep one. Stylistically, it does a surprisingly decent job of Scott-style microhumor, though while it's clearly trying to mimic a non-AI writing style, it definitely slips into recognizable LLM patterns occasionally, not unlike a British actor playing an American who can't quite keep all of their 'r's rhotic. Pangram, of course, has no trouble identifying the writing as 100% AI.
Comparing it with the 2023 essays, the difference is pretty remarkable- it's easy to forget how often GPT-4 could slip into complete incoherence. I don't think we're quite at the point that the 2023 market would resolve to 'yes'- but I don't know, it feels like we're getting pretty close.