u/Gil_berth

This study gives us some insights on the effects of AI assistance:

* People who use AI assistance show a sharp drop in performance when AI is removed.

* People also show a drop in persistence and motivation, they are more likely to give up.

* LLMs are "short term collaborators": "extraordinarily helpful in the moment, but indifferent to what that help does to the person receiving it over time", in contrast to humans, that optimize for long term objectives, and can encourage "independent development".

The study asks what would be the long term effects of using AI assistance? Since these negative effects already appear in a short time frame. Many companies are forcing AI usage, aren't they deskilling their workforce and therefore losing a competitive edge? What about the people that voluntarily give up their skills to LLMs? Like programmers who claim that they only prompt now, how sharp is the drop in performance for these people? Imagine a year or more only prompting an LLM, can you even judge the output of the LLM anymore?

When there is an outage in some LLM company, you go to the LLMs subreddits and what you see is sad: people are helpless, they can't do anything without the robot. Some claim they have years of experience but apparently they can't do anything without Claude now. Is this the future of intellectual work? A bunch of people dependent on the availability of data centers?

https://preview.redd.it/92ob9d477r0h1.png?width=2285&format=png&auto=webp&s=5b0f8c1767315617d50eb30a00036c3a1b9a92a9

Well, it seems the day when an LLM becomes more expensive than a traditional developer is coming sooner than we expected.
Screenshot with preview – 12 days of use, ~900 premium requests

How to check: Github Account Settings -> Billing and Licensing -> Premium Request Analysis -> Preview your billing impact

Anthropic called it "too dangerous to be released", Dario said it was a "step change", but curl's creator, Magnus Daniel Stenberg, doesn't think that Mythos is anything to write home about. In his own words:

"My personal conclusion can however not end up with anything else than that the big hype around this model so far was primarily marketing. I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing."

Mythos presented 5 "confirmed" security vulnerabilities to the curl's team, but 3 were false positives and 1 was a bug. The only real security vulnerability was of low severity. So an 80%+ rate of hallucination, a far cry from the 90%+ rate of success that Anthropic was claiming.

The curl's team routinely uses LLMs and other tools to scan for security vulnerabilities. In the end, I think LLMs will be just another tool on the cyber security's tool box, one that can find some vulnerabilities, but not others, and always with human supervision. Or course, when is projected that this year the investment on LLMs will reach almost 3 trillion dollars since 2022, you have to ask, is it worth it?

Mythos is 10x more expensive than its predecessor, and it seems that the disminishing returns are already starting to appear. By how much will LLMs need to scale to keep meaningfully improving? Do we really need to build a Dyson Sphere as Sam Altman said?

I enter the /vibecoding sub, and I'm astonished: "650k vibe coders", that is more than many programming subreddits combined. Are there more vibe coders than coders now? I don't know, but with so many people vibe coding is safe to say that there is a lot of software being produced. I know that many people vibe code dashboards, simple websites and fitness trackers; but, where are the highlights? I want to see the outliers, the best that vibe coders can produce. I tried to ask there, but the post was eliminated, so I'm asking here to see if there someone that can show me the best of the best: the most original, groundbreaking and innovative vibe coded projects that the vibecoding community have produced so far. Honestly, I'm extremely curious, I'm ready to be surprised.

In the last months, many companies have been laying-off people citing AI. This can't be because LLMs are replacing people, they are not reliable enough for this, so CEOs are saying that AI makes their employees more productive, therefore, they don't need as many employees as before to keep the same productivity; but, isn't this a weird strategy long term? Would not they be out competed by the companies with more employees by sheer productivity force? Some talk about the trade-off of employees cost, but would not this be offset by capturing a greater pie of the market? If you remain stagnant while your competitors have a multiplier, isn't this a death sentence for your company? I don't know, it seems weird to me that they say: "Thanks to AI, our employees are more productive, so we are going to get rid of the gains in productivity by firing a lot of people"

Let's put some scenario forward: let's say you fire 80% of your workforce and your productivity remains the same thanks to AI, but your competitors don't fire anyone and they 10x their productivity, how can you compete like that? Would not your company be completely wiped out in the long run? Let's put the example of a video game company: Let's say company "A" fires 80% percent of their employees and now can deliver a game every 3 years with a fraction of the cost, but company "B" doesn't fire anyone and now can churn out a game every 3 months with the same quality, who will win more money in the end? Change "game" with "features and quality of life improvements" and you have yourself another example.

We see this happening with the LLM companies, they are hiring like crazy, it doesn't matter how productive the LLMs make their employees, they need to keep pace with the competition, so firing people is stupid.

Now, there is also the "illusion of productivity" with LLMs. Since you can't trust this non-deterministic software, you have to check its output, offsetting the gains in productivity. This is a kind of version of the known Amdahl's law: a system speed up is limited by its weakest link. LLMs speed up production of text, but all the other bottlenecks remain the same, so in the end the overall increase in performance is lower than expected.

In the last months, many companies have been laying-off people citing AI. This can't be because LLMs are replacing people, they are not reliable enough for this, so CEOs are saying that AI makes their employees more productive, therefore, they don't need as many employees as before to keep the same productivity; but, isn't this a weird strategy long term? Would not they be out competed by the companies with more employees by sheer productivity force? Some talk about the trade-off of employees cost, but would not this be offset by capturing a greater pie of the market? If you remain stagnant while your competitors have a multiplier, isn't this a death sentence for your company? I don't know, it seems weird to me that they say: "Thanks to AI, our employees are more productive, so we are going to get rid of the gains in productivity by firing a lot of people"

Let's put some scenario forward: let's say you fire 80% of your workforce and your productivity remains the same thanks to AI, but your competitors don't fire anyone and they 10x their productivity, how can you compete like that? Would not your company be completely wiped out in the long run? Let's put the example of a video game company: Let's say company "A" fires 80% percent of their employees and now can deliver a game every 3 years with a fraction of the cost, but company "B" doesn't fire anyone and now can churn out a game every 3 months with the same quality, who will win more money in the end? Change "game" with "features and quality of life improvements" and you have yourself another example.

We see this happening with the LLM companies, they are hiring like crazy, it doesn't matter how productive the LLMs make their employees, they need to keep pace with the competition, so firing people is stupid.

Now, there is also the "illusion of productivity" with LLMs. Since you can't trust this non-deterministic software, you have to check its output, offsetting the gains in productivity. This is a kind of version of the known Amdahl's law: a system speed up is limited by its weakest link. LLMs speed up production of text, but all the other bottlenecks remain the same, so in the end the overall increase in performance is lower than expected.

During six months(and maybe more) Openai's very expensive next token predictor has had an undesirable quirk that makes it mention goblins, gremlins, raccoons and other fantastic beasts in weird places where it shouldn't. After putting all their (human) intelligence to work on finding the cause, they concluded that this is a side effect of reinforcement learning in training for a "Nerdy" personality: "The rewards were applied only in the Nerdy condition, but reinforcement learning does not guarantee that learned behaviors stay neatly scoped to the condition that produced them".

This makes me wonder: since "reinforcement learning does not guarantee that learned behaviors stay neatly scoped", what others(not so obvious) side effects of the heavy use of reinforcement learning are there in Chatgpt and other LLMs? These side effects could be negative and very hard to circumvent, so no prompt engineering could save us. Imagine, for example, in programming, where some RL'd behavior is good for some tasks but horrible for others, and it doesn't matter how much you prompt the model, you can only reduce the chance that the model doesn't do the negative behavior, but it will do it eventually.

In the end, Openai claims it had to retire the "Nerdy personality" to stop the creatures from appearing, but couldn't do it in time for the last iteration of Chatgpt. Openai even admits that the goblins: "...are also a powerful example of how reward signals can shape model behavior in unexpected ways, and how models can learn to generalize rewards in certain situations to unrelated ones".

What surprises me the most of all of this is that Openai is admitting in this blog post some serious limitations of LLMs and the reinforcement learning that they apply to them, but at the same time is confident that this unreliable and expensive technology is very close to super intelligence.