u/Disastrous_Room_927

It’s well known that corporations and groups have paid people to sit around astroturfing all day, and in general we know just how insidious and effective fake grassroots movements can be in manipulating public opinion.

The thing that actually worries me about LLMs is that they’re practically purpose built for astroturfing - it’s trivial to configure and fine tune them to write in specific styles and push certain ideas, and and that was the case early on for them. And you don’t need anything near a frontier model to be convincing to a non-trivial number of people. Basically the next stage of algorithmic/targeted content.

What concerns me is that there’s a lack of insight into how LLMs are being used to “organically” spread ideas/shape opinions. But like, do we have any idea what the PR firms that were astroturfing before LLMs are doing now? To what extent AI companies are using their own resources to sell themselves and AI in general? Or steer conversations on social media away from topics they want to keep on the down low? Or derailing actual exchanges to prevent certain ideas or information from spreading organically?

123 years ago, Charles Spearman published a paper titled “General Intelligence: Objectively Determined and Measures”. This paper basically formalized the loose concept of intelligence so that it could be studied empirically. Modern theories of intelligence grew out of this, as did the popular notion of it we have today.

It doesn’t just bug me that the people working in the AI “field” seem completely unaware that over a century worth if research on intelligence is out there, it bugs me that there’s seemingly no interest in giving the concept of AI an empirical treatment.

I can’t say I’d expect anything else from the tech industry, it’s just incredibly problematic that some half baked notion of intelligence is being used to justify shoving LLMs into everything. Or that the people who don’t see an issue with this have the world’s economy in a stranglehold.

Another shower thought here: how convenient is it that search engines have gone to shit, given that the vast majority of people are using LLMs as a glorified search engine? I won't say that it's gone to shit because of AI given that it started going downhill awhile ago, but there's literally no incentive to return Google search (for example) to what it was before it was polluted by SEO and sponsored links because it might reveal that LLMs aren't doing much better of a job of retrieving information than a little google-fu can.

And when it comes to code, I've found myself doing exactly what I used to do finding code chunks on Stackexchange and modifying them to suit my needs. I got sick and tired of trying to explain to an LLM what I wanted and just use it to prompt for syntax I don't care to memorize.

This particular result made it's rounds on reddit a week or two ago with headlines like "GPT-5.6 Disproves Statistics Conjecture in 90 Minutes, Exposing Flaw in 130,000-Citation Method" or "GPT-5.6 AI disproves 20-year statistics conjecture with proof". The conjecture concerns a method that was covered when I was in grad school for stats, and I wanted to make a comment because the coverage is rustling my jimmies.

The TL;DR here is that the Benjamini–Hochberg procedure is used to control the False Discovery Rate when repeatedly performing hypothesis tests (important because without correction, the probability of a false positive increases with the number of tests you perform). I won't go into any details other than to say that as with most other test/procedure/method that people are familiar with, the actual coverage/level of control is only guaranteed under the assumptions they were derived under. The BH procedure assumes independent p-values, so it's not guaranteed to control the False Discovery Rate when p-values have some kind of dependence (think correlation).

The first headline is straight up bullshit because the conjecture in question argued that coverage is guaranteed even under a specific condition where that independence assumption is violated. In other words the conjecture pertains to how the procedure works when it isn't used as designed, this would be like calling a flat head screwdriver flawed because it doesn't work as well as a phillips for... phillips screws. The second headline is what you'd expect at this point: they're using the age of the problem to make the problem sound more difficult or important.

The reality of the situation is a lot less interesting:

• Roughly speaking, less than a dozen researchers have been seriously looking at this conjecture over the last 20 years, and maybe a third of them have made a sustained effort to try any prove it analytically.
• When the author says "Before this work, a positive answer was widely believed", he's talking about the small group of researchers discussing the topic, not the field of statistics as a whole. The BH procedure itself is ubiquitous, but most people in the field wouldn't be aware of this particular conjecture concerning it.
• My hot take here would be that if anything, this is an example of a small group of researchers leading themselves leading themselves down a rabbit hole without adequate justification. Ask a statistician in a vacuum and they might tell you that control/coverage shouldn't be assumed, and that simulation studies aren't a substitute for a theoretical guarantee. I think this is a great example of why - prior work reflected the fact that no counter example had been found, which would've been the case even if nobody had been looking for one or if the researchers involved were looking in all the wrong places. But like... this isn't the first time this has happened, it turns out that it's difficult to find a counterexample when something is robust to violated assumptions.
• AI was used in a very familiar way here. the claim being made was universal so only one counterexample needed to be found, and we're talking about an outcome that was already plausible and can be verified easily. It's the perfect setup for a structured search.

Long story short, the headlines are desperately pushing the "AI did something humans couldn't" angle when the reality is usually just that it did something humans just haven't done for whatever reason. AI did a thing, a lot of people are getting sick of the implication being that it did so because the problem was beyond human capability. The age of the problem or intuitiveness of the result doesn't say much about AI when we're talking about problems that hardly any researchers have looked at, or at a small group of researchers that might need to take some mushrooms and get a fresh perspective on things.

A comment on the statistics sub perfectly captures my sentiment:

>The most surprising thing to me about this result is that it was open and apparently not assumed to be false.

I'm mildly hungover so I'm struggling with how to phrase this, but AI hype is killing the vibe for things I've put a great deal of time and energy into. I was intrigued by it at first because I went to grad school both for ML/stats and for cognitive science, but found my self saying "that's all there is to that?" at every turn when I started digging into. I figured that I had some catching up to do given that I graduated in 2019, but what I discovered is that:

• From a math perspective very little has changed since I was in school. There's some LLM specific jargon, but a lot of it is just obfuscating things that aren't any different than elsewhere in ML.
• Beyond the terms themselves, nothing of substance from cog sci is being used. At best they're being applied to things that vaguely resemble a pop-psych understanding of some concept, but again, much of it is obfuscating math that is pretty unremarkable. They're borrowing terms that are defined and characterized by empirical data in cog sci, but I haven't seen a single citation of relevant contemporary research.
• This all seems to be compounded by the fact that the conversation about any of this is very one-sided. It isn't a conversation about AI/ML in general, it's a conversation about it according to the Silicon Valley circlejerk. It's very homogeneous in that almost everyone involved is coming from a CS/software development background, it's self referential, and heavily influenced by rationalism/effective altruism.

Those things aren't what's making me cynical at face value, what's really driving that is the amount of people with legitimate credentials getting caught up in this. From my perspective, many of them ought to be even better positioned to see through the marketing noise than I am, so I end up either a) questioning my own understanding of ML, b) assuming that money is behind what they're saying, not their actual expertise or c) wondering if this is is just hubris from people that made something impressive happen and decided that they could literally do anything.

Anyways, I'm just rambling about some things that have been rustling my jimmies. Curious if anyone else here has a similar background and is having a hard time not getting burnt out by all of this.

While I think we're all aware of the "safety' labs that focus on P(doom) or some other sci-fi nonsense, I think it's worth calling out safety evaluations that supposedly target real world use. For example:

>Deployment simulation is a new approach to pre-deployment risk assessment that helps frontier labs and evaluators predict how language models may behave in the real world and understand the risks they pose before deployment. It complements existing safety evaluations, red-teaming, and targeted analysis by adding a more production-like prediction layer that can improve estimates of deployment behavior, reduce evaluation-awareness effects, and make pre-deployment predictions checkable after release. Used alongside traditional evaluations, Deployment Simulation can help make model risk assessment more realistic, more quantitative, and more useful for deployment decisions.

The elephant in the room here is that this is like looking at how often people wear seat belts, drink and drive/text instead of like... looking at how often people get injured when they do/don't do these things.

And in spite of saying that "It complements existing safety evaluations", one thing you'll notice is that those evaluations are more of the same - they're equating risk with a rather narrow subset of behaviors deemed undesirable, but they aren't looking at the downstream impact of AI in a concrete way (whether or not those behaviors are present). Actual safety research would frame these behaviors as possible risk factors, but wouldn't deem them as such... without even quantifying the risks associated with them. So much of this alignment research begs the question in that it assume the target AI should be aligned to is relevant in the first place, and that addressing this solves the problem of alignment.

I'm highlighting this not because I expect companies like OpenAI to actual care, but because this is right out of the playbook used by big tobacco/oil/pharma for decades. That play has always been to a) manufacture concern for tangential issues that are convenient for a company b) direct a substantial amount of effort toward addressing them, and c) put on a show about how much you care about or are doing to solve said issues, hoping that people . Big tobacco, for example, put a great deal of effort into research on indoor air quality in an effort to keep attention away from secondhand smoke. And for big oil the play in recent years is to stop outright denying climate change, but to make a show supporting reforms and policies that have a questionable impact (both to emissions and their bottom line).

Anyways, this isn't exactly isolated or new behavior from the tech industry, it's just worth calling out any time you see it:

https://jacobin.com/2023/05/big-tech-artificial-intelligence-lobbying-academic-research-influence-big-tobacco

https://www.holyrood.com/news/view%252Cai-companies-adopting-playbook-of-big-tobacco-and-big-pharma-to-influence-policy

https://www.humanetech.com/podcast/weaponizing-uncertainty-how-tech-is-recycling-big-tobaccos-playbook

Is that they've been phoning it in entirely on the very concept of intelligence, to the point that discussions around it resemble our understanding of intelligence before we started studying it using empirical approaches way back in 1904 - that's when Charles Spearman released a paper titled "General Intelligence," Objectively Determined and Measured outliniing a way of defining and measuring intelligence empirically. He coined the phrase 'general intelligence' in this paper because the entire point here was to isolate the underlying capacity driving performance on a range of tasks, and specifically not raw performance on a variety of tasks.

As he put it:

>A few words may now be said concerning the eventual meaning attachable to the result which we hope to obtain. To put it briefly, the usual direction of inquiry is in the present work reversed. The customary procedure consists in determining some matter of research subjectively, say, "Perception," "Attention," "Imagination," "Fatigue," etc., and then ascertaining its relation to other similarly pre-determined psy- [p. 258] choses or neuroses. Here, on the contrary, although every effort has been made to render the mental phenomena as unequivocal and significant as possible, yet in the beginning not the least note is taken of any psychological import beyond such as is barely necessary to define the subject of discussion in the most positive and objective manner; while the structure of language necessitates the continued use of such terms as Discrimination, Faculty, Intelligence, etc., these words must be understood as implying nothing more than a bare unequivocal indication of the factual conditions of experiment. For the moment we are only inquiring how closely the values gained in the several different series coincide with one another, and all our corrections are intended to introduce greater accuracy, not fuller connotation; the subjective problems are wholly reserved for later investigation. It is no new thing thus elaborately to deal with and precisely measure things whose real nature is concealed from view; of this nature, for instance, is obviously the study of electricity, of biology, and indeed of all physical science whatever.

Spearman was low key saying that the usual approach risked begging the question - researchers essentially began with psychological categories like “attention” or “intelligence,” assigned tests to those categories, and then treated the resulting scores as measurements of the very constructs they had assumed. If a definition is instead supposed to follow from and be justified by data, you now have a theory/hypothesis that can be tested and falsified.

I think the irony is that 'general intelligence' entered scientific psychology as a warning against the exact kind of reasoning that is common in discourse about AI. The term AGI doesn't directly follow from the psychometric usage of GI, it was coined to differentiate between performance on a narrow domain and human-like performance on a broad range of tasks. Which totally misses the point (that was apparent 12 decades ago) that intelligence is a causal factor in performance, not performance itself. Or that you actually have to do science to conclude that your reasoning benchmark actually measures something that can be called reasoning.

Anyways I could write forever here but I'll just cut it short because I really just want to bitch about how unbelievably shallow any talk about intelligence itself has been with AI. I'd say they're 12 decades behind psychology at this point (which has a reputation for having a low bar as it is), but I don't want to imply that an attempt was made.

Mind boggling that this was released only 3 years later, the jump feels like when I went from a T5 to an R10. I loved my mark I, but in the way you’d love a beat to shit ‘85 S10 pickup with 300,000 miles on it. The only complaint about moving the the mark II I have is that I have a ton of settings to remap, lol.

An example from a government report from the 80s in Canada:

>[AI] systems could be a solution to problems that cannot be dealt with using current data processing methods. They will certainly become a way to preserve and disseminate a vast range of expertise. The value of experts, therefore, will be amplified when their expertise is made available to all. [AI] systems will be the new Gutenberg revolution.

Even better are the wonderfully naive timelines:

>In the early 1990s we will begin to see systems with sophisticated natural language interfaces and the technology will begin to proliferate. By the end of the 1990s we should be able to make use of systems capable of self-learning.

Hot take: AI today is what people hoped AI was going to be 40 years ago. We didn't learn our lesson when those promises took a hell of a lot longer than anticipated and made even more outlandish promises. At this point you have to wonder if people never learn, if this mindset is ingrained in the culture of the tech industry, or if what we're seeing now is being carried over by influential names that were getting their start around that time.

AI as a broad concept is something that is uniquely suited to my background. Back in the day I got a BS and MS in experimental psych, and after a brief stint in the real world I went back to get a BS in math and then went back to grad school for ML and stats.

I'm genuinely interested in AI as a subject, but instead of reigniting interest in cognition (it's bothered me for a long time that I don't use my psych background for much of anything), this whole episode is actively killing my interest in other things I'm passionate about. I work in a consulting role doing a mix of software development and ML research, so I'm naturally in the same boat as a lot of other people watching LLMs and agentic AI seep into everything. I've tried to be a voice of reason being the sole ML guy on the team, but it's gotten so bad that we're losing bids to people that are using LLMs as fucking fraud detection algorithms.

On top of that, PC gaming has always been a pastime of mine and we all know what a clusterfuck that is while manufacturers deciding consumers don't exist anymore. On top of THAT, I work as a photographer on the side and suddenly everyone thinks editing pictures the way people have for the last few decades looks too much like gen AI (I actually like editorial looking shots, but it's kinda absurd that it's becoming the bar for what people think is genuine). Bonus points: I was planning to go for PhD in ML before all of the genAI hype started ramping up, now that sounds like the opposite of what I want to do.

Anyone wanna to grab a few Airstreams and set up a commune somewhere in the high desert? LLMs haven't ruined my dream of going off grid and setting up a pirate radio station yet.