u/Rieux_n_Tarrou

He was already good at debugging

Question: As much as technology has always advanced, why do people think [Data Poisoning Tech] cant scale either?

Technical Reasons

The golden rule of data science is Garbage in, Garbage out. Anyone whose taken an intro course to DS, let alone a professional, knows that unglamorous bulk (80%+) of machine learning is data cleaning/munging. LLMs were trained on the entire frikkin internet, and before they became viable, a Herculean effort was made to ensure they don't spew garbage. Of course there are hilarious examples of failure in this regard (see: Microsoft's Tay chatbot), but in general (i.e. accounting for edge cases/situations) LLMs have managed to distill the the internet (which is mostly trash) into conversational models that perform better than 80% of humans.

Economic Reasons

Let's say there are 10x as many Poisoning Aficionados as there are members in this sub, so 100,000 hobbyists and activists around the world utilizing the latest and greatest open source poisoning fountains. Now, for everyone one of these folks, there are 10 people getting paid full time jobs to un-poison and enrich data for next-gen model training. Ranging from data scientists, to engineers, to gig workers who spend their days maximizing the signal-to-noise ratio in data. They do this thru manual and automated methods. Hundreds of Open Source tools for every Poisoning tool. Projects that we know about and don't know about. Simply put, the modern global economy is directed towards the improvement of AI, because AI extracts value from data better than humans can extract value from data.

Game-theoretic Reasons

Adversarial attempts to "trick" AI only make AI stronger. Remember GANs? Two models pitted against each other: one generates fake data, the other tries to figure out if the data is fake or not. They just let them run for trillions of cycles; the end result is that the generating model becomes really really good. This was over a decade ago and was the foundation for the Gen AI explosion of the past 5 years. I would be 0% surprised if it turned out this sub is actually an attempt to crowd source adversarial behavior from naive humans. Certainly many of the comments in this thread and others are bots, and certainly much of the code that's posted here is AI-generated. It's the same concept of white hat hacking: "Show us what you got, throw your best at us; it will be excellent training data to harden our models against adversaries"

What if the Poisoning Succeeds?

Let's imagine there was a breakthrough in Poisoning technology, and after spending billions of dollars on data centers, water, and compute, OpenAI unveils its latest model, GPT 6.9. Turns out, 6.9 has the reasoning and communicating ability of the average Redditor. Oh no! What now? Well, with literally a click of a button, 6.9 is shut down and 6.8 is re-deployed. Then OpenAI will deploy a few more billions of dollars towards figuring out how 6.9 got poisoned, clean up the poison, and retrain (more energy, water, data centers, etc). Then the mythical Poisoning Nuke is invented and the entire internet gets exploded with brain-rot data (even more so than now). Training new models is somehow impossible, and humans who consume digital information are more likely to be poisoned than not. Is this the future that Poisoners want? Short answer: Yes, because it is an orientation towards technology that is fundamentally irrational and anti-competitive: "burn everything to the ground, regardless of the cost to innocent life, because I can't deal with a changing world."

What are we Poisoning, Exactly?

Ok, you published Fake News so that LLMs make people stupider. Ok, you published broken code so that coding potentially useful software takes longer. What about Medical AI models? You going to publish fake MRIs and XRays so more people die? You going to publish fake Driving Data so Autonomous Vehicles kill more people? Garbage Weather Data so that Natural Disasters cause more casualties? Fake Genomic Data so that disease breakthroughs are pushed back another decade? You see how insane this is? All because, what, a bunch of butthurt white collar workers can no longer rest on their laurels and are forced to adapt to a changing economic landscape?

A Sane, Rational Path Forward?

Make no mistake, AI is an egregore of the Kraken **(**some dude on Reddit was the first to realize it about 7 years ago). The Singularity is nigh, that much cannot be denied. In the meanwhile, a healthy human gets happiness from being of value to those around them. Human Connection, Honesty, Competence, Integrity, Hard Work, Trust. The formula never changed, and it can't be poisoned. What has changed is that Data has become the driving force of value in our world. To be clear, I have no love for Big Tech/Late-Stage Capitalism. I don't trust them one iota. But to the extent they improve AI, I benefit, because using AI as a tool allows me to be of greater value to people, and that makes me happy. The Conversation needs to shift from AI bad/good to Data Ownership. To that end, I am a big proponent of Decentralization (self-hosting, blockchain, Network States) and Open Source. You can't un-summon the Kraken, but you can nurture your own mini-kraken to protect and enhance your values, and those of the people you care about.