r/singularity

This is something I've been saying for a long time. But it's interesting that it's coming from the slavemaster general himself... ​​​

As a math PhD student, I've been having a number of alarming conversations with my colleagues, none of which came to any concrete conclusion. I'm interested in some different takes, so for what it's worth I'll just share my thoughts and the thoughts of some of my colleagues.

About me:

1. I started my PhD in February
2. It is in mathematical physics
3. I'm not an AI die hard crank nor am I completely anti AI

Scenarios for future maths research

Scenario #1: this moment right now is just the initial shock, AI is going to improve but as it finishes the human material for its training it's going to reach a plateau in terms of the quality of its output. At that point, when the dust settles, us mathematicians will learn to rely on its improved capabilities, publish more human-written papers, train the AI on those, then catch up and so on in a sort of tidal-wave dynamic. I think this is the most likely scenario honestly: if AI starts to train on its own material, which regardless of its correctness is exposed usually very very badly, its output is bound to get worse.

But I know yall don't believe in this, this is r/singularity after all. So let's move on

Scenario #2: let's just pretend that AI becomes so good that it basically becomes an unknowable mathematics oracle. Is there still a role for mathematicians in this world?

Here's my take: mathematics has always been a completely masturbatory endeavor. It is something we only do out of curiosity. Until now we've always been able to get some funding because we've been hiding behind the fact that yeah this theorem is completely useless now, but 150 years down the line some physicist is going to try to model something that needs exactly that and so we contribute to science and the well-being of humanity, that's how maths departments make the case for getting their taxpayer money.

I've always seen a lot of hypocrisy in this. When people used to talk about the Navier-Stokes problem in a public lecture, it was always described as important because fluids are central in engineering and planes and shit.

The Navier-Stokes problem was completely 100% useless for any kind of application. We've always known that. It is a mathematical masturbatory question about convergence of a PDE. Now that it has been solved by AI though, suddenly everyone had a sudden surge of honesty and admitted that it's completely useless, just to downplay the size of this result.

This is bad communication and it's the reason I absolutely despise public lectures. They're lies made up to sell you a product - in this case, a mathematics course at your uni.

-Edit: I don't want you to misunderstand: it is absolutely true that often what physicists or scientists need is (some - not all) mathematical structures that were invented decades earlier. I'm just arguing that that's not what maths research is about.

This is true for theoretical physics too mind you.

Application is not what anyone is in (pure) maths for, and I say that as I'm doing my PhD in mathematical physics and string theory.

If in this hypothetical future the physicists get their hands on this mathematical oracle that can suddenly out of nowhere tell them what is the mathematical structure that models that specific phenomenon, then our excuse for getting funded disappears, and that is everyone's worry at the moment.

As a community, our main task right now is to be introspective and ask ourselves why it is that we do maths in the first place, and do it honestly, not just finding talking points to get grants. I have my own personal answer as to why I do maths, everyone does, but there is no community level answer, and I think that's the main issue. Once we figure that out, we can ask ourselves whether AI really undermines that.

I see this scenario becoming real only if somehow the AI becomes much better at mathematical exposition. I've heard some might argue that simplifying an argument to get good exposition shows an understanding of a sense of "elegance" and clarity that the AI cannot develop. I don't buy this argument, I think it's a huge cope, but my gut feeling is still that tech just can't improve forever and some kind of plateau (whether it's computing power or something architectural like having enough high quality data anymore or something else) has to manifest itself at some point.

About Terry Tao's post (edit - apparently this is not Terry's take, it was a repost of someone else's opinion. Everything I say here is about the author, not necessarily Terry)

I hate his take. I'm a mathematician and he doesn't speak for me. I think it is completely hypocritical to say "mathematicians didn't ask you to do this". Maths is for everyone, and in particular it must be for OpenAI too. It's an absolutely hideous opinion that exemplifies everything that I find bad and fake about this community, and he should feel ashamed for having said that so openly.

That said, the way OpenAI handled this was also very bad.

For one, there's the usual issue of not citing the relevant literature, which is IP theft, but AI companies are not new to this kind of shitty behavior so I shouldn't be surprised.

Then, I don't see the value in dropping 700 results 500 of which were just low hanging fruits waiting for a PhD student to solve. That is just being a bully for the purpose of showing that the AI's raw throughput is more than that of a human, and I don't think anyone needed proof of that. Until we completely reform the way PhD's are handled, there is really no point in doing these sorts of stunts, this is just eating our work. The results are not hard not prestigious, we use these low hanging problems just to build the bones of future researchers.

Also, most of them are unchecked and literally the day after release people already found mistakes in the proofs using Astra. They couldn't even be bothered to ask Astra to check the results, and that just shows how much they care about the maths itself: it's clear that no human has actually read through all the slop. Clearly this can't be the future of research.

I actually buy the case for the Lean-formalized bigger results like the quasi-Riemann hypothesis. I think it's incredibly sick, but again a human will have to go through the results to check that actually the lean code is reflecting the natural language paper, which is not guaranteed.

If they had only released the 100 biggest papers but had put any care in writing and formalizing them, I don't think anyone would have any sensible reason to criticize them, and I myself would just dismiss the mathematicians' response as just being emotional. They would've still have made headlines and I would've been all for it, while by doing this stunt they just gave talking points to the haters. This is not how we (or anyone, including AI) should do research.

There's one last thing I want to point out: regardless of which scenario plays out, this tech is here to stay, and we should not pretend otherwise. We should all adapt to use it in our day to day research. However, to me it is really uncomfortable to give all this power to a private company. In a perfect world, AI is completely nationalized and instead of stealing IP we all contribute to it in a transparent and institutionalized process. Now, I'm not naive, I know this is not how things will play out in the short term, but it's really the only reason why even I feel a bit of an ick when I use the codex for my own research. I don't want all research to only become prerogative of whoever happens to have the computing power, that is a very dangerous slippery slope towards a cyberpunk dystopia.

I'm curious to hear other takes on this. I'm very disappointed by the maths community response, I really hoped that we would react better than the artists, but in the end it turns out we're all human. At the same time, I think OpenAI should've handled this differently. Overall, I'm stoked about the tech, but I'd much prefer it if they had someone who actually gave a shit about maths running this research.

Also I should mention that I really don't buy their figure of "3 hours of chatGPT pro", I think that's all marketing.

Have recent advances shifted your feelings and what do you think is in the horizon?

Discussions about AI Math in the sub have been quite heated lately, and I’ve been following them every day. I am not a mathematician, in fact I come from the linguistics field. But I do want to chime in and share some thoughts.

Basically, there is this one question I cant shake out of my head: why are people so focused on the idea that mathematics is not about proofs, but more about understanding? Of course, I agree that mathematics is more than getting the right answer. But I can't help feeling that some of the arguments in this debate are based on assumptions that haven't been clearly discussed. That is, what exactly do we mean by understanding? How do you define it?

Suppose a mathematician spends ten years proving a conjecture. They go through countless failed attempts, discuss ideas with colleagues, and eventually publish a paper. Throughout this process, they undoubtedly develop a deep understanding of the problem. But who actually benefits from that understanding? First and foremost, the mathematician, and perhaps the colleagues who worked with them. Later researchers don't need to go through those same ten years of exploration. They can read the proof, follow the reasoning, recognize the underlying mathematical structures, and develop their own understanding. Isn't that how academic knowledge has always been passed on?
So if AI produces the same proof in a few hours, complete with detailed reasoning, why should we call it "an answer without understanding"?

I agree that discovering a proof yourself is certainly different from reading someone else's proof. The former doesn't just tell you that a particular method works. It also develops your ability to find methods in the first place. The failed attempts, unexpected observations, and changes of direction may help you solve other problems later. That's a fair point. But couldn't an AI-generated paper provide similar benefits to future research? It can show the mathematical community which methods work, reveal unexpected connections, suggest new directions, and pass those ideas onto future generations of mathematicians. Researchers may find inspiration in an AI-generated proof and develop theories that the AI itself never explicitly proposed. Shouldn't the value of a paper depend on what its readers can learn from it, rather than how many failed attempts its author went through?

I know that human research can lead to unexpected discoveries. But why should we assume that AI-generated proofs cannot contain equally unexpected ideas? Some people seem to assume that AI can only rearrange information from its training data. But even if that were true in a particular case, we would still need to explain why combining existing knowledge in new ways cannot produce genuinely new mathematical insights. After all, how much of human mathematical creativity also involves reorganizing and extending existing ideas? Of course, this doesn't mean every AI-generated proof is original or insightful. It still needs to be evaluated case by case.
 
I wonder what you think.

I've noticed a pretty alarming trend lately in the AI subreddits and across the internet as a whole. I think it is one that we are also seeing reflected in the real world when it comes to AI safety / alignment researchers being ousted left and right.

The internet seems to have broken into two camps, one of those absolutely opposed to AI, and one of those that are blindly for it.

If anyone strays from one of those two schools of though, they get ostracized by both groups.

It seems to be almost impossible for anyone to talk about both the benefits AND the detriments to the rapidly advancing pace of AI, because doing so results in both the AI crowd and the Anti-AI crowd deciding you are against both of them and tearing you apart.

From a safety perspective, this is apalling. You can be 100% for a technology while also wanting steps to be taken to ensure proper alignment and to minimize the harm to people from those technologies. Yet doing so seems to make people think you simultaneously love AI and also want to see it completely shut down.

The most publically facing example I can think of is how AI safety and alignment researchers are being sacked by major AI companies (mainly OpenAI) over what appears to be them wanting more steps to be taken to pace the frontier. These are people that have dedicated YEARS of their lives to AI research, so obviously they support the tech and want to see it succeed, yet because they aren't willing to throw safety away for the sake of progress they are getting axed.

Has anyone else noticed or felt this? I feel like I am going crazy because everywhere I see people discussing AI it is either entirely positive or entirely negative with no inbetween.