u/telephantomoss

This is inspired by a recent comment about some people understanding entire fields. So I'm wondering what it could even mean to understand an entire field. Certainly the average mathematician doesn't satisfy that. I suspect most mathematicians do not have an entire field mastered. Again, what does that even mean?

I propose one possible interpretation. Let's just take something that is arguably a "field of math". I'll use probability theory as an example since it's what I'm familiar with. It's a big field. So the best pretty theory experts really understand the entire field though? It's a big field. Probably even then best experts still have a long list of results they have never heard of. They'll likely literally know everything in their little sub discipline, but the *entire* field of probability theory?!?

I could be wrong though.

Also, this is likely simply asking too much. Rather than literally knowing every result and every proof, maybe we should set the bar at something like: they can read an arbitrary new-to-them result in that field and understand it nearly instantly and to be able to breeze through the proof and then explain it without much study. That is probably feasible for a really smart expert, but I'm not really sure. This is less than asking them to produce a fully rigorous proof, more like a satisfactory sketch.

I'm not at all an example here. I know very little compared to such folks. I suspect my level of knowledge is not that unusual though, even if somewhat on the low end. But one can know orders of magnitude more than me and still not approach the entire field of probability theory.

I home at least some find this question interesting.

I claim that AI (e.g., the most powerful frontier models) is now less error prone and faster than the vast majority of college professors at most, if not all, undergraduate and graduate college math. This is not of course universally and absolutely true, but should be viewed as a testable statistical claim.

One should be as skeptical of textbooks and published papers and course notes as they are of AI math.

Roughly a year ago, the vast majority of people still thought AI could do symbolic computations when in fact it was already writing Python code in the background to use actual symbolic engines. At that time, it was already quite proficient at a room of undergrad math. Its error rates have continued to drop.

I also claim that AI is now more useful that the best majority of human referees. Again, this is to be understood at a testable statistical claim. But if you take a typical research paper submitted for publication, AI will find the errors (not just typos) faster and more accurately and will explain than better than a human referee.

These are all easily testable claims but aren't really worth the effort to test.

Is anyone not yet convinced of this, and if that is the case, why not? Explain the particular area of math you are working in where AI is still very error prone. And also be sure you are using frontier models like Sol 5.6 on Ultra thinking. Maybe Fable/Mythos 5 counts too, but I'm not as familiar with them.

I have been asking ChatGPT about my average usage level (Plus subscriber). It said I use about 500k visible tokens but about 7 million total tokens and about 8 total hours of GPU-compute in a month (relative to a single GPU).

My usage has been pretty heavy for the past few months, lots of technical math and data related stuff, light programming, document processing.

It said that I am a fairly heavy user, relative to most Plus subscribers. I tend to write long prompts about technical matters and lots of tool use.

I'm curious what others usage levels are and if these numbers seem reasonable.

This post is about the relevance of coding skills/proficiency/knowledge in the age of AI. I want to gather perspective that might be useful for anybody whether undergrad or grad students in math or a related quantitative field, or those looking for jobs, or even current researchers whether in academia or industry. The main focus isn't directly about teaching or advising, but that is a very relevant aspect here. I am looking for perspective from people with all manner of mathematical experience and life stages.

The main question is: We used to say that everyone should learn to code. What is the current status of this guidance?

Partly I want to know what to tell my students. But also I want to think about it for myself as I continue along on my own path. I'm also just interested in this from a purely sociological standpoint.

My thought is that everyone should still learn to code. I mean, you still need to understand how code works and to be able to read it. I find myself coding a lot with AI. I've generated more code in the past year than I have the past 10 years probably. Of course, I have done very little coding from scratch in that year though. However, I still find myself reading the code to understand the broad picture of it and editing little bits and pieces. Sometimes I ask AI to do a complete pass over the entire code and fix what I want. Sometimes I ask it for a snippet for me to paste manually. Sometimes I just edit it purely manually when it is a language and syntax that I understand, especially when it would take more effort to prompt the AI, wait for it, and the risk of it messing up. I mean, I do want to understand as much as possible, and sometimes I want the challenge of figuring out how to paste in a little patch. But it is also about economic efficiency.

I have spent many long hard hours coding from scratch in various platforms and languages, so I can read almost any language and understand a lot of what is going on. I cannot imagine lacking that skill! I cannot imagine lacking an understanding of what it means to instantiate an object, give it a name, and dynamically access its methods and properties. I can't imagine one can be truly very effective at using AI to code when they don't have that basic understanding. Am I wrong about that?

I mean, everyone (say, students who want to get a math degree and get an industry job) should still learn to compute basic derivatives and integrals even though most will never do that in a real world job, right? Doing that work will generate conceptual understanding and intuition which carries real value I would argue. They will almost certainly be doing things that involve integration. Not to mention, having a scientific understanding of reality is indispensable, and that involves understanding how areas, volumes, rates, flux, etc relates to integral, even just conceptually without the need to actually set up experiments and do computations.

However, maybe this changes what we should concentrate on when we learn to code? I mean, you probably need to learn to prompt AI effectively for creating code from scratch and for uploading and editing code. That's much easier than learning to code from scratch though and you don't really need textbooks and classes for that. You can learn to be effective with AI coding by using AI to actually learn the syntax, of, say, Python. I have been wanting to learn Python for a long time, and I finally am starting to understand it by using AI to write Python code. I was so confused about py vs py3 vs python vs python3, and whether to execute in OS terminal vs python terminal, etc. But AI helped me get it all set up and I can execute python code effectively now finally after so many years of neglecting it. Can I write it from scratch? No. But I intend to have that ability eventually---even if it isn't necessary, but by the time I use Python for a few years, I'll have some basic code-from-scratch ability at least. But, more importantly, I'll be able to read and edit it effectively.

Any constructive thoughts here are appreciated, and especially getting as many different perspectives as possible would be great. Thanks.

***Note I got an alert on this possibly being about career guidance, but it isn't that in my view. It could be construed as that since it is partly about me wanting to know what to tell my students, but that's not really the focus or only aspect though. I think the general math forum is the best place to grab a bunch of different perspectives on this issue. So I'm going to go ahead and post it and see what happens. It's clearly a very relevant question for all math folks.

Here's a hypothetical experiment that will show the weakness of AI. Take all the best models right now or whichever models you want. Feed them a question and talk then to solve it and write a paper. Maybe even solve a decent problem without giving them the argument. Assume it's something reasonably novel, creative, whatever. Maybe like not a top Annals level result, but just an average publication worthy working mathematician paper. Have the AIs work on it for a year, checking their work and training it, without human intervention except to ensure they keep running, no human text input etc. World they produce s correct paper with correct and satisfactory details?

It probably would look correct as it would match the grammar and structure, but there would probably be errors where the argument requires inference of novelty that is not in the existing literature in it's training data.

Bruh: AI says this.

Derp: But it's wrong all the time.

Bruh: It literally cited it's source right here. It's literally just quoting the literature directly (processed via LLM/transformer/processes).

Derp: but it LiEz.

Bruh: Here is the link. Look at the paper yourself.

Derp:... it lie though! take downvotes!