u/Evipicc

There's a huge number of advancements literally underway ***right now***. On the power (and compute) efficiency side then there's silicon photonics and wetware. On the structural side some of the AI groups area already saying there's more to it that using 'just' transformers.

HOW inference and spitting out an answer works is what changed I think earlier this year, could have been last year. The 'looping' (NOT the correct term, it's RLVR/Thinking/Test-Time-Compute etc) in training AND responses is what kicked off this major surge in data center construction, but like anyone will point out, this kind of scaling is still in raw compute, and not sustainable. It works, but it's a ham-fisted method. Blah blah AI bubble blah blah, these companies are using the fast and loose money while it lasts to get infrastructure that won't go away when financials change.

Silicon photonics (look up Intel's Loihi 3, or Lightmatter) is an absolute gamechanger if we manage to get the point that the compute itself is photonic at scale. You're looking at multiplexed, neuromorphic, analog and binary, low power and higher speed compute and interconnects. A massive change. It would also deviate from consumer hardware competition and be its own specialized thing for a time, which starts to push the consumer PC parts market back in line, sort of, but we all know prices don't really just go back down overnight. Right now it's a real challenge to create a light based parallel for HBM, high bandwidth memory.

Another one is truly curated data, which is an RSI goal (recursive self improvement), or a manually curated data set. Right now these models are basically trained on all data that exists, but not all data is good, and it's time consuming, and expensive. Sifting through to throw out garbage and repeat data means training inference are drastically lighter, making an impact again.

Add all of that together and we're still looking at another multiple orders of magnitude in compute efficacy in the near future, some of it on existing hardware, some of it on a new breed of machine. I say multiple orders of magnitude because photonics specifically can do 10,000 times as much 'stuff' at 1/100th the power (their own reports, the real changes and efficacy will have to be proven, of course.

That's why these companies are scrambling to get so many data centers built, because the models inside them are going to shrink in their compute load over time, so the same data center (while there is a churn to the actual compute modules) is going to stand for a long time. The companies working on photonics are trying to make 'plug and play' the goal, so the modules just slot in to existing racks, which is objectively the right call.

There's also SSM (State-Space-Models) but I'm personally not educated on that. Supposedly it's one of the things that goes beyond transformers. Maybe both run in tandem, maybe it's the new breed, only an actual ML engineer would be able to answer that.

MoE/MoA, reaching out to sub-models that are more finitely trained on just the one thing they know... that's a whole new and active field of research now too. The orchestrating LLM actually doesn't need to train on more than just 'language', and the sub-models report back results instead. Lets you simultaneously run multiple things, concatenate them, and bring back a better answer. Also brings up network methodologies that aren't being used because what if some company in Zimbabwe trains and runs the perfect cooking recipe model, at a data center local to them, and questions about that are just always routed there. Suddenly every other LLM (or other architecture) on the planet doesn't need to ingest any cooking recipe training data. Do that across more topics and you start to get into the Torrent style AI model, kind of like a peer to peer system. This is already happening, sort of, in multi-agent-marketplace systems, but isn't really there yet. Discoverability is protocols are a weak point.

One thing I always gotta rant on is the pseudo religious bullshit... Some form of consciousness isn't a necessity for useful function. Full stop. It's just not. We have very little understanding of how our own works, so attempting to say it's not artificial intelligence because it's not 'tHiNkInG fOr ReAl' is one of the stupidest things I've ever heard. A calculator doesn't need to tHiNK to be right. the other one is that he arbitrary and constantly moving goalposts of AGI and ASI are completely worthless, all that matters is what it can do.

We live in a ridiculous time, and all that we're seeing now is literally the first 1% of what's coming. The world already isn't ready for what's already been launched, let alone what's coming.

Edit: Quantum computing has some minor implications in the compute stack of AI, but it's noisy and problematic. Not really worth mentioning today. In niche research apps (like protein research in pharmacology) it matters, but a typical user won't benefit from what's out there right now. A comment pointed out the re-configuration issues with silicon photonics, which are real, and the same applies to Quantum components tenfold.

There's a huge number of advancements literally underway right now. On the power (and compute) efficiency side then there's silicon photonics and wetware. On the structural side some of the AI groups area already saying there's more to it that using 'just' transformers.

HOW inference and spitting out an answer works is what changed I think earlier this year, could have been last year. The 'looping' (NOT the correct term, it's RLVR/Thinking/Test-Time-Compute etc) in training AND responses is what kicked off this major surge in data center construction, but like anyone will point out, this kind of scaling is still in raw compute, and not sustainable. It works, but it's a ham-fisted method. Blah blah AI bubble blah blah, these companies are using the fast and loose money while it lasts to get infrastructure that won't go away when financials change.

Silicon photonics (look up Intel's Loihi 3, or Lightmatter) is an absolute gamechanger if we manage to get the point that the compute itself is photonic at scale. You're looking at multiplexed, neuromorphic, analog and binary, low power and higher speed compute and interconnects. A massive change. It would also deviate from consumer hardware competition and be its own specialized thing for a time, which starts to push the consumer PC parts market back in line, sort of, but we all know prices don't really just go back down overnight. Right now it's a real challenge to create a light based parallel for HBM, high bandwidth memory.

Another one is truly curated data, which is an RSI goal (recursive self improvement), or a manually curated data set. Right now these models are basically trained on all data that exists, but not all data is good, and it's time consuming, and expensive. Sifting through to throw out garbage and repeat data means training inference are drastically lighter, making an impact again.

Add all of that together and we're still looking at another multiple orders of magnitude in compute efficacy in the near future, some of it on existing hardware, some of it on a new breed of machine. I say multiple orders of magnitude because photonics specifically can do 10,000 times as much 'stuff' at 1/100th the power (their own reports, the real changes and efficacy will have to be proven, of course.

That's why these companies are scrambling to get so many data centers built, because the models inside them are going to shrink in their compute load over time, so the same data center (while there is a churn to the actual compute modules) is going to stand for a long time. The companies working on photonics are trying to make 'plug and play' the goal, so the modules just slot in to existing racks, which is objectively the right call.

There's also SSM (State-Space-Models) but I'm personally not educated on that. Supposedly it's one of the things that goes beyond transformers. Maybe both run in tandem, maybe it's the new breed, only an actual ML engineer would be able to answer that.

MoE/MoA, reaching out to sub-models that are more finitely trained on just the one thing they know... that's a whole new and active field of research now too. The orchestrating LLM actually doesn't need to train on more than just 'language', and the sub-models report back results instead. Lets you simultaneously run multiple things, concatenate them, and bring back a better answer. Also brings up network methodologies that aren't being used because what if some company in Zimbabwe trains and runs the perfect cooking recipe model, at a data center local to them, and questions about that are just always routed there. Suddenly every other LLM (or other architecture) on the planet doesn't need to ingest any cooking recipe training data. Do that across more topics and you start to get into the Torrent style AI model, kind of like a peer to peer system. This is already happening, sort of, in multi-agent-marketplace systems, but isn't really there yet. Discoverability is protocols are a weak point.

One thing I always gotta rant on is the pseudo religious bullshit... Some form of consciousness isn't a necessity for useful function. Full stop. It's just not. We have very little understanding of how our own works, so attempting to say it's not artificial intelligence because it's not 'tHiNkInG fOr ReAl' is one of the stupidest things I've ever heard. A calculator doesn't need to tHiNK to be right. the other one is that he arbitrary and constantly moving goalposts of AGI and ASI are completely worthless, all that matters is what it can do.

We live in a ridiculous time, and all that we're seeing now is literally the first 1% of what's coming. The world already isn't ready for what's already been launched, let alone what's coming.

Edit: Quantum computing has some minor implications in the compute stack of AI, but it's noisy and problematic. Not really worth mentioning today. In niche research apps (like protein research in pharmacology) it matters, but a typical user won't benefit from what's out there right now. A comment pointed out the re-configuration issues with silicon photonics, which are real, and the same applies to Quantum components tenfold.

I went on an absolutely incredible all day date in the city I flew in to for an interview yesterday, and it's kind of turned me inside out.

She was amazing, and beautiful and funny and intelligent, but we both clearly established it was just spending the day together. Half as a favor to me to help me see the town to see if it's a good place to live, for me and my kids, half just for fun.

We talked the whole time, without any difficulty keeping conversation going, we had fun going around town, going to a museum, walking along the river, going to a bar and just talking for like 3 hours. She held my hand practically the whole time... When she left the hotel room this morning I asked her if I'd see her again if I did move there and she said probably not. Which that's fine, it was just casual, which isn't the problem, i don't feel rejected or anything, in fact it was probably one of the best days I've had in a decade or more...

Coming down from that I'm crashing pretty hard. I just don't want to be alone anymore.

I lost my wife to suicide the day after Christmas in 2019 after 14 years together, and I just want love again. I know exactly what it can be and I want it back so badly...