
Why probabilistic code generation scares me for actual hardware
kind of a late night thought while fighting with some embedded C for a feedback loop setup.
Everyone in the broader tech space is completely obsessing over automated code generation tools right now, but ngl, using purely probabilistic models for anything that deals with physical systems feels like a massive safety hazard. if a model has a tiny chance of hallucinating a pointer mistake or an invalid state transition, that is basically a broken actuator or a ruined prototype in our world. We need absolute guarantees, not just "palausible" outputs
I was reading an interesting post about how benchmarks are finally shifting toward strict formal verification instead of just checking if the code compiles. seeing systems progress toward machine-readable logical frameworks where things are mathematically proven before deployment makes way more sense for safety-critical stuff
idk, are any of you guys actually trusting automated code for actual controler implementation yet or are we all still writing every single line by hand out of sheer paranoia?