Will Deep Learning give way to some other form of AI like SVM etc?
I just can't get this hardware lottery concept that we are bound to iterate what fits the hardware (gpu) best. Transformers fit GPU wonderfully and scales beautifully and thus next generation of GPUs treat transformer as first class citizen and thus transformer gets even better and so on.
Also almost all deep learning models are based on back propagation which means entire model's parameters need to be updated at the same time which is partly why we can't have models learn like humans do (on the go)
I know there's test time training/continual learning but can it be done as good as animal brain with GPU as a substrate?
I can't get this idea that as long as SIMD or dataflow architecture are substrate of deep learning it has inherent ceiling and will thus be replaced by other AI especially for robotics and edge and become like SVM/tree-based methods. Useful in certain scenarios but no longer a center stage
what do you think?
- i worded it weirdly i meant will deep learning be replaced by something new like svm was replaced by deep learning (although svm/tree based methods still has their area of strength like tabular datasets - i don't think deep learning will completely disappear either)