u/Responsible-Fox-4933

I have been teaching myself ML by building everything from raw math no heavy libraries like PyTorch, just NumPy/CuPy and derivatives worked out by hand. Wanted to share where I've landed and get some outside perspective on where to take it.

The most recent thing I worked on was a curvature-aware optimizer, using Rayleigh quotient estimates of the Hessian eigenvalues to adjust the learning rate based on loss landscape curvature instead of just time-based schedules. I documented 6 versions with different architectures. The best one (V3) actually beat my baselines on some synthetic N-dimensional terrains, but it fell apart on spherical and Rastrigin terrains, and on real data (MNIST, CIFAR-10) it consistently underperformed a plain Adam + cosine annealing baseline. I've frozen that repo for now, my conclusion is that compressing all the curvature + gradient information into one scalar learning rate was the wrong way to go, and a per-parameter approach might be the actual fix, but I haven't built that yet.

Repos:

• Optimizer study: github.com/flackojodie/2nd-Degree-Optimizer-Fail-Study
• CNN from scratch: github.com/flackojodie/ConvolutionalNeuralNetwork-puremath
• Logistic regression (foundational): github.com/flackojodie/LogisticRegression-puremath

Before that I built a CNN from scratch in CuPy for a 10-class dog breed classifier — hand-derived backprop, a custom activation function, Squeeze-and-Excitation blocks, im2col convolutions as a part of the "puremath" family of repos which are more of a running journal of everything I was learning at the time than a polished project.

Honestly at this point I don't have a clear next target. Options I'm weighing are going back to fix the optimizer with a per-parameter approach, moving on to build a transformer from scratch, or diving deeper into the math side before building more. If anyone's got opinions on what's actually worth pursuing here, or related work I should be reading, I'd take it.