I feel like a low parameter model on maximum thinking.
Took the CS480 exam. At first glance, I might get a 0.
Never seen these questions before. Didn't know what they were asking, didn't know where to start.
But then I started reading word by word. Each word went into my context window. Every time a new word appeared, I attended to it with all the context from the words before.
y_i * f_w(x) = w? What does that even mean? I only recognize y_i * ŷ_i = 1. Are these related? Is this a typo? Wait — w = 1 is the answer, so is it saying y_i * f_w(x) = 1? Is this a trick question?
Wait, let's try an example. x_1 = -1, y_1 = -1, x_2 = 1, y_2 = 1.
Hmm. I don't understand what I'm supposed to do with that.
What if I substitute the values in?
Oh. y_i * w * x_i = 1 * w * 1 = w. I see. (-1) * w * (-1) = w. Ah, that makes sense.
……
d(L)/d(a). Okay, I know what d(L)/d(s) is — it's g. But then I need d(L)/d(s) · d(s)/d(r) · d(r)/d(h) · d(h)/d(z) · d(z)/d(a). Wait, how do I even differentiate matrices?
Oh — 2d) says d(L)/d(z) = g(0, 1, 1)^T. So d(s)/d(r) · d(r)/d(h) · d(h)/d(z) needs to match g(0, 1, 1)^T. Let's pattern match and get the answer.
wait, my dimensions are wrong. a is a scalar, so how did I get a 3x3 matrix? I did something wrong. Let me reconsider..
I walked out of the exam not knowing up from down, with zero confidence in any of my answers. But as we talked through the problems afterward, each one made more and more sense.
A large model, even at its lowest thinking level and using x100 less tokens, would have one-shot every problem. Instead, I iterated on every word, absorbed context from future questions to the current one, double checked, used examples, triple checked, wow.
I've never seen this type of thinking in myself ever. I'm so confused. I used to have a top-down approach. I look at a problem, think of possible solutions from the highest level, then start writing.
But now, I start writing without knowing where I'd even end up. Yet through pattern matching, was able to get to the correct result. This autoregressive behavior I exhibited was incredibly fascinating, and many standard deviations away from what I believe my normal thought process is.
How is it possible to write a solution but only understand your solution after you've written it?
This is so odd to me.
I feel like I trained a mini gpt model inside my brain. If my brain doesn't know where to start, a minigpt part of my brain, which feels completely detached from my brain, starts outputting continuously. My brain then makes sense of the output from the smaller model and completes a final polished and correct answer.