Login

u/Willwaste63

r/NepalStock r/GlobalOffensive r/linuxmemes r/mapporncirclejerk r/pytorch r/Pokhara_University r/NepaliFood r/duolingo r/NepaliFutballJerks r/IndiasGotLatent r/CinemaNepal r/IndianTeenagers r/technepal r/soccercirclejerk r/NepalSocial r/MLQuestions r/fulbo r/ProgrammerHumor r/learnmachinelearning
▲ 120 r/learnmachinelearning

Building a LLM from scratch was easier than i thought.

So i was following a resource called Visuara Ai, and they explained basic intuition of building a GPT162M from scratch.

I previously had built a transformer which was not quite successfull, GPT was fun to build, i implemented everything in torch. I implemented NN, Layer Norm, Multi head attention . Everything was easy except for that Masked Multiple head attention part, it threw error so many times while managing heads and masking. Only API call I had used was tiktoknizer, i also had build it from scratch but it wasnt so efficient I had used a recursive loop for more sequence recognition in a word.

And yeah i also used Autograd so it is not completely from scratch. And still training on Tiny stories over 50k stories for 5 epochs.

reddit.com
u/Willwaste63 — 6 days ago