BTW the model is 15M param (Not-ordinary transformer), and is pre-training as we speak it's only has about 800 steps of it's max 20k training steps.

No SFT and all. I just wanted how the model stability holds, if you ever worked on pre-training LLMs is this what you see as well. (Mathematically makes sense to me as acc is about: 0.17, but I want to you know be sure, this one is expensive compute and as an independent researcher I have more to lose if I see it failing. So it is for me big loss if did not work out.)

When testing the scaled down 1k, 10k & 100k param architecture on set patterns the model showed high intelligence. Only trained on couple of steps <500 and the model learned the multiplication scheme taught to it in all test sizes and the 1k variant was perfect till it was trained but started failing as the model input was increased and was held out/never shown that data in training run (it did 64/100 on those unseen tests, still good considering a vanilla Transformer ~600k params did less than that) ; 10k and 100k showed sparks of supreme intelligence per param (outperforming pattern held out training by upto 10M digits more than it was ever trained on... the model was trained to multiply till 10000, it multiplied till `10000-(12 zeros more)` with 100% accuracy even surpassing CPU computation which is off by some float points. 10k/10k score for both 10k and 100k model. Idk how but 100k model somehow made a logical explanation on it's own for addition. It was able to add using multiplication.

I am really seeing this as something; this 15M param model as we speak outperforms Qwen-3-4B-base on this same training data in terms of same hyperparameter checks.

For training dataset being ~1.05B tokens of high quality general domain data, science/creative writing/maths/general school knowledge.

For what I can see the model is pattern recognition beast. Like it learns like crazy and at crazy fast speed. I was training it's 1M param model, you will not believe it, it learned the entire tinystories dataset which has like 2M rows (repetitive and close to `Once upon a time` types I know... since LLMs are normalised output machines "generalization" is obvious once saturation is reached.), back to the experience so it learned the format in 500steps (not accurate or too coherent) but dammit the model was really close (like even assumed the next character name perfectly) to the training data it never even get too see. those 500 steps were of 64k samples out of 2M samples.

This is why I am trying to scale as much as my budget allows me to and test this model. If it fails I may be a fool; I can only find out that after words (I may already be a stupid fool already) 😄

So if you see something strange help me please don't be afraid to ask questions apart from architecture details I can give you all the knowledge.

https://preview.redd.it/j6if0jxm8h2h1.png?width=1054&format=png&auto=webp&s=2c5be301908e4861c515b5f22d1f72974606d264

I started learning generative AI thinking most of my time would go into understanding models.

Ended up spending time on completely different things.

One day I was reading about prompts, then embeddings, then vector databases, then RAG, then trying to understand why a model was giving weird outputs even though everything looked fine.

I also realized building something yourself feels very different from watching tutorials. I'll watch a 20 minute video and think "okay that looks straightforward", then spend the next few hours trying to figure out why something isn't working.

Not complaining or anything, I actually like it. I just didn't expect the learning process to go like this.

Curious if anyone else had the same experience or if I just went down a weird path.

I need to write an essay, and to be real, I suck at this kind of stuff. I’m more into technical fields, so writing pieces where you have to express your opinions is definitely not my vibe. Any tips on how to get better? My main problem is that my sentences feel completely disconnected, like they're from different papers. I have ideas in my head and want to blend them nicely, but the final result is just a mess. I also make a lot of grammar errors, but that’s an easy fix with a couple of rounds of proofreading

Autoregressive LLM world models factorize next-state generation left-to-right, preventing them from conditioning on globally interdependent anchors (tool schemas, trailing status fields, expected outcomes) and yielding prefix-consistent but globally incoherent rollouts. MDLMs' any-order denoising objective sidesteps this by learning every conditional direction from the same training signal. Empirically, fine-tuned MDLMs (SDAR-8B, WeDLM-8B) surpass AR baselines up to 4x their total parameter count on BLEU-1, ROUGE-L, and MAUVE across in- and out-of-domain splits, with lower Self-BLEU and higher Distinct-N confirming reduced prefix mode collapse. GRPO training on MDLM-generated rollouts shows up to +15% absolute task-success gains over AR generated training on held-out ScienceWorld, ALFWorld, and AppWorld across 1.2B–7B backbones (LFM2.5, Qwen3, Mistral) in a zero-shot transfer setting.

Genuine question,
how do you take some architectural decisions like the size of the neural network and the whole set of hyperparameters.

I get that there's brute forcing and hyperparameter search (which sometimes, really, it's a LOT), or some notes in literature regarding the choice of activations or loss based on context, but how would one really target some specific design choices when starting to explore efficiently, especially in terms of number of layers and latent space dimensions.

I appreciate your time, will take every tip into account