r/neuralnetworks

What is a overparameterized network?
▲ 9 r/neuralnetworks+1 crossposts

What is a overparameterized network?

I got this paragraph from Claude, could someone please explain this and verify if it's a real thing or hallucination:

Overparameterization isn't just about final capacity, it's about the optimization process itself. A wide, overparameterized network gives gradient descent a much friendlier loss landscape — more paths downhill, fewer bad local minima, room to explore before committing. The "core" only emerges as a byproduct of that search happening in a much bigger space than it needs to end up in. Strip the space down first and you've removed the thing that let the search work.

Conversation: https://claude.ai/share/8813a637-c327-4d0c-b120-def27e5203d5

u/basafish — 2 days ago
▲ 1 r/neuralnetworks+3 crossposts

The attention mechanism, simply explained.

Demystifying AI Attention Mechanics in under 90 seconds! ⚡

Queries, Keys, and Values don't have to be abstract.

u/automi_ai — 3 days ago
▲ 30 r/neuralnetworks+3 crossposts

I built 48 interactive ML visualizations where the algorithm actually runs in your browser — drag the data and watch it resolve live (free, no signup)

I kept running into ML explainers that were either a pre-rendered animation or a wall of notation, so I built the version I wanted to learn from: 48 concepts where the algorithm genuinely executes in your browser and you can interfere with it while it runs.

Everything computes for real no canned GIFs, no server calls. The math is hand rolled TypeScript with ~500 tests behind it, so you can grab a data point and watch the model resolve immediately.

A few that seem to click for people:

- k-means — drag the starting centroids into one corner and watch it get stranded in a bad local minimum. Step through assign/update one phase at a time.
- ROC & AUC — scrub the threshold from 1 to 0 and the curve draws itself while the confusion matrix updates in lockstep. Set positives to 1% and watch AUC still look great while precision-recall collapses.
- Decision trees — see the greedy *search*: every candidate cut it scored, and the one it kept, with the Gini drop as live math.
- CNN — draw a digit and a real trained MNIST network classifies it (weights trained offline, exported to JSON). Rotate the digit 90° and confidence collapses, because convolution shares weights across space, not orientation.
- Lasso — step through coordinate descent and watch a coefficient snap to exactly zero when it falls inside the threshold.

Every page also has a deliberate "now break it" control. I think you learn more from watching a perceptron oscillate forever on non-separable data than from watching things succeed.

Each concept also has the plain-language intuition, the math with every symbol decoded, and a worked example traced on real numbers.

Free, no signup, nothing gated: https://www.mlvisualization.com

I'd genuinely rather hear which explanations don't land than which ones do — happy to fix them.
mlvisualization.com
u/mtyrgh006 — 6 days ago

ML partner

Im currently learning deep learning with deep mathematical proofs and building it using maths only, like how at each epoch weights gets learned and backpropogated , how we use different gradients for optimizations and how they shift momentum , mechanistic inter. of transformers , reverse engineering dl models .If some1 is interested in this kinda stuff DM.

reddit.com
u/Notsoboringi — 6 days ago
▲ 9 r/neuralnetworks+1 crossposts

Has AdamW replaced Hessian-Free for Deep RNN?

In 2010, Martens invented deep Hessian-free methods. In 2011, Sutskever collaborated with Martens to apply it successfully to arbitrary RNNs (not special LSTMs, GRUs etc.)

In 2020, NERSC, Lawrence Berkeley National Laboratory, publish applications of combining the method with AdamW. Did everyone went quiet about the method invented by Sutskever and Martens 15 years ago and just forgot about it, or did it become SoTA optimizer secretly used for MoE router nonlinearities?

https://arxiv.org/abs/2006.00719

u/BrianCannard — 6 days ago

A cry for ST-GCN help

I will be honest. I don't have much time (around a week) and I need to create ST-GCN neural network for my classes, but I know next to nothing about it and while using AI create "some" model, by looking at results, I see that it has nothing do to with ST-GCN and propably have data leakage.

So to what I need to create is a STGCN model that will predict bike usage in public bike sharing system (like Citi Bike in NYC) by using agregated data from 1 hour windows (last 5 hours should be used to predict result from next hour)

Currently I have data:

- List of station

- Usage over 3 month time (I am using data gathered from my city, it doesn't have public data, can only gather current state via API)

- Number of features for each station (like number of available bikes, temperature, rain, wind, distance to metro station etc.)

- Adjacency matrix for stations

Data is agregated for 1 hour windows where bike availability is an average from that hour.

Data is segregated in 3 sets, training, validated and test that are separate.

So from what I know, it looks like I have all the data I need, but I have honestly next to 0 idea how to build ST-GCN. What I am asking is something that will explain how to build it in python, because I don't know what to do or even when to start with it.

reddit.com
u/Zestyclose-Jacket568 — 9 days ago

Absolute beginner here: What are the easiest, most intuitive resources to start learning Deep Learning / Neural Networks from scratch?

I’m completely new to Deep Learning and Neural Networks, and to be honest, the whole topic feels a bit overwhelming with all the math and heavy jargon out there. I want to build a solid foundation, but I’m looking for resources that start from the absolute basics something that focuses on building visual intuition and practical concepts first, before diving deep into complex math or code.

For someone starting from point zero:

What are the easiest books, video series, or courses you would recommend?

Are there any interactive tools or visualizations that helped you click with how a neuron actually learns?

What was the single resource that made the "aha!" moment happen for you?

Thanks

reddit.com
u/Sad-Actuary-7603 — 11 days ago

Programmierer für eine Neue KI/Neuronales Netzwerk gesucht.

Hi.

Ich suche Leute die mir beim Programmieren einer KI mit selbst erstellten Neuronales Netzwerk helfen.

Ich möchte fürs erste gerne was darüber lehren wie das alles funktioniert und evtl. gerne meine eigen KI bauen.

Dafür brauche ich aber Hilfe von Leuten die davon Ahnung haben und mir dabei gerne helfen wollen.

Mein aktueller Programmierung stand ist nicht sehr hoch da ich bis jetzt nur Grundlagen in meiner Schule hatte.

Und ich habe selber eine App entwickelt die nicht anspruchsvoll war.

Deswegen suche ich Leute die mir gerne dabei helfen wollen und die auch wirklich davon Ahnung haben und mir gerne helfen wollen.

Fals ihr Interesse habt meldet euch gerne bei mir.

Dann kann ich euch auch meine Vorstellung mitteilen.

reddit.com
u/DifferenceFriendly83 — 11 days ago

Building a Transformer from scratch + benchmarking optimizations — Open to feedback & using AI to learn

Hi! I'm working on "attention-evolution-50m" — implementing a Transformer

decoder-only from scratch in PyTorch and systematically measuring modern

optimizations. This is explicitly a learning project.

Full transparency: I'm using Claude to help me understand concepts, review

code, and avoid pitfalls. Not reinventing the wheel — understanding it.

Roadmap (flexible):

- Phase 1: Basic Transformer (done)

- Phase 2: Robust training with checkpoints/logging

- Phases 3-7: RoPE, Flash Attention, KV Cache, Mixed Precision, LoRA

- Phases 8-11: Benchmarks, visualizations, ablation studies

Looking for:

Someone who wants to learn this deeply. Open to changing direction, trying

different approaches, pivoting based on what we discover. 4-8 weeks, flexible

pacing. Comfortable using AI as a learning tool.

Real talk:

This might not be optimal — it's educational. I'll make mistakes and track

them. If you have better ideas, I'm genuinely open. The goal is understanding

first.

Everything is documented so you can follow the reasoning.

Interested? Drop a comment.

reddit.com
u/Volverman222 — 13 days ago