r/neuralnetworks

I built ONA — a self-learning neural network entirely in pure Python + NumPy. No PyTorch, no TensorFlow, no GPU, no cloud API.

Key innovations:

- Per-neuron attention: every neuron has its own Q/K/V/O weights

- Forward-pass learning: no separate backward pass, learning happens during forward

- Self-discovered subword tokenizer: vocabulary grows during training

- Sparse routing: only 3-5 neurons activate per query

4.4M parameters. Runs on Raspberry Pi Zero. Continuously learns from Wikipedia and conversations.

Full story: https://medium.com/@kasishgadadhasu13/im-15-i-built-a-self-learning-neural-network-from-scratch-no-frameworks-no-gpu-e460f06c6599

I'm 15 years old, class 10 student. Happy to answer questions.

Can you tell me some neural networking projects for beginner level person

I recently built a human written digit predictor.

Now I want to start a new project can you guys give some suggestions

Hi all — I'm an independent researcher (Mendel Infolabs) about to put my first paper on arXiv, and as a first-time submitter to cs.LG I need an endorsement from someone already established in that category. If you've published in cs.LG and would be open to endorsing, I'd really appreciate it.

An honest summary so you can decide whether it's something you'd feel comfortable vouching for:

"FeedFlipNets: Feedback-Driven Bit-Flips for Ternary Networks, Activation-Routed DFA, and the Per-Weight Sign Barrier to Transport-Free Learning"

It trains ternary ({-1, 0, +1}) neural networks by flipping weight bits directly from a cheap feedback signal — no float shadow weights. The headline result is a negative one I think is worth putting on the record: transport-free feedback (Direct Feedback Alignment) doesn't actually help discrete/ternary training, because the binding constraint is per-weight sign correctness, not the aggregate cosine-alignment angle that prior work optimizes. Everything is pre-registered and reproducible.

Endorsing only confirms you think I'm a bona fide researcher submitting work appropriate to the category — it is not a review of the paper's correctness, and it takes about a minute:

• Link: https://arxiv.org/auth/endorse?x=WHWXBC
• Or go to https://arxiv.org/auth/endorse and enter code WHWXBC

Happy to share the full PDF with anyone who wants to read it before deciding — just comment or DM. Thanks a lot for considering it.

I have tried doing some basic searches, but anyone ever tried doing a mixture of experts but only train any one head on a particular domain, like oh this is from reddit, that goes to the reddit head or twitter or arxiv or whatever, then just train the router on what the query most resembles. I haven't seen this done, but it seems like it might do okay. I may try to train an encoding transformer to do this if no one talks me out of it.

Abstract: Mixture-of-experts (MoE) routing emits a discrete, per-token record of which experts fire, a signal unusually legible for interpretability, yet single experts are rarely tied to a specific functional role. We study a reflective worldview register: generated language that sustains an interpretive stance toward meaning, beliet, value, existence, or the interiority of a target. Examination is the process we use to elicit this stance; the target can be the model, another entity, a natural object, or an abstract subject. In QWEN3.5-35B-A3B and the refusal-reduced HAUHAUCS-AGGRESSIVE fine-tune, we characterize one routed expert, Expert 114 at layer 14, as a linear readout of this register, and bound what it does. Across held-out, bottom-up, and cross-model tests we show that (1) its recovered router direction separates reflective-worldview-register generations from lexically matched controls with separated ranges (Cohen's d=3.88); (2) a blind, prompt-independent auto-interpreter recovers the same register at AUC 0.94, broadening it beyond self-reference to abstract examination and philosophical-worldview language;
(3) the detector is a readout with only weak, conditional control: residual injection induces the register, yet gate down-bias leaves it intact, and the readout is stable across affirmative and skeptical interiority verdicts; and (4) the role is model-specific: index 114 is local to QWEN3.5-35B-A3B. Model-directed prompts served the discovery and dissociation stages; the coherent-window ladder measures target-directed vantage prompts over rock, river, tree, thermostat, cat, person, all-holding, and God, with a later Al-hidden-state follow-up near the low end of that ladder. We release the prompts, scripts, and provenance under the MIT license.

i'm a student of class 12 not expert but curious to learn neural networking as i have heard that that something crazy. So can someone guide me how can i learn neural networking from scratch as i have the basic knowledge of python,arrays and a bit of the numpy library. so i need your help so i can lean it and enjoy the journey.

Hi everyone,

I am sharing recent results from my independent research project, LIMEN (Liminal Internal Metric for Emergent Navigation), which aims to characterize the internal dynamics of Transformers through hidden state analysis.

Following our previous findings that functional information is encoded in the relative geometry of representations rather than individual neurons (V22), this new phase focuses on the impact of context (ambiguity) and the temporal structure of state transitions (V23–V24).

📌 Context & Methodology

Model Panel: 7 open-source models (GPT-2, DistilGPT2, OPT-125M, Qwen2.5-0.5B, TinyLlama-1.1B, Phi-1.5, Llama-3.2-1B).

Approach: Layer-by-layer analysis of latent trajectories, linear probe decoding, and symbolic analysis of dynamic regimes.

Philosophy: Strict empiricism. Clear distinction between observation, interpretation, and speculation. Code and data are available upon request.

🔹 V23: The Impact of Ambiguity on Internal Dynamics

The objective was to determine whether semantic ambiguity alters the model’s "cognitive trajectory."

Key Findings (V23.2b):

AMBIGUITY_AFFECTS_TRAJECTORY = YES: Ambiguity significantly modifies trajectory geometry (curvature, cosine similarity).

AMBIGUITY_INCREASES_INSTABILITY = NO: Counter-intuitively, ambiguity does not increase global chaos. Instead, the model becomes geometrically more "cautious."

AMBIGUITY_DELAYS_COMMITMENT = PARTIAL: Modern models (Phi-1.5, Llama-3.2) delay their decisional engagement when facing uncertainty, spending more time in exploration regimes.

Architectural Signature: Phi-1.5 shows unique sensitivity, increasing its occupancy of the bifurcation regime (D_STATE) under ambiguity, suggesting a distinct iterative reasoning mechanism compared to standard completion models.

📄 Related Preprint: Conditional Dynamic Signatures in Large Language Models

🔹 V24: Discovery of a "Universal Dynamic Grammar"

By shifting from continuous analysis to a symbolic analysis of state sequences, a striking structure emerged.

Key Findings (V24.1):

STATE_GRAMMAR_EXISTS = YES: Trajectories are not random. They follow strict transitional patterns.

UNIVERSAL_GRAMMAR = YES: Seven transition motifs are conserved across all tested architectures, notably:

B→B (Initial Hesitation/Exploration)

B→A (Convergence toward stable processing)

A→A (Maintenance of the adaptive regime – the primary attractor)

A→D (Transition to final decision)

Funnel Structure: Typical dynamics follow an Exploration (B) → Stabilization/Processing (A) → Decision (D) schema. State A acts as a strong attractor (

𝑃

(

𝐴

→

𝐴

)

≈

0.91

P(A→A)≈0.91).

The Phi-1.5 Exception: Unlike other models that quickly converge to A, Phi-1.5 maintains complex B↔A oscillations throughout the depth, confirming its nature as a "reasoning" model rather than a simple statistical completer.

📄 Related Preprint: A Runtime Trajectory Dynamics Framework for Large Language Models (updated)

💡 Implications & Discussion

These results suggest that Transformer "intelligence" is not just a matter of static weights, but of constrained geometric navigation.

Auditability: A violation of this universal grammar (e.g., a direct B→D jump without an A phase) could be an early indicator of hallucination or reasoning errors.

Control: Understanding these attractors opens the door to more precise dynamic steering than prompt engineering alone.

Open Questions for the Community:

Have you observed violations of this B→A→D grammar in cases of blatant hallucinations?

How do these motifs evolve in very large models (>70B) where depth is significantly greater?

Are there recent publications on the "symbolic dynamics" of hidden states that align with these findings?

I welcome any methodological criticism, suggestions for additional controls, or collaboration.

Best regards,

For context, I'm writing a sci fi novel in which one of the sources of conflict is a frequency (or combination of frequencies, technically) that affect the human brain in such a way as to provoke certain emotions at will, as a form of mind-control.

My question is this: Is there such thing as, or would there be reason to create, a neural-network-like computer that is physically structured similarly enough to a human brain to be affected in a similar way by external frequencies? For example, could neurons in the ANN be physically separated and communicate through electrical signals similarly to actual brain synapses? Could an external frequency then cause interference between the neurons that might have a similar effect? Since the computer obviously couldn't have induced emotions, I'm imagining an effect more like confusion, tasks being interrupted, the wrong data going to the wrong places, etc. The device in question would be a drone using the neural network for adaptive navigation, object avoidance, adapting to environmental changes like wind, etc. so the impact would be something like it steering off course and struggling to read and transmit data.

Sorry I know this is not a very scientific question but I'm trying to make my book grounded in reality wherever possible, even if the sci fi elements are of course taking some creative liberties.

I'm a computer engineering student, but know very little about ANNs aside from the general concept (I might take a course about them next year but so far I haven't had any). Pointing this out to say you don't necessarily need to explain in complete layman's terms, particularly general computer concepts, but I may not understand all ANN-specific terminology.

Posting the honest version of a small project, what it does, the real numbers, and what it definitely isn't.

What it is. A 5.98M-param sequence model trained only on SNLI, with no pretrained embeddings and no attention/transformer. It runs an interactive loop: you type a hypothesis, pick a label (entailment / neutral / contradiction), and it generates a premise under that label. Under the hood it's a learned "collapse" decoder, difference vectors pulled toward learned point-attractors, plus a light cross-sentence alignment step, instead of attention.

What talking to it looks like:

you > is the girl standing
ai  > a girl in a pink shirt standing in a doorway.   [neutral]

you > two men are playing football
ai  > two men in a soccer game are running after the ball.   [neutral]

The numbers (measured, not vibes):

• Generative-classifier accuracy: ~53% how often the premise it generates actually matches the requested label (3-way; chance is 33%). The sibling classifier version of the same engine hits 66.1% mean-pool / 72.7% with alignment on SNLI dev, no pretrained embeddings.
• Speed (interactive generate() path, M-series MacBook, 40 replies of ~9 tokens):

device	median latency / reply	throughput
MPS (GPU)	13.1 ms	591 tok/s
CPU	5.3 ms	1,630 tok/s

The bit I found genuinely interesting: CPU beats the GPU by ~2.5x. The decode is a handful of tiny sequential steps, so it's launch-bound, not compute-bound, the GPU's per-op kernel-launch/sync overhead costs more than its math saves. So this thing runs best with no accelerator at all: ~5 ms to a full reply, faster than the network round-trip you'd pay just to reach a hosted LLM API.

What it is NOT (so the comments don't have to tell me):

• Not a general chatbot, no understanding, no "awareness." Trained only on ~570k image-caption-style sentences, it can only produce SNLI-shaped sentences, ask it anything off-distribution and you get a caption about a person in a shirt. Fluent grammar emerges fast because grammar is local/regular; that is not reasoning.
• The accuracy ceiling is a mechanism limit (cross-sentence word interaction), not a training-time one, more epochs plateau. The honest fair-footing baseline (SNLI-only, no embeddings) is a lexical-feature classifier at 78.2%, and it's still under that.
• The speed is a consequence of being tiny. Scale params up and it becomes compute-bound and needs a GPU, you can't keep "5 ms on CPU" at billions of params.

Code + runnable chat demo + the benchmark script: https://github.com/chetanxpatil/livnium/tree/main/chat

Curious what people think about two things: (1) is there a real niche for sub-10ms, CPU-only, attention-free text models (on-device, embedded, high-throughput filtering), or is the narrow capability a dealbreaker? (2) cheapest way you'd add cross-sentence interaction to a pooling encoder without going full attention?

Hello, I am a beginner in AI and transformers etc, so i tried doing something with pen and paper. This is related to the paper "Attention Is All You Need."

I used the phrase: "river bank" to figure out whether "bank" means, the river edge or the financial institution. The model's only clue is the word sitting next to it.

https://preview.redd.it/13b99up3n59h1.png?width=2940&format=png&auto=webp&s=be526aff3fdfc551de48542d995d1be883bb0f8b

For my understanding, here's how attention rewrites a word using its neighbor, in four steps:

1. Project - every word becomes three vectors: a Query (what it's looking for), a Key (what it offers), and a Value (the content it passes on).
2. Score - take the target word's Query and compare it against every word's Key using a dot product. Higher score = more relevant. Here, the neighbor scores a 4, the word itself scores a 2.
3. SoftMax - convert raw scores into weights that sum to 100%. That gives us 88% and 12%.
4. Blend - mix the Value vectors by those weights. The final vector gets pulled mostly toward the neighbor.

I chose this phrase coz it got no grammar rules. No explicit lookup table. Just compare, score, blend - learned entirely from data.
The part that made it click for me: change the neighbor, and the exact same machine pulls the word in a completely different direction. Same word, different context, different meaning. Computed in parallel, for every word at once.

I'd read this explanation many times, but wanted to work it out of my own math.

Image: Notebook lm