u/Loose_Literature6090

Hi everyone,

I built H64LM, a research project to better understand modern LLMs by implementing one from scratch in PyTorch.

Instead of relying on high-level training frameworks, I implemented the core components myself attention, MoE routing, normalization, and the training loop.

Features

• 249M-parameter Transformer
• Grouped Query Attention (GQA)
• Sparse Mixture-of-Experts (8 experts, Top-2 routing) with 3 auxiliary routing losses
• SwiGLU, RoPE, RMSNorm
• Sliding-window attention
• Mixed-precision training, gradient accumulation
• Custom training loop (no Trainer abstractions)
• Checkpointing and resume support

The included checkpoint was trained on a subset of WikiText-103 to validate the pipeline end-to-end, not to be a strong model it's visibly overfit past epoch 10 (best val PPL ~40.5).

Known limitations are documented in the README, including batch-size-1-only generation and no true DDP (falls back to DataParallel).

GitHub: https://github.com/Haiderkhan64/H64LM

Feedback on the implementation or architecture is very welcome.

I wanted to see how much of the real-time infrastructure I could build myself instead of relying on managed services.

It currently supports:

• Posts with multi-image uploads
• Nested comments and replies
• Communities
• Community-based feeds
• Real-time messaging
• WebRTC voice/video calls

Stack: Next.js 14, TypeScript, MongoDB, Redis, Clerk, and UploadThing.

The part I enjoyed building the most was the real-time infrastructure. I wrote a custom WebSocket server that runs alongside Next.js instead of using a managed provider. Chat uses Redis Pub/Sub for message fan-out, optimistic UI updates with reconciliation, reconnect handling, and read receipts. WebRTC signaling also runs over the same WebSocket connection.

There are still a couple of limitations:

• WebRTC signaling isn't multi-instance safe yet.
• It requires a persistent Node server, so it isn't compatible with Vercel's serverless deployment model.

If you've built or scaled WebSocket-based systems before, I'd really appreciate your thoughts. I'm especially interested in how you'd approach horizontal scaling or whether you'd change anything about the current architecture.

Repo: https://github.com/Haiderkhan64/threadsApp