r/LLMeng

US4 V6 — Apple Edition. Universal State Runtime for local LLM inference on Apple Silicon (M1..M5+). C++17/20 + MLX + Metal + NEON + ANE

https://github.com/wesleysimplicio/us4-v6-simplicio-apple

Suppose a PM shipped a care coordination agent. Week one, patient says "I've been getting chest pain in the evenings." Agent logs the note and demo looks great. Week three, same patient comes back "should I be worried about that pain again?" Agent replies: "What pain?"

By default, agents forget everything the moment a turn ends. If you want continuity, you build it yourself:

• Context window: everything the model sees right now, fast, free to use, but has a token budget. As conversation gets longer the oldest turns fall off. When the session ends, everything disappears.
• Scratchpad: working memory that survives across loop steps within a single task. If Patient says "book my follow-up and refill my prescription." Agent writes a note, calls calendar tool, updates note as it completes it. Without this, the agent forgets what it already did and repeats what its supposed to do once. Simplest implementation is a JSON object the agent reads and writes every turn.
• Vector store: At the end of each conversation, the agent summarizes the important parts. In our example things like diagnosis, medications, follow-up dates, embeds it and stores it with a patient/user ID. Next session, before replying, it searches the archive. So when needed that note flows back into the context window. Now the agent has continuity across sessions.

Thus Memory is a product decision, not a model feature. Your job is designing what gets summarized, what gets stored, what gets retrieved.

You can checkout this video from SkillAgents YT for more details. Subscribe for similar content.