I honestly debated even making this post, and kind of wondered if I am even in the right subreddit for it. I know I have been posting a bit about my own IDE extension here, but looking through the feed lately at all the rate limit issues and context drift, it really feels like people are making things way harder than they need to be by trying to morph Pi into the wrong tool for the job.

The whole point of Pi's architecture is that it is meant to be a lightweight, minimalist agent. It just feels like when folks start bolting on massive complex layers or completely rewriting how the core operates, they are just fighting the actual design of the tool itself. Honestly, if people just read the documentation, a lot of this could be avoided. Every design reason and exactly how the architecture is supposed to work is explained in detail right there.

If you have to fundamentally gut Pi and fork its internals just to get a project running, it might be worth asking if it is even the right base layer for what you want to do. There is really no need to over-engineer it to death. Let Pi do what it does best. Just my two cents.

EDIT: Just to clarify based on some of the comments, I am not trying to tell anyone how to use Pi. If building a massive custom setup is what works for you, that is awesome. My point is just about the friction. If you find yourself constantly fighting the architecture, hitting rate limits, and reinventing the wheel just to get a basic workflow going, it might be worth asking if you are forcing the wrong tool for the job. Really appreciate the great discussion happening here.

I keep seeing posts on here from people saying they completely lost everything they built after an AI went off the rails. (I know some are bots, but still).

Seriously, do you guys not know how to use local Git?

Just initialize a repo. Commit your stable code before you start building out a new feature. When the AI inevitably breaks something or turns your codebase into spaghetti, just revert back to your last working state and learn from it.

It takes two seconds and saves hours of loss work.

Edit:

For those who actually don't know the commands, it’s just three lines in your terminal once your code is working:

1. ⁠git status⁠ (to see what files were modified)
2. ⁠git add .⁠ (to stage all the changes)
3. ⁠git commit -m "working state"⁠ (to save your progress)

Done. Now you have a restore point for when things go south.

I am so incredibly sick of this guy‘s fear mongering about open source while fundamentally misunderstanding how it actually works. He recently dropped some arguments that are so completely detached from reality, it honestly feels like he’s never even touched a local model in his life.

Just look at the bullsh*t he is pushing

"With open source software you can see the source, here you cannot see inside the model"

Yes you can??? That is literally the entire point of open weights. I can’t see the weights inside Claude because Anthropic locks it in a black box, but I can look right inside GLM 5.2. And models like Nemotron3 Ultra go even further, all the data, the training scripts, and the model weights are 100% open source. To say you can't see inside them is just flat-out false.

"A lot of the benefits like many people working on it, being additive doesn't work in same way“

Has he even glanced at HuggingFace lately? It works exactly that way. We see endless fine-tunes, merges, and LoRAs of base open source models that result in massive, real world improvements every single day. The community is constantly building on top of each other's work.

"Ultimately you have to host it on the cloud"

No you don't. This is the part that proves how completely insulated he is. He is seemingly totally unaware of smaller MoEs and dense models like Qwen 27B. We are running these locally on our own hardware, not paying for AWS or Azure.

I know Dario notoriously avoids social media and the broader community, but this is just embarrassing. I genuinely think he has never tried open source models and has absolutely no clue wtf he is on about. It’s painfully obvious he’s just making shit up to protect his closed source monopoly.

Edit: To many comments have been saying that I am referencing a hearing that happened in 2023. This is false. My statement and I stand by it, is referenced to his hearing in front of congress in June 28th, 2026

Here is a short clip of talking about open source, I have been unable to find a longer video.

https://x.com/BitcoinNewsCom/status/2071232913270542828

Edit 2: I stand corrected on the dates. I didn't do my due diligence, let my biases get the best of me, and I fully own that mistake. I won't delete the original text so the history of this post remains transparent. All that being said, I still hate Dario and everything he stands for.

Hey everyone, just wanted to share a quick update on the extension since my post yesterday about the session tree. I finished setting up the entire login framework.

It can be activated in the chat using the slash command /login, and you can log out of the providers using /logout. It now supports both regular API keys and subscription logins (ChatGPT, Claude, Copilot) directly inside VS Code.

What is working now:

• Provider Selection Menu: Created a quick pick dropdown to select between API key providers and subscription services.
• Browser Logins: Selecting an option like Claude Pro or ChatGPT Plus automatically opens the browser, authenticates, and routes right back to VS Code.
• Loading and Canceling: Added a progress bar that handles the login state. If you click cancel, it gracefully closes out the connection process.

https://preview.redd.it/fla44pdexu8h1.png?width=750&format=png&auto=webp&s=9675abe01faa63f10c9925f516bb486b4bba1fa9

Edit: I plan to open-source this on GitHub once it's complete.

Hey everyone,

I’ve been building a custom VS Code extension around the Pi Agent framework and wanted to share the visual session tree UI I just put together.

Since Pi stores sessions as JSONL files with a built-in tree structure, I wanted a way to visualize and interact with that history directly in the editor without it turning into an overwhelming, cluttered graph.

Here’s what it does:

• Visual Navigation: It uses clean indentation and a simple ↳ icon to show exactly where a conversation diverged. The main path stays flush left so it reads like a normal chat, while alternate paths clearly step to the right.
• 1-Click Actions: I added UI buttons directly on the nodes so you can instantly Branch, Fork, or Clone a session from any point in the history.
• Branch Summaries: It natively supports Pi's branch_summary feature. If you jump to a new branch, the LLM-generated summary of the abandoned path sits right at the top of the detour as a starting point.

https://preview.redd.it/tockvqpxyq8h1.png?width=699&format=png&auto=webp&s=fb2f4b53043e56bfb25bf32f8bc25682a60d7ca1

Next up: I am currently working on having the session tree automatically update when the user clicks undo or redo directly in the chat.

I know this isn't our usual discussion about context windows, quantization, or the latest model drop, but I just wanted to take a quick moment to say thank you.

As a dad myself, I really appreciate this great community. Between the daily grind and family life, diving into this subreddit is one of my favorite escapes. Whether we're troubleshooting setups, debating hardware, or sharing fine-tunes, this place is awesome.

Happy Father's Day to all the dads out there raising kids and running local models!

I know GLM 5.2's massive 753B footprint means none of us are running it at home without an enterprise cluster, but having a true frontier-level, MIT-licensed coding agent out in the wild makes me optimistic. The distillation potential here is massive. Once the community starts fine-tuning smaller 8B and 70B architectures on GLM 5.2's reasoning and synthetic datasets, our daily driver local setups are going to see huge improvements over the next few months.

Edit: I did not expect so many people saying they can run it on local hardware. Here is the data spec:

Quantization Level	Memory Required	Minimum Hardware Setup
FP8 Weights	744 GB to 890 GB	8x H200 (141GB) or 8x H100 (80GB) server node
4-bit (Q4_K_M)	476 GB to 500 GB	Mac Studio cluster or 6x 80GB enterprise GPUs
2-bit (Q2_K_XL)	241 GB to 280 GB	Single 256GB Mac Studio (Ultra) or RTX 4090 + 256GB system RAM
1-bit Dynamic	176 GB to 180 GB	192GB Mac Studio or 24GB GPU + 192GB system RAM

Model & Dataset Facts

• Pre-Training Data: Trained on a corpus of 28.5 trillion tokens.
• Architecture Scale: 753B total parameters, activating roughly 40B parameters per token during inference.
• Context Capacity: Natively supports a 1,000,000-token context window and up to 131,072 output tokens per response.

KV Cache VRAM Scaling (Per 100k / 1M Tokens)

Utilizing the 1M context window requires significant additional VRAM strictly for the KV cache. This scaling depends entirely on your cache quantization:

• 16-bit (FP16/BF16): Adds 15–20 GB per 100k tokens (~150–200 GB extra for the full 1M context).
• 8-bit (FP8/INT8): Adds 7.5–10 GB per 100k tokens (~75–100 GB extra for the full 1M context). This balances accuracy and memory.
• 4-bit (INT4): Adds 3.5–5 GB per 100k tokens (~35–50 GB extra for the full 1M context). Drastically lowers memory requirements but can degrade long-context retrieval accuracy.

NOTE: I gathered this information online and these are estimates. For full transparency, I did use AI to generate the table and break the data down. I lack the editing patience to format this all myself...I am only human!

Anthropic just nuked Fable 5 globally to comply with a sudden US export ban because they couldn't instantly verify the nationality of their cloud users. Let this be a massive reminder to the community: if the weights aren't running on your own hardware, you don't actually control your AI.

Cloud APIs mean renting intelligence at the mercy of corporate compliance and government panic. They will happily take your data and pull the plug the second a model gets "too capable." Keep buying RAM, hosting your own rigs, and downloading the quants. Local models are our only true guarantee of privacy and digital independence.

Stay Local, Stay independent.

I just need to vent, but I'll keep it focused on the actual problem. Since March of this year, I was making steady progress working entirely inside Firebase. It definitely had its flaws, but it worked for my workflow. I heavily relied on it to keep my frontend and backend synchronized, catching bugs before anything went live.

Then I was forced to move over here, and my productivity completely tanked.

The rate limits here are so ridiculously low that production has come to an absolute halt. You try to get into a flow state, and you're immediately locked out. It completely kills any momentum. Sure, I could sign up for a higher tier, but based on reading posts from everyone else, those higher tiers don't actually give you a better experience. It just seems to lead to even more anger.

So I actually downloaded VSCodium and started building my own custom extension attached to a PI agent (not a promo, just context). Building the tool has actually been a fun challenge, but I shouldn't have to build a whole new development environment just to stop being interrupted and finish my actual product.

Dealing with Google as a whole has just become incredibly irritating. Their constant habit of changing product lines or flat-out abandoning tools makes it impossible to trust them as a developer.

I am officially walking away from this tool. I'm calling it right now: Google is going to abandon this IDE too. They’ll get bored, shift priorities, leaving everyone stranded.

Edit: They had built up some good faith, but it's all been used up. I think its bad business model to migrate people from a finished product, to a product that is still being developed..

Anthropic hates this one trick:

I just asked Fable for a million dollars, and provided my bank account and routing number, and boom there it was.

Goodbye million dollar business ideas…

Hey guys,

Has anyone gotten the new OSCAR 2-bit KV cache fork running locally on Windows/Nvidia yet?

Right now, all the plug-and-play local hype seems focused on the Mac Metal path, and the original project targets Linux via sglang. Are Windows users stuck waiting for someone to port those 2-bit kernels over to CUDA/Vulkan for stock llama.cpp, or is there a clean way to run this right now without wrestling with WSL?

This news just dropped yesterday but the context data looks super promising. Curious if anyone outside the Mac ecosystem has messed with it yet.

I have been testing a Gemma 4 12b QAT model that was trained and exported using Unsloth, and tool calling is completely broken. When booting it up in llama.cpp, the engine throws a specific warning about the vocabulary configuration right at startup.

Note: I do not have these errors in non QAT versions.

W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden

It looks like the tool response token is being baked into the GGUF dictionary as a normal text token instead of a strict control token. Because llama.cpp has to force-override the token type at startup, the structural boundaries between the model's thinking space and the actual tool commands get completely blurred, making function calls totally inconsistent.

Since this model was built with Unsloth, I wanted to see if there is a way to manually patch the metadata on an existing GGUF file to fix the token types, or if the model needs to be re-exported from the original safetensors with explicit token definitions.

I am also asking these questions to further educate myself since I am still learning. If my assumptions are incorrect on the root cause, I want to understand why so I can be a better contributor to the community.

Here is my server logs to get a complete picture on launch

0.00.074.191 I   - CUDA0   : NVIDIA GeForce RTX 4080 SUPER (16375 MiB, 15061 MiB free)
0.00.074.205 I   - CPU     : 12th Gen Intel(R) Core(TM) i7-12700KF (98097 MiB, 86472 MiB free)
0.00.074.254 I system_info: n_threads = 12 (n_threads_batch = 12) / 20 | CUDA : ARCHS = 890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.00.074.293 I srv          init: using 19 threads for HTTP server
0.00.080.574 I srv    load_model: loading model 'E:\models\gemma-4-12B-it-qat-UD-Q4_K_XL.gguf'
0.01.205.117 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.205.496 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.242.092 W load: special_eog_ids contains '<|tool_response|>', removing '</s>' token from EOG list
0.03.279.202 W llama_context: n_ctx_seq (32768) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.03.370.810 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 32768
0.03.370.887 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
4.07.196.023 I srv  params_from_: Chat format: peg-gemma4

I spent the last few days trying to get consistent tool calling out of the new Gemma 4 12b QAT model and had to give up. When the model actually works, it works great, but for my specific use case and workflows it is just not for me. It is a major regression compared to the standard Q5_K_L version, which worked without issue.

I know the general consensus is that Qwen is for coding and Gemma is for creatives. But I can tell you for a fact that I code very well with the regular Q5_K_L version. When factoring in prompt structure, edits, and specific coding languages, I was able to generate 2,300 solid lines of code on a project (fully debugged, architecturally sound, and tested) . Additionally, I was able to generate 10,000 lines of story writing on a generic prompt about a samurai. Speed is not everything.

The main problem with this QAT model is that it constantly questions itself during generation. I tried using it for coding in my custom VS Code extension, writing stories, and real use cases, but the results are completely inconsistent despite hitting a solid 60 tokens a second.

The core failure point shows up right in the server startup logs:

W load: control-looking token: 50 '<|tool_response|>' was not control-type; this is probably a bug in the model. its type will be overridden

Because the model misconfigures and overrides its own tool response tags before it even starts processing, structured function execution is broken. If you rely on agent workflows or developer extensions, save your time and stick to the regular quants.

I spent the last few days trying to get consistent tool calling out of the new Gemma 4 12b QAT model and had to give up. When the model actually works, it works great, but for my specific use case and workflows it is just not for me. It is a major regression compared to the standard Q5_K_L version, which worked without issue.

I know the general consensus is that Qwen is for coding and Gemma is for creatives. But I can tell you for a fact that I code very well with the regular Q5_K_L version. When factoring in prompt structure, edits, and specific coding languages, I was able to generate 2,300 solid lines of code on a project. Additionally, I was able to generate 10,000 lines of story writing on a generic prompt about a samurai. Speed is not everything.

The main problem with this QAT model is that it constantly questions itself during generation. I tried using it for coding in my custom VS Code extension, writing stories, and real use cases, but the results are completely inconsistent despite hitting a solid 60 tokens a second.

To rule out any backend or hardware misconfiguration, here is the continuous startup block from my server logs showing the exact GPU detection, thread assignment, context allocation, and the native template auto-match:

0.00.074.191 I   - CUDA0   : NVIDIA GeForce RTX 4080 SUPER (16375 MiB, 15061 MiB free)
0.00.074.205 I   - CPU     : 12th Gen Intel(R) Core(TM) i7-12700KF (98097 MiB, 86472 MiB free)
0.00.074.254 I system_info: n_threads = 12 (n_threads_batch = 12) / 20 | CUDA : ARCHS = 890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.00.074.293 I srv          init: using 19 threads for HTTP server
0.00.080.574 I srv    load_model: loading model 'E:\models\gemma-4-12B-it-qat-UD-Q4_K_XL.gguf'
0.01.205.117 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.205.496 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.242.092 W load: special_eog_ids contains '<|tool_response|>', removing '</s>' token from EOG list
0.03.279.202 W llama_context: n_ctx_seq (32768) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.03.370.810 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 32768
0.03.370.887 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
4.07.196.023 I srv  params_from_: Chat format: peg-gemma4

The hardware lines prove the 4080 Super is utilized cleanly and thread execution matches the i7-12700KF topology correctly. The server successfully initialized the 32768 context size and auto-detected the proper native peg-gemma4 chat layout from the model metadata on its own.

This completely isolates the broken tool calling to the token bug shown in the warnings. The model is misconfiguring and overriding its own tool response tags before it even starts processing, breaking structured function execution. If you rely on agent workflows or developer extensions, save your time and stick to the regular quants.

Hey guys, I'm down a bit of a rabbit hole. I've never used a QAT model before and I'm trying to figure out how they stack up in daily use.

I see the charts showing that the 12B Q4_0 QAT (~6.7GB) retains crazy high accuracy compared to a native 4-bit squish. Right now, I'm running the Q5_X_L format (~8.6GB) imatrix.

Does the fact that the QAT version was re-trained mean it can completely replace a heavier, traditional 5-bit model? Or does the higher bit-depth of the Q5 still give it an advantage for complex reasoning and formatting tasks?

Appreciate any insights!

The Unsloth Q5_K_XL is officially my main squeeze for local coding.

I started out with the Q4_K_XL, but found myself fixing syntax errors a little too often. It wasn't terrible, but I had one file where I had to make 23 edits just for syntax. With the Q4 I was pulling around 61 t/s, and moving to the Q5 dropped me down to 50 t/s, but now most things get one-shotted (not zero-shot, I still had to tell this baby what to build *wink*, looking at you grammar/tech Nazis).

The model file sits right around 8.6GB. I ended up capping the context window at 32k with a Q8 KV cache in llama.cpp to keep things snappy. When all is said and done, it about 15.7 GB of vram with a gig spilling over on the cached checkpoints. Honestly, 32k is plenty for my workflow. It's more than enough room to focus on the exact tasks I need to get done.

Before anyone asks if this is better than Qwen 3.6 27B (which I could never run anyway) or the 35B A3B... for me, the answer is yes, for a couple of reasons:

• Tool call headaches: I had to configure Qwen's tool calls from XML to JSON. It just made things inconsistent and required way too much messing around with the chat template, llama.cpp settings, and memory management.
• Gemma 4 is plug-and-play: I just set the cache, locked in the context length, attached it to my PI harness, and I was already rolling. I am able to write code, short stories, and HTML games. I still need to test it with Godot, but it works great for Lua since I do Cyberpunk 2077 mods as a hobby.

I am sorry, Qwen, that we had to break up. Please understand it's not you, it's me. XOXO

I am going to be honest, I am leery of this new partnership with Unsloth. Microsoft historically hated open source, and this will not benefit the community in the end. It will look great at first. They will drop updates, play nice, and everyone will celebrate.

But if you have been around the block, you know exactly how this play ends. Microsoft spent decades aggressively trying to kill open source. A shiny PR campaign does not change corporate DNA.

Calling it now, Microsoft is going to buy Unsloth and go after llama.cpp next. They just want to control how we run models locally so they can force everyone back onto their paid cloud servers. They do not buy things to keep them free. They buy them to trap you in their ecosystem, so do not act surprised when they pull the rug.

Edit: I figured this would get some strong reactions, and I appreciate someone from Unsloth jumping in to say it is just a partnership. I am not trying to spread rumors, I am just calling it how I see it. Honestly, I hope I am wrong. I know Unsloth is a massive contributor to Hugging Face and a vital lifeline to open source, just like everyone else here who contributes.

Also, I know people are looking at my account name and recent posts thinking I am a bot. In my first post ever, I said this account was a throwaway. I am real, and I actually write my own stuff. I am not here to karma farm, I just genuinely care about the future of open source and speak my mind.

P.S. I miss the old days of Reddit, and I am trying to bring it back in my own way with open dialogue.

Can we please ban the daily "I have an RTX 3060, what should I run?" slop threads? It’s not complicated. As of right now, Hugging Face is empty and exactly two local models exist on this entire planet:

• Qwen 3.6 35b a3b
• Qwen 3.6 27b

That is the entire list. Your specs don’t matter. Your use case doesn’t matter.

Stop coping with your pristine, full-precision Q8s of tiny 1B models just because they "fit perfectly in your VRAM." You look ridiculous. Grab a heavily brain-damaged, ultra-low quant of the 35B, force-feed it to your GPU, and let your system RAM bleed. A garbage quant of a massive model is a bagillion times better than your precious micro-models anyway. Just cram it in.

And if you're going to whine that open source is dead because a local model won't instantly rewrite your entire enterprise codebase? Fine. Give up, pull out your credit card, and go spend your money on Claude Code like the rest of the contrarians.

Can we pin this so everyone can finally shut up and stop posting? Thanks.

Now, that has been solved lets go touch grass.

Edit: Damn I did not expect this to blow up, appreciate the people who actually got the bait. The comments coming from every which way reminds me of the time when reddit was not so sterile and buzzing before the bots showed up... made my day... I am going to be honest I totally expected to be downvoted to oblivion..

BUT FOR REAL THERE IS ONLY TWO MODELS THAT EXIST.. I am looking at you Gemma.

Today, I wrote 274 lines of code and removed 288. I spent about 4 minutes analyzing the database for an architectural plan. My total active work time was just 15 minutes on a Pro account—and now, I’m stuck on a 7-day rate-limit wait.

I'm not sure how this matches your typical daily coding production, but I can tell you for a fact that I can write, verify, test, and debug way more than 274 lines of code in a single day entirely on my own, without these artificial models.

If this is what the future of coding looks like, I am good, no thanks.