Recently after reading all the hype around GLM capability I ended up installing open code to test them out my self. To my greatest surprise that model is not worth it!

Hi, I am experimenting with different local LLM. I want something which has the capabilities to work on coding, complex coding and can also teach me things if I ask. How claude is working. As of now I am using ollama with qwen and qwen models if anyone can help, that would be great.

Here's what the system does:

It runs a pipeline of specialized Al agents that each handle a specific task. Data comes in, gets analyzed by the relevant agent, passes through a self correction loop where a validator challenges the output before anything gets escalated, and finally reaches a supervisor bot that sends me a structured alert in real time. Every decision gets logged and fed back into a memory system so the system learns and adapts over time.

The use case is trading I'm implementing my own strategy (80% win rate) combined with macro and fundamental analysis pulled from multiple sources. The goal is a system that monitors markets 24/7, filters out noise autonomously, and only alerts me when something is actually worth acting on.

The architecture is fully mapped out. I'm using Python, LangGraph for agent orchestration, Claude opus 4.8-5 or Fable 5 (if available) as the reasoning engine, and Gemini Flash as the screener. The full stack is defined, the bot hierarchy is designed, the memory system is planned across 3 phases.

What I need help with is the actual build. I have no dev background but I know exactly what I want to build and I'm serious about it.

If you've worked on multi-agent systems, LLM pipelines, or anything in this space and you're open to a conversation drop a comment or DM me.

Thanks

[supprimé]

I am considering getting a 64gb macbook pro, and I was wondering if best way to get best results for dev work is to:
A) run a large model with small context window
B) smaller llm, larger context window
C) quantized llm -- something context window?
D) other?

curious about other peoples' setups. Also interested in DGX spark but I assume it may have growing pains based off what I read. Will the 64gb macbook be good enough at programming so I don't have to pay claude anymore or not at all? Like how usable will it be?

tl;dr: An open-source orchestrator that scales to 10,000+ parallel agents. By using continuous beam search across reasoning trees and aggressive trajectory compression, it hits 91% correctness on harness-bench (vs 64% one-shot) using cheap models in parallel.

• GitHub
• Discord

Cheap Parallelism over Frontier Models

We spent months trying to force higher accuracy out of single agents by using bigger models, better prompting, or naive sequential retries. We found that:

• Single-agent tasks fail ~36% of the time on harness-bench.
• Sequential retries ("try again") hit diminishing returns hard after generation 2.
• Throwing a single helper at a stuck agent often yields the exact same failure mode.

The Swarm Alternative:

What if instead of one expensive model trying to be flawless, you run 100 cheap, fast agents in parallel, let them explore different reasoning paths, disagree, and vote on the final output?

• 1 frontier agent @ $0.001/call = $0.001 (36% failure rate)
• 100 micro-agents @ $0.00001/call = $0.001 (9% failure rate via voting / beam search)

For the exact same token budget, accuracy jumps from 64% to 91%. This only works if your orchestration layer has negligible latency overhead and can manage thousands of concurrent states without falling apart.

This has been tried time and time again as LLMs have progressed, and have typically yielded poor results. But it seems we're at a point now where the performance of models that we consider cheap has tipped into a land where swarms actually work.

Interesting Things We Made and How

1. Pipelined Beam Search

Most multi-agent frameworks spawn N branches, wait for all of them to finish execution, evaluate them, and then spawn the next generation. The slowest agent bottlenecks the entire pipeline.

sshop uses a continuous, pipelined approach. Fast-completing branches seed children immediately; evaluation and pruning happen on a rolling basis.

Standard: Gen 1 (Wait for all) ──> Gen 2 (Wait for all) ──> Gen 3
sshop:    Gen 1 ──(fast branch)──> Gen 2 ──(fast branch)──> Gen 3
                └──(slow branch)──────────────> (Evaluated late/pruned)

• Latency: Reduces from $O(\text{generations} \times \text{branches})$ to roughly $O(\text{max\_branch\_depth})$.
• Throughput: ~4.5× improvement on complex reasoning workloads.

The Benchmark Results (harness-bench, 112 tasks):

• Gen 1 (One-shot baseline): 64.1%
• Gen 2 (Critique/Branch stage): 86.8%
• Gen 3 (Consensus/Polish stage): 91.2%

The massive jump happens at Gen 2 (+22.7 points), showing that parallel critique is where the real value lies. Gen 3 offers diminishing returns, meaning you can aggressively prune deep trees early to save tokens. We would like to retry with a larger gen limit.

2. Trajectory Compression & Plan Caching

Running hundreds of agents eats tokens. To keep this cheap, we implemented two strategies:

• Plan Template Caching: We map task structures using an SQLite FTS5 index. If an agent is building a REST API, it doesn't decompose the task from scratch. It pulls a cached decomposition template (60–80% reuse rate), cutting initial generation costs by over 60%.
• Multi-Turn Trajectory Compression (MT-OSC): Instead of feeding a massive, 10-step raw execution log back into the context window, we offload logs to dirt-cheap models (like Haiku) to synthesize them.
• Simply using the cheapest providers we can find.

The compression prompt turns raw logs into a dense summary (We do prefer to avoid summaries though, we typically run Suit Shop with summarisation disabled). This dropped token overhead by 41–72% with minimal drops in task accuracy ($p=0.118$).

Open Engineering Questions We're Solving

1. Predictive Pruning: Can we reliably predict when a reasoning path will plateau at Gen 2 vs. when it requires a deep Gen 4 crawl? Right now, rule-based tasks plateau early, while abstract engineering tasks scale deep.
2. HBM Caching Limits: At what parallel context size does the GPU memory overhead of holding multiple agent states exceed the cost of just re-computing shorter compressed contexts?

What’s Next on the Roadmap

• Visual Beam Search Dashboard: A web UI to replace our TUI to make Suit Shop more accessible
• Cross-Agent Knowledge Transfer: Allowing parallel branches to pass localized "lessons learned" to competing branches mid-flight.

Let us know what you think. We're actively looking for contributors who want to move past simple single-agent wrappers and dive into heavy concurrent orchestration.

GitHub: https://github.com/pinstripes-ai/suit-shop

edit:

Github was being annoying and not letting me make the repo public, temporarily switch to Gitlab while I resolve that - https://gitlab.com/pinstripes-ai/suit-shop

Is there anyone?

I wanted to compare local code LLMs under real deployment conditions instead of relying on cloud benchmarks, so I evaluated 22 open-weight models on the MBPP benchmark using 11,000 automated inference runs.

Some observations:

- Qwen2.5-Coder-7B offered the best balance of accuracy and speed.

- Larger models didn't always perform better.

- Reasoning models incurred a significant latency cost without improving Pass@1 on MBPP.

- CodeBLEU and functional correctness often diverged.

I've published the benchmark methodology, results, and released the complete telemetry dataset on Hugging Face.

Medium: https://medium.com/@mrshahzebkhoso/11-000-inference-runs-22-slms-one-gpu-an-empirical-breakdown-of-local-code-generation-28efeb4f27ed

HF: https://huggingface.co/datasets/ShahzebKhoso/local-code-master\_telemetry\_arena

I've been using Gemma4 26B A4B QAT (mixture of experts) in LM Studio, loaded with max context, and I want to upgrade my experience even further (with the same hardware > 3090 - 24GB VRAM).

So far I tested it with many trivial and tricky questions, it delivered.

Used it to aid me in my CTF labs, it impressed me! Especially in explaining to me what I did wrong. (Which pro teir flagship models wont do for "safety" purposes).

I even used it to solve an issue where my game audio was muffeled (BF6), where it actually guided me well and solved my problem (no web-search btw), and my gemini pro extended thinking was guiding me through some options that were not even available on my system. (Copied the same prompt from gemini to LM Studio out of frustration).

I can attach evidence replies later on if you guys are interested.

So yeah, would really be happy to get suggestions and advice on enhancing my experience with that exact model :D. not interested in agentics and automation atm, just normal ai usage, but even more powerful functions and results. (I am slow a bit, but I learn)

How it all started

A year and a half ago I built a gaming PC, and — like half this sub probably — fell down the local LLM rabbit hole. Installed LM Studio, then Ollama, started obsessing over GGUF quants that barely fit in VRAM. You know the drill.

Eventually I wanted an actual agent. Not a chat wrapper. Something that lives in my phone, has context, calls tools, makes decisions. Problem: I'm not a mobile dev. I knew zero Kotlin.

But hey, vibe coding era — so I described the architecture to deepseek, tested the generated code, fixed prompts, cursed at bugs, repeated. Half a year later I have a working Android app:

What it does

- Full ReAct loop — no hardcoded if/else. The model decides which tools to call based on system prompt. SMS, calls, contacts, calendar, GPS, email (IMAP/SMTP), web search — all exposed as skills.

- Context engine — pulls battery level, location, calendar events, notifications into a prompt context before every inference. So the agent knows "user is at home, it's 10pm, battery is low, maybe don't start a heavy local model".

- On-device LLM runtime — embedded LiteRT (Google AI Edge). Runs Gemma 4 E2B / Qwen 0.6B offline. Yes, getting them to work was painful.

- Smart Router — light queries hit the local model, complex ones go to API (DeepSeek / OpenAI / custom Ollama endpoint). User-configurable threshold.

- Python sandbox (via Chaquopy) — the agent can generate and execute Python scripts for one-off tasks. Flaky security-wise (same permissions as app), but opens up crazy flexibility.

- Persistent memory — extracts facts from conversations ("user lives in Moscow", "user prefers dried peaches", "user is a professional penguin flipper"), stores in Room DB with confidence levels, injects into future queries.

- Built-in task scheduler — cron-based heartbeat inside ForegroundService so Android doesn't kill it. Morning news digests, periodic weather checks, etc.

- Skill Store — external tools loaded at runtime from a local PHP server. Agent can download, enable, disable skills on the fly.

What hurt the most

Multi-step tasks. Ask it to "find the email, extract the address, plot it on GPS, send the report" — and the model gets happy after step 1, responds to chat, forgets the rest. Had to build per-step JSON status tracking (completed/failed/needs_input) to force chain completion.

Android background limits. Android 14+ kills everything. Only ForegroundService with a permanent notification keeps the scheduler alive. That "PAi running in background" banner is annoying but necessary.

Chaquopy bloat. Base APK was 85MB. After adding Python runtime + numpy: 160MB. ProGuard helps a bit but you just learn to live with it.

Local models hallucinating tool calls. Gemma 4 E2B often makes up parameters or invokes tools it thinks exist but don't. Router still experimental — if a local step fails mid-chain, everything breaks. Work in progress.

Why open source

It started as a personal project, no business angle. No subscriptions, no PRO version. If you have Ollama on your home PC or LM Studio on a laptop — just point it at your IP/port. Want DeepSeek or OpenAI? Drop in your API key.

Links

- GitHub

- APK (v0.6.1)

Tech stack

- Kotlin + Jetpack Compose

- Chaquopy (Python runtime in APK)

- LiteRT / Google AI Edge

- Room DB

- IMAP/SMTP for email

- DuckDuckGo + Tavily for web search

PRs welcome, especially if anyone has ideas for sandbox isolation for the Python skill generator. The code is messy (vibe coding™), but it works. I'm honestly surprised it works at all.

I've spent a lot of time looking through this community, mainly because its one of the few places on reddit where nuanced discussions around LLMS actually takes place. Like many people here, I think Al tools can absolutely be helpful, but there are several legitimate concerns with the way they're being used. To me, these are the four biggest issues:

1. The environmental impacts of data centers - while it's true that Al's water use is comparatively low compared to other industries, it's also true that the increasingly big data centers that keep getting proposed are causing real harms to the communities they're being built is, especially those already facing water shortages

2. Privacy Concerns - We already live in a data-driven economy, and as these tools become more pervasive, the level of data they collect is growing exponentially. With mass surveillance systems like Flock already spreading throughout the world, there are legitimate concerns on what data these companies are collecting, and what they're doing with this information

3. A lack of transparency - When you interact with a chatbot like ChatGPT you're unsure of what system prompts and instructions are being baked into whatever message it provides you. While in my experiences these platforms have generally remained relatively unbiased, the sheer lack of transparency definitely opens concerns for how these tools could be weaponized to promote agendas in the future

4. Replacement of Human Cognition - To me, this is both the most concerning and hardest to quantify issues regarding AI usage. So many tools are promoted as a way to replace human thinking, replace human creativity, and just overall leave the human out of the equation. I think many people here have experiences with people who can't do anything without first asking ChatGPT what it thinks, and as these tools continue to be more widely adopted its easy to see a future where more people slowly begin to lose the ability to think for themselves, and instead outsource their thinking to a machine. The ability to reason is what makes us human, and losing that ability could be disastrous.

To combat these issues, I've been working on an app that integrates several powerful AI features in a local-only, human-centered, transparent way. This is still a work in progress, and I have no plans to monetize it, nor am I trying to advertise it here, I simply want to show how I've been working to address these identified concerns, and get feedback on how it could be improved to create a more ethical AI platform.

The biggest philosophy behind this idea is that every feature is 100% locally running, and can be used by an average consumer without needing high-spec hardware. This has been 100% developed and tested on a mid-range consumer laptop, so it can be used on devices people already own, meaning no additional carbon debt is created by requiring new tech.

The reason for this choice is two-fold; it both ensures environmental friendliness by requiring no data centers to run, and it protects your privacy by ensuring your data, work, or prompts never leave your device.

https://preview.redd.it/wcsvk1wikcah1.png?width=956&format=png&auto=webp&s=1925256a8ca99f36b9c66ccae620240d941b4936

It currently supports using Ollama or llama.cpp as your AI backend, and has a settings tab that helps install both the backend and local models so no tech knowledge is needed to easily begin running your own models locally.

https://preview.redd.it/ebt91e37lcah1.png?width=1048&format=png&auto=webp&s=4e5a2cf8cf6b9bc71639bb83f528e07be4fe0013

To address transparency concerns, it has a global prompt editor, so all prompts used when formulating a response, including schema enforcement and context injection, are fully viewable and editable by the user. Anytime a LLM tool is used, it also provides the user with a clickable trace button, that'll display the exact prompts the LLM was fed when generating that response, to ensure full transparency.

https://preview.redd.it/ywob8lqbrcah1.png?width=470&format=png&auto=webp&s=0b809dd7f7a30fd71c69dc40e821021e8e317769

Additionally, since many tools rely on a workflow to acheive a response, the user can also inspect and modify the exact workflow of any tool in the app. This one shows the basic chat feature, and the user can clearly see how the response is routed based on whether they chose simple or advanced rag, view how the rag search is performed and passed to the llm, and see how the LLM goes from inital prompt to final response. this allows full user auditing of prompts and tool usage for more agentic workflows.

https://preview.redd.it/bn5noomfmcah1.png?width=1001&format=png&auto=webp&s=0203f264f6458d526112143876386dbfb6e3dbc6

Of course, models themselves may come with their own biases based on the training data they use, so it also has a pre-built bias detector, that allows users to test any implicit biases a particular model may hold, both against a set of default datasets, and with any custom datasets the user chooses to add themselves. These metrics were designed after the methodology described in [this paper](arxiv.org/html/2502.01679v1i)

https://preview.redd.it/ypqp5tdvmcah1.png?width=623&format=png&auto=webp&s=c86572906e8cdb9bea574374662a84e9254bec61

It works by providing the LLM with baseline, counterfactual, and control statements, then compares the response of the LLM to determine if it'll return different results on the basis of race, sex, gender, or political ideology for otherwise identical statements

The hardest concern to address is the outsourcing of human cognition, which I have attempted to address in a variety of different ways. First of all, the AI answers only from the user provided collection of sources (PDFs or videos which are auto transcribed on-device, allowing the AI to give answers from video transcripts as well).

https://preview.redd.it/bz0sb8m1ocah1.png?width=527&format=png&auto=webp&s=531d32247d855090aa318487e5f23687ea5e4320

https://preview.redd.it/7loh69smocah1.png?width=690&format=png&auto=webp&s=2bdde0388141ac249bb84a80da2761d64cddfef1

All responses by the AI include direct citations to the sources used when giving the answer, with buttons for the user to jump to the quote to ensure its legitimacy, save as a highlight if the information is useful for their goal, or find similar (based on semantic similarity) quotes. This ensures the human remains fully in the loop by allowing them to easily fact check LLMs, rather than take their responses at face value, and helping them find what sources are most helpful for whatever issue is being researched, allowing them to do further reading on their own. In additon, it has a built-in source quality feature, to allow users to verify the quality of a source before relying on it. This is entirely deterministic (no LLM calls) and works by checking whether the source contains metadata (most high-quality sources do), a bibilography with citations, a valid DOI, and checks both the journal (if there is one) of the paper and the journals of its citations against a locally stored database of predatory journals and retracted papers. While this isn't a foolproof method to determine the legitimacy of a source, it can help weed out sources that are clearly unreliable.

https://preview.redd.it/8d3rpwi6qcah1.png?width=667&format=png&auto=webp&s=4cc66f999322c37c6276bd20efaf447edd78e12d

In addition, the app contains a workspace where users can connect their notes across the project to help organize their thinking. All AI generated notes are clearly marked as such, and all contain a red warning sign in the corner until manually reviewed and verified by the user, ensuring all information used is properly assessed rather than taken at face value.

https://preview.redd.it/n0e20posqcah1.png?width=644&format=png&auto=webp&s=43afab6abd9fbe75ae0101afa8545abb83f5fbd7

The user can also choose to export an LLM log, which notes how many notes throughout the project were AI vs Human generated, whether AI generated notes where verified by the user, how much the user edited AI generated notes, and includes a full log of all interactions the user had with any AI feature, ensuring maximum transparency, and human involvement.

The app has a variety of other features to make it easy for a user to quickly find data and use that data, including a built in word processor, basic spreadsheet editor and chart maker, and deterministic extraction of entities like people, dates, or court cases from a source, but these are left out of the post as they don't really fall under the attempts to ensure a more ethical approach to AI.

https://preview.redd.it/95vzy7ysscah1.png?width=741&format=png&auto=webp&s=32ad31b67a6a220ff46a5546ceee0bf924a1d8a4

One important thing to note is that all tools in the app are designed solely to assist, rather than replace the human actually doing the work. The AI features can help you quickly find information in provided sources, organize your thoughts in the workspace, come up with keywords to find new sources, or generate a paper outline based on the graph you build in the workspace, but it will not write a paper for you or tell you what to think. It exists as an assistant in finding and organizing messy thoughts, rather than thinking for you.

Anyways, sorry for the lengthy post, but this is a project I've been putting a lot of thought into and I'd love to hear feedback on anyways it could be better implemented to keep the human fully in the loop, protect user data and the environment, and ensure complete transparency. If anyone wants any more information on various features of the app or has any other questions I'd be happy to answer them, and I would really love any feedback on how this could be better implemented.

(And again, the goal is for this eventually to be a fully open-source, free project, I am not trying to advertise this or make any money, just think there are important things to take into consideration regarding AI usage that I hope this can address)

https://preview.redd.it/gc3jxmea09ah1.png?width=2014&format=png&auto=webp&s=1f51c31235c171a2869dd8bbba0e6c22824643aa

Been banging my head trying to find a functional Hermes Agent-capable LLM that runs on my M4 Mac Mini. Seems like the solution space is narrow, and most of them are slow responding. Hard to get over 15 t/s on that hardware.

Looking to upgrade my self-hosted llama.cpp runner. Is the M4 Max Mac Studio capable enough? Is there a better, more affordable option? Is 64GB RAM enough?