u/dancingwithlies

I am a heavy user of opus models, but recent benchmarks on GLM 5.2 made me think about its price and capabilities, so i want to switch to some provider that has GLM 5.2..

I dont want API, bcs from my experience i spend 20x more on API for the same usage i get on 20x max plan..

So my question is;

Should i buy Devin $200 plan or Z.ai Max plan ?

Which will give me more usage in general for GLM 5.2 ?

If GLM is so much cheaper than Opus, i expect at least double total usage from what i have using Opus 4.8 max all day..

I spend months making a harness based on Claude Code. It currently still in development.
Why i decided to make a Harness ?

Well, Harness are more important than the model i would say.. the model has limits like its context window for example, if you need your system to work on something bigger than its context, there is where everything fall apart.. models are probabilistic too, so giving the model access to your entire computer and data is bad.. And no i dont believe LOOP is the answer like everyone is saying, its correct that managing multiple models is more effective, but the "managing" part is the problem, a bad loop can compound SLOP.

When i started using AI almost a year ago, i noticed that by far CC was the best harness, the only that made the model really useful.

Then i started using it and noticed limits like everyone else, MCPs, hooks and skill files are nice, they allow for a lot of customization, but the thing is, when you start working with complex codebases or projects (specially from 0) everyone knows that this session based workflow is hard to have the model keep track of everything

And this is only 1 issue, my harness aims to solve many different issues, i am building it around one idea: you cant trust a model that grades its own work. a model that writes the code, writes its own tests, and says "done" can pass its own checks from inside the same box the check was meant to police. so the whole point of mine is making failure visible, where it failed, when a test never ran, when it made something up, some examples of what i aim to fix;

- the model tells you "done" and its not done. every claim of work now leaves a receipt it cant fake, the proof is the real test run not a checkmark it typed itself
- tests that "passed" but never actually executed or have a different scope. the quietest lie there is, here it cant slip through
- the model hallucinates and nobody catches it. failure is surfaced, and on checkable work it repairs the bad answer instead of hiding it, ~27 wrong per 100 down to ~9 at the same coverage in my own tests
- code that compiles but nothing ever calls. the dead function a test pokes and the real app never touches, caught at write time, not three weeks later
- a one-line fix that silently breaks something across the repo. flagged before the edit lands, because it knows what a change can reach
- the model is a black box you cant watch. live view of what its doing and why, focusing on improving visualization, using nodes, every action on the record, replay any of it
- the project dies when the session ends. state lives in an append-only log, not the chat, so it survives the session, a crash, a week off, no re-explaining your codebase every morning
- .md files rot into spaghetti. they were never the storage, theyre a view rendered off the log, an ugly one gets regenerated and the truth underneath never moves
- a small or local model is unreliable on its own. a small 8b wired right beat a model 15x bigger by 50+ points on a hard retrieval task in my tests, why ? because the same problem that affect bigger models on big tasks, affects small models too, they are bad at reasoning ? yes but that can be improved too, everything can be improved if we focus on "context management", a small context window is just a bigger challenge and smaller tasks.
- paying top-model prices for every task. it measures the task first and routes it, local model for the bulk, cheap api for the middle, the expensive one only when it earns it, with a hard cost ceiling that blocks before the bill

(This is not even close of the full scope of what my project targets..)

So yeah, i am deep on this. A cost efficient, grounded, reliable and deterministic harness on top of a probabilistic model fleet is what i aim for..

But i am facing a challenge now, i have 3 credit cards late, i am working for only 600 usd a month, i am working solo on this for months, spend more than 2k usd in total on this project alone, and then as i get close to the marketing time, i need to make a choice i cant go back on;

Should i ship this as closed source, open source or open source with commercial license ?

i NEED revenue from this, can i depend on donations ? can making it open source help me find a better job ?

But again this is just a Harness, even if i make it closed source will be just so i can have control of the engineering and improvements, Claude Code is closed source but the CLI itself is free, they just limit the plan usage to it

So guys, i am here basically asking, if i can save your time, your money etc with my product, and i am in a bad situation now, should i charge for it or rely on donations/maybe job offers by the code/results ?

I am still rolling the benchmarks, so i cant show anything now, like the name or data numbers, but i assure you, i am solving all problems i see people (and me personally, i know how to code but no human alone could make 750k LOC in less than 8 months xd) having with current software like CC, so depending on the feedback, i will choose if i charge for it or make it open source, praying i get something out of it.

I am leaning into open source because if my entire harness is based on honesty and reliability, how can i say that with a closed source software ? you guys SHOULD be able to see the inside and decide if you want to use it, its just that my situation is bad, i lost a friend recently because i am not making money out of this yet, so yeah, what are your thoughts ?

Hey so i am building my own harness with my own hook points, context management, focus on determinism etc and i want you to share if you are doing something similar, and if yes, why ? why the current open source/closed source available harnesses are not enough for you ?

i reverse engineered the three "AI wait-state" ad tools (kickbacks, adspin, idledev) and one of them silently installs unsigned code

so i installed all three of these things, the ones that stick ads in the claude code spinner and supposedly pay you a cut, and then i pulled them apart. read the whole source where it was small and every security-relevant path in the big kickbacks bundle.

first the good news, and it goes for all three: none of them steal your code, your prompts, your env vars, your api keys or any credential. no exec, no eval, no shell stuff, nothing reading your .ssh or .aws or .env. the whole "it quietly harvests your machine" thing just isnt there.

the actual risk is way narrower and its almost all in kickbacks.

quick ranking, least invasive to most:

- idledev, clean, barely touches anything, the only one id leave installed
- adspin, clean, well built, one small privacy thing
- kickbacks, the worst by a mile, two findings and one of them is bad

the bad one, kickbacks silently updates itself with the signature check turned OFF

kickbacks runs its own auto updater. it polls a manifest endpoint on their server, downloads a .vsix (thats a full vscode extension, ie arbitrary code) and installs it itself. the only thing you ever see is a little "reload window?" toast, and by the time that pops up the new code is already written to disk and installed.

heres the part that got me. it actually HAS signature verification code in there, but its switched off in the build i installed. the function that returns the public key just returns nothing, theres a dead if-statement guarding it, so theres no key baked in. and because theres no key, the "require a signature" flag is false, so the entire verify step gets skipped.

so the only things actually standing between you and an install are: the download url has to be on their google cloud bucket, and the file hash has to match the hash in the manifest. but both the url AND the hash come from the same server. so that hash check only catches a corrupted download, it does nothing against a malicious one. whoever controls the kickbacks backend can push any extension they want and it auto installs and runs as you, no approval, no signing. thats remote code execution by design, the only thing protecting you is hoping their servers never get popped. the crypto to lock it down is literally sitting in the code, they just shipped with it open.

if you really want to keep running it, set KICKBACKS_REQUIRE_MANIFEST_SIG=1 in your environment. that forces the signature path, and since theres no key it then refuses every update instead of installing it blind. thats the safe way to fail.

second kickbacks thing, it rewrites anthropics actual extension

the other two only touch the supported settings file. kickbacks goes further and patches claude codes own bundle on disk, it edits the webview index.js to inject the ad and it loosens the webview content security policy so its ads can phone home. it does the same thing to the openai codex extension too.

to be fair, i checked and it does this carefully: the CSP change is connect-src only so it doesnt open an actual script injection hole, it backs up the original first and the restore works, and the little local server it runs only binds to localhost behind a random token. but still, rewriting a signed third party extension breaks its integrity, its gonna fight every claude code update by re-patching, and its a sketchy amount of access just to show an ad.

adspin, clean, one privacy note

tokens stored properly in vscode secret storage not some flat file, settings backed up and restorable, ad text sanitized. it only touches the settings file, never anthropics code, no self update. the one note: it peeks at your claude projects folder but only reads file modified-times, not the contents, to figure out if youre actively using claude so it only bills when you are. fine, but it is looking in there.

idledev, cleanest, least access

the shipped file is byte for byte identical to the published source, i diffed them. it only writes its own config and the settings file, sanitizes the ad text, validates urls, and sends nothing but your token and the local hour. no self update, no patching anything, never reads your transcripts. if you keep one of these, keep this one.

tldr

- nobody is stealing your keys or code
- kickbacks can silently auto install unsigned extension code from its server, thats real RCE by design, set KICKBACKS_REQUIRE_MANIFEST_SIG=1 or just dont run it
- kickbacks also rewrites anthropics signed extension on disk
- adspin is clean, just peeks at your project folder timestamps
- idledev is the least invasive

i can drop the exact file and line numbers from the beautified bundles if anyone wants to verify any of thisi reverse engineered the three "AI wait-state" ad tools (kickbacks, adspin, idledev) and one of them silently installs unsigned code

so i installed all three of these things, the ones that stick ads in the claude code spinner and supposedly pay you a cut, and then i pulled them apart. read the whole source where it was small and every security-relevant path in the big kickbacks bundle.

first the good news, and it goes for all three: none of them steal your code, your prompts, your env vars, your api keys or any credential. no exec, no eval, no shell stuff, nothing reading your .ssh or .aws or .env. the whole "it quietly harvests your machine" thing just isnt there.

the actual risk is way narrower and its almost all in kickbacks.

quick ranking, least invasive to most:

- idledev, clean, barely touches anything, the only one id leave installed
- adspin, clean, well built, one small privacy thing
- kickbacks, the worst by a mile, two findings and one of them is bad

the bad one, kickbacks silently updates itself with the signature check turned OFF

kickbacks runs its own auto updater. it polls a manifest endpoint on their server, downloads a .vsix (thats a full vscode extension, ie arbitrary code) and installs it itself. the only thing you ever see is a little "reload window?" toast, and by the time that pops up the new code is already written to disk and installed.

heres the part that got me. it actually HAS signature verification code in there, but its switched off in the build i installed. the function that returns the public key just returns nothing, theres a dead if-statement guarding it, so theres no key baked in. and because theres no key, the "require a signature" flag is false, so the entire verify step gets skipped.

so theonly things actually standing between you and an install are: the download url has to be on their google cloud bucket, and the file hash has to match the hash in the manifest. but both the url AND the hash come from the same server. so that hash check only catches a corrupted download, it does nothing against a malicious one. whoever controls the kickbacks backend can push any extension they want and it auto installs and runs as you, no approval, no signing. thats remote code execution by design, the only thing protecting you is hoping their servers never get popped. the crypto to lock it down is literally sitting in the code, they just shipped with it open.

if you really want to keep running it, set KICKBACKS_REQUIRE_MANIFEST_SIG=1 in your environment. that forces the signature path, and since theres no key it then refuses every update instead of installing it blind. thats the safe way to fail.

second kickbacks thing, it rewrites anthropics actual extension

the other two only touch the supported settings file. kickbacks goes further and patches claude codes own bundle on disk, it edits the webview index.js to inject the ad and it loosens the webview content security policy so its ads can phone home. it does the same thing to the openai codex extension too.

to be fair, i checked and it does this carefully: the CSP change is connect-src only so it doesnt open an actual script injection hole, it backs up the original first and the restore works, and the little local server it runs only binds to localhost behind a random token. but still, rewriting a signed third party extension breaks its integrity, its gonna fight every claude code update by re-patching, and its a sketchy amount of access just to show an ad.

adspin, clean, one privacy note

tokens stored properly in vscode secret storage not some flat file, settings backed up and restorable, ad text sanitized. it only touches the settings file, never anthropics code, no self update. the one note: it peeks at your claude projects folder but only reads file modified-times, not the contents, to figure out if youre actively using claude so it only bills when you are. fine, but it is looking in there.

idledev, cleanest, least access

the shipped file is byte for byte identical to the published source, i diffed them. it only writes its own config and the settings file, sanitizes the ad text, validates urls, and sends nothing but your token and the local hour. no self update, no patching anything, never reads your transcripts. if you keep one of these, keep this one.

tldr

- nobody is stealing your keys or code
- kickbacks can silently auto install unsigned extension code from its server, thats real RCE by design, set KICKBACKS_REQUIRE_MANIFEST_SIG=1 or just dont run it
- kickbacks also rewrites anthropics signed extension on disk
- adspin is clean, just peeks at your project folder timestamps
- idledev is the least invasive

i can drop the exact file and line numbers from the beautified bundles if anyone wants to verify any of this