About to buy one of these things for a local LLM build.
Curious what people have / or are currently paying for a used one in the Canadian market? i.e. Canadian dollars.
I ask because the asking prices are often inflated, and I don't want to overpay (obviously).
Thank you, you very kind people!
I am considering investing in some local AI hardware but to make decision on what to get, I need to figure out the context size of my code base.
I saw there are calculators for this. e.g. tiktoken. I see reports that you need like 8xH100 gpus to support 256k context window in some places. What's that about?
Has anyone here used local AI models on a relatively large code base? How do you know if the model's context windows is large enough or not?
EDIT:
Normal / Naive designed transformers are O(n^2) in space complexity (quadratic growth in the amount of memory required). i.e. Without any tricks the amount of VRAM required for 256k context length would be hundreds of GB of VRAM it seems.
I think that's why I read the 8 x H100's as a requirement on some site last night.
But Qwen uses some special tricks to make it ~ O(n) space complexity, so 32GB VRAM actually works for a 32k Context.
I am still looking for something that can actually work with code bases. i.e. Not just single file apps, not just single file bash scripts. But something where I can give it access to my codebase, give it a spec for a new feature, hit a button, then 2 hours later get a working feature with little or no bugs.
Does that exist yet? Money is no objects at the moment, I am purely looking for something that actually works (and is local) at the moment.
I have the money, I just need to know it works before I shell out the dollars for it.
I've tried Qwen 3.6 27b on a 32GB RTX 4500 PRO on a remote pod, but the pod keeps going down..
If anyone knows of a reliable one I can test on?
- - - - - - -
EDIT 1: Budget <= $100k.
EDIT 2 @ 9:25pm EST time
I finally was able to get a rented one working with a RTX 5090 32GB + Qwen 3.6 27b.
While its certainly VERY helpful, its no SWE replacement (by a long shot). However I am easily 3-10x faster for coding tasks. So its well worth purchasing the card for my self to use it seems. Obviously I won't be using it 24/7 so I might rent out the compute to others when I am not using it or something. Anyone know a place in Toronto I get buy one these things on the cheap?
Ever been reprimanded for laughing at work?
I almost was—on multiple occasions. It turns out that despite the "culture" claims many employers make, they don’t actually want you to have a good time. This is especially true if you’re enjoying yourself while they aren't.
The painful truth: In every job where joy was punished, the pattern was identical: weak leadership. These are managers obsessed with status symbols who resort to "status slaps" the moment they feel threatened. If you speak with confidence, you aren't met with feedback—you’re met with personal insults, often staged in front of the team to "assert dominance."
That isn't modern leadership. It’s predatory behavior. People don't follow these types because they respect them; they stay because of a paycheck, and even then, they’re looking for the exit.
Anyone else experience this?