Hello Everyone,
I am thiniking to upgrade my laptop to MSI Raider GE68. Here are the Specs & price.
I want to keep the spending under 2K CAD.
Any opinion?
I am completely beginner in AI, but I am aware that I can't build a laptop worth competing with ChapGPT.
Your input will be valuable.
If you are working/developing Local Models, please do share your learnings.
I read it everywhere: fp8 quantization and fp8 kv cache are being used.
This is good for Gemma but Qwen will run perfectly fine with 4-5 bit.
I ran it in fp, 8bit and 4 bit and it is behaving very similar.
The difference is night and day - at fp8 you can barely fit 27B on a 4000$ gpu.
At 4bit it runs on a 3090 - and due to the smaller context size it runs faster , less vram IO needed
So do yourself a favor and run Qwen 3.6 in smaller bit quantizations
Join r/LocalTextToSpeech for local TTS models, voices, benchmarks, and setup notes.
Free scripts, tools and help posted regularly.
Contribute and help others, or get help.
I have been benchmarking Qwen3.6-27B and Qwen3.6-35B-A3B locally through llama.cpp, with GitHub Copilot Chat (Vscode Insiders needed) used as the frontend harness.
I am using Claude Opus, GPT 5.5 and Qwen 3.6 (27B) a lot in the past weeks.
The reason for Qwen is proprietary code areas where remote inference is not an option as it would leak the code out. And as long as you don't task it to write a complex cuda graph, it performs well.
Qwen 27.B is at Sonnet 4.6 if you combine it with a high value system prompt - or between Sonnet 4.5 and Sonnet 4.6 without.
Copilot Chat is an excellent harness for this kind of setup. You get the IDE integration, agent flow, tool calling UI, file context, and normal coding workflow, while the actual model is your own local llama-server endpoint.
All of this works while being LOGGED OUT of the Github Copilot account - as that is not affordable in pricing anymore.
This is a practical configuration guide for people already comfortable with llama.cpp, GGUFs, VRAM budgeting, and long-context local inference.
Main focus:
Recommended GGUFs:
27B:
Qwen3.6-27B-UD-Q4_K_XL.gguf
or
Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL
35B-A3B:
Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
If memory is tight on the 35B-A3B model, drop to a smaller Unsloth Dynamic quant:
Qwen3.6-35B-A3B-GGUF:UD-Q3_K_XL
If even that is tight, use UD-Q3_K_M or UD-Q3_K_S.
For the 35B model I do not recommend KV-cache quantization. Run the normal cache and keep the context sane. the 35B model is MoE and very low on kv-cache
For the 27B model, I do highly recommend:
--cache-type-k q4_0
--cache-type-v q4_0
Recent llama.cpp KV-cache improvements make q4_0 much more usable here. The 27B model handles q4_0 KV cache very well in my testing - almost identical to FP in evaluation results.
>What changed: llama.cpp added something like Hadamard rotation to kv-cache which shuffles the tensor distribution in a higher dimensionality and allows quantization superblocks to function.
Because Copilot is a very good harness - beating Codex, Cursor, Claude in my opinion
Vscode Insiders is needed to get the openAI compatible endpoint (to interface the model)
You get:
The important part is that Copilot Chat is only the harness. The model is served locally through llama-server.
It allows MUCH more control over settings, we do not just use MTP drafting. We use a combination of context and MTP drafting which can lead to 300+ tokens/sec on the 27B model. MTP is a medium speedup (1.5x) but once the model is paraphrasing source code from thinking or prefill the ngram draft speedup can reach 6x or more.
So the stack is:
VS Code Insiders
↓
custom OpenAI-compatible model config
↓
llama.cpp llama-server
↓
local Qwen3.6 GGUF
This is the shape I used for VSCode Insiders:
[
{
"name": "WSL",
"vendor": "customoai",
"models": [
{
"id": "qwen3.6-27b",
"name": "QWEN-27B-WSL",
"url": "http://172.27.211.123:1234/v1/chat/completions",
"toolCalling": true,
"vision": true,
"thinking": true,
"maxInputTokens": 165000,
"maxOutputTokens": 15000
}
]
}
]
Adjust the URL to your own llama-server host, in WSL you'll see it by entering ipconfig or ifconfig. port you can choose of course.
The input and output tokens need to be adapted to your context setting.
The id must match the llama-server id.
For local-only setups this is usually one of:
http://127.0.0.1:1234/v1/chat/completions
http://localhost:1234/v1/chat/completions
http://<WSL-IP>:1234/v1/chat/completions
If your Copilot Insiders build expects the newer custom endpoint shape, use the same model block but switch the provider shape accordingly. The key fields are the endpoint URL, model id, tool calling, thinking, and max token limits.
This is the 27B style I recommend.
CTX=150000
PARALLEL=1
HOST=0.0.0.0
PORT=1234
MODEL=/models/Qwen3.6-27B-UD-Q4_K_XL.gguf
/usr/src/llama.cpp/build/bin/llama-server \
-m "$MODEL" \
--ctx-size "$CTX" \
--flash-attn on \
--batch-size 1024 \
--ubatch-size 1024 \
--parallel "$PARALLEL" \
--host "$HOST" \
--port "$PORT" \
-ngl 99 \
--threads 8 \
--threads-batch 8 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.00 \
--presence-penalty 0.00 \
--jinja \
--chat-template-kwargs '{"preserve_thinking": true}' \
--reasoning-format none \
--reasoning-budget 16000 \
--slot-save-path /kv_cache/ \
--props \
--metrics \
--checkpoint-every-n-tokens 1024 \
--ctx-checkpoints 64 \
--perf \
--spec-default \
--spec-type draft-mtp \
--spec-type ngram-map-k4v \
--spec-ngram-map-k4v-size-n 16 \
--spec-ngram-map-k4v-size-m 24 \
--spec-ngram-map-k4v-min-hits 1
For the MTP-specific Unsloth repo, use:
MODEL=/models/Qwen3.6-27B-MTP-UD-Q4_K_XL.gguf
or the HF shorthand if your build supports it:
-hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL
The important part is the drafting chain:
--spec-default
--spec-type draft-mtp
--spec-type ngram-map-k4v
--spec-ngram-map-k4v-size-n 16
--spec-ngram-map-k4v-size-m 24
--spec-ngram-map-k4v-min-hits 1
MTP gives useful speedup, but leave VRAM headroom. In practice I budget roughly +1 to +2 GB VRAM headroom for the MTP/drafting path and related buffers. If you are right on the edge, reduce context before blaming the model.
At q4_0 KV cache, every extra 1 GB of free VRAM is roughly another 13k tokens of 27B context, before runtime overhead.
If you are tight in vram, remove only the MTP part as ngram drafting is free.
You can also just use `mod-ngram` as an alternative to the more complex k4v map.
This part matters.
I use:
--jinja
--chat-template-kwargs '{"preserve_thinking": true}'
--reasoning-format none
--reasoning-budget 16000
The reasoning-format none is important for Qwen3.6 because it avoids bad stop behavior and broken multi-turn thinking state during long coding sessions.
Copilot Chat was created to hide thinking from you (proprietary GPT models) but you want to see the thinking usually. So this solves both issues.
I also keep:
--reasoning-budget 16000
This gives the model room to think, but avoids runaway reasoning loops eating the whole session.
For 35B-A3B, I recommend being more conservative.
CTX=100000
PARALLEL=1
HOST=0.0.0.0
PORT=1234
MODEL=/models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
/usr/src/llama.cpp/build/bin/llama-server \
-m "$MODEL" \
--ctx-size "$CTX" \
--flash-attn on \
--batch-size 1024 \
--ubatch-size 1024 \
--parallel "$PARALLEL" \
--host "$HOST" \
--port "$PORT" \
-ngl 99 \
--threads 8 \
--threads-batch 8 \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.00 \
--presence-penalty 0.00 \
--jinja \
--chat-template-kwargs '{"preserve_thinking": true}' \
--reasoning-format none \
--reasoning-budget 16000 \
--slot-save-path /kv_cache/ \
--props \
--metrics \
--checkpoint-every-n-tokens 1024 \
--ctx-checkpoints 64 \
--perf
No q4_0 KV cache here - the sub 4B active parameters need barely any VRAM anyway.
I recommend keeping 35B-A3B below roughly:
110k context
The model can be pushed past 200k context, but in my testing it becomes more likely to fall into reasoning loops. Once that happens, the session usually does not recover cleanly. Start a fresh session.
The upside of the 35B model is extreme performance, as in hundreds of tokens without any drafting enabled.
You CAN use drafting on top, mod-ngram, MTP and other drafting can be added for more speed but those will need a careful balance (that I have not tested yet)
So my practical 35B rule is:
35B-A3B: stay below 110k if you want stable coding behavior.
27B: can go as high as it fits, but below 150k is where it feels strongest.
Using LM Studio is possible but you need to use a few tricks and it won't achieve the same top-tier performance.
LM Studio does not support our chained drafting, but it supports MTP.
Everything else is similar to llama-server, you'll not have the same max performance but it works well.
You can always just install the latest llama release binaries, and use the commandline to load the model from the lmstudio models directory.
These are practical planning numbers, not hard guarantees. Actual fit depends on:
Recommended cards:
24 GB: RTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10
32 GB: RTX 5090, RTX 5000 Ada, Tesla V100 32GB
Approximate context fit with full GPU offload:
| VRAM | Example NVIDIA cards | Practical context |
|---|---|---|
| 16 GB | RTX 4060 Ti 16GB, RTX 4080 Laptop 16GB, RTX 5080 16GB, RTX 5070 Ti 16GB, RTX A4000 16GB | Not recommended for full 27B UD-Q4_K_XL offload. Use smaller quant or partial CPU offload. |
| 24 GB | RTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10 | ~45k-60k with MTP, ~60k-75k without MTP |
| 32 GB | RTX 5090, RTX 5000 Ada, Tesla V100 32GB | ~140k-160k with MTP, ~160k-180k without MTP |
For 27B, q4_0 KV cache is the difference between normal local context and huge local context. It is the main reason this setup is viable.
On a 5090 you have enough VRAM to supply 2 sessions in parallel with both model types.
Or you could run one fast model for context summarization and 27B for code.
Recommended cards:
24 GB minimum for useful GPU-resident contexts
32 GB strongly preferred
Approximate context fit:
| VRAM | Example NVIDIA cards | Practical context |
|---|---|---|
| 16 GB | RTX 4060 Ti 16GB, RTX 4080 Laptop 16GB, RTX 5080 16GB, RTX 5070 Ti 16GB, RTX A4000 16GB | Not recommended for full 35B-A3B Q4. Use Q3 or partial offload. |
| 24 GB | RTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10 | ~40k-50k |
| 32 GB | RTX 5090, RTX 5000 Ada, Tesla V100 32GB | ~100k-110k recommended; more will fit but stability drops |
The 35B-A3B model is very good, but I would not treat it as a “just max the context” model. Keep it tighter.
If you have the VRAM: Instead of large context, consider multiple sessions with limited context, so you can have 2 or 3 chats simultaneously.
Once llama-server is running:
curl -s http://127.0.0.1:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Reply with exactly: local ai works"}],"max_tokens":16}' \
| jq -r '.choices[0].message.content'
Expected output:
the model responds to your input
If your server is inside WSL or another host, replace 127.0.0.1 with the server IP.
(Invoke-RestMethod `
-Uri "http://127.0.0.1:1234/v1/chat/completions" `
-Method Post `
-ContentType "application/json" `
-Body '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Reply with exactly: local ai works"}],"max_tokens":16}'
).choices[0].message.content
Expected output:
local ai works
My current practical ranking:
27B:
Best long-context local coding model in this setup that is close to Sonnet 4.6
Use q4_0 KV cache.
Use MTP if you have the headroom.
Strongest below 150k context, but can go much higher if memory allows.
35B-A3B:
Excellent quality but will fail on hard tasks
Do not use KV-cache quantization.
Keep below ~110k context for best stability.
Can go above 200k, but reasoning loops become more likely.
If it loops, start a new session.
For Copilot usage, I prefer exposing a conservative maxInputTokens in the JSON, even if the server can technically run higher. For example:
"maxInputTokens": 165000,
"maxOutputTokens": 15000
If you set wrong context here you'll get issues serverside, so make sure that matches.
I had cases where the server went OOC (out of context) when getting too close to the max context so I'd leave a little room. copilot seems to not follow this very strictly.
If you want the most practical Copilot-local setup:
Use Qwen3.6-27B UD-Q4_K_XL
Use llama.cpp server
Use q4_0 KV cache
Use preserve_thinking
Use reasoning budget
Use Copilot Insiders as the harness
Use MTP only when you have VRAM headroom
If you want the stronger but more conservative model:
Use Qwen3.6-35B-A3B UD-Q4_K_M
Do not quantize KV cache
Stay below ~110k context
Drop to UD-Q3_K_XL if memory is tight
This is the first local setup I have used where Copilot feels like a serious frontend for a fully local long-context coding model instead of just a toy endpoint test.
I have tested this on terminal use, debugging, and massive codebase development - it works just like Sonnet 4.6.
Anyone hate the cookie banners ? Those are absolutely nothing in comparison to what is about to come.
The AI Act requires lots of things, many people know it requires every AI modified or generated audiofile to be metadata tagged and fingerprint-watermarked from August on (13M$ fines)
But the Act says a lot more than that and it means all good open source models are affected.
Providers of AI systems generating synthetic text must make outputs machine-readable and detectable as AI-generated, using technical solutions that are effective, interoperable, robust and reliable (code of conduct says 2 layers)
Very simple OSS models are exempt, so the old llama 7B might be fine, but models of "systemic risk" are not exempt. Qwen 3.6 series, Deepseek Flash, GLM, Kimi are all systemic risk GPAI models.
The AI act says that also TEXT output must be machine detectable, it only suggested "statistical methods" alongside cryptographic tagging (where possible) of the metadata (c2pa etc).
So if chatGPT offers you a PDF ot TXT download -> that must have crypto signed metadata.
But 2 layers must be present, not one. And they have to be "robust". A simple "AI Generated" (the label that must be added into every AI image from August on) is not enough, the text itself must be statistical watermarked.
Similar to laser printed paper, that carries little invisible dots. Of course that means serious degradation in AI quality.
Who is affected? Anyone who offers a tool or provides a service that can be accessed by a EU citizen. Even if it's just a single EU citizen being a tourist in Malaysia or using a VPN.
LM-Studio, ollama, llama.cpp, vllm, chatgpt, claude, codex, copilot, opencode, cursor, stable diffusion Web UI, Huggingface and so on.
Any AI application that modifies or generates AI images, AI text, AI music, AI speech must use the EU stamp "AI modified" or "AI generated", must metadata seal the files, must watermark the content. Must also provide risk assessment §50 documentation, compliance documentation, detection service.
Addition: for text the "AI Generated" stamp is only needed when "informing the public" like news and can be left away by shifting the risk to a human reviewer.
Addition: Documentation is not forced by the law directly but providers subject to Article 50 practically must keep evidence showing their marking, labelling, detection, feasibility decisions, exemptions, and robustness testing - or risk non-compliance when audited.
Any failure to do so is fined at 13,000,000 Euro (about 15 million USD) OR a few percent of the annual income (whatever is higher ).
They also have a huge code of conduct, you are voluntarily asked to follow - but if you don't your risk of being fined is seriously increased. That code adds a ton of more horrific additions, often technically very implausible or openly contradicting.
You could say your Qwen 3.6 or gemma 4 12B does not fulfill the requirements of "GPAI with systemic risk" - maybe right if you ignore some benchmarks .. but that moves the 13M$ fine risk on you if you become a provider or distributor. So if the EU believes it IS systemic risk AI you'll face the fines at court as soon as they can catch you.
But what if you are not in the EU ? You are still fully affected.
Any service that offers anything into the EU is affected, including every open source project and model.
Any service that offers something to an EU citizen who is just a tourist, or uses a VPN is liable.
You want to code your nice new startup ? You better ensure no european can touch it.
Or you better never put a foot into the EU and never grow the business too large.
That means every providers out there must be compliant. Every open source tool.
The regulation is so extremely invasive while simultaneously it is ridiculously shortsighted.
99% of all images on the internet will have AI content within the next years, all of them need to be watermarked visibly.
99% of all websites will be AI generated in text and code.
99% of all movies will have AI content, every marketing and sales clip.
99% of all marketing and sales voiceovers will use AI.
So this is like the cookie banner, everyone warning you that they must use a cookie to serve you a website and you can accept all cookies or just all needed cookies - just that it's going to be in your face everywhere.
It's a cookie banner on text, image, video and sound.
Source: https://eur-lex.europa.eu/eli/reg/2024/1689/oj
I've taken a look at how projects are reacting to that, and some already did.
Speech & Music
Elevenlabs - Teamed up with Google: SynthID for all (a strong AI based watermark from Google)
Demodokos Foundry - support answered that they work on a light compliance
Suno - Audible Magic content ID,support ignored me - no official statement on AI Act yet. (likely silent compliance)
Udio - no official statement, support ignored me (likely silent compliance)
Images and Text
Open AI/ChatGPT - SynthID+c2pa comes to Codex, API, ChatGPT with "OpenAI Verification tool"
Google/Gemini - SynthID for image/audio/text/video announced and expanding into C2PA and more
Adobe - Full C2PA hashing and Firefly gets watermarked
Microsoft Copilot/Designer - C2PA, visual and audible watermarks in 365, cryptographic signing of images
Meta - visible labels, IPTC metadata, invisible watermarks,C2PA hashes for Meta AI Images
Stable Diffusion - "working on implementing content credentials” with Adobe/CAI/C2PA
Anthropic Claude - Intends to sign the EU General-Purpose AI Code of Practice (everything and MORE)
Transformers code - Text watermarking: https://huggingface.co/docs/transformers/v4.46.0/en/internal/generation_utils#transformers.SynthIDTextWatermarkingConfig
Open Source
I went through github of vLLM, Ollama, llama.cpp, lmstudio, openrouter and they appear pretty unprepared
The Omnibus "postponing" text:
>Parts of the AI Act have been recommended to be "postponed" but nothing I wrote is incorrect and delaying it by a few months wouldn't be reason to not be concerned.
The Digital Omnibus text approved by Parliament would delay Article 50(2) watermarking / machine-readable marking obligations for pre-existing releases to 2 December 2026, but as of now it still requires formal Council adoption before it enters into force.
If the text is adpoted - Article 50(2) machine-readable marking / watermarking for synthetic audio, image, video, text
For systems released on the market before 2 Aug 2026, compliance is delayed to 2 Dec 2026.
For any new developments the rule applies on 2nd August 2026.
Most other Article 50 transparency duties Still effectively 2 Aug 2026.
This includes pre-existing projects
Source: https://www.consilium.europa.eu/en/press/press-releases/2026/05/07/artificial-intelligence-council-and-parliament-agree-to-simplify-and-streamline-rules/
Corrections and Clarifications:
>- Fines under §50 for watermarking/warning are 13 million Euro (or 3% revenue) whatever is higher not 35M.
- Fines for prohibited use are 35M or 7%, whatever is higher
- Watermarking is not 2 layer forced by law, only by code of conduct, you can use one layer if it is robust but you'll increase risk of non compliance significantly.
- For end-users the obligation to inform about AI Text is less strict, so not every website text falls under it
- It is undefined which open models will fall under "systemic high risk" but if you assume a model is NOT covered the non compliance risk is immediately there.
- EU citizenship alone is not the trigger. A genuinely non-EU, EU-geoblocked service has much lower AI Act risk, but risk remains if it is effectively offered into the EU, knowingly serves EU-located users, or produces outputs intended for use in the EU.
ChatGPT Pro 5.5 reviewed