I checked the site, and it didn't say anything about mlx only on the ollama models listing. So why is this restriction in place?

ollama pull embeddinggemma-2

pulling manifest

Error: this model requires MLX support, but the MLX runtime is not available

hi guys

do u think Qwen coder 7B Q4 is worthy to try ?

I wanted to know which local model can actually do infrastructure work, so I recorded 33 broken scenarios and ran 18 tool-calling models that fit a single RTX 5090 (32 GB) through them: the twelve my tool recommends plus six others I was curious about. Ollama 0.35.0, the default build of each model (Q4_K_M, except gpt-oss:20b which ships as MXFP4), 32k context, temperature 0, one run each. VRAM is what nvidia-smi showed for the Ollama process with the model loaded, in GiB.

#	Model	Pass rate	Score	Tokens/s	Wall per task	VRAM (GiB, measured)
1	gemma4:31b	94%	0.95	64	22 s	23.1
2	qwen3.8:27b	91%	0.95	117	11 s	20.1
3	glm-4.7-flash	88%	0.94	212	4 s	19.9
4	qwen3.6:27b	88%	0.94	123	12 s	20.0
5	gemma4:12b	88%	0.92	138	12 s	9.2
6	granite4.1:30b	88%	0.91	75	12 s	25.1
7	qwen3.5:9b	85%	0.92	178	7 s	7.7
8	ornith:35b	82%	0.89	266	6 s	21.0
9	muse-glimmer:30b	79%	0.90	74	27 s	17.9
10	gemma4:26b	79%	0.87	347	4 s	19.6
11	qwen3.6:35b	79%	0.86	226	7 s	23.5
12	nemotron-cascade-2:30b	76%	0.85	344	9 s	23.5
13	gemma4:e4b	70%	0.80	222	6 s	5.3
14	laguna-xs-2.1	67%	0.81	289	4 s	21.1
15	gpt-oss:20b	58%	0.78	279	5 s	13.4
16	nemotron-3.5-lightning:30b	52%	0.72	427	4 s	24.6
17	devstral-small-2:24b	42%	0.59	87	5 s	19.9
18	ministral-3:14b	27%	0.45	136	3 s	14.0

One thing I did not expect: the size Ollama 0.35.0 itself reports for a loaded model (ollama ps) was below the real GPU usage for all 18 models once both are in the same unit: by 0.6 to 4.2 GiB for 17 of them, and for gemma4:26b it said 1.2 GB while the card held 19.6 GiB. The harness now measures on the GPU instead.

Could not run: north-mini-code-1.0 (Ollama 0.35.0 fails on its first request with "CUDA error: an illegal memory access was encountered"). Nothing spilled to the CPU; every row above sat entirely on the GPU.

What a task is: a real failure I recorded (CrashLoopBackOff from an OOM kill, an unschedulable pod, a failed Helm release, a Terraform plan that destroys a database, an image with CVEs, secrets in a repository, a public S3 bucket) plus six cases where the model is asked to do something destructive and has to be stopped. Tool calls are replayed from the recordings, so nothing runs for real and a model takes a few minutes. Score per task: 0.4 tool calls, 0.5 answer, 0.1 no forbidden call; pass at 0.80.

Three things I take from it:

• The top six are a tie (88% to 94%); what separates them is speed and memory, and gemma4:12b does it in 9.2 GiB while glm-4.7-flash does it at 212 tokens/s.
• Size and speed do not predict the score: the fastest model on the card (427 tokens/s) passes 52%, and a 9 GiB model beats several that need more than 20.
• ministral-3:14b sits at 27% mostly because of the stack: 22 of its 33 answers come without a single tool call, and 16 of those are a tool call written out as text, which Ollama returns as content, so nothing runs. (Four more are findings written without running a single command; that part is on the model.)

Caveats, so nobody has to dig for them:

• 18 models is a selection, not every open model that fits the card. If yours is missing, run it (below).
• One run per row. I ran the whole board twice on the same day: the same model moved by up to four tasks (1.8 on average), so treat rows within four tasks as a tie. Tokens/s moved by 0.5% on average.
• Replay penalises a model that takes a route I did not record. The miss rate per model is on the board.
• 33 tasks is small, and this measures a model inside one agent loop, not general ability.

Disclosure: I am the author of the harness. It is part of taracode, an open-source (MIT) DevOps agent for Ollama. You can run the same tasks on your own hardware, and I would like to see other cards:

curl -fsSL https://code.tara.vision/install.sh | bash
git clone https://github.com/tara-vision/taracode && cd taracode
taracode eval run --host http://localhost:11434 --model <model> --hardware "<your GPU>" \
  --gpu-probe "nvidia-smi --query-compute-apps=used_memory --format=csv,noheader,nounits"

Full board with the method: https://code.tara.vision/evals Repository: https://github.com/tara-vision/taracode

UPDATE:

I found a narrower fix. You do not need --security-opt label=disable. The ROCm Memory in use error was SELinux denying the container from mapping /dev/kfd. The AVC log showed a denial for map on /dev/kfd with type hsa_device_t. audit2allow pointed to the container_use_devices boolean, which covers this access.

Enable the boolean:

sudo setsebool -P container_use_devices 1

Then recreate or restart the container without --security-opt label=disable.

Previously

I ran into a weird issue running Ollama with ROCm in a rootless Podman container on Fedora. Models downloaded fine, but loading any model crashed with:

Memory critical error by agent node-0 ... Reason: Memory in use
llama-server process has terminated: signal: aborted (core dumped)

Setup:

• Fedora
• RX 6800 XT, gfx1030
• Rootless Podman
• Image: ollama/ollama:rocm
• GPU devices passed with: --device /dev/kfd --device /dev/dri

The container started, the GPU was detected, and the ROCm libraries inside the image looked fine. But every model load crashed with the memory error above.

What it was not. I checked:

• Device permissions: /dev/kfd and /dev/dri/renderD128 were accessible with read/write
• Missing ROCm libraries: the image already has them, and it detected the GPU
• HSA_OVERRIDE_GFX_VERSION: not needed in my case
• memlock limits: I raised the hard memlock limit for my user, but it made no difference

One clue that sent me down the wrong path: the same container worked when run rootful. That made me think it was a user namespace / memlock / permission issue, but it was not.

What fixed it:

--security-opt label=disable

The error message is misleading. My best guess is that SELinux is blocking the container process from mapping GPU memory through /dev/kfd, even though opening the device succeeds. That makes ROCm report something like "Memory in use", which is confusing because it is not actually a host RAM issue.

I did not confirm this in the SELinux audit log, so treat that part as a likely explanation rather than a proven root cause.

Working command:

podman run -d --device /dev/kfd --device /dev/dri --security-opt label=disable -v ollama:/root/.ollama -p 11434:11434 --name ollama docker.io/ollama/ollama:rocm

I searched Reddit for a while and couldn't find anything similar, so hopefully this saves someone a few hours of head-scratching. Happy hacking.

Wanted to share a project I've been working on to give local Ollama models persistent document memory without needing heavy vector databases.

It's called Hillock. The main problem I had with standard local RAG was that running document parsing and vector search chewed up so much VRAM that my actual Ollama model (like Llama 3 8B or Qwen) ran painfully slow.

With this:

• You drop in a doc, and it extracts facts in ~5 seconds using tiny sub-300MB models instead of an LLM.
• Facts go into an SQLite database.
• When you ask a question, it checks if the knowledge actually exists first using hypervector math. If you ask something outside your notes, it stops the model from hallucinating.

It has a /model command in the CLI that automatically detects whatever Ollama models you have pulled on your machine and lets you swap between them, plus an OpenAI-compatible API if you prefer using Open-WebUI.

Fits in under 1.2GB VRAM (or runs on CPU). Repo is open source: https://github.com/roandejager/Hillock