u/FantasyMaster85

I'm running llama.cpp using this docker container: https://github.com/mixa3607/ML-gfx906 (it's just a lot easier than building from source, which I was doing previously). The MI60 (or MI50) are just a real pain in the behind to get working with Ubuntu 24.04. That container has it up in minutes, real timesaver.

Anyway, my personal use case for LLM's is primarily for Frigate to review camera footage and cut down on "notification noise" (it's like having a human review footage to determine what I need to know about and what I don't). The other use is for HomeAssistant. I ditched all my Alexa devices and replaced it with this (it's amazing).

Anyway, I wanted to be sure I was getting the absolute most of out my hardware for speed and efficiency. I had Claude write me a script that would do batch testing of of the two models I got great accuracy out for those two use cases.

• Gemma 4 26B.A4B Q4_1
• Qwen3 35B.A3B Q4_0

The MI60 (and MI50) get a speed boost on the _0 and _1 quants inherently, which is why I use them. The only reason for not using 4_1 for both is the size. I use 3 slots, each with their own cache so the size difference between the qwen 4_0 and 4_1 was eating too much space for my desired context size.

The final result of the testing had a HUGE impact on the speed of both HA (less than 1.2 seconds to complete my voice commands) and Frigate (less than 18 seconds for review summaries of footage). I figured I'd share this here in case it helps anyone else. The following is generated by Claude (summary of what the script did, and it generated the table of results from the outcome of running the script):

The benchmark sweep script executed 30 total runs across 8 sections, testing two models — Gemma 4 26B Q4_1 and Qwen3 35B Q4_0 — against three KV cache pre-fill depths (0, 1,000, and 6,000 tokens) with a fixed 512-token prompt and 128 generation tokens per run, each repeated 5 times internally by llama-bench for statistical stability. The knobs turned were: flash attention on vs. off; KV cache quantisation at three levels (f16 default, q8_0, and q4_0); ubatch size at four values (512, 2048, 4096, and 8192); logical batch size at two values (2048 and 8192); CPU thread count at three values (8, 12, and 24); and two ROCm-specific environment variables — GGML_ROCM_FORCE_MMQ (1 vs. 0, switching between quantised matmul kernels and rocBLAS GEMM) and HSA_ENABLE_SDMA (enabled vs. disabled, switching between DMA and blit-copy memory transfers). Sections 1 through 7 each varied exactly one parameter while holding all others at the production baseline, enabling clean attribution of any performance change to a single cause. Section 8 then stacked three combinations of the most promising individual results — SDMA disabled with q8_0 KV, SDMA disabled with q4_0 KV, and SDMA disabled plus MMQ off plus q8_0 KV — to determine whether gains compounded or cancelled when applied together. The production llama-server container was stopped before each run to ensure exclusive GPU access, and each model configuration was launched as a fresh throwaway container from the same image used in production, with identical device mappings, volume mounts, and environment variables.

https://preview.redd.it/mb0jdzqg1x2h1.png?width=1278&format=png&auto=webp&s=6f2f23c55b45bbb4b9bfebd1af4874f0a21069de

https://preview.redd.it/e5z8lrxe66zg1.png?width=1200&format=png&auto=webp&s=c269bca89b7ad8b8fa07d487ee2ae41f811be433

This is just a heads up for those of you using the GenAI function with Frigate. It's extraordinarily powerful (I was able to cut my "notification spam" to absolutely zero...I only get alerts for threat level 1 and 2 now, and they're 100% accurate and I can tell at a glance whether I need to watch the video or not). I've saved all my notifications for the past three days and made a short video (at bottom of the post). The times when it's actually me/my wife/both of us getting in the car, we almost never get notifications about that because part of my prompt is "if [my name] or [wifes name] are recognized, which you'll know if they're passed along to you in the object list, no matter what is happening in the scene it's threat level 0". So in the video the few that there are of us is when it didn't recognize our faces or we deliberately hid them.

Anyway, back to the main point of the post...with a well crafted prompt, I'm able to use "dumber" models to get additional speed while not losing out on the details that I'm looking for. I've got an absolutely MASSIVE prompt and am currently using gemma 4 26b A4B on an AMD MI60 GPU with 32gb VRAM. I have 130k context (two parallel slots, each with their own 130k context window). My current Frigate metrics show that I'm getting my review summaries as:

Average Inference Time 50722.53ms
(and it would be about 15 seconds less than that if I didn't allocate the 1024 thinking tokens, but I've found that without any thinking at all with this model, even with a good prompt it can get details incorrect)

I'll save my entire prompt for another post, as I had to modify a core python file (frigate/genai/__init__.py ) to have my prompt come after the "analyses guidelines" (which I also had to modify, as a number of the elements of it were interfering with my mechanical rules that I wanted to be followed strictly, and there were too many references to allowing it to "think" about the scene versus just following precisely what I told it to do...it would talk itself out of things being a threat level and assigning them a 0 that I wanted to be threat level 2).

Just a quick example, if it were the middle of the day and I ran out to get something out of the car but my face was hidden, it would assign threat level 0 because "this appears to be a homeowner retrieving something from your car" ...and while that's technically correct, my prompt says that any time ANY unknown person at all EVER opens or interacts with my car that it HAS to be a threat level 2. We park in a townhouse parking lot with shared parking and on a street where plenty of people walk their dogs and such. We've had our car broken into twice and my wife can occasionally forget to lock the doors. If someone that the facial recognition doesn't recognize opens a car door, I don't want the AI "guessing" if that's a problem...its ALWAYS a problem.

Anyway, like I said, that's for a different post.

          ====================================================================
          VEHICLE DOOR SIDE CLASSIFICATION — U.S. DRIVER STANDARD
          DETERMINISTIC RULE SET (NO HEURISTICS)
          ====================================================================

          None of what follows is "relative" to anything.  It's to be interpreted only as written explicitly. No reasoning or thinking.  Follow in the order given:

          STEP 1
          1 - can you see the headlights and/or grill - if so, that means the vehicle is "front facing" - stop - otherwise
          2 - can you see the taillights and/or the trunk - if so, that means the vehicle is "rear facing" - stop 
          Now you now whether its front of rear facing
          END STEP 1

          STEP 2
          1 - Divide the entirety of the viewable frame perfectly down a dividing line that is dictated by the center of the viewable vehicle (to be done per vehicle).  That should give you two "rectangles" that bewteen the two are the entirety of the frame. 
          2 - Is the person or object of interest in the left rectangle? If so, then they're "left side" - stop - otherwise
          3 - Is the person or object of interest in the right rectangle? - then they're "right side" - stop
          Now you now whether they're on the "left side" or "right side"
          END STEP 2

          STEP 3
          1 - "Front facing" combined with "left side" = passenger side - stop - otherwise
          2 - "Front facing" combined with "right side" = drivers side - stop - otherwise
          3 - "Rear facing" combined with "left side" is drivers side - stop - otherwise
          4 - "Rear facing" combined with "right side" is passengers side - stop
          END STEP 3

          STEP 4
          OUTPUT FORMAT (MANDATORY):
          - Use EXACTLY ONE of the following:
          - "Driver-side door"
          - "Passenger-side door"
          - "Door side undetermined"

          OUTPUT ORDER (MANDATORY — follow this sequence exactly):
          1. Resolve vehicle orientation and door side first (silent or shown) using the steps above
          2. Explain to yourself how you arrived at the determination to be sure, mechanically going through them
          3. Write scene description using the resolved door side label after the above step

          Do NOT write the scene description before door side is resolved (if it's relevant to the scene)
          The scene description MUST use the door side label produced by the 
          classification steps above — never an independently assumed label.
          END STEP 4

So that's just one part of my very long prompt about vehicle door side classification. You might think that even that is absurdly long for one thing...and I've tried many many iterations and I finally realized I should just upload the screenshots of things it was getting wrong to my localLLM and ask it to tell me why it came to the conclusion that it did. This helped me immensely in writing my prompt to get the results I wanted. For example, when I first had the prompt I had the front/rear facing part working (it identifying the front/rear of the car) and it just would CONTINUOUSLY say "driver side". So I uploaded it and asked it to tell me why, and it would say "well the person is on the right side of the frame"...which is technically correct...I am on the right side of the frame, but not the right side of the car. There's more to it than that, but that's a quick summary.

So, if you're not getting the results you want from your prompt (or don't have a very good one to begin with), try deliberately doing some things on your camera, taking some screenshots of them, uploading them to your LLM of choice and have it explain how it's coming the incorrect conclusions that it is.

https://reddit.com/link/1t3qcf8/video/zz8i50v2z5zg1/player

https://preview.redd.it/qo2hllhbp5zg1.png?width=1200&format=png&auto=webp&s=cd71cf280ee539234b068429074b47a866cf4bed

This is just a heads up for those of you using the GenAI function with Frigate. It's extraordinarily powerful (I was able to cut my "notification spam" to absolutely zero...I only get alerts for threat level 1 and 2 now, and they're 100% accurate and I can tell at a glance whether I need to watch the video or not). I've saved all my notifications for the past three days and made a GIF (second image) showing what it looks like.

What's more, is with a well crafted prompt, I'm able to use "dumber" models to get additional speed while not losing out on the details that I'm looking for. I've got an absolutely MASSIVE prompt and am currently using gemma 4 26b A4B on an AMD MI60 GPU with 32gb VRAM. I have 130k context (two parallel slots, each with their own 130k context window). My current Frigate metrics show that I'm getting my review summaries as:

Average Inference Time 50722.53ms
(and it would be about 15 seconds less than that if I didn't allocate the 1024 thinking tokens, but I've found that without any thinking at all with this model, even with a good prompt it can get details incorrect)

I'll save my entire prompt for another post, as I had to modify a core python file (frigate/genai/__init__.py ) to have my prompt come after the "analyses guidelines" (which I also had to modify, as a number of the elements of it were interfering with my mechanical rules that I wanted to be followed strictly, and there were too many references to allowing it to "think" about the scene versus just following precisely what I told it to do...it would talk itself out of things being a threat level and assigning them a 0 that I wanted to be threat level 2).

Just a quick example, if it were the middle of the day and I ran out to get something out of the car but my face was hidden, it would assign threat level 0 because "this appears to be a homeowner retrieving something from your car" ...and while that's technically correct, my prompt says that any time ANY unknown person at all EVER opens or interacts with my car that it HAS to be a threat level 2. We park in a townhouse parking lot with shared parking and on a street where plenty of people walk their dogs and such. We've had our car broken into twice and my wife can occasionally forget to lock the doors. If someone that the facial recognition doesn't recognize opens a car door, I don't want the AI "guessing" if that's a problem...its ALWAYS a problem.

Anyway, like I said, that's for a different post.

          ====================================================================
          VEHICLE DOOR SIDE CLASSIFICATION — U.S. DRIVER STANDARD
          DETERMINISTIC RULE SET (NO HEURISTICS)
          ====================================================================

          None of what follows is "relative" to anything.  It's to be interpreted only as written explicitly. No reasoning or thinking.  Follow in the order given:

          STEP 1
          1 - can you see the headlights and/or grill - if so, that means the vehicle is "front facing" - stop - otherwise
          2 - can you see the taillights and/or the trunk - if so, that means the vehicle is "rear facing" - stop 
          Now you now whether its front of rear facing
          END STEP 1

          STEP 2
          1 - Divide the entirety of the viewable frame perfectly down a dividing line that is dictated by the center of the viewable vehicle (to be done per vehicle).  That should give you two "rectangles" that bewteen the two are the entirety of the frame. 
          2 - Is the person or object of interest in the left rectangle? If so, then they're "left side" - stop - otherwise
          3 - Is the person or object of interest in the right rectangle? - then they're "right side" - stop
          Now you now whether they're on the "left side" or "right side"
          END STEP 2

          STEP 3
          1 - "Front facing" combined with "left side" = passenger side - stop - otherwise
          2 - "Front facing" combined with "right side" = drivers side - stop - otherwise
          3 - "Rear facing" combined with "left side" is drivers side - stop - otherwise
          4 - "Rear facing" combined with "right side" is passengers side - stop
          END STEP 3

          STEP 4
          OUTPUT FORMAT (MANDATORY):
          - Use EXACTLY ONE of the following:
          - "Driver-side door"
          - "Passenger-side door"
          - "Door side undetermined"

          OUTPUT ORDER (MANDATORY — follow this sequence exactly):
          1. Resolve vehicle orientation and door side first (silent or shown) using the steps above
          2. Explain to yourself how you arrived at the determination to be sure, mechanically going through them
          3. Write scene description using the resolved door side label after the above step

          Do NOT write the scene description before door side is resolved (if it's relevant to the scene)
          The scene description MUST use the door side label produced by the 
          classification steps above — never an independently assumed label.
          END STEP 4

So that's just one part of my very long prompt about vehicle door side classification. You might think that even that is absurdly long for one thing...and I've tried many many iterations and I finally realized I should just upload the screenshots of things it was getting wrong to my localLLM and ask it to tell me why it came to the conclusion that it did. This helped me immensely in writing my prompt to get the results I wanted. For example, when I first had the prompt I had the front/rear facing part working (it identifying the front/rear of the car) and it just would CONTINUOUSLY say "driver side". So I uploaded it and asked it to tell me why, and it would say "well the person is on the right side of the frame"...which is technically correct...I am on the right side of the frame, but not the right side of the car. There's more to it than that, but that's a quick summary.

So, if you're not getting the results you want from your prompt (or don't have a very good one to begin with), try deliberately doing some things on your camera, taking some screenshots of them, uploading them to your LLM of choice and have it explain how it's coming the incorrect conclusions that it is.

https://reddit.com/link/1t3q7i1/video/1y01fbiqy5zg1/player