r/OdysseusAI

▲ 25 r/OdysseusAI+1 crossposts

8GB VRAM + 32GB RAM - how far can I realistically go with local LLMs?

Hi, I'm new to running LLMs locally and I'm trying to figure out what my laptop can actually handle.

Specs:

  • RTX 5060 Laptop GPU — 8GB VRAM
  • 32GB RAM (31.2GB usable)
  • Ryzen 7 260 — 8 cores / 16 threads
  • Radeon 780M iGPU

I'm a bit confused about how far I can go with bigger models.

I know something like a 30B model won't fit in 8GB VRAM, but from what I understand I can use system RAM as well and offload part of the model to the CPU/RAM.

So what would be the realistic limit with this setup? Could I use something around 20B–30B at Q4, or does it become painfully slow once too much of it is running from RAM?

Also, how much difference does Q4/Q5/Q6 make in this situation?

I'm mostly interested in understanding what my laptop is capable of and what kind of local AI stuff I could reasonably use it for. Any tips from people with similar hardware would be appreciated.

reddit.com
u/Zestyclose-Bison-340 — 9 days ago

Odysseus + FreeCAD + RobustMCP + Local Ollama — finally working!

Hey everyone!

After a few days of hammering away at both Odysseus and FreeCAD in Docker through Portainer, and going through a ton of failed attempts, I've finally got it working.

I can now ask my Odysseus agent to create things for me in FreeCAD live, using models running locally through my Ubuntu server's Ollama instance.

I've currently tested lfm2.5 and qwen3.6. So far, lfm2.5:8b has given me the most success. It's definitely still a work in progress and nowhere near perfect, but it's working well enough to be genuinely useful.

Hopefully this helps someone else trying to get the same setup running.

1. Set up Odysseus from the main branch

I cloned the current main branch and built the Docker image myself:

sudo git clone -b main https://github.com/odysseus-dev/odysseus.git /opt/odysseus
cd /opt/odysseus

Important: Before building the image, I had to make a small change to requirements.txt to pin MCP to a version compatible with the FreeCAD toolset:

sed -i 's/^mcp$/mcp>=1.28,<2/' requirements.txt

Then build the image:

docker build -t odysseus:main .

>

2. Persistent storage

I use persistent storage for everything important that affects the experience.

Make sure all of the bind-mount directories exist before deploying the stack from Portainer.

You'll also need to add the .env contents to your Portainer stack and set your admin username/password before deploying. Otherwise, you can end up having to wipe the installation, including the directories used by the persistent volumes.

I've included an example environment configuration below.

3. Docker Compose stack

This is the compose stack I'm currently using:

services:
  odysseus:
    container_name: odysseus
    image: odysseus:main
    restart: unless-stopped

    ports:
      - "${ODYSSEUS_PORT:-7000}:7000"

    extra_hosts:
      - "host.docker.internal:host-gateway"

    environment:
      TZ: ${TZ:-Europe/Oslo}
      OPENAI_API_KEY: ${OPENAI_API_KEY:-}
      OLLAMA_BASE_URL: ${OLLAMA_BASE_URL:-http://host.docker.internal:11434}
      SEARXNG_INSTANCE: http://searxng:8080
      CHROMADB_HOST: chromadb
      CHROMADB_PORT: "8000"
      DATABASE_URL: sqlite:///./data/app.db
      AUTH_ENABLED: "true"
      LOCALHOST_BYPASS: "false"
      ODYSSEUS_ADMIN_USER: ${ODYSSEUS_ADMIN_USER:-admin}
      ODYSSEUS_ADMIN_PASSWORD: ${ODYSSEUS_ADMIN_PASSWORD:-}
      ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-http://localhost:7000}
      SECURE_COOKIES: ${SECURE_COOKIES:-false}

    volumes:
      - odysseus_data:/app/data
      - odysseus_logs:/app/logs
      - odysseus_ssh:/app/.ssh
      - odysseus_huggingface:/app/.cache/huggingface
      - odysseus_local:/app/.local

    depends_on:
      chromadb:
        condition: service_started

      searxng:
        condition: service_healthy

    networks:
      - ai_cad


  chromadb:
    container_name: chromadb
    image: docker.io/chromadb/chroma:latest
    restart: unless-stopped

    environment:
      ANONYMIZED_TELEMETRY: "false"

    volumes:
      - chromadb_data:/data

    networks:
      - ai_cad


  searxng:
    container_name: searxng
    image: docker.io/searxng/searxng:2026.5.31-7159b8aed
    restart: unless-stopped

    environment:
      SEARXNG_SECRET: ${SEARXNG_SECRET}
      BASE_URL: http://searxng:8080/

    volumes:
      - searxng_data:/etc/searxng

    healthcheck:
      test:
        [
          "CMD",
          "python",
          "-c",
          "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/', timeout=10)"
        ]
      interval: 20s
      timeout: 10s
      retries: 20
      start_period: 60s

    networks:
      - ai_cad


  ntfy:
    container_name: ntfy
    image: docker.io/binwiederhier/ntfy:latest
    restart: unless-stopped

    command: serve

    environment:
      TZ: ${TZ:-Europe/Oslo}
      NTFY_BASE_URL: http://ntfy
      NTFY_LISTEN_HTTP: :80

    volumes:
      - ntfy_cache:/var/cache/ntfy
      - ntfy_data:/var/lib/ntfy

    networks:
      - ai_cad


  freecad:
    container_name: freecad
    image: lscr.io/linuxserver/freecad:latest
    restart: unless-stopped

    shm_size: "4gb"

    environment:
      PUID: ${PUID:-1000}
      PGID: ${PGID:-1000}
      TZ: ${TZ:-Europe/Oslo}

    ports:
      - "${FREECAD_HTTPS_PORT:-3001}:3001"

    volumes:
      - cad_data:/config
      - cad_projects:/projects

    networks:
      - ai_cad


  freecad-mcp:
    container_name: freecad-mcp
    image: spkane/freecad-robust-mcp:0.6.2
    restart: unless-stopped

    depends_on:
      - freecad

    network_mode: "service:freecad"

    user: "0:0"

    entrypoint: ["/bin/sh", "-c"]

    command:
      - |
        set -eu

        python - <<'PY'
        from pathlib import Path
        import re

        path = Path(
            '/opt/venv/lib/python3.11/site-packages/freecad_mcp/server.py'
        )

        text = path.read_text()
        changed = False

        # ------------------------------------------------------------
        # Patch FastMCP transport security to allow Docker hostname
        # ------------------------------------------------------------

        transport_import = (
            'from mcp.server.transport_security '
            'import TransportSecuritySettings'
        )

        if transport_import not in text:
            pattern = re.compile(
                r'(from mcp\.server\.fastmcp(?:\.server)? '
                r'import FastMCP\s*)'
            )

            text, count = pattern.subn(
                r'\1\n'
                'from mcp.server.transport_security '
                'import TransportSecuritySettings\n',
                text,
                count=1
            )

            if count != 1:
                raise SystemExit(
                    'Could not add TransportSecuritySettings import; '
                    'upstream image layout changed.'
                )

            changed = True

        # Check for the actual hostname entry rather than a one-line list.
        if '"freecad:*"' not in text:
            pattern = re.compile(
                r'mcp\s*=\s*FastMCP\(\s*'
                r'name="freecad-mcp",\s*'
                r'lifespan=lifespan,\s*'
                r'\)',
                re.MULTILINE
            )

            replacement = '''mcp = FastMCP(
            name="freecad-mcp",
            lifespan=lifespan,
            transport_security=TransportSecuritySettings(
                enable_dns_rebinding_protection=True,
                allowed_hosts=[
                    "localhost:*",
                    "127.0.0.1:*",
                    "freecad:*",
                ],
                allowed_origins=[
                    "http://localhost:*",
                    "http://127.0.0.1:*",
                    "http://freecad:*",
                ],
            ),
        )'''

            text, count = pattern.subn(
                replacement,
                text,
                count=1
            )

            if count != 1:
                raise SystemExit(
                    'Could not apply FreeCAD MCP transport-security patch; '
                    'upstream image layout changed.'
                )

            changed = True

        # ------------------------------------------------------------
        # Existing HTTP compatibility patch
        # ------------------------------------------------------------

        if 'mcp.settings.host = "0.0.0.0"' not in text:
            pattern = re.compile(
                r'mcp\.run\(\s*# type: ignore\[call-arg\]\s*\n'
                r'\s*transport="streamable-http",\s*\n'
                r'\s*host="0\.0\.0\.0",\s*# noqa: S104\s*\n'
                r'\s*port=config\.http_port,\s*\n'
                r'\s*\)'
            )

            replacement = (
                'mcp.settings.host = "0.0.0.0"  # noqa: S104\n'
                '        mcp.settings.port = config.http_port\n'
                '        mcp.run(transport="streamable-http")'
            )

            text, count = pattern.subn(
                replacement,
                text,
                count=1
            )

            if count != 1:
                raise SystemExit(
                    'Could not apply FreeCAD MCP HTTP compatibility patch; '
                    'upstream image layout changed.'
                )

            changed = True

        if changed:
            path.write_text(text)
            print('Applied FreeCAD MCP compatibility patches.')
        else:
            print('FreeCAD MCP compatibility patches already present.')

        PY

        exec /opt/venv/bin/freecad-mcp

    environment:
      FREECAD_MODE: xmlrpc
      FREECAD_SOCKET_HOST: 127.0.0.1
      FREECAD_XMLRPC_PORT: "9875"
      FREECAD_TIMEOUT_MS: "30000"

      FREECAD_TRANSPORT: http
      FREECAD_HTTP_PORT: "8000"


networks:

  ai_cad:
    name: ai_cad
    driver: bridge

    ipam:
      config:
        - subnet: 172.16.0.0/16


volumes:

  odysseus_data:
    name: odysseus_data
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/odysseus_data

  odysseus_logs:
    name: odysseus_logs
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/odysseus_logs

  odysseus_ssh:
    name: odysseus_ssh
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/odysseus_ssh

  odysseus_huggingface:
    name: odysseus_huggingface
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/odysseus_huggingface

  odysseus_local:
    name: odysseus_local
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/odysseus_local

  chromadb_data:
    name: chromadb_data
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/chromadb_data

  searxng_data:
    name: searxng_data
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/searxng_data

  ntfy_cache:
    name: ntfy_cache
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/ntfy_cache

  ntfy_data:
    name: ntfy_data
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/ntfy_data

  cad_data:
    name: cad_data
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/cad_data

  cad_projects:
    name: cad_projects
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/docker-storage/container/cad_projects

4. Example .env

TZ=Europe/Oslo

PUID=1000
PGID=1000

ODYSSEUS_PORT=7000
FREECAD_HTTPS_PORT=3001

ODYSSEUS_ADMIN_USER=admin
ODYSSEUS_ADMIN_PASSWORD=PUT_A_LONG_RANDOM_PASSWORD_HERE

SEARXNG_SECRET=PUT_A_LONG_RANDOM_SECRET_HERE

OPENAI_API_KEY=

OLLAMA_BASE_URL=http://host.docker.internal:11434

ALLOWED_ORIGINS=http://localhost:7000
SECURE_COOKIES=false

Once the directories are created and the .env is configured, deploy the stack through Portainer.

There are still a couple of steps before everything is ready.

5. Install the RobustMCPBridge FreeCAD addon

This is the ridiculous one-liner that installs the RobustMCPBridge workbench inside the FreeCAD container:

docker exec freecad bash -c 'rm -rf /tmp/freecad-addon-robust-mcp-server /config/.local/share/FreeCAD/v1-1/Mod/RobustMCPBridge && git clone --depth 1 https://github.com/spkane/freecad-addon-robust-mcp-server.git /tmp/freecad-addon-robust-mcp-server && mkdir -p /config/.local/share/FreeCAD/v1-1/Mod && cp -a /tmp/freecad-addon-robust-mcp-server/freecad/RobustMCPBridge /config/.local/share/FreeCAD/v1-1/Mod/ && rm -rf /tmp/freecad-addon-robust-mcp-server'

Yep, that's a hell of a line. 😂

It simply:

  1. Removes any previous copy.
  2. Clones the addon repository.
  3. Creates the FreeCAD Mod directory if necessary.
  4. Copies RobustMCPBridge into the FreeCAD workbench directory.
  5. Cleans up the temporary clone.

After running it, restart the FreeCAD container through Portainer. The RobustMCPBridge workbench should then appear in the FreeCAD workbench dropdown.

6. Connect Odysseus to FreeCAD

Once everything is running, you should be able to access:

  • Odysseus: http://YOUR_SERVER_IP:7000
  • FreeCAD: http://YOUR_SERVER_IP:3001

In Odysseus, go to:

Settings → Integrations

Add the FreeCAD MCP connection there.

Name: FreeCAD

Type: Streamable HTTP

URL: http://freecad:8000/mcp

Once you've added it, I recommend restarting the containers in the stack so everything comes back up cleanly.

After startup, check the MCP integration and make sure the expected 83 tools are available.

7. Connect your local Ollama models

I'm assuming you already have Ollama running locally.

In Odysseus:

Settings → + Add Models → + Add Local Models (Endpoint)

Click the ... button and use the scan option. It should find your Ollama instance through the configured endpoint.

I personally ran a test chat first to make sure Odysseus could communicate with Ollama before trying to start the FreeCAD MCP server through the RobustMCPBridge workbench.

And that's it.

At this point I can have a locally running model in Odysseus interact with FreeCAD through MCP and actually create CAD geometry for me.

It's definitely not perfect yet, but after fighting with this for several days, seeing the agent actually create things in FreeCAD is pretty damn cool. 😄

If anyone spots something I've done wrong or has suggestions for improving the setup, I'd be very interested in hearing them.

reddit.com
u/OkComb3954 — 8 days ago

New to AI setups

I have only been playing around with claude and chat gpt as a free user and a generic corporate person without tech or coding knowledge at all. so i am learninng how to set up a local ai workspace with odysseus.

I have just set a workspace for odysseus to use using /workspace pick and i wanted to test this out by having it create a text file with some text it it however it keeps thinking half way and completely stops. and the file isnt created whats wrong here

u/Alarmed-Cap3523 — 12 days ago

How do I go about increasing file size limits?

I tried increasing the values in upload_limits.py but it still errors out about the file size limit. OpenWebUI doesnt have that problem and the local models work fine with files above 10mb so I dont think the problem is about performance.

Setup- WSL docker / windows 11

reddit.com
u/Bowbowjowjow — 11 days ago