u/Jrodna

I run both a home server on my local network and a vm in AWS that both run docker containers. I'm trying to monitor the cloud based containers from my home server using cadvisor, prometheus and grafana, but I can't get the docker container traffic to route through tailscale. Does anyone know what kind of setup would work for this?

Prometheus always returns errors like

Error scraping target: Get "http://100.125.152.52:62882/metrics": context deadline exceeded

and

Error scraping target: Get "http://100.125.152.52:62882/metrics": dial tcp 100.125.152.52:62882: connect: no route to host

From what I can tell the cadvisor part is working, hitting its IP or tailscale hostname on the cadvisor port in my browser gives me a page of metrics

Cloud docker-compose:

services:
  #... Other containers
  # Cadvisor is for monitoring our docker containers
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    privileged: true
    ports:
      - "62882:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    restart: unless-stopped

Home server docker-compose:

services:
  tailscale:
    image: tailscale/tailscale:latest
    hostname: tailscale-observability
    environment:
      - TS_AUTHKEY=&lt;my auth key&gt;
      - TS_EXTRA_ARGS=--advertise-tags=tag:server --snat-subnet-routes=false
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_USERSPACE=true
    volumes:
      - ./tailscal/state:/var/lib/tailscale
    devices:
      - /dev/net/tun:/dev/net/tun
    cap_add:
      - net_admin
      - net_raw
    restart: unless-stopped
    user: 0:568
  prometheus:
    depends_on:
      - tailscale
    network_mode: service:tailscale
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/rules:/etc/prometheus/rules
      - ./prometheus/prometheus_data:/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --web.enable-lifecycle
    restart: unless-stopped
    user: 0:568
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    volumes:
      - ./grafana/grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=&lt;my admin password&gt;
      - GF_USERS_ALLOW_SIGN_UP=true
    restart: unless-stopped
    user: 0:568
    network_mode: service:tailscale
networks: {}

Extra notes:

- Cloud VM has tailscale installed on the host system not using a docker container

- Home server is running truenas community edition 25.04 and tailscale can not be installed on the host machine, only in a container

Any help would be greatly apreciated.

Edit:

Figured it out,

I had to set the TS_ACCEPT_DNS=true env variable so it would use the tailscale dns and not the default docker setup.

Routing docker container traffic through tailscale