r/kubernetes

How do I troubleshoot this? I don't know where to start. Longhorn is failing to attach four volumes

How do I troubleshoot this? I don't know where to start. Longhorn is failing to attach four volumes

I shutdown two nodes in my cluster to redo cooling, and when I started them up, four of my volumes failed to attach and sent their pods into a crash loop.

What does this screen in Longhorn mean and where do I go from here? Sorry this is so under-researched, I don't know what I don't know.

The ArgoCD error message for the Immich server is:

AttachVolume.Attach failed for volume "pvc-2a76683c-bb9d-4b3b-be27-ceb5e84190ff" : rpc error: code = DeadlineExceeded desc = volume pvc-2a76683c-bb9d-4b3b-be27-ceb5e84190ff failed to attach to node pi-4 with attachmentID csi-bd4715e7b94274c679a216eeabbd99737527401fcb85c782424488ed797d04d9

and the error for the Immich valkey is:

AttachVolume.Attach failed for volume "pvc-af59781f-b394-4242-b529-89cf3bb463a0" : rpc error: code = DeadlineExceeded desc = volume pvc-af59781f-b394-4242-b529-89cf3bb463a0 failed to attach to node pi-4 with attachmentID csi-39c27266d5941a83c0612a23170c8f16f13ddf0fb268495720abd4022213794d
u/ferriematthew — 5 days ago

looking for the team's K8s web-based UI

i'm looking for a good web-based UI for k8s that allows my team developers to see a cluster by authenticating with their OIDC credentials. They also need to be able to evict a single pod.

I was using kite, but that button to evict a single pod is missing. I do my daily driving with lens, but I'm not interested in paying over $300 a year per person for commercial lens. and it doesn't do anything with Oidc. I have heard headlamp, but haven't tried it yet.

what would you recommend?

i'm also curious if maybe, I'm asking the wrong question.

reddit.com
u/simonsimone1980 — 5 days ago
▲ 34 r/kubernetes+1 crossposts

Networking a 3-Node Kubernetes Homelab with Cilium, Gateway API and Cloudflare Tunnel

Part 2 of my Talos homelab (3 OptiPlex Micros, all control planes). Part 1 got the cluster up, this one is networking.

I moved the running cluster from Flannel + kube-proxy to Cilium without rebuilding it. The default-deny policy that Flannel quietly ignored is now actually enforced, and Hubble shows the drops.

Then I gave it two front doors:

- an internal Gateway on a LAN IP (Cilium LB IPAM, no MetalLB) with a Let's Encrypt wildcard cert, so lab apps get real HTTPS at home

- a public Gateway behind a single Cloudflare Tunnel, so nothing is port-forwarded on the router

An app picks its door with the route's parentRef, which keeps lab stuff from going public by accident.

Write-up with all the manifests and commands: https://blog.prateekjain.dev/networking-a-kubernetes-homelab-cilium-gateway-api-and-cloudflare-tunnel-on-talos-362f548a2f3d?sk=d454cd4eaff4ce8a8fd0cd8e6b3d82f6

Curious if anyone else is using Cilium's Gateway API instead of a separate ingress controller.

u/root0ps — 4 days ago

What are the Kubernetes security gaps people keep missing in production?

I’ve been looking into Kubernetes hardening, and a lot of the advice out there focuses on checklists and compliance. But existing clusters don’t always follow those recommendations, and security gaps can stick around for years.

Things like overly permissive RBAC, privileged workloads, missing network policies, weak secrets management, untrusted images, poor audit logging, or outdated clusters.

For those running Kubernetes in production:

  • What security gaps do you commonly find in existing clusters?
  • Have you seen any security incidents involving Kubernetes? What went wrong, and what can we learn from them from a cybersecurity perspective?
  • Which 2–3 security controls make the biggest real-world difference?
  • Are CIS Benchmarks useful in practice, or do you rely more on your distribution’s hardening guidance?
  • What becomes hardest to secure and maintain as a cluster grows?

Like ,basically, what are teams actually getting wrong when it comes to securing Kubernetes in production?

reddit.com
u/Ashveli — 5 days ago
▲ 30 r/kubernetes+2 crossposts

Webinar on Podman Desktop as Docker alternative - covering daemonless containers + Kubernetes pods

A webinar on Podman Desktop for teams looking for Docker Desktop alternatives. Covers migration, Kubernetes alignment, and a preview of sandboxed AI agent development.

Key takeaways:

  • Drop-in Docker compatibility with a daemonless security model.
  • Native pod + K8s manifest support for local-to-prod consistency.
  • Enterprise fleet management and support.
redhat.com
u/product-marketer2017 — 6 days ago

gateaway API - please help: gateway stuck in programmed=false

Hi everyone,

please help, I'm unable to configure cilium + gateway API... the gateway just is stuck in the "programmed=false" state. Tried so many things but no success, also checked this post but if I'm correct, my current version should have the fix included:

https://github.com/cilium/cilium/pull/46350

Nodes:

NAME                         STATUS   ROLES           AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                          CONTAINER-RUNTIME
n1.k8s.net   Ready    control-plane   3d13h   v1.37.1   192.168.78.3   <none>        AlmaLinux 10.2 (Lavender Lion)   6.12.0-211.61.1.el10_2.x86_64 (amd64)   cri-o://1.37.2
n2.k8s.net   Ready    control-plane   13h     v1.37.1   192.168.78.4   <none>        AlmaLinux 10.2 (Lavender Lion)   6.12.0-211.61.1.el10_2.x86_64 (amd64)   cri-o://1.37.2
n3.k8s.net   Ready    <none>          13h     v1.37.1   192.168.78.2   <none>        AlmaLinux 10.2 (Lavender Lion)   6.12.0-211.61.1.el10_2.x86_64 (amd64)   cri-o://1.37.2

Cilium:

cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator          Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium                   Running: 3
                       cilium-operator          Running: 1
                       clustermesh-apiserver
                       hubble-relay
Cluster Pods:          6/6 managed by Cilium
Helm chart version:    1.20.2
Image versions         cilium             quay.io/cilium/cilium:v1.20.2@sha256:2939231d0d3e3ebddcd80fffa168b7ddcc78fdf0dc864d1c8c126ff523c54f01: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.20.2@sha256:64d8798350e8569b8e7622563fed6e44dce2625f311e4651b774816516c744fc: 1

GatewayClass:

kubectl describe gc
Name:         cilium
Namespace:
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: cilium
              meta.helm.sh/release-namespace: kube-system
API Version:  gateway.networking.k8s.io/v1
Kind:         GatewayClass
Metadata:
  Creation Timestamp:  2026-10-10T07:23:46Z
  Generation:          1
  Resource Version:    592430
  UID:                 2d99e2fe-ab22-451f-8f71-f09968df4e52
Spec:
  Controller Name:  io.cilium/gateway-controller
  Description:      The default Cilium GatewayClass
Status:
  Conditions:
    Last Transition Time:  2026-10-10T07:24:02Z
    Message:               Valid GatewayClass
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
  Supported Features:
    Name:  BackendTLSPolicy
    Name:  GRPCRoute
    Name:  GRPCRouteNamedRouteRule
    Name:  Gateway
    Name:  GatewayAddressEmpty
    Name:  GatewayFrontendClientCertificateValidationInsecureFallback
    Name:  GatewayHTTPListenerIsolation
    Name:  GatewayInfrastructurePropagation
    Name:  GatewayPort8080
    Name:  GatewayStaticAddresses
    Name:  HTTPRoute
    Name:  HTTPRoute303RedirectStatusCode
    Name:  HTTPRoute307RedirectStatusCode
    Name:  HTTPRoute308RedirectStatusCode
    Name:  HTTPRouteBackendProtocolH2C
    Name:  HTTPRouteBackendProtocolWebSocket
    Name:  HTTPRouteBackendRequestHeaderModification
    Name:  HTTPRouteBackendTimeout
    Name:  HTTPRouteCORS
    Name:  HTTPRouteDestinationPortMatching
    Name:  HTTPRouteHostRewrite
    Name:  HTTPRouteMethodMatching
    Name:  HTTPRouteNamedRouteRule
    Name:  HTTPRoutePathRedirect
    Name:  HTTPRoutePathRewrite
    Name:  HTTPRoutePortRedirect
    Name:  HTTPRouteQueryParamMatching
    Name:  HTTPRouteRequestMirror
    Name:  HTTPRouteRequestMultipleMirrors
    Name:  HTTPRouteRequestPercentageMirror
    Name:  HTTPRouteRequestTimeout
    Name:  HTTPRouteResponseHeaderModification
    Name:  HTTPRouteRetry
    Name:  HTTPRouteRetryBackendTimeout
    Name:  HTTPRouteRetryConnectionError
    Name:  HTTPRouteSchemeRedirect
    Name:  ListenerSet
    Name:  Mesh
    Name:  MeshClusterIPMatching
    Name:  MeshHTTPRouteBackendRequestHeaderModification
    Name:  MeshHTTPRouteNamedRouteRule
    Name:  MeshHTTPRouteQueryParamMatching
    Name:  MeshHTTPRouteRedirectPath
    Name:  MeshHTTPRouteRedirectPort
    Name:  MeshHTTPRouteRewritePath
    Name:  MeshHTTPRouteSchemeRedirect
    Name:  ReferenceGrant
    Name:  TCPRoute
    Name:  TLSRoute
    Name:  TLSRouteModeMixed
    Name:  UDPRoute
Events:    <none>

Gateway:

kubectl describe gtw
Name:         nodeport-gateway
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2026-10-10T07:35:16Z
  Generation:          4
  Resource Version:    607512
  UID:                 f6b3b043-d072-4e6c-8f37-bf28cb37a1b2
Spec:
  Gateway Class Name:  cilium
  Listeners:
    Allowed Routes:
      Namespaces:
        From:  Same
    Name:      web-gw-80
    Port:      80
    Protocol:  HTTP
Status:
  Conditions:
    Last Transition Time:  2026-10-10T08:53:31Z
    Message:               Gateway successfully scheduled
    Observed Generation:   4
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
    Last Transition Time:  2026-10-10T08:53:31Z
    Message:               Gateway waiting for address
    Observed Generation:   4
    Reason:                AddressNotAssigned
    Status:                False
    Type:                  Programmed
  Listeners:
    Attached Routes:  1
    Conditions:
      Last Transition Time:  2026-10-10T08:53:31Z
      Message:               Resolved Refs
      Observed Generation:   4
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2026-10-10T08:53:31Z
      Message:               Listener Accepted
      Observed Generation:   4
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2026-10-10T08:53:31Z
      Message:               Address not ready yet
      Observed Generation:   4
      Reason:                Pending
      Status:                False
      Type:                  Programmed
    Name:                    web-gw-80
    Supported Kinds:
      Group:  gateway.networking.k8s.io
      Kind:   HTTPRoute
      Group:  gateway.networking.k8s.io
      Kind:   GRPCRoute
Events:       <none>

Happy for any input, thanks a lot!

reddit.com
u/therealmcz — 5 days ago

Sugestions for single NFS server with multiple IPs

Open to any suggestion on my home lab architecture.

Goal:
Obtain a little bit of throughput when using NFS storage class since I have a 10 GB nics available on all servers ( no switch tho ) and a full SSD raid in the NAS for it only.

Scenario:
I have 1 nas with 2 10GB cards. They are configured respectively as 10.1.0.1 and 10.0.0.1 and 2 nodes running a hypervisor connected on this 10GB cards for NFS only in a direct straight cabling connection ( no switches ) . ( everything else goes through 192.168.1.x network )

My NFS definitions on worker nodes differ according to the node running since the NFS server is available on each server in different IPs.

During the inicial setup, all works because I know in advance where the worker node will run and I configure it properly but in case of moving a vm from a host to another the a crash happens.

This is my home lab setup, I totally understand a 10 GB switch will solve this issue entirely but I'm trying to find a way without one.

Open to whatever load balancer / vIP / floating IP / DNS / nfs HA / hosts file ideas. I'm looking for something simple and elegant. No need of expansion to multiple host because I'm limited to 2 hosts anyways.

Thank you all.

Update:

The nas is a xpenology and the Link agregation mode Balance-SLB doesn't seams to be working in the expected way. That said it doesn't seams to be a viable option in this case.

https://preview.redd.it/ywt8mdlnvouh1.png?width=799&format=png&auto=webp&s=a9a67df41eb85fd1176218e92287ba3d595a8539

reddit.com
u/OldAsk3025 — 4 days ago

Teams that moved from LLM APIs to self-hosting: was it worth it?

Infra engineer here, trying to figure out when self hosting LLMs actually makes sense vs just paying for the APIs. every blog post says "it depends" lol

If you've done it (or looked into it and bailed), would love to hear:

  • how big was your API bill when you started thinking about it? what pushed you over the edge
  • what did it really cost once you add up GPUs, idle time, and the engineering hours?
  • what was the most painful part? cold starts, autoscaling, OOMs, model quality, getting paged at 2am
  • if you went back to APIs, what made you switch back

ballpark numbers are totally fine. I'll put together a cost / decision writeup from the replies and post it back here

reddit.com
u/One_Mention_5385 — 13 days ago

I can run a kubernetes cluster fine but cant read the operator code, time to learn go properly

3 years as a platform engineer. i can debug a crashlooping pod half asleep and write helm charts nobody else wants to touch, but every time i open the source of an operator we run i bounce straight off it. our team wants to start writing our own controllers next year so im learning go for real this time, employer covered. shortlist is ardan labs, boot dev, plus one of the linux foundation cert tracks. where did go start making sense for people coming from the ops side?

reddit.com
u/NetNarrow8938 — 14 days ago

Built a hands on tutorial about basics of CRDs on Iximiuz Labs

I've had so much fun and have learnt a lot about kubernetes through iximiuz labs, this was an attempt to create a hands on tutorial of what a CRD is and how can one build their own. It links to a bigger How to Build an Operator tutorial at the end.

Looking forward to see what the community thinks of it!

u/benne-masale — 14 days ago

Monthly: Who is hiring?

This monthly post can be used to share Kubernetes-related job openings within your company. Please include:

  • Name of the company
  • Location requirements (or lack thereof)
  • At least one of: a link to a job posting/application page or contact details

If you are interested in a job, please contact the poster directly.

Common reasons for comment removal:

  • Not meeting the above requirements
  • Recruiter post / recruiter listings
  • Negative, inflammatory, or abrasive tone
reddit.com
u/AutoModerator — 14 days ago

Suggest good resources to Kubernetes

I'm a L1 Linux System Administrator. I'm new to container and container orchestration.

I want to learn Docker and Kubernetes right from scratch.

Can you please suggest me right resources (Free+Paid) to learn and hopefully pass CK A?

Also let me know any right set-up to practice any videos to create and setup the environment to practice as I can't afford KodeKloud, Killersh etc.

reddit.com
u/sathya619sk — 13 days ago