r/openshift

The tokenomics of self-hosted LLMs | Red Hat Developer
▲ 16 r/openshift+1 crossposts

The tokenomics of self-hosted LLMs | Red Hat Developer

Paying a cloud API per million tokens is straightforward until the invoice arrives. But when you move to self-hosted large language models (LLMs), the economics flip upside down: you pay for raw compute regardless of whether your GPUs are processing requests or idling.

For platform engineers and machine learning operations (MLOps) leads, tokenomics, the economics of how tokens are produced and what they effectively cost, requires looking at both what you spend and how much you serve.

developers.redhat.com
u/Rhopegorn — 17 hours ago
▲ 35 r/openshift+1 crossposts

One kernel feature, 93% system throughput gone: A Red Hat Enterprise Linux 10.2 kernel regression and how to mitigate it | Red Hat Developer

TL;DR — During performance validation of Red Hat Enterprise Linux 10.2, we identified a kernel scheduler regression that reduced system throughput by up to 93% on multi-NUMA instance types. The regression is bimodal: Workloads either run at full speed or experience severe CPU work starvation. Eliminate this throughput collapse at runtime without rebooting by disabling the NI_RANDOM scheduler feature:

echo NO_NI_RANDOM > /sys/kernel/debug/sched/features
Copy snippet

developers.redhat.com
u/Rhopegorn — 2 days ago

OpenShift and item-level restore

Hello all,

I've gotten the fun task of evaluating RH OS for my company's use cases and I am currently in the phase of exploring backup solutions. Specifically, for the virtualization part of the story, namely fileservers.

Now, I've already setup K10 and it's snapshotting happily, but when I try to follow the procedure for item level restore, I keep bumping into the same issue, regardless of the operating system (tried Fedora from template and WS2022 from sysprepepd image).

I've been working with Claude on this, so I asked it to write up a summary, which is found below:

---------------------------------

Start-of-Summary

Environment

  • OpenShift Container Platform: 4.22.8
  • OpenShift Virtualization (CNV): matching 4.22.x
  • Storage: ODF (OpenShift Data Foundation) / Ceph RBD, storage class ocs-storagecluster-ceph-rbd-virtualization
  • Kasten K10: reproduced identically on v8.5.8 (OLM/Marketplace install) and v9.0.3 (Helm install)
  • VM disks: volumeMode: Block, accessModes: [ReadWriteMany] (required for live migration / failover support)
  • Guest OS: reproduced on both Windows Server 2022 (NTFS) and Fedora (ext4) — not OS-specific

What fails

Any FileRecoverySession against a restore point sourced from a volumeMode: Block + accessModes: [ReadWriteMany] PVC fails within ~1 minute, regardless of guest OS:

{"message":"getBlkidProperties","function":"kasten.io/bmd/go/flr/session.(*objectAnimatorBase).getBlkidProperties",
"file":"kasten.io/bmd/go/flr/session/animator_common.go:49",
"fields":[{"name":"devOrImagePath","value":"/mnt/v0s0/volume.img"}],
"cause":{"message":"exit status 2"}}

Full chain: getBlkidPropertiesActivation errorActivateMountObjects → 0 new objectsMount fails → failed mountfailed to start session.

Ruled out as the root cause:

  • Guest OS / filesystem type (Windows/NTFS and Linux/ext4 fail identically)
  • Partition table complexity (not tested exhaustively, but ruled lower-priority once OS/filesystem was excluded)
  • Kasten version (identical failure on 8.5.8 and 9.0.3)
  • Export completion status (export reports Complete in all failing cases)

Supporting evidence pointing toward the export/mount pipeline rather than blkid itself: Kanister artifact detail for a failed case showed UploadSize: 0 B for the main VM disk object despite Complete status, and total exported object storage size was on the order of ~100 KB for a disk with several GB of actual guest data.

End-of-Summary

---------------------------------
I did get it working via a Veeam B&R agent, but that's just messy for large-scale management.

I did also get it working via restoring the snapshot to a different Namespace. The VM boots normally and I can restore files via interactive login session.

However, I'm stuck on understanding whether this is how admins are expected to restore files in a RHOS Virtualization environment, what I may have done incorrectly and what kind of a workflow is typical for people using it daily to backup fileserver VMs.

My colleagues are already in the "this is crap" camp, and I'm finding it hard to prove them wrong.

Thanks for any opinions, experiences and advice.

reddit.com
u/chypsa — 3 days ago

noobaa without odf

I have a situation where I may need the noobaa operator without ODF, this customer has no ODF subscription and doesn't plan to get one, their storage is on a SAN and there's a CSI.

they need an on prem S3 solution and given all the various option noobaa on it's own, backed by a CSI managed block pvc seems to be a viable option, has anybody done this already?

reddit.com
u/Zestyclose_Ad8420 — 7 days ago

Requirements for migration VMware to OpenShift VE

Hello,

Have I understood/interpreted the documentation correctly that, in order to use the built-in VMware to OCP VE migration tooling, one would need to:
- first download the VMware-vix-disklib from the Broadcom website
- this needs to be uploaded to a thing called "the registry" in OpenShift
- this "registry" is an S3 object based repository?
- this "registry" does not exist out of the box on a bare metal cluster?
- Out of the box, when using a bare metal 4.22 OCP VE cluster against regular (supported) external block or NFS storage using it's CSI, there is no S3 storage and hence, no registry?
- In such an environment, due to the lack of S3 and thus that repository, there is no way to use the built in VMware to OpenShift VE migration tooling?

Above is my conclusion, based on what i've found. But I might have missed something, allowing the use of the built-in VMware to OCP VE via another way?

Kind regards,
Steve

reddit.com
u/SteveTheSneaky — 8 days ago

Arbiter on ARM

Howdy, has anybody attempted to run the OCP Arbiter with an ARM? I have two x86 servers for the actual nodes. I was hoping to use a pi5 with 16GB of RAM to act as the Arbiter for this lab.

reddit.com
u/Topless_Mopar — 9 days ago
▲ 11 r/openshift+6 crossposts

Mitigating the risk of diagnosing live Linux system with AI tools

This article explores an alternative to directly troubleshoot production Linux systems with AI tools by using the sos command and using AI to analyze sosreports instead. I think is an interesting read:

https://medium.com/@linuxjedi2000/the-agentic-ai-risk-issue-on-linux-environments-fd5c55cedcc5?sharedUserId=linuxjedi2000

I know that this subject is very controversial and would love to read your point of view on the subject.

u/jlrueda — 11 days ago