TL;DR: Linux doesn't delete a file that a process still has open. Oracle holds its control file, and every online redo member always opens through it. So for a short, glorious window after that, all of it was still sitting in /proc. We pinned the file descriptors, copied them back byte-for-byte, and opened the database without CREATE CONTROLFILE or RESETLOGS. Zero data loss, 16 minutes down, one entirely new grey hair.

The setup

2-node RAC on an Oracle Database Appliance X9-2 HA. The HA stands for High Availability, which I can now confirm I tested more thoroughly than the vendor intended.

The bit where I ruin my own morning

sudo rm -rf <dbname>/redo. Wrong directory. That directory held one production database's online redo logs, its standby redo logs, and — the good part — its one and only control file.

Both instances were open and serving users at the time. Datafiles were on a different volume and survived, which is the single piece of luck in this entire story.

Yes, we had backups. No, I did not want to be the guy who found out whether they worked.

The ten seconds that actually saved us

Here's the thing nobody thinks of while their stomach is somewhere near the floor: rm deletes a directory entry, not a file. The inode is freed only when the link count reaches zero, and no one has it open. An Oracle instance holds its control file, and every online redo member opens continuously via LGWR.

The files were still there. Intact. Readable through /proc/<lgwr_pid>/fd/.

But it's a melting ice cube — the second LGWR exits, it's gone. Crash, `shutdown ', reboot, anything.

Find the descriptors:

sudo lsof +L1 2>/dev/null | grep '/path/to/redo/dir'

As root, not as oracle. The oracle binary is setuid, which clears the process dumpable flag, so the kernel hands /proc/<pid>/fd/ ownership to root. Being the same user is not enough. Learning this at speed was a treat.

Note the fd numbers on the ora_lgwr_ line. They differ per node. Read them on each node, don't assume.

Pin them with something that isn't the database:

sudo setsid bash -c 'exec 200</proc/<LGWR_PID>/fd/<ctl> \
  201</proc/<LGWR_PID>/fd/<log1> 202</proc/<LGWR_PID>/fd/<log2> \
  203</proc/<LGWR_PID>/fd/<log3> 204</proc/<LGWR_PID>/fd/<log4>; \
  echo $$ > /tmp/inode_holder.pid; while :; do sleep 3600; done' \
  </dev/null >/dev/null 2>&1 &

setsid detaches it from your terminal so it outlives your SSH session. Do it on both nodes.

Twenty minutes later, the instance died on its own — SMON couldn't open the control file, ORA-00210, instance terminated, database down. The pin held. The files were still readable. That one command is the reason this is a war story, not a resume update.

Copying it back

Two rules:

1. Zero database processes running. Copy a control file while CKPT is writing to it, and you get a beautifully torn, completely useless control file. ps -ef | grep -E "ora_[a-z0-9]+_<SID>" | grep -v grep | wc -l → must be 0.

2. Verify exact byte counts with stat, not ls -h. ls -h rounds, and "8.0G" is not a checksum.

   sudo dd if=/proc/<HOLDER_PID>/fd/200 of=<control file path> bs=1M status=none sudo dd if=/proc/<HOLDER_PID>/fd/201 of=<redo group 1 path> bs=8M status=progress

   ...and the rest

   sudo chown oracle:asmadmin <files> sudo chmod 640 <files> stat -c '%s %n' <files>

Then STARTUP MOUNT, check V$LOG — our sequence numbers came back identical to pre-incident — confirm the datafiles are all there and need no recovery, and ALTER DATABASE OPEN. Crash recovery chewed through the restored redo like nothing had happened.

Final proof: ALTER SYSTEM SWITCH LOGFILE and ARCHIVE LOG CURRENT. If ARCn can open redo by path and archive it, you're actually back and not just optimistic.

Standby redo logs were a write-off — nothing held them open — but recreating them is a two-minute j0b and, honestly, felt like a rounding error by that point.

Lessons, some of them expensive

• Do not shut down a database whose files you just deleted. Every instinct screams, "shut it down cleanly before this gets worse." That instinct will delete your data for real. The correct response to a catastrophe here is to do absolutely nothing, very quickly.
• You can't DDL your way out. Once the control file path is gone, every new foreground process dies with ORA-00210. No ALTER DATABASE ADD LOGFILE. RMAN wants a snapshot control file and hits the same wall. A restart is unavoidable — which is precisely why the pin has to come first.
• An abort It is survivable if the redo is pinned. Crash recovery has everything it needs once the files are back where they belong.
• Multiplex your control files across different disk groups. We had exactly one, plus single-member redo groups, all on the same volume. With normal multiplexing, this would have been a boring online repair instead of a Sev 1. That part is entirely on us.
• A standby at 1-second apply lag is a lovely thing to have in your back pocket. Never needed it. Knowing it was there is what made attempting the clever fix feel responsible rather than reckless.

The part I keep thinking about

The highest-value action of the whole incident cost ten seconds and one command. Everything after it — the analysis, the careful copying, the verification — only mattered because someone grabbed those file descriptors before the instance died.

Anyway. Go check whether your control files are multiplexed. Seriously, alt-tab, I'll wait.

Had one of those “everything is broken except the thing I’m staring at” afternoons.

Setup was simple: two vLLM boxes running behind LiteLLM in Docker, exposing a single OpenAI-compatible endpoint to my apps.

The backend was fine. These were the actual problems:

1. LiteLLM was still using cached config. I changed the backend from keyless to using an API key, updated LiteLLM, restarted vLLM, and kept getting 401s. Turned out LiteLLM was still holding the old keyless deployment. `docker restart litellm` fixed it straight away. If you use LiteLLM’s DB-backed model config, restart LiteLLM after changing keys, base URLs, or params.

2. The model name sent to vLLM was wrong.

   The app uses the LiteLLM alias, but vLLM sees whatever comes after `openai/`. That value must match `--served-model-name`. If it doesn’t, you can get 404s through LiteLLM while direct backend curls still work. `curl backend:8000/v1/models` is the quickest sanity check.

3. `localhost` was inside the container.

   If LiteLLM is in Docker, `api_base: http://localhost:8000\` points at the LiteLLM container itself. Not your host, not your vLLM service. Use the real hostname/IP, `host.docker.internal` plus `host-gateway`, or the Docker bridge gateway.

Small extra: don’t use `systemctl is-active` to decide whether vLLM is ready. Active just means the process started. It may still be loading weights or compiling kernels. Poll `/v1/models`.

Every error looked like it was coming from the layer underneath. It wasn’t.

Lost a chunk of time last week on LiteLLM + vLLM errors that looked like backend problems, but weren’t.

Setup: two vLLM backends, LiteLLM in Docker, one OpenAI-compatible endpoint in front.

This is what bit me:

1. Cached LiteLLM deployment.

   Changed the backend to require an API key, updated LiteLLM, restarted vLLM, still got 401s. The missing step was restarting LiteLLM itself. `docker restart litellm` cleared the old keyless config. If your LiteLLM models are DB-backed, restart after changing auth, URLs, or params.

2. Wrong model string going to vLLM.

   LiteLLM’s `model_name` is just what your apps call. The string after `openai/` is sent to vLLM and must match `--served-model-name`. If they differ, you get 404s through the proxy even when direct curls work. Hit `backend:8000/v1/models` and use the name it returns.

3. Docker `localhost` mistake.

   `api_base: http://localhost:8000\` from inside the LiteLLM container means LiteLLM is calling itself, not your host. Use the actual backend address, `host.docker.internal` with `host-gateway`, or the bridge gateway.

Also worth mentioning: don’t treat `systemctl is-active` as “vLLM is ready”. It can be active while still loading weights or compiling kernels. Poll `/v1/models` and only call it ready when that answers.

remain active while weights are still loading or kernels are being compiled. The annoying part is that each error points you at the wrong layer. So yeah, check the proxy before blaming the backend.