u/kofi_Average5837

EDIT (correction): As u/Practical_Board_1810 points out in the comments, on a stock EL9 install rsyslog is enabled and populates /var/log/secure, so the file-based setup works fine there — my original framing was too broad. The zero-parses gotcha applies to journald-only setups: minimal/cloud images, containers, or boxes where rsyslog is absent or disabled. journald acquisition works in both cases, which is why the role still defaults to it.

A few people in my fail2ban thread suggested CrowdSec for handling repeat offenders, so I moved my mail/web boxes over. Install went fine, services green — and a day later, cscli decisions list was empty while the box was visibly getting hammered.

The gotcha: RHEL 9 ships sshd logging to journald, not /var/log/secure. Most CrowdSec writeups are Debian-flavored and point file acquisition at paths that don't exist on EL9, so cscli metrics shows zero lines parsed and no scenario ever fires. The fix is a journald source:

# /etc/crowdsec/acquis.d/sshd-journald.yaml
source: journalctl
journalctl_filter:
  - "_SYSTEMD_UNIT=sshd.service"
labels:
  type: syslog

If cscli metrics shows 0 lines for sshd, you have this problem. "Service started" and "logs being parsed" are different claims — verify with metrics, not systemctl.

Second lesson, carried over from my fail2ban days: whitelist your own IP before the engine's first start, not after. CrowdSec replays logs on startup — my own flaky-VPN SSH retries from earlier in the week counted against me. A whitelist in /etc/crowdsec/parsers/s02-enrich/ does it:

name: my/admin-whitelist
whitelist:
  reason: "admin IPs"
  ip:
    - "YOUR.ADMIN.IP"

Two more EL9-specific notes: firewalld is nftables-backed, so use crowdsec-firewall-bouncer-nftables (the iptables one fights the native backend). And I keep a coarse GeoIP pre-filter (firewalld ipset + ipdeny zone files) in front so CrowdSec spends its cycles on the interesting attackers — just create the ipset with a bumped maxelem, because cn + ru together clear the 65536 default.

I packaged the whole thing — engine, nftables bouncer, journald acquisition, whitelist-first (the role hard-fails if you leave the placeholder IP), optional country blocking — into a small MIT role: https://github.com/arhab194/crowdsec-rhel

For those running CrowdSec: do you use the console/community blocklist or keep it purely local? Anyone been burned by the community list false-positiving a legit customer IP?

Most fail2ban writeups stop at the [sshd] jail, so I wanted to share the setup I actually run on my mail/web boxes — and two defaults I learned the hard way.

1. Whitelist your own IP before you enable it. Early on I turned fail2ban on, fat-fingered my SSH login a few times over a flaky connection, and banned myself off my own box. Now the very first thing the playbook does is drop my admin IP into ignoreip. If you take one thing from this post, it's this: put your management/VPN/home IP in the whitelist before the first run, not after.

2. bantime = -1 is a trap at scale. Permanent bans feel satisfying but the iptables/nftables set just grows forever and you can never age stale hosts out. The cleaner pattern is a recidive jail: normal jails ban for an hour or a day, and anything that keeps coming back gets escalated to a long but finite ban. You get the "stop knocking" effect without an unbounded ban list.

Beyond SSH, the boxes were getting hammered on Postfix SASL auth and Apache scanners far more than on sshd, so the playbook enables jails for SSH, Postfix SMTP, Postfix SASL, and Apache (auth / bad-bots / noscript). It also persists bans across restarts and verifies the jails actually loaded with fail2ban-client status — because "service started" and "jails running" aren't the same thing.

Targets RHEL / AlmaLinux / Rocky 9. It's a single idempotent Ansible playbook, MIT-licensed, here if it's useful to anyone: https://github.com/arhab194/fail2ban-rhel

Curious how others handle repeat offenders — recidive, an external blocklist/CrowdSec, or just permanent bans and periodic pruning?

The "Copy Fail" kernel LPE (CVE-2026-31431) is a good reminder that kernel CVEs are annoying to automate well: you have to patch, reboot to actually activate the new kernel, but you don't want to reboot 400 hosts that didn't get a kernel bump. Here's the pattern I use so the reboot is conditional and I get audit evidence out of the same run.

Record the running kernel, patch security-only, then reboot only if it changed:

- name: Running kernel before

command: uname -r

register: kern_before

changed_when: false

- name: Apply security updates

ansible.builtin.dnf:

name: "*"

security: true

state: latest

register: patch

- name: Newest installed kernel after

command: rpm -q --last kernel

register: kern_after

changed_when: false

- name: Reboot only if the kernel actually changed

ansible.builtin.reboot:

msg: "Activating patched kernel"

when: kern_before.stdout not in (kern_after.stdout_lines[0] | default(''))

Then scan AFTER the reboot (not before — otherwise your report reflects the pre-patch state and you chase findings that are already fixed):

- name: OpenSCAP eval -> dated ARF + HTML

command:

argv:

- oscap

- xccdf

- eval

- --profile

- xccdf_org.ssgproject.content_profile_stig

- --results-arf

- "/var/log/evidence/arf-{{ inventory_hostname }}-{{ ansible_date_time.iso8601_basic_short }}.xml"

- --report

- "/var/log/evidence/report-{{ inventory_hostname }}-{{ ansible_date_time.iso8601_basic_short }}.html"

- "/usr/share/xml/scap/ssg/content/ssg-rhel{{ ansible_distribution_major_version }}-ds.xml"

register: oscap

failed_when: oscap.rc not in [0, 2]

changed_when: false

Two gotchas that have bitten me:

- security: true needs the repo to publish updateinfo metadata. On minimal/custom mirrors it's missing and dnf silently patches nothing. Check `dnf updateinfo list security` on one host first.

- Dated evidence paths let you diff week-over-week and hand an auditor proof of remediation instead of "trust me."

I packaged the whole thing (targeted-CVE mode, canary rings, version-aware datastream) into a small MIT role if it's useful: https://github.com/arhab194/rhel-kev-patch

Curious how others here sequence scan-vs-reboot — do you scan pre-patch to prove the finding or post-patch to prove the fix?