u/CapiCapiBara

Anybody started dabbling with LLMs (your flavour) and Fortigate configs?

I was wondering if they could be useful in the following contexts:

- running a check against a baseline configuration (settings sync between multiple FGTs)

- looking for and report sub-optimal or unsafe settings as per current best practices

- looking for signs of past compromises (excess local admins, other undesired modifications as per published past PSIRT / CVE indicators)

Are we near a useful level of maturity LLM-side, or still too many hallucinations and lack of consistency?

[EDIT: lots of interesting suggestions incoming... looks like an hot topic!]

As per title, we started adding ISDB lists as local-in-policy block on all new setups and some consolidated old installation.

Code is simple enough:

config firewall local-in-policy
    edit 90
        set intf "wan1"
        set dstaddr "all"
        set internet-service-src enable
        set internet-service-src-name "Malicious-Malicious.Server" "Tor-Exit.Node" "Tor-Relay.Node" "Botnet-C&C.Server" "Spam-Spamming.Server" "Censys-Scanner" "Shodan-Scanner" "Phishing-Phishing.Server" "Proxy-Proxy.Server"
        set dstaddr-negate disable
        set action deny
        set service "ALL"
        set service-negate disable
        set internet-service-src-negate disable
        set schedule "always"
        set status enable
        set comments ''
    next
end

Issue is, it never seems to hit, either with wan1/wan2 or "virtual-wan-link" (SD-WAN) interfaces:

diagnose firewall iprope show 00100004 90
idx:90 
pkts:0 (0 0 0 0 0 0 0 0)
bytes:0 (0 0 0 0 0 0 0 0)
asic_pkts:0 (0 0 0 0 0 0 0 0)
asic_bytes:0 (0 0 0 0 0 0 0 0)
nturbo_pkts:0 (0 0 0 0 0 0 0 0)
nturbo_bytes:0 (0 0 0 0 0 0 0 0)
flag:0x0

LLMs can't solve this with the prompts we could think of... we tried and tried ending up in a rabbit hole of diag commands, to no avail.

[EDIT] - I will update this post with the LLM reply about the above (final score: Human -1 - LLM - 0):

>
You're right, my previous suggestion was wrong. According to the article, the correct group for custom local-in policies is:

>00100001 [ CUST_LOCAL_IN ] — Custom local-in policies.

>For reference, the other local-in related group is 0010000e [ IMPLICIT_IN ] — all default local-in policies (i.e. the built-in ones FortiOS adds automatically, not the ones you configure manually). fortinet

>And 00100004 that I previously suggested is actually 00100004 [ ENC_FWD ] — all forwarding policies, which is the standard firewall policy table — completely wrong for local-in. Apologies for the earlier mistake.

Before reinventing the wheel... anybody checked for sure, and in which way, if these blocklists get any hit?