u/WarmAd6505

I’ve been experimenting with supervised AI-assisted pentesting in authorised lab environments, and I’m curious how people here think it should fit into platforms like Hack The Box.

There’s obviously a big difference between:

“Explain why this HTTP response is interesting”

and

“Here’s the target — enumerate everything, exploit it and give me the flag.”

The second might get you a solve, but I’m not convinced it teaches you much.

What I’ve found more interesting is using an agent almost like a structured pentesting partner:

- keep track of observations

- turn them into hypotheses

- suggest the cheapest useful test

- preserve command output and evidence

- challenge assumptions when a path goes nowhere

- require an actual verification step before calling something vulnerable

But leave the human responsible for understanding why each test makes sense.

That feels closer to the methodology HTB tries to teach, particularly in paths like CPTS where enumeration, evidence and reporting matter just as much as running the exploit.

I also think there should be a hard distinction between assistance and spoilers.

For active boxes/challenges, an AI system shouldn’t have access to walkthroughs, leaked solutions or previous solve data any more than a human learner should.

So I’m curious:

Where would you personally draw the line?

Would you use AI for:

- explaining tool output?

- maintaining notes?

- suggesting hypotheses?

- generating commands you then review?

- automating enumeration?

- exploitation?

- reporting?

And at what point do you think it stops helping you learn and starts solving the box for you?

No active-box spoilers please.

I've been experimenting with AI-assisted pentesting in authorised lab environments, and one of the biggest problems is false confidence.

An LLM can produce a very convincing vulnerability explanation without actually proving anything.

I've found it useful to force every suspected vulnerability through a simple pipeline:

Observation → Hypothesis → Cheapest test → Verification → Evidence

1. Observation

Record what you actually saw.

For example:

"GET /api/orders/123" returns your own order.

Don't immediately write:

«Possible IDOR vulnerability.»

At this point you've only discovered an endpoint.

2. Hypothesis

Turn the observation into something falsifiable:

«The server may rely on the object ID without checking whether the authenticated user owns the requested order.»

That is something you can actually test.

3. Cheapest useful test

Before reaching for complicated tooling, test the smallest change that could disprove the hypothesis.

For example, in an authorised lab:

Change only the object identifier while keeping the same authenticated session.

The important part is controlling variables.

4. Verification

A "200 OK" by itself proves very little.

You need to establish whether the response contains data belonging to another authorised test account or otherwise demonstrates the access-control failure.

For injection testing, the same principle applies.

A database-looking error isn't automatically SQL injection.

A payload appearing in a response isn't automatically exploitable XSS.

The test should demonstrate behaviour that distinguishes the vulnerability hypothesis from normal application behaviour.

5. Preserve the evidence

For web/API findings I normally want enough information for another tester to reproduce it:

- exact endpoint

- HTTP method

- relevant headers/session context

- original request

- modified request

- relevant response

- expected behaviour

- observed behaviour

- reproduction steps

Screenshots are useful, but raw requests/responses are often much more valuable.

The useful rule

I think this is especially important when using AI agents:

Never let “the model believes it found a vulnerability” be the success condition.

The success condition should be something externally verifiable.

The model can generate hypotheses all day.

Only the evidence gets to promote one into a finding.

For anyone using AI during CTFs, labs or authorised pentests: have you found a better workflow for keeping hallucinated findings out of your reports?

I’ve been experimenting with supervised AI-assisted pentesting in authorised lab environments, and one question keeps coming up:

At what point should we actually accept an AI-generated finding as valid?

LLMs are very good at producing something that sounds like a vulnerability report.

That is obviously not the same thing as proving the vulnerability exists.

For a human pentester, I’d normally want enough evidence that somebody else can independently reproduce the issue.

For an agent, I think the bar should be at least as high.

For example, depending on the finding:

- exact request and response

- reproduction steps

- affected endpoint/parameter

- observed versus expected behaviour

- evidence showing impact

- clean verification request

- screenshots where useful

- relevant tool output

- enough context for another tester to reproduce it

I’m increasingly sceptical of AI pentesting benchmarks that simply count “vulnerabilities found”.

If the model says:

«This endpoint appears vulnerable to IDOR»

that should be worth zero until it actually demonstrates unauthorised access to another object and preserves the evidence.

Same for injection.

Generating a payload isn’t finding SQL injection.

Getting an error isn’t necessarily finding SQL injection.

You need a reproducible behavioural difference that supports the hypothesis.

I also think the agent shouldn’t be the sole judge of whether its own work constitutes proof.

Ideally there is some deterministic or independently reviewable layer between:

hypothesis → test → evidence → validated finding

The other interesting question is false negatives.

An agent that reports 15 genuine vulnerabilities and misses five is arguably much more useful than one that reports 20 but five of them collapse under manual verification.

For people who actually pentest:

What minimum evidence would you require before accepting a vulnerability found by an AI agent?

Would a raw request/response pair be enough?

Would you require an independent verification step?

And should the standard be different for something like reflected XSS versus an access-control or business-logic flaw?

I built Violin for Hermes can you use it during OSCP? It's the way the world's going, isn't it a little behind.

I've just released Violin v3.1.0 🎻

This release is mostly benchmark and guard work, not another pile of prompts.

The benchmark now runs Hermes end-to-end and scores what it actually proved, not what sounds convincing in "report.md".

- Executed request/response evidence is checked against the endpoint, method and decisive proof.

- Proof must link back to a validated hypothesis and canonical "FIND" file.

- Execution receipts are HMAC-signed and bind evidence files by SHA-256, so edited artifacts fail verification.

- The guard now stops target work when evidence is not being recorded as you go, and checks excluded URLs and paths inside command payloads.

- Docker, CI and known-good/known-bad scorer calibration are included.

Release:

https://github.com/Strategic-Automation/violin/releases/tag/v3.1.0

I'd appreciate people trying to break the scorer and guard. Can you make weak proof pass, good proof fail or get the workflow stuck?

I'm not looking for “nice update” comments. If it is overbuilt, unsafe or wrong, tell me.

Violin is now at v3.0.1 on master.

It is a Hermes-native profile for supervised, authorised penetration testing, with:

• 31 methodology and vulnerability playbooks

• An execution guard at the target boundary

• Evidence-backed findings and reporting

• Structured scoping and approval gates

• No additional credential broker or provider lock-in

Install:

"hermes profile install https://github.com/Strategic-Automation/violin"

I’m looking for Hermes users and penetration testers to test the installation and engagement workflow and report where the guard, evidence capture, or reporting process creates friction.

I have been working on a framework to solve HTB, I am ana experienced pentester who moved into AI development about 6 years ago but hasn't done much work pentesting and wanted to see what I could achieve with AI and pentesting with 9b models and free endpoints. I managed to get to level 41 on HTB in 2 days with my violin profile for Hermes. Its still not perfect but it's improving everyday check it out and raise issues. Share anything you do different or ideas. Oh and smash some boxes. Just look up Violin.