I’m working on a lightweight audit layer for knowledge graph operations proposed by LLMs, mainly for GraphRAG and agentic workflows. I’d like a reality check from people who have maintained KGs, ontology pipelines, entity resolution systems, or graph ingestion workflows.
The workflow I’m designing around is:
An LLM or external pipeline proposes a graph operation.
The operation is parsed into a normalized expression.
A static preflight check estimates impact, required capabilities, and cost.
The system rejects it, queues it for human review, or allows downstream execution.
Before routing proposed writes to human review, I’m thinking of using an audit record with roughly these fields:
- Audit Record ID
- Upstream Event ID
- Source System
- Target Graph / Namespace / Tenant
- Operator / Agent ID
- Occurred At / Received At
- Correlation ID
- Normalized Expression Text
- Expression Hash
- Parse / Validation Status
- Ontology / Schema Version
- Policy / Preflight Rule Version
- Preflight Decision
- Impact Summary
- Required Capabilities
- Estimated Cost / Budget Estimate
- Derived Action: Allow / Pending Review / Reject
- Provenance Pointers, such as source URIs, document IDs, or evidence snippets
- Review Status and Justification, if applicable
For people dealing with ontology maintenance, GraphRAG pipelines, KG ingestion, or entity resolution review:
What critical fields would you expect to see before trusting this enough to put in front of a human reviewer?
I’m especially unsure about two design points:
Should ontology / SHACL validation results be embedded in this same audit record, or recorded as separate validation events?
Even for small, low-impact graph writes, is it better practice to strictly separate the “LLM proposal” event from the “human approval” event?
(Context: this is for a small open-source prototype I’m building called CogLang, but I’m not trying to promote the project here. I’m mainly trying to stress-test the audit schema before locking in the human-in-the-loop review shape.)