u/Willy__Wonka__

Hi everyone,

I am building a legal RAG project for New Zealand tenancy questions and would love feedback from people who have worked on RAG evaluation, domain-specific retrieval, or legal/regulated-domain QA.

The project is called Astraea.cpp (or Astraea for Python). The practical product is a tenant-facing Q&A tool for NZ tenancy law.

Current architecture:

- legislation-first RAG
- Residential Tenancies Act and Healthy Homes Standards indexed
- Tenancy Tribunal decisions indexed
- official Tenancy Services guidance manually ingested
- source-type-aware retrieval: legislation, official guidance, and cases are retrieved separately
- deterministic statute routing for important sections
- soft vector anchors when no route fires but legislation retrieval is confident
- local LLM generation with citations
- context/debug output showing what the model actually saw

I also have a dataset of 300 verified real-world tenancy Q&A pairs. The answers are strong practical advice, but they do not always include legislation sections or Tribunal citations. So I am thinking of using them as a "practical advice floor", not as the final legal gold standard.

My current evaluation idea:

1. Keep the original Q&A pairs as style/usefulness references.
2. Add gold annotations for each post:
   - issue labels
   - relevant RTA / Healthy Homes sections
   - official guidance where applicable
   - Tribunal/court decision where useful
   - expected legal rule
   - must-include practical steps
   - must-not-say unsafe advice
3. Score model answers on:
   - issue identification
   - legal correctness
   - citation support
   - practical usefulness
   - tone/readability
   - no harmful advice
   - no fake citations
4. Use two tiers:
   - Tier 1: at least as useful as the human practical answer
   - Tier 2: better than the human answer because it adds legislation, official guidance, and case grounding

The big question I am thinking about:

Should every golden example include legislation + official guidance + relevant Tribunal decision, or should court decisions only be required for fact-heavy questions where case comparison is actually useful?

I am also interested in ideas around:

- better metrics for legal RAG
- how to evaluate citation usefulness rather than just citation presence
- how to avoid overfitting to one adviser style
- how to build a good "must not say" safety set
- how to judge answers when the human reference is useful but not citation-heavy
- whether fine-tuning on enriched answers is worth it, or whether RAG + better evaluation is enough

The goal is not to imitate the human answers exactly. The goal is to preserve their practical usefulness but make the system more legally grounded and verifiable.

What would you improve in this setup?

I built a local-first RAG demo for New Zealand residential tenancy law:

https://tenancy.localrun.ai

It retrieves from 31,000+ public Tenancy Tribunal decisions and live Residential Tenancies Act sections, then generates an answer with cited sources. It is not legal advice, and the site asks users not to enter private or identifying details.

The local stack:

• Qwen3-8B-Q5_K_M via llama-server
• single RTX 4060 Laptop GPU
• Qdrant + PostgreSQL retrieval
• live legislation extraction from legislation.govt.nz
• Redis cache for web verification
• FastAPI + SSE streaming
• Cloudflare Tunnel

Runtime: Qwen3-8B-Q5_K_M via llama-server on an RTX 4060 Laptop GPU, ctx-size 5120, --parallel 1, about 59 tok/s.

The interesting part was retrieval, not the generator.

A few benchmark/engineering findings:

• Qwen3-8B Q5_K_M gave the best local generator tradeoff versus Q4/Q6
• Qwen3-Embedding-0.6B improved exact-gold retrieval versus nomic in my benchmark
• generic BM25 hybrid search hurt broad legal questions
• generic cross-encoder reranking often promoted semantically dense but legally less useful chunks
• deterministic statute routing was needed because casual queries like "my carpet is worn" do not naturally match formal section titles like RTA s49A
• I added route-specific legislation allow-lists to stop unrelated sections from contaminating answers
• debug mode shows rewritten query, retrieved chunks, live RTA anchors, context budget, and citation-to-source links

Repo:

https://github.com/jwongso/nz-legal-rag

I am especially interested in feedback on the embedding and reranking choices for legal retrieval.