We needed a RAG AI system for a 50 person professional services firm. Internal knowledge bot to answer policy questions, project history queries, and client engagement guidelines. Tested 4 different approaches over 3 months.
Approach 1: Off the shelf chatbot platform with document upload Cheap, fast to set up, terrible at anything requiring synthesis across multiple documents. Good for simple FAQs, bad for anything that needed the AI to connect information from two or three sources.
Approach 2: OpenAI API with basic vector database Better results but requires a developer to maintain. Every time documents were updated someone had to re-run the embedding pipeline. Not feasible for a non-technical team.
Approach 3: Custom RAG with proper chunking strategy and metadata tagging This is where quality jumped. The difference between chunking documents by character count vs by semantic meaning is enormous. Results that were frustrating in approach 2 became accurate in approach 3 once the chunking was done properly.
Approach 4: Managed RAG service with a knowledge management layer Best overall for a business that does not have in-house AI engineers. The knowledge management interface meant non-technical team members could update documents and the system would handle re-embedding automatically.
What we settled on: Approach 4. The ongoing cost is higher than approach 1 but the accuracy and maintainability difference is worth it for us.
Key findings: Retrieval quality mattered more than model choice. The same GPT-4o model gave completely different quality outputs depending on how well the retrieval was set up. Metadata tagging was underrated. Being able to filter retrieval by document type or date made a big difference for queries where recency mattered.
Evaluation loops are not optional. You need a way to measure what the AI is getting wrong before you can improve it.
Happy to get into specifics on any of these.