u/Strange-Release3520

My introduction to evaluation has not been very pleasant. To me it is a choice between investing time or spending money.

By that I mean you can either:

1. Spend a lot of time hand-crafting Q/A pairs and then perform manual checks to see if your system retrieves the correct chunks. And you repeat this for every single query. Suppose my dataset has 100 Q/A pairs, is this not a lot of work for a single person to do?
2. Use LLM-as-a-judge which automates much of what I described but you run into API rate limits pretty quick. I did figure out a workaround where I evaluate my dataset in batches (e.g: 5 questions per run) but again, it takes time.

I am not trying to complain here. I believe there is a more efficient way to run evaluation but I am new to RAG which is why I do not know proper evaluation techniques. Most metrics on DeepEval use LLM-as-a-judge which comes with API costs but I think I could try RAGAS and it's deterministic evaluation metrics.

Is evaluation in itself a time consuming process?

I need direction and guidance, advice would be much appreciated because I have basically put my project to a halt.

Basically the title. Unless you're paying for API services how are you supposed to evaluate your RAG application?

And by expensive I mean you have to spend some amount of money for an API service to evaluate your system against your golden dataset. I created a 45 Q/A pairs set and no free API could handle it which makes sense but I had to try.

How do you guys test your RAG apps with a lot of Q/A pairs? Are most people using paid API's here or is there a workaround? Please help.

For context I have 45 Q/A pairs and I am trying to test my retrieval but since my API's are free I am running into rate limits.

Beginner here. I can't imagine one building production RAG without a dataset to evaluate it with. For instance how do you know ensemble retriever works better than some other type of retriever? Or have people actually built good systems without evaluation?

It's been only a month since I got into RAG and I started by watching a couple of tutorials to understand the basics. After a while I thought I know enough to build a RAG system which handles documents, Youtube transcripts and web links but boy have I ran into issues of all kinds.

It is very easy to understand how RAG works but building something for production is difficult. Chunking and RAG evaluation are my biggest headaches so it would be really helpful to hear from someone who's done RAG for production. Which actually works, and by that I mean something people can use.