Struggling with LLM Re-Ranking in Our Product Recommendation System – Any Advice?
Hey fellow data enthusiasts,
I've been experimenting with using LLMs for re-ranking in our product recommendation system. We already have collaborative filtering and popularity-based algorithms in place, but real order data shows that almost half of our users end up buying products ranked beyond position 20. And keep in mind, our first page only shows 4–5 items.
Here’s what I’ve tried so far:
- Directly re-ranking the top 100 candidate products using an LLM. Unfortunately, due to attention limitations, the results were sometimes worse than the original ranking. The model tends to push popular items back, even though users clearly exhibit herd behavior.
- Feeding the model user demand signals and profiles, scoring each product individually. This was a mixed bag: sometimes it correctly promoted the products users wanted, sometimes the opposite. Overall, performance slightly lagged behind the original ranking.
- Hierarchical / group-wise re-ranking. For example, protecting the top 10 items while re-ranking items 11–100. This gave a modest +2pp lift in conversion.
A big challenge is that most of our users are new, so we have very little behavioral history to analyze, and even the data we have is noisy.
I’m curious if anyone has suggestions on:
- Other techniques to improve LLM-based re-ranking under low-data / new-user scenarios
- Using methods like GraphRAG or vector embeddings to enhance re-ranking effectiveness
Any thoughts or references would be greatly appreciated!
If you want, I can also draft an even punchier, highly upvotable Reddit version with a more casual/humorous tone that emphasizes the “half of users buy stuff past rank 20” pain point—it could increase engagement. Do you want me to do that?