u/Conscious-Talk-751

I have tried doing some basic searches, but anyone ever tried doing a mixture of experts but only train any one head on a particular domain, like oh this is from reddit, that goes to the reddit head or twitter or arxiv or whatever, then just train the router on what the query most resembles. I haven't seen this done, but it seems like it might do okay. I may try to train an encoding transformer to do this if no one talks me out of it.