u/Dry_Weakness_5721

I’m building a small model where the output is not a price target, but probability for few stats:
undervalued
fair value
overvalued
I am stuck on how to set the prior.
My first idea was to take historical companies which are somewhat comparable and estimate the prior from that. But I feel this can create selection bias because deciding what is “comparable” itself can change the result.
So would it be better to:
start with a broad base rate and let the features update it, or
make the prior from a matched universe based on sector, size, valuation etc?
I’m still learning this stuff, so maybe I am thinking about the problem wrong.
How would you approach this?

I’m working on a small research project around AI assisted investment research.
Instead of building an LLM that simply says BUY / HOLD / SELL, I’m testing an agent that maintains a probability distribution over possible states:
Undervalued
Fairly valued
Overvalued
Deteriorating
The agent then updates those beliefs when new evidence arrives and chooses an action based on both probability and the consequences of being wrong.
I’m particularly interested in three questions:
How should priors be constructed?
Should the prior come from historical comparable companies, sector/base-rate data, factor models, or something else?
How would you evaluate calibration?
If an agent says “60% probability of undervaluation,” what would you consider a meaningful test that this 60% actually means something?
How should asymmetric loss influence the decision?
For example, a 60% chance of being right may still be a bad BUY decision if the downside of being wrong is much larger than the upside.
I’m deliberately not trying to prove that an LLM can generate alpha.
The research question is narrower:
Does explicitly representing uncertainty + asymmetric decision costs produce better decisions than simply choosing the highest-probability state?
I’d especially appreciate criticism from people who have worked with Bayesian models, systematic investing, portfolio construction, or decision theory.
What am I getting wrong?