u/durlabha

I’ve my opinions but wanted community take .

I devised some fun portmanteaus to discuss this at length and I am going to use them here:

Genie ontology (I'm calling it genealogy).

Palantir ontology (I'm calling it paleontology).

Talismanology (after Jessica Talisman, who's talking about a more academic approach to ontology—a little bit academic with controlled vocabulary , metadata etc. The whole ontology pipeline, you can look it up.)

Is anyone here running an actual calibration chain on their judges?

Human panel as the primary standard, tracked agreement rate, forced recalibration when the model version bumps or the input distribution shifts.

Or is everyone shipping on raw judge scores and hoping?

Need to freely distribute about 2500 copies of my book . Is there a reasonably priced printing press here ?