Rock paper scissors except its Femboy, Incel, Goth girl
Gothgirl beats femboys due to the goth girls dominant and femboys submissive nature.
Incel Beats Goth girl
And Femboy beats incel because incels get sexually confused and manipulated by femboy bussy.
So the whole goal behind IQ tests is to try to approximate your G factor as a measure of your general intelligence. Which is still just a psychological theory. Practically none of the general IQ questions are domain specific. Meaning being really good at IQ tests does not guarantee that you will be good at solving real problems in real world domains or getting a decent job.
On top of that even if you have a high IQ and that IQ is accurate to your general cognitive ability it also doesn't determine if you'll ever even utilise it. You could be 140IQ failing CS grad that plays videogames and uses chatgpt and coursera. Never making it past a real interview.
So why measure IQ when you can measure general academic knowledge via GRE or ACT etc. Or tests for specific domains.
You could argue that the point of IQ tests being non-domain specific is the exact feature in which why they were created. But, it doesn't justify it as a useful metric.
Your performance in your career and academics is not dictated by your IQ.
also according to the theories around how IQ works its suppose to be something you cant improve. So arguments around using it as a metric to strengthen weak areas fall apart.
Being an expert in a specific domain is infinitely more valuable and useful than having a high IQ score.
So now once you have your IQ metric there is nothing to do with it. The only "thing" you can do with it is attach it to your identity and use it to feed your ego.
So I took the WASI-II the other day, however I wasnt told my final scores. Still if someone could calculate an estimation. I believe I have most of the information to calculate. I was told I got all of the matrix subset questions correct. In block designs I got everyone but the last one. similarity, should be around 100% since I knew all the words. Vocabulary I got most of them theoretically since I knew the words except the last 3 words.
So Block design -> max-1
Matrix -> max
similarity -> likely max
vocabulary -> max-3
If the verbal is too vague, at the very least my Performance IQ should be calculable
It seems to me that AI hallucinations are by far the biggest problem currently plaguing SOTA LLMs. However, despite that I see very little focus on the issue by proprietary companies. It seems like they are focusing on just getting the best looking flashy bench mark scores. And pumping out AI models that can breakdown in real-world performance.
I would go as far to say that a stupid AI model that never lies or makes stuff up when it doesn't know would be more valuable then an extremely smart AI model that hallucinates 1% of the time.
I think most people are quick to reduce the problem to just that the AIs base performance/size is the only predictor of hallucinations. I think theres a strong correlation obviously, but even an unintelligent human wouldnt fabricate things the way small LLMs do. Theres something about the training objective or architecture that creates hallucinations.
Ive seen some attempts and hypothesis around how hallucinations could be reduced. For example, Chinese researchers discovering H-Neurons, a small subset of neurons in an LLM that cause hallucinations. Theoretically a form of ablation could be used post training as a patch for hallucination behaviour.
Some say its due to the training objective. If AI is trained to get as many questions right its literally incentivised to guess when it doesnt know for a chance of getting it right. In this case a potential solution would be to not penalise saying "I dont know" and only penalise wrong answers. An AI trained this way might have a lower benchmark score. But it would be much more reliable and honest when it doesnt know something instead of fabricating information.
I also saw the potential of EBM(energy based models) reducing hallucinations. Because they dont just settle on one token, they iterate until they are confident enough.
Another potential idea I proposed previously was a small 1-8 bits of meta data on each token giving its source context and truthfulness during pre-training. which would natively give the LLM an idea of truth.
Engram also looks decent at preventing some basic knowledge based hallucinations
I think that the true solution to this problem could be any of these all of these and even something not listed here. But, the core issue persists. The objective of AI research isnt fundamentally aligned with the most important characteristic to have in an AI, which is honesty and information integrity. And, companies will continue to produce models that score high on benchmarks but fail in ridiculous ways.