[D] Can complex statistical methods be used to make weak findings look stronger than they really are?
I have held this opinion for a long time. I'm wondering what people think.
I keep running into the same thing in medical papers: the first stats you see are already the fancy ones. Mixed models, tons of covariates, custom link functions, the whole works. And I’m always thinking: what the heck did the data look like before making all those assumptions?
I want to see the boring, simple analysis first. Means. A scatterplot. A simple comparison. An effect size. Let me see if there is a signal before making multiple assumptions, multiple adjustments, complex analyses. If the basic and complex analyses agree, great. The model probably adds something useful.
But if there’s nothing much there until the fifth layer of adjustment, that’s when I start getting suspicious. Is this just p-hacking due to known publish-or-perish incentives?
Not because complex models are bad. Sometimes they’re exactly what the question requires. My conspiracy-theory antenna goes off when the paper never shows the simple version at all.
Thoughts? Does anybody have insights into this? I'm genuinely curious whether I'm just overly skeptical of much of the biomedical research literature, or whether basic descriptive and inferential analyses still play a critical role and should be included in every research paper.