u/David_Henry_Smith

Let's try an experiment:

Take a piece of writing that you're proud of -- be it a paper or a grant -- and feed it into an LLM. Ask it, "Please review this."

Since LLM defaults to sycophantic mode, you'll most likely get showered with praises. You get a confidence boost, and everyone is happy.

Next, take the same piece of writing and feed it again to the LLM in a new session. This time, ask it, "Find all flaws. Be critical."

The few criticisms that are insightful will be served along with several other hallucinated nonsense. On balance, not bad. this is still useful for improving your work, and all is good in the world.

Now, a reviewer takes the work that you had tirelessly crafted, feeds it to an LLM, and uses the second prompt, receives a truckload of ammunition, and picks a few bullets of different calibres to include in his report. Since this is not his first rodeo, he types everything out, paraphrasing everything, without understanding much of anything. Time is short, after all.

You get the comments back on your submission. Rejected. Ah well. Let's see what the reviewers have to say...

What in the name of... is all this nonsense?

You contact the editor or program officer to voice your concerns, but the technical details of the comments are lost on them.

What would you do?

P.S. Those are double dashes above, not em dash

I am a tenure-track assistant professor working in computational biology, and I've been fairly lucky with grants. I just got back the reviewer's comments for my grant this year, and I am now convinced that a grant reviewer is really unqualified to review grants with any statistics. (The other reviewer gave me perfect scores across the board, and I only had two reviewers this year.)

I proposed a new Bayesian model for modelling genomic data, and he raised three "fatal flaws" with my model:

1. p(theta) = 1 is wrong.
2. p(x | theta) = Binomial( x | x + y, theta) is wrong because x is a random variable and it cannot be a distributional parameter.
3. hyperparameters should be fixed.

For reference, the simplified version of my model is:

Given read counts x_j (alternative) and y_j (reference) at a locus j, we estimate theta \in (0, 1) under the model:

p(x_j | theta) = Binomial( x_j | x_j + y_j, theta), p(theta) = 1

(I've probably oversimplified the model. It is now, of course, just the standard binomial model familiar to most statisticians. The real model quite a bit more complicated/novel...)

Point 1. p(theta) = 1 is valid for my model, because theta is in (0, 1). When you integrate p(theta) = 1 from 0 to 1, you get 1, which means this probability density function (pdf) integrates to 1 over the support of the pdf. Further, since p(theta) = 1 >= 0 for all theta, this function satisfy the non-negativity constraint. Therefore, p(theta) = 1 is a proper pdf for my model. The properties of a pdf is routinely taught in any introductory course in mathematical statistics... not sure why the reviewer is calling this equation a "fatal flaw." (Yes, in Bayesian statistics, you would often see p(theta) \propto 1 instead when theta is real, which is not the case here.)

Point 2. Under the binomial distribution, the total count is fixed. Here, x + y is clearly the total count, and it is of course fixed. x is a random variable, y is a random variable, but x + y is fixed. This is the standard assumption for the binomial distribution taught in introductory statistics. On a related not, a true Bayesian statistics expert would not object to a parameter being a random variable, because that's exactly what hierarchical models use.

Point 3. Of course, hyperparmeters are fixed. The vast majority of hyperparameters in Bayesian models are fixed. Most papers assume that you know that hyperparmeters are fixed, unless specified otherwise. Really not sure why this is a "fatal flaw". My grant is read by biologists as well, and I didn't want to use the word "fixed", which most people don't understand.

This reviewer has been reviewing my grants every year, and he (or she) usually give mediocre scores (I guess experts in "Bayesian statistics" are harder to find). All other reviewers seem to be computational biologists in different subfields, and they don't comment on the details of the statistical model. This year, this person is particularly critical, and the panel member really took this person's comments to heart.

Last year, I showed preliminary data that convincingly showed that my other statistical model works, and this reviewer said he don't believe me because the data is better than anything he's ever seen and accused me of not explaining why my model works. On the contrary, I had written half a page on the key trick of my model (accounting for measurement error), and even dedicated a whole figure panel to illustrating the novel approach. This reviewer somehow missed all of this. Luckily, the panel member ignored him last year, and my previous grant was funded. This year, I wasn't so lucky.

I don't know who this reviewer is, and I am not sure whether this reviewer has beef against me. He seems to know some basic Bayesian concepts, like priors and hyperparameters, but he is also making basic math mistakes and wrongly accusing me of being confused and my models of being fatally flawed, based on his poor training in basic statistical concepts, as evidenced above.

So, for people writing statistics grant applications, what strategies would you use to exclude an unqualified reviewer from reviewing your application and commenting on the statistical details of your grant application?

---

Additional context: For this particular grant agency, grants are always sent to people overseas (to avoid conflict of interest). And the reviewers are *paid*. What multiple former members that served on the review panel noticed is that reviewers from developing countries have an unusually high rate of accepting a request to review a grant (presumably due to the financial incentives). So, unqualified reviewers have a financial incentive to review grants for the agency, even when they have no business reviewing the grants. I suppose one way of appearing like an expert is to be hypercritical.

---

When I was in the US, I also had my K99 application reviewed by a "Bayesian expert" who really is just a former physicist who dabbled with some Bayesian statistics, and he seems to think a determinantal point process prior is "basic" stuff. It wasn't that hard to guess who the reviewer was, since he was the only person on the K99 special study section who had anything to do with "Bayesian statistics."

Maybe I should stop putting down "Bayesian statistics" as a keyword on my grants?

I am an tenure-track assistant professor working in computational biology, and I've been fairly lucky with grants. I just got back the reviewer's comments for my grant this year, and I am now convinced that a grant reviewer is really unqualified to review grants with any statistics. (The other reviewer gave me perfect scores across the board, and I only had two reviewers this year.)

I proposed a new Bayesian model for modelling genomic data, and he raised three "fatal flaws" with my model:

1. p(theta) = 1 doesn't make sense.
2. p(x | theta) = Binomial( x | x + y, theta) is wrong because x is a random variable and it cannot be a distributional parameter.
3. hyperparameters should be fixed.

For reference, the simplified version of my model is:

Given read counts x_j (alternative) and y_j (reference) at a locus j, we estimate theta \in (0, 1) under the model:

p(x_j | theta) = Binomial( x_j | x_j + y_j, theta), p(theta) = 1

(I've probably oversimplied the model. It is now, of course, just the standard binomial model familiar to most computational biolgists. The real model quite a bit more complicated/novel...)

Point 1. p(theta) = 1 is valid for my model, because theta is in (0, 1). When you integrate p(theta) = 1 from 0 to 1, you get 1, which means this probability density function (pdf) integrates to 1 over the support of the pdf. Further, since p(theta) = 1 >= 0 for all theta, this function satisfy the non-negativity constraint. Therefore, p(theta) = 1 is a proper pdf for my model. The properties of a pdf is routinely taught in any introductory course in mathematical statistics... not sure why the reviewer is calling this a "fatal flaw." (Yes, in Bayesian statistics, you would often see p(theta) \propto 1 instead when theta is a real number, which is not the case here.)

Point 2. Under the binomial distribution, the total count is fixed. Here, x + y is clearly the total count, and it is of course fixed. x is a random variable, y is a random variable, but x + y is fixed. This is the standard assumption for the binomial distribution taught in introductory statistics.

Point 3. The vast majority of hyperparameters in Bayesian models are fixed. Most papers assume that you know that the hyperparmeter is fixed, unless specified otherwise. Really not sure why this is a "fatal flaw". My grant is read by biologists as well, and I didn't want to use the word "fixed", which most people don't understand.

This reviewer has been reviewing my grants every year, and he (or she) usually give mediocre scores (I guess experts in "Bayesian statistics" are harder to find). All other reviewers seem to be computational biologists in different subfields. This year, this person is particularly critical, and the panel member really took this person's comments to heart. Last year, I showed preliminary data that convincingly showed that my other statistical model works, and this reviewer said he don't believe me because the data is better than anything he's ever seen and accused me of not explaining why my model works. I wrote half a page on the key trick of my model (accounting for measurement error), and even dedicated a whole figure panel to illustrating the novel approach. This reviewer somehow missed all of this. Luckily, the panel member ignored him last year. This year, I wasn't so lucky.

I don't know who this reviewer is, and I am not sure whether this reviewer has beef against me. He seems to know some basic Bayesian concepts, like priors and hyperparameters, but he is also making basic math mistakes and wrongly accusing me of being confused and my models of being fatally flawed, based on his poor understanding of basic mathematical concepts.

So, what strategies would you use to exclude an unqualified reviewer from reviewing your grant?

---

Additional context: For this particular grant agency, grants are always sent to people overseas (to avoid conflict of interest). And the reviewers are *paid*. What multiple former members that served on the review panel noticed is that reviewers from developing countries have an unusually high rate of accepting a request to review a grant (presumably due to the financial incentives). So, unqualified reviewers have a financial incentive to review grants for the agency, even when they have no business reviewing the grants. I suppose one way of appearing like an expert is to be hypercritical.

---

When I was in the US, I also had my K99 application reviewed by a "Bayesian expert" who really is just a former physicist who dabbled with some Bayesian statistics, and he seems to think a determinantal point process prior is "basic" stuff. It wasn't that hard to guess who the reviewer was, since he was the only person on the K99 special study section who had anything to do with "Bayesian statistics."

Maybe I should stop putting down "Bayesian statistics" as a keyword on my grants?