r/AskStatistics

• Which models do you use most frequently, and for what tasks?
• Which models have delivered the most practical value versus being primarily academic?
• How important are classical statistical models compared to modern ML methods?
• Are tree-based models still dominant, or is deep learning becoming more prevalent?
• If you were starting over today, which models would you prioritize learning?

Industry practitioners are invited to comment on any of the above. Thanks in advance.

Hi,

Suppose I have a neural network classifier C, based on, e.g., a CNN or Transformer.

And suppose further that I have a modification, called M, of C that I hypothesize that the accuracy of C should be better.

I can afford to run experiments for N runs (e.g., N=5) for C and C+M.

What test statistic should I use to demonstrate that the modification shows 'significant' improvement?

Moreover, for each configuration (C or C+M), should I report standard deviation (stddev) of accuracy or standard error (stddev/sqrt(5)) ?

From the context, I have often seen ML papers report stddev but some also report stderr.

Also, I have typically seen those papers that perform multiple runs do not perform any statistical tests to quantify the improvement of the methods they propose. I find this trend discerning.

Thank you very much in advance for your answer!

I’ve seen some people saying cronbachs is overused and doesn’t actually measure consistency. Trying to see if or when that’s the case and if alternatives like omega is an option?

Hi all,

I’m evaluating the effect of an intervention on future healthcare utilization using propensity score matching (PSM) with a matched control group.

After matching on 11 variables, the intervention and control groups are largely balanced, though two variables remain slightly imbalanced. To account for this, I fit a post-matching logistic regression model including those variables as covariates.

In addition, I’m observing that two other variables, while well balanced between groups, remain statistically significant predictors of the outcome in the regression model. These variables were expected a priori to be strong predictors of utilization, so the direction and significance make sense. However, I had initially expected that balancing them through matching might attenuate their significance.

This leaves me with a model that includes four covariates: two addressing residual imbalance and two that are balanced but strongly predictive of the outcome. Including these predictors improves model fit (lower AIC) and attenuates the intervention estimate toward the null; excluding them makes the intervention effect statistically significant and increases AIC.

My current understanding is that balance across groups does not eliminate a variable’s association with the outcome, and that seeing these variables remain significant is expected in a doubly robust framework.

I’d appreciate your perspective on the following:

• Is it appropriate to retain outcome-predictive covariates in the regression model even if they are balanced after matching?
• Is their statistical significance expected, reflecting within-sample associations rather than residual imbalance?
• Are there concerns about over-adjustment or redundancy when including variables already used in the propensity score model?
• In practice, how do you approach variable selection post-matching (e.g., all matching variables vs. a parsimonious subset based on fit or substantive importance)?

TL;DR: Some covariates are balanced post-matching but remain strong and significant predictors in the outcome model. I want to confirm that I’m interpreting this correctly and not over-specifying the regression.

Thank you so much!

I am analysing some data related data and what to check how it would relate to different demographic variables like employment status, marital status, etc.
Both employment and marital status in the data have four categories (eg. single, married, divorced, widowed). I want to see their association with clinical variables like onset, frequency (both continuous). What would be the appropriate analysis for this?

I read a article and it used the sentence a) and i cant wrap my head around it. Don't get if it's wrong or mainly confusingly written. Simplified this is roughly what its about

The relative risk is 0.33 for group A compared to placebo. Wouldnt line a) be wrong?

a) group A has rougly 0.33 times lower risk compared to placebo

b) A is effective compared to placebo with rougly 67% lower risk in group A

Is a) correct by what I'm seeing in the article? Wouldn't a) imply that the relative risk is 0.67 or 67% as it says 0.33 times lower risk? and thus implying that the reduction is 0.33 times placebo?

I fail to understand and can’t find any relevant courses (class is still tbh) online. I can find slot of stats 101 in khan, and was actually 2 units in. I’m not the best with math so I’m taking an alt class my colleges are now offering, pass either this sociology “stat for soc sci” course or statistics.

Can anyone show me a sample question? I know for stats I can just paste a graph and ask for the median mode etc. In this course is it more written or explain this and that? If so idk how this is supposed to be easier. I enjoyed a logic class but I struggled with that one. Just want to make sure I can study before taking this sociology for stat soc science course at my local college. How far is it from statistics?

I've been reading a couple of different statistics textbooks (mostly about regression), and I've noticed that while the theory is mostly the same between them, some of them tend to give different kinds of practical advice. For example, I was reading Regression and Other Stories, by Gelman et al., and it seems like he's just come up with stuff I've never heard of.

In the section on hypothesis testing, he writes about how he doesn't like "type 1" and "type 2" errors, and instead uses "type magnitude" and "type sign" errors. I have never heard of these types of errors, and it almost feels like Gelman is just making it up. He makes some arguments in their favor that seem reasonable, but I'm a bit uneasy accepting advice about something when nobody else I've ever spoken to or read has ever so much as mentioned it (something as huge as Kutner et al's Linear Models textbook never mentions this). And yeah, I know that Gelman is more Bayesian than classical, but my impression is that a lot of statistics is based off of rules of thumb that have been accepted because of years of successful application.

Gelman is just one example, but I hear about all kinds of other "rules" like this that I've never seen in any book. When I search a problem online, I'll get a stackexchange thread about how one type of statistical test is better than another, based on some reasoning I've never heard of ("Welch's test is more powerful for this kind of data, see this simulation").

Even if these approaches are reasonable, I'd like to apply practices that don't require me to take it on faith that an author somehow knows better than decades' worth of practical experience. Of course, they could be right, but the last thing I want is to have to justify to an angry employer why my analysis was wrong, and having to explain that instead of using a tried-and-true method, I followed an ad-hoc practice that someone only came up with a few years ago. Should I just stick to classical textbooks or something, or am I just being too pretentious about it?

And what does this mean logically like why is it happening?

Hi all,

I have 2 sets of questions, (i) is about selecting the ideal method and (ii) is how to report the optimism, discrimination and validation of the approach. Ideally I would also like to report OR, CI, and p-values that meaningfully reflect my selection strategy (i) . I am working using R. I am ok with this being an exploratory / early look needing further validation.

I'm working on a prediction project. My original plan was to use a penalised regression system, ideally LASSO in order to have a select number of variables to report on as the most "unambiguously" predictive. However I've received the data and there are a very small number of events (9 out of n = 90), and 65 variables of interest.

I appreciate that (i) with such small event numbers there is the risk of loss to noise,(ii) there is a significant risk of collinearity in the variables further compounding loss.

(i) Is LASSO (or alt penalised regression) still useable with these numbers? 9 seems very small and 65 variables is a lot. I am working with the team to reduce these numbers in a sensible fashion

(ii) If a penalised regression method still holds, then would bootstrapping to assess the stability of the selected variables (selected >90% of the time considered stable) be suitable coupled with n/2 subsampling for internal validation (>50% stable) of the final model be appropriate (or even doable, given the small event numbers)

(iii) Finally to use a package like hdi in order to obtain OR, CI, and p-values that are aware of the original selection method / n of variables

Many thanks!

i intend to major in statistics and my school has given me these requirements for a laptop on the website but i feel like they might be overkill

OS: windows 11 (64 bit) / macOS 13 or higher
processor: intel i-series (11th gen or newer) / amd ryzen 6000 series or higher / apple m2 or higher
ram: 16+ gb
storage: 500+ gb ssd

would it be possible for me to just use an iPad or a chromebook for stats? my iPad Pro is only a few years old and i don’t want to just toss my chromebook. as far as i can tell there aren’t any specific windows only or mac only things that i need

edit: looking at the replies im not gonna use my ipad for stats so should i get a macbook or windows or does it not matter

Currently I have a knowledge of Calculus I and II, and would like to self study Statistics over the summer since I haven't taken a class in it yet.

With the rise of ai/agent systems, its became very hard and important question to evaluate these systems, can we create a mathmatical framework that can evaluate any system given Task, i don't know how to do this , i have some hypothesis, Let say any System S is built using n subcomponent systems, which can be dependent or independent of each other,

What we say when we mean evaluation E of system S is what are the chances this system will fail P(S will fail), if we know this probability and if its less than some threshold t then we usually say this system is good,
Now S is built using n subcomponents ( S1, S2,...Sn)
Lets define a random variable X= S will fail
X= U{k=1 to n} Sk

P(X)= inclusion exclusion principle over Sk=> we need to know 2^n probabilities to be sure

Is my reasoning correct?
Can someone eval this?

This is feel is the most important question of this century!

My professor was introducing us to point biserial correlation in a course of using spss and he said it’s too hard for us to understand that all the previous class students couldn’t understand it right

I would appreciate any guidance on understanding it and what’s so hard about it ?
Is there any free simple sources that i can use to understand it ?

He said even AI can’t help you with that, that’s why i am concerned with what the source that i would use!

Hello, i want to compare two medians and quantify the difference if it is a large, small, or minimal change. Would I do this by dividing the difference of the medians by the standard deviation?

If not, how do I know if the difference between medians and meaningful or not. Not looking for P values, but will it be necessary?

Mixed parametric and non-parametric tests within same study — is this valid?

Hi r/statistics (or r/AskStatistics),
I'm writing up a psychology thesis and have a methods question I'd love some input on.
I'm comparing two conditions (problem-focused vs solution-focused news) on seven emotion ratings using independent samples t-tests. For each variable I checked normality with Shapiro-Wilk and homogeneity of variance with Levene's, and applied the following decision rule consistently across all seven emotions:
Normal distribution + equal variances → Student's t
Normal distribution + unequal variances → Welch's t
Non-normal distribution → Mann-Whitney U
This means most emotions were analysed with t-tests/Welch's, but two (Joy and Calm) were analysed with Mann-Whitney due to significant Shapiro-Wilk results.
My question is: is it valid to report these side by side in the same results table, and can I meaningfully compare the pattern of findings across emotions even though they used different tests? I'm not directly comparing effect sizes across tests (Cohen's d vs rank-biserial r) but I am drawing general conclusions about which emotions showed larger vs smaller differences between conditions.
Any input appreciated!

I understand the textbook definitions but in layman’s terms I can’t wrap my head around it. In a regression or a CFA, what does a DF actually refer to?