“All work presented is my own, and I have followed all rules for collaboration. I have not used generative AI on this project.”
I think the authors mean that statistical significance being used less means we will rely less solely on the p-value for statistical significance and use statistical thinking in a more contextual way. An example of this would be if you find a small p-value for something with a relatively small difference 9it might not mean anything in the real world, like finding a significant difference between the weight of 2 ants… who cares. It’s just more about being more thoughtful and open to thinking of possibilities besides the answer the p-value gives us.
Using an arbitrary number like p < 0.05 to declare results significant or not leads to a possible false sense of certainty or maybe the absence of an effect you thought was stronger than it is. Labeling something as statistically significant can lead people to believe that the association is true and important, and labeling something not statistically significant can mislead people to think the association is useless and false. The problem with this is that the p-value alone can’t tell us this information. A p-value of 0.049 and 0.051 are treated as categorically different under the 0.05 rule, when in reality the actual difference in evidence they provide is minimal.
Based on the information, I strongly agree with what the author is saying, especially for scientific publishing. Relying on statistical significance as a gatekeeper for reporting and highlighting findings is bad for the integrity of scientific publishing and research. Instead of relying on the p-value, researchers should be more open and honest with all of their results regardless of statistical significance. This could be through providing the size of effects, more confidence intervals and just considering the broader context of the research.
I agree strongly again that the one size fits all approach to statistical inference is an inappropriate expectation. The authors explicitly state that it is impossible to find one majestic solution to replace the role of statistical significance and p-values. Scientific research is often exploratory and sometimes uncertain, and a one-size-fits all approach is an inappropriate expectation given the varying nature of research questions, data and contexts. Using thoughtful and contextual approaches seems to be the most fitting.
Statistical thoughtfulness to me seems to represent a deep and comprehensive engagement with statistical aspects of research that goes beyond the application of formulas and the obtaining of statistical significance. It involves a critical and well rounded approach to design, analysis and interpretation grounded in the context of the research and the understanding of uncertainty. This can be done by clearly stating the research questions, reporting p-values as continuous quantities instead of significant or not, and explicitly discussing limitations and holes in research.
I think the main problem these scientists are talking about is the way we label things and the language we use. When an average person who may not have statistical background hears that the findings of a study are not significant, they are going to take the researcher’s word for it and probably not understand the full context of the problem. The words significance and confidence can be misleading because they lead to overconfident claims and possibly misleading results. I agree with using the term compatibility instead because this way we think of p-values as measuring the compatibility between the hypothesis and the observed data, and don’t rely on them for statistical “significance”. There is a persistent confusion between statistical significance and actual importance that this use of language causes.
A quote that stuck out to me is in section 1 on page 5, that says, “Don’t believe that your p-value gives the probability that chance alone produced the observed association or effect or the probability that your hypothesis test is true”. This quote stuck out because it directly addresses a very prevalent and fundamental misunderstanding of what a p-value actually represents, and I was blindly following what I had been taught about p-values until I read this paper. The notion that the null or alternative hypothesis are true or false based on the p-value is a long running misconception that leads to limited and often misinterpreted information. It made me think about how deeply ingrained this incorrect interpretation is and how much it might skew our research findings across various fields of study.