I read all the articles linked to above and many that they link to. They highlight a troubling issue that affects not only food science but arguably every hard and soft scientific and engineering discipline. People publish an awful lot of sloppy statistics, either as a result of their motivation to find what they want to find or just simply their innocent ignorance of the requirements of rigorous statistical analysis. It got to the point long ago that I cringe whenever I read about analysis results being reported with p values. Such results are descriptive but, by definition,
posteriori, and, as the cited articles show, deserving of skepticism, given the prevalence of p hacking, whether unscrupulously or naively motivated. If you really want me to believe you, establish an
a priori criterion (e.g., an alpha value such as 0.05 for a t-test or paired t-test) and stick to it.
Don't even get me started about people performing the wrong type of statistical test, i.e., the one type they know how to do or the one they copied from someone else who seemed to know what to do, rather than the type that is appropriate for the data and the hypothesis they are trying to test.
Another pet peeve of mine is when multiple conclusions are reported for a single data set based on multiple statistical tests conducted at a given confidence level. There are robust ways (not usually taught in the first or even second college statistics course, and I am referring to those beyond first-semester Tukey tests, by the way) for conducting multiple statistical tests and for determining what overall confidence level can be attributed to the results of multiple tests at any given level of confidence for a single data set. As a simple example, four separate statistical tests conducted at a 95% confidence level have an overall simultaneous confidence level of not 95% but rather 95% to the fourth power = 81%. If you want to have 95% confidence in the results of four independent statistical tests at the same time, you have to conduct those tests at individual confidence levels of 98.75%. This is a mistake that I have witnessed often and believe that many researchers in many fields make routinely out of ignorance of correct statistical practices rather than an intent to deceive.
I remember a meeting with an influential research sponsor who took issue, fiercely, with our commenting on several points on which the data did not support a conclusion. "Don't tell me what you don't know," I remember him saying, "tell me what you know!" That would make for a better one-paragraph press release, but, sorry, no, that's not how hypothesis testing works. Just because something can't be proven with the data available doesn't mean for sure that it couldn't be proven with other data. So when a hypothesis fails a statistical test, all we can say in good conscience is what we can't prove, not that we've proved the opposite. This is hard, though, for the public and the media to grasp.
As a result, I'm a little troubled by the conclusion reached by the author of the Mother Jones article. The real solution is not to just be more skeptical of research findings. At first, that seems like a step in the right direction, but upon reflection, it strikes me that it's only marginally less random than being no less skeptical of research findings. The real solution is to educate yourself better, or seek out better advice, about what constitutes rigorous statistical analysis and how to recognize signs of analysis techniques and reporting methods that are not rigorous. I don't hold my breath, however, about the public or the media (other than the 538 blog, for example) wanting to go to that trouble.
Last edited by
Katie on September 28th, 2018, 1:24 pm, edited 1 time in total.
"Your swimming suit matches your eyes, you hold your nose before diving, loving you has made me bananas!"