Last year, after many years of p-value abuses, the american society of statistics in a revolutionary move published an instruction on how to use p-value. YOU HAVE TO read this before hand, if you ever gonna use p-value. But, here I want to pay attention to a more general mentality, which I believe is the caused of this misuse of p-value.
Since you may not have enough time to glance over the paper, let me just briefly summarize my understanding in few words, although this does not replace the whole manual. It basically says that the value of p-value for large p, does not have any significance by itself. Whenever, p is less than a particular threshold (say <0.05) then we can say that it is less likely your data with a true null hypothesis. Or more scientifically, you can strongly reject the null hypothesis, that is, there is an effect (null hypothesis is assuming that there is no effect). However, the size of effect is not determined by p-value. Additionally, you should always consider that the unusualness of your data.
In modern science, especially, during last 50 years numerical value became more important than ever. Previously, we had qualitative and quantitative understanding, now the former is losing the race to the quantitative universe. Crudely speaking, for me quality is not necessarily non-mathematical or numerical, but something that you can not specify it with a finite set of numbers. IQ, SAT score, GPA are just a few examples of many, which is a single number to specify Intelligence, or Academic preparedness, or academic valuation of someone (which of course, is a sham). In this particular universe, only (finite) number matters, and it usually take while to come up with a good MEASURE which turns multi-variate systems into a single (or a few) number. In particular, dealing with large set of events and data, is one of those areas that turning knowledge into number is usually hard. So the measure, such as p-value come to rescue!!
Now, all scientist in life sciences, environmental sciences, and psychology who does not want to spend enough time to study to understand statistics with underlying assumptions, just take the equation and leave the rest (it is always said: the devil is in the detail). Few years ago, there was a research paper titled: “Why Most Published Research Findings Are False” which was really disappointing, later on there was very depressing while courageous statement in Nature article in 2014, by Steven Goodman (statistician@Stanford) stating that: “The wake-up call is that so many of our published findings are not true.”. In the light of these findings, most likely we should rethink and reevaluate all those suggestions by FDA and other agencies, which are heavily based on those studies! The depth of disaster is so deep that nobody wants to even scratch the surface.
Anyway, I do suggest not only restricting the daily usage of p-value, but also leave the whole mentality that everything can be turned into a single number! Sometimes, it is just more than that. I don’t say we should not try to understand things quantitatively, in contrast, what I am saying is that sometimes quantifying with a single number is over simplification and underestimate the reality.