rise and fall of p-value: a lesson to be learnt.

Last year, after many years of p-value abuses, the american society of statistics in a revolutionary move published an instruction on how to use p-value. YOU HAVE TO read this before hand, if you ever gonna use p-value. But, here I want to pay attention to a more general mentality, which I believe is the caused of this misuse of p-value.

Since you may not have enough time to glance over the paper, let me just briefly summarize my understanding in few words, although this does not replace the whole manual. It basically says that the value of p-value for large p, does not have any significance by itself. Whenever, p is less than a particular threshold (say <0.05) then we can say that it is less likely your data with a true null hypothesis. Or more scientifically, you can strongly reject the null hypothesis, that is, there is an effect (null hypothesis is assuming that there is no effect). However, the size of effect is not determined by p-value. Additionally, you should always consider that the unusualness of your data.

In modern science, especially, during last 50 years numerical value became more important than ever. Previously, we had qualitative and quantitative understanding, now the former is losing the race to the quantitative universe. Crudely speaking, for me quality is not necessarily non-mathematical or numerical, but something that you can not specify it with a finite set of numbers. IQ, SAT score, GPA are just a few examples of many, which is a single number to specify Intelligence, or Academic preparedness, or academic valuation of someone (which of course, is a sham). In this particular universe, only (finite) number matters, and it usually take while to come up with a good MEASURE which turns multi-variate systems into a single (or a few) number. In particular, dealing with large set of events and data, is one of those areas that turning knowledge into number is usually hard. So the measure, such as p-value come to rescue!!

Now, all scientist in life sciences, environmental sciences, and psychology who does not want to spend enough time to study to understand statistics with underlying assumptions, just take the equation and leave the rest (it is always said: the devil is in the detail). Few years ago, there was a research paper titled: “Why Most Published Research Findings Are False”  which was really disappointing, later on there was very depressing while courageous statement in Nature article in 2014, by Steven Goodman (statistician@Stanford) stating that: “The wake-up call is that so many of our published findings are not true.”.  In the light of these findings, most likely we should rethink and reevaluate all those suggestions by FDA and other agencies, which are heavily based on those studies! The depth of disaster is so deep that nobody wants to even scratch the surface.

Anyway, I do suggest not only restricting the daily usage of p-value, but also leave the whole mentality that everything can be turned into a single number! Sometimes, it is just more than that. I don’t say we should not try to understand things quantitatively, in contrast, what I am saying is that sometimes quantifying with a single number is over simplification and underestimate the reality.

 

Advertisements

Most scientific papers are probably wrong!

I remember talking to a friend of mine mentioning the fact that I am really skeptic about the results of the studies in the multi-variable fields with no underlying theories, such as life sciences, social sciences, and earth sciences. My argument is as following, there are many parameters to change, and since we have no theory to compare the experimental predictions with, we could easily get misled by the results. In fact, in most of these fields we can’t even answer the basic questions, such as how many parameters are there to estimate the right sample size!

Recently, I came across this article in newscientists which basically proves that many scientific results are probably wrong! What I like about physics, in general, is that we have theoretical guidelines which help us to better understand the experiments. Of course, our theories are based on many approximations and assumptions, however, it is still unreasonably accurate. In fact, the main difference between theoretical physics and applied mathematics is how to lay down different approximations. This is one of the main reasons that, I believe, we have a steady progress in physics, and sorts of a random walk in other fields. By the random walk, I do not mean that we are not making progress, of course, we are. However, there is no sense of direction in what we are doing, at least by an outsider like me!

رعایت قانون اختیاری نیست

اخیرا موضوعی رو خوندم که توش اشاره داشت که اما بنینو، وزیر امور خارجه ایتالیا در هنگام ورود به ایران اصرار داشت که بی‌ حجاب وارد شه. صرف نظر از اینکه آیا شخص بنده با روسری اجباری موافق یا مخالفم، به نظرم کار ایشون یخورده غیر منطقی‌ و خارج از شئونات دیپلماتیک بوده، و در واقع احساسی‌ عمل کرده. از آونجایی که برخی‌ هموطن‌های ما هم دچار این مباحث میشن، دوست دارم که نظرم رو راجع به این موضوع بیان کنم.

خانوم اما بنینو باید این مساله رو درک کنه که حجاب در ایران از بحث فقهی‌ و مذهبی‌ خارج شده و به بحث قانونی تبدیل شده، و رعایت قانون بر همگان بویجه افرادی که وارد ایران یا هر کشور دیگه‌ای میشن لازم الاجراست. مثل اینکه، بر فرض داشتن سلاح تو کانادا غیر مجازه، و یه آمریکایی بگه من می‌خوام با کلتم وارد بشم.

در ادامه ظاهراً اشاره شده بود که وقتی‌ که دیپلماتهای ایرانی‌ به کشورهای دیگه میرند، کشور میزبان به احترام اونها مشروب روی میزا نمی‌گذارند. این نشان فرهیختگی اون کشوراست، ولی‌ باید باز اشاره کنم که گذاشتن شراب روی میز یه بحث عرفی در خارج از ایران هست، نه یه بحث قانونی، در حالیکه پوشیدن حجاب یک بحث قانونی در ایران هست.

اینکه حالا چرا حجاب اجباریه، و چجوری می‌شه عوضش کرد و غیره کاملا بحث‌های جدایی هستند که در جای خود باید نقد بشند. ولی‌ این نکته مهمه که درک کنیم، در هر کشوری زندگی‌ می‌کنیم، یک سری قوانین هست که ما با اونا مشکل داریم، ولی‌ مخالفت ما به ما حق عدم رعایت قانون رو نمیده. بلکه ما باید در چهارچوبهای قانونی تلاش کنیم که اون قانون به سمت بهتر شدن حرکت کنه.

Are we affected by Electromagnetic field around us?

A close friend of mine, Alireza, sent me a link to a recent documentary on the effect of electromagnetic field on human and other living bodies life. You can watch the movie here.

This is a great effort to show the recent and old studies on this issue. However, there is no interview with critics to hear both sides of stories. Here I would like to present my view on this as well, as following:

1) it is not hard to assume that we are synchronized with our environment, from visible senses such as colour, smells, etc, and of course non-visible senses such as magnetic field.

2) It is very bad habit of industrial world that they expose society to something before they make sure their mid or long term problems. For example, cigars, plastic, and most less-comntroversial of all asbestos!

3) It is very bad habit of people of industrial world to trust to what their governments say without inquiring enough evidences. However, all of us know that government does not fund many long term researches, including chronic health related problems. Particularly, if this is not straightforward to see it immediately.
4) To me, cancer is simple; some cells get irritated, and then get out of control biologically leading to cancer. So to speak, any constant modulation of a group cells would cause a cancer. So, it is very likely constant exposure to EMF would lead to severe health problems such as cancer!
However, strong statement in this movies about extinction of birds and bees and so on, does not seem very obvious to me. Since the same type of evidence are being used for global warming and so on.  Or even I do remember that while ago breast cancer was also was associated with increase exposure to plastic, etc.
I believe that we need to be cautious, and we need to be more critical what we use and what we eat.
Stay Well,

Incentives suppress productivity for creative jobs!

I like to encourage you all to watch the following TED talk in which the speaker present some scientific findings showing that incentive worsen the productivity for creative jobs.

At the end of this talk, speaker present some ideas for enhancing productivity for creative jobs, namely, Autonomy, excellence, and purpose. So, the idea is that we should help people excel in their field of interest and then let them to work freely, if we are really looking for increasing productivity.

Noting that the university supposedly should be a house of creativity, these advices mean revolutionizing how the current universities operate. As I mentioned in my earlier post,  the current style of university, particularly, graduate studies not only does not improve creativity, it actually diminish creativity, as we can see. The current style of university even fail to help student excel in their field of interest by preoccupying them with some knowledge without providing enough time and resources for students to deepen their understandings.

I would expect from the universities to have a courage to do at least as good as google, in providing 20% of free time to graduate students to work on something that they feel is important and working in the group that they want to.

Education is designed to suppress creativity!

After years of being involved in educational system, I came to this believe that the educational system is designed to suppress creativity. One of the most important aspects of creativity is that it depends on one particular person physical, mental, and societal characteristics. So, it is something that one could  and would do while someone else might do it in other way. So to speak, the particular creativity is not transferable among people (it is like finger print).

On the other hand, once you are designing a system, you cannot afford to respect diversity. This is the downside of every system, that is, it has to consider every of members of society indistinguishable from one another in making laws. The foundations of any system is based on something  in common in most people or something that should be enforced, anyway. The education system is no exception in this regard. It is designed to deliver some basic materials necessary to do some tasks in a given time. It does not depend on any of students characteristics, and it does not care about those features at all.

To my understandings, the educational system is designed the same way as the highway traffic system. The speed limit, and the deriving rules, are designed in a way that is independent of the type of particular vehicle. This may have some justifications for cars, but for human and particularly education of human being, I don’t think so.

Educational system is not optimized to teach every students how to enrich their abilities and talents, but, it is completely the other way around. It is based on society needs and characteristics. There are some tasks in the society needs to be done, and someone should do it, so educational system is optimized so that it can train enough number of people to do those tasks. To support my words, I encourage you to read UC Berkeley mission statement as one example. So, educational system is designed in a way to uniformize all the students, however, make sure that they receive some minimum education set by some standards. There is no more into it.

I should further mention that this problem resists to stay even in the graduate education system. Even there, usually PhD projects are defined before the students arrival, and basically, the nature of project defined for students does not depend on the students abilities, as it depends on the funding and possibly the job market.

Furthermore, usually the laws of any particular system are enforced below the average response, since after all this law should be respected by everybody not matter what. So, even within this education system, it slows down the fast-paced students. For fast-paced students it is like deriving a Porsche in Canadian highways, where still you can not go faster than 100 km/hr legally.

The conclusion is that if you are different, and if you can understand materials quickly, basically the educational system does not work well for you. The longer you stay in the system the more you learn to forget your own features and replace it with what you have been thought. It is no accident, it is designed this way. This problem is more obvious by looking at the grading system, where all the students actions boils down to a number, independent of how students have done it! (For example, you can solve a mathematical problem in the most ingenuous way, and receive the same mark as someone who has done it in the most doll way.)

I should mention that there are people in academia who remained creative, and people often use some of their creativity in solving problems, but my point is that the educational system is not based on students creativity. To be honest, I think if we want to have a central educational system, this is the price we have to pay (I keep open the possibility of not having central educational system).

I do believe that for higher education (especially PhD programs) it is possible to define students-depend type educational path. So the job of university and the corresponding advisor should be understanding students abilities, and then mutually define a project suitable for him/her, and then recommend him to surpass his abilities through the project (usually it is the other way around, i.e., there is a defined project and faculty members are looking for someone to do it).

At the end, I encourage you all to view the following TED talks  on the education :

http://www.ted.com/talks/ken_robinson_says_schools_kill_creativity.html.

http://www.ted.com/talks/andreas_schleicher_use_data_to_build_better_schools.html

James Randi TED talk against Psychics.

Recently, my wife posted a link on a TED talk presented by James Randi against psychic claims. Here is the link (with persian subtitle): http://www.ted.com/talks/james_randi.html

That is very true that with our current scientific knowledge we can not understand everything, and even those things that we (think that we) understand do not provide a good picture of some of the observations.

However, what Randi is suggesting is very pleasing. He is claiming that if anyone could provide some measurable psychic effect, he will give him/her 1 million dollar. As one of the measurable experiment that he has in mind is to make predictions about something like a coin flip. Say they flipped coins many times and psychic person guessed correctly 52% of the time. Given enough trials, that percentage would be enough. Or in one other place he (Randi) asks, if someone just by looking at picture could say the person is dead or alive is another testable measure to win the prize. The experiments look very easy to perform and all are based on deviation from fair random selection.

That is true that there are many things that we do not know, and there are many things that we see and can not explain, but this should be the source of our curiosity and not the source of our deception. Especially, his (Randi’s) emphasize on the role of mass media in spreading “not tested claims” in a nice and fancy cover is very important to notice.

A friend of mine reminded me that there are some experiments suggesting some unexplainable effects, for example PEAR lab in Princeton, however some evidences in some experiments has not much in common with what psychic claims. What I understood is that people like Randi are against those people who makes claims about some (deterministic) “super/unnatural power” while they are not even trying to prove it. Sadly, there are many people who fall for these people. By paying to those people we are supporting them, so they get more powerful and once they have all the power, it is very hard to stop them. For example, homeopathic people should try to explain why he did not fall into deep sleep after taking 22 pills (assuming those pills are what he claims to be).
There is a huge difference between experimenting something and practicing something. In my opinion the path is rather clear. We should move toward understanding what we don’t know based on what we know. However, all society wide practices should be based on what we know, or at least we think we know.
I should have also mentioned that some of the skeptics make it hard (and sometimes impossible) for others to pursue new directions which may deviate from the main stream experiments. However, the presence of experiment such as PEAR lab experiment, shows that there are ongoing experiments in this direction (however not many). Dogmatism is against any progress in understanding and we should stay away from that. However, we should not get confused between being open minded and supporting balloonies. I believe we should continuously reexamine what we consider as “knowledge” and keeping eye for new developments.

Misuse of academic metrics!

Today, a friend of mine sent the following link about uselessness of academic metric for evaluating an applicant: http://www.sciencemag.org/content/340/6134/787.full

To briefly summarize, this note says that the scientific metric such as IF (impact Factor) is/was not intended for measuring a scientist but it is only a measure of academic journal. This note also warns about the destructive misuses of academic metrics.

Although I completely agree with those comments,  what is the solution then? We all have heard the same story from leading scientists mentioning that we should not do this or that. However, I did not come across any solution offered by these experts. Assume you are in a hiring committee, and there are 200 applicants (which is relatively low). What would you do in order to establish who is the best? Even reading the applicant files, with current measure, would take weeks. Without those metric, what is the other option?
I believe that the problem with metric arises not because other people do not understand that those metric are useless, but since they are THE ONLY practical measures. In my opinion, we need to reduce the total number of applicants! That means North America, should stop its PhD making factory. The total number of job positions (academic and industry) per area should be around the same number of PhD graduates, i.e., around the same number of PhD students admitted per year.
University in the present form are like inconsiderate parents, who make a lot of childs per year and has no time; first to make sure every child gets enough care, second to make sure that every child will end up somewhere decent (left aside a happy life).
Please let me know if anyone has a better solution!
PS: There is another interesting note on the science about how to encourage new science here:   http://www.sciencemag.org/content/320/5878/849

NSERC substantial Postdoctoral drop!

Just recently I came across NSERC website where the statistics of last year scholarships and awards is posted:http://www.nserc-crsng.gc.ca/NSERC-CRSNG/FundingDecisions-DecisionsFinancement/ScholarshipsAndFellowships-ConcoursDeBourses/index_eng.asp?Year=2012

Those who study in Canada have heard about the several budget cuts during last year and a year before. However, I am surprised this budget cut mostly affected postdoctoral program compared to graduate scholarship program. Particularly noting that CGS D worth almost the same amount as PDF money wise. For example just in 2012, only 98 PDF (~7.8% of applicants) were awarded while 233 CGS D scholarships (~14%) were awarded. Not to mention the other 426 PGS (~26%) awards which worth more than 200 PDF money wise. In terms of money, the NSERC has funded more than 400*40K for graduate students, while less than 100*40K for PDF. More importantly, finding a postdoc position is relatively much harder than PhD position.
There are also other funding opportunities such as MITCAS or IPS (industrial postdoc). Furthermore, the industry partners (especially in Canada) are not well aware of such programs, and when I contact them they get all confused about what to do, and finally decide not to go through with it. They simply do not have corresponding person for such new program, and it turns to be hassle for secretary to read through all the conditions , requirement, policy, etc. So, they prefer not to do it. I should mention for cutting edge researches which is rather far from present industry, especially in Canada, it is impossible to find an industry partner.
I understand that government desires to have more industry related experiences, but just cutting funds all at once does not help in that direction much. I believe that funding more graduate students, while not supporting them for their career does not seem reasonable (at least to me as a student). Particularly, such a large drastic budget cut at once would lead to huge confusion between students, as I can see in my friends and classmates. More importantly, the other national labs as big as CSA (Canadian space agency) has also suspended their postdoc program which makes things even more chaotic.
In my opinion, the current policy would transform universities to industrial R&D, which is a free ride for industries. Low pay students works for years with government budget for industrial projects. I believe that if a faculty member wants to work on something which is mostly related to present industry, he or she should also receive the money from the related industry instead of government. On the other hand, government should supports those fields who can not receive funding from industry by their nature. The bottom line is that industries should invest on the present and the government should invest on future.

PhD mass production.

If you are in higher education for sure you are well aware of the whole publish or perish policy. Or the competitive academic position, or small number of industry position for PhDs, etc. The question is when this problem started to appear?

For me this topic is similar to “food mass production”.  Chemical nutrients, pesticides, and recently bio-engineered sources all came to help to have a successful food production, possibly to remove world hunger problem. Now we know that not only we did not diminish the world hunger, we lost all the taste of fruits and vegetables. Not to mention the possible side effects. All these problems can be translated in one to one correspondence to academic mass production.

The funny part is that these two processes seem started around the same time. It suggests the fact that, these ideas came from the same master mind. Nowadays, universities work like a factory, which aim to engineer new ideas. Sure, we get more people to learn and practice some stuff, but the question is this efficient or useful in making new ideas?

This type of approach to universities may somewhat work for applied fields, where students trained to acquire some expertise. However, this does not work for abstract sciences, such as philosophy, physics, chemistry, mathematics, etc where pure idea is the main concern. I should add that these abstract sciences are the main hub of future developments. The current approach of universities would lead to the rise of new technologies, and fall of abstract sciences, i.e., fall of future technologies. Now that I have spent more than 7 years in Canada, I can feel more than ever such problems.

World leaders should know that, fundamental sciences which only relies on the deep intellectual abilities are far different from applied fields which mostly relies on practice. Don’t get me wrong, I do not say that engineers are not geniuses, actually some of them are. What I am saying is that with no intelligence it would not be possible to create an idea with practice. (By ideas I do not mean Apple or Microsoft, I mean Newton laws of gravitation, relativity, or Euclid laws of geometry, etc)

In my opinion, for abstract/fundamental sciences university should act as a fertile environment to provide resources such as books, experts (faculty members), labs to help new ideas grow. You can not enforce new ideas, and you can not engineer new ideas. You can only provide environment and let new ideas emerge.

PS: This post was triggered by the recent article sent by my beloved wife. http://www.guardian.co.uk/higher-education-network/blog/2012/may/24/why-women-leave-academia?CMP=twt_gu

Additionally I like this link which is somehow related to this topic:  http://www.wired.com/opinion/2013/05/so-you-know-that-10000-hours-makes-an-expert-rule-bunk/

Recently, Macleans published something interesting while disappointing for PhDs in Canada:

http://www2.macleans.ca/2013/06/03/an-academic-dead-zone/