I love the idea of science
Science is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe. It is an ideal, and if you have paid any attention to what is happening on these web pages the application of science based process is what I have been harping about.
Sadly the problem with science is “not enough science”. Often bad ideas are entrenched by old scientists, invested in their obsolete ideas, unwilling to embrace new information. The following PDF is by Sean Last and Ryan Faulk.
Robert Wilson agreed:
“Every fact of science was once damned.
Every invention was considered impossible.
Every discovery was a nervous shock to some orthodoxy.
Every artistic innovation was denounced as fraud and folly.
Everything that is man-made and not given to us by nature, is the concrete manifestation of some man’s refusal to bow to authority.
We would be no more than the first apelike hominids if it were not for the rebellious, the recalcitrant, and the intransigent.’’
~Robert Anton Wilson, 1932–2007~
The problem of peer review:
In 2005, three MIT graduate students Stribling, Aguayo and Krohn wrote the program SCIgen to generate fake papers. As of 2013, at least 16 SCIgen papers have been found in Springer journals.The program SCIgen is available on the internet free to download and use by anyone.
The fact that journals accepted a junk article is not interesting. What is interesting is that journals run by Sage, Elsevier and Wolters Kluwer all accepted bogus paper. These sorts of findings have been replicated by the British Medical Journal by Fiona Godlee in 1998.
In 2008, Fiona Godlee submitted papers for peer review with glaring errors which were subsequently published. Her paper “What errors do peer reviewers detect, and does training improve their ability to detect them? Fiona Godlee et. al. , 2008 : can be found here. The takeaway is that the peer review process does not guarantee that the study is relevant, useful or true.
Strangely studies showing that most studies usually aren’t replicable have been replicated many times. Scott Armstrong from The Wharton School wrote a scathing evaluation of what peer review is like in his paper “Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation”. Scott listed a number of problems all which suggest that the process of generating peer reviewed papers lack a level of capability maturity.
-Reviewers lack relevant credentials
-Reviewers often work anonymously
-Reviewers often do not get remuneration
-Reviewers on average spend two to six hours in reviewing a paper
-Yet they often wait for months before doing their reviews
-Reviewers seldom use structured processes.
-They are not accountable for following proper scientific procedures.
-Reviewers’ recommendations often differ Cicchetti (1991).
Lacking a process, lacking metrics, lacking training, the process is operating at the Initial stage of capability maturity.
What can be done? It is amazing the number of recommendations that can be found using google scholar “peer review reform“. I will leave the recommendation to someone more qualified, and the implementation to someone who really cares; suffice to say people who do not understand the importance of capability maturity in workflow process control, measurement and improvement do not understand its importance in research. No surprise.
The above mentioned examples are from an attached article written by Sean Last and Ryan Faulk.
A short quote about Great Scientists worth emulating
Five geometers—Clairaut, Euler, D’Alembert, Lagrange and Laplace—shared among them the world of which Newton had revealed the existence. They explored it in all directions, penetrated into regions believed inaccessible, pointed out countless phenomena in those regions which observation had not yet detected, and finally—and herein lies the imperishable glory—they brought within the domain of a single principle, a unique law, all that is most subtle and mysterious in the motions of the celestial bodies. Geometry also had the boldness to dispose of the future; when the centuries unroll themselves they will scrupulously ratify the decisions of science.
- François Arago, Eloge of Laplace (1842) as quoted by Dirk Jan Struik, A Concise History of Mathematics (1948)
Back to Kidney transplant