Every day we are bombarded with stories of medical “breakthroughs”, cures for cancer and claims that substance X causes disease Y. But how do we know which reports to trust? Are we being fed the truth or has it been bent beyond all recognition by PR departments, journalists and headline writers? Below is a rough guide to getting to the bottom of news stories about medical research.
Where is the study reported?
Whether the study has been reported on television, radio, print or online, you must first judge the quality of the media outlet on which the story is presented and whether bias may be present. Gone are the days in which “mainstream” media was more reliable than online media but you can still choose a news source you know to be trustworthy. Sites such as Newstrust and mic can help you identify these. Examples of trustworthy sources include the New York Times, Reuters and BBC News. In addition, The Conversation is an independent source of news and analysis written by academics and research scientists. It features topical stories and corrects misinformation emanating from less rigorous media sources. Specific medical research institutes may also have blog posts written by their own experts in the field.
At what stage is the study?
Sometimes researchers may issue a press release before they start their research, when research is in progress or before their work has been reviewed and published. These situations are the least reliable. Ideally we will know whether the researchers are the first to find such results (they will definitely say if this is the case). If so, it’s possible that a single study may not be able to replicated by others. As a research area matures, researchers will spend time writing a “systematic review” or conducting a “meta-analysis”. Systematic reviews are “studies of studies” which identify, appraise, select and synthesize all high quality research evidence relevant to a specific research question and make conclusions about whether sufficient evidence exists to support a research finding. Among the most trusted examples of these are Cochran reviews. New treatments for diseases must go through these processes before entering the clinic.
What organism has been studied?
This may sound like a strange question but medical research often starts with human cells in a dish. Showing that a drug can kill cancer cells in a petri dish does not mean it will kill cancer in a real person. And new drugs require testing on animals before humans and surprisingly, news reports may sometime fail to mention whether the study is of mice or men.
Is sensationalism used?
Studies labelled “breakthrough” and “cure” rarely are. Research takes years and the vast majority of research is achieved in small advances. See also “what stage is the study at” above. Check the language used; researchers often suggest that their research “may lead to” a cure.
How thorough and critical is the news report?
A bonus if an independent expert is quoted; this provides a valuable independent assessment. Lazy news sources may not even speak to the original researchers and will quote the original press release verbatim, so if in doubt look up the study.
Go to the source
Another bonus if the source of the news story is quoted. Journalists may include just the journal name but if you are lucky, the story will link to the article itself. However, there are a couple of barriers to overcome once you have come this far. Firstly, if the study is not freely available, you can search for it. Most journals are, however, still pay-per-view but at least you will be search for and read the summary or “abstract” for free. Once you have done this, there are excellent guides that can help you understand and evaluate research papers.
“But I don’t have time to do all this checking!”
In this case, someone may have already done the checking for you. The searchable Behind the Headlines from the UK National Health Service’s NHS Choices web site is the one site you need. It breaks down each story into sections of where the story came from, what kind of study it is, what the basic results were and how they were interpreted. Each short article ends with a conclusion summarising the strengths and weaknesses of the study followed by links to the original research article and examples of how the mainstream media has covered the story.
Finally, if I were offering two pieces of advice: start skeptical and if you disagree with the way the story is covered, write a comment on the story’s web site or complain about it. After all, every news outlet should share the motto of the New York Times: “All the News That’s Fit to Print”.
Acknowledgements
Many thanks to Anne Byrne for proof reading
Dr Chromo: Nature Boy 