“New drug cure cancer”

“Aspirin could reduce heart attack risk”

“Eating breakfast can help you lose weight”

Health headlines like these flooding the news, but, often contradicting each other.

So how can you figure out which is the true health concern or an actually promising remedy, and which less conclusive?

In medicine coverage, there’s often a disconnection between news headlines and the scientific research they cover.

That’s because a headline is designed to tease attention—it’s more effective when it contains a big claim.

On the other hand, many scientific studies produce meaningful results when they focus on a narrow & specific question.

The best way to bridge this gap is to look at the original research behind the headline.

We’ve demonstrated a simplified research scenario for each of these three headlines to test your skills.

Pay attention intently for the explanation of the first study; analyze at the headline to figure out the flaw.

Assume all the information you need to recognize the clickbait heading flaw is included.

Let’s start with this hypothetical scenario:

A study using mice to test a new cancer drug.

The study includes two groups of mice, one treated with the drug,
the other with a placebo.

At the end of the experiment, the mice that receive the drug are cured, while those that received placebo one are not.

Can you find the problem on this headline:

“Study found new drug cure cancer”

Let’s review. Since the subjects of the study were mice, we can’t draw conclusions about human disease based on this research.

In real life, early research on new drugs and therapies is not conducted on humans.

If the early results are promising, further clinical action conducted to determine if they hold up in humans or not.

Now, you’ve warmed up, let’s try a trickier example:

A study about the aspirin impact on heart attack risk.

The study randomly divides a pool
of men into two groups.

The members of one group take aspirin daily, while the others take a daily placebo.

By the end of the experiment, the control group suffers more heart attacks significantly than the group that took aspirin.

Based on this situation, what’s wrong with the headline:

“Aspirin could reduce heart attack risk”

In this case, the study shows evidence that aspirin reduces heart attacks in men because all the participants were men.

But the conclusion “aspirin reduces risk

of heart attacks” is overly broad;

we can’t assume that results found in men would applicable to women.

Studies often limit participants based on geographic location, age, gender, or many other factors.

Before these findings can be generalized, similar studies need to be conducted on other groups.

If a headline makes a general claim, it should draw its evidence from a diverse body of research, not just one study.

Can you take your skills from the first two questions to the next level?

Try this example about the impact of eating breakfast on weight loss.

Researchers recruited a group of people who had always skipped breakfast and ask them to start eating breakfast every day.

The participants include men and women of various ages and backgrounds.

For a year-long period, participants lose five pounds on average.

So what’s wrong with the headline:

“Eating breakfast can help you lose weight”

The people in the study started eating breakfast and lost weight — but we don’t know that they lost weight because they started eating breakfast;

Maybe, under weight observance inspired them to change their eating habits in other ways.

To put aside the possibility that some other factor caused weight loss, we would need to compare these participants to a group who didn’t eat breakfast before the study and continued to skip it during the study.

A headline certainly shouldn’t claim the results of this research are applicable for general.

And if the study made such a self-claim without a comparison group, you should question its credibility.

Now that you’ve battle-tested your skills on these hypothetical studies

and those headlines, you can test them on real-world news.

Even when full papers aren’t free, you can often still find summaries of experimental design and results in freely available abstracts, or even through the text of a news article.

Individual studies have results that don’t necessarily correspond to a catchy headline.

Big conclusions for human health issues require lots of accumulated evidence over time.

But, in the meantime, we can keep on top of the science, by reading past the headlines. God will

 “It’s not always easy to be both interesting and accurate, but … its better than being exciting and wrong.” –Maria Konnikova

This paper is the second part of the discussion about Learning to Avoid Clickbait. You can see the previous discussion here.

Now, let’s continue our discussion.

“This vitamin can reduce your risk of heart disease.”

“Eating chocolate reduces stress in students.”

“New drug prolongs lives of patients with a rare disease.”

Following is a simplified research scenario for each of these three headlines to test your skills.

Pay attention intently for the explanation of the first study; analyze at the headline to figure out the flaw.

Assume all the information you need to recognize the clickbait heading flaw is included.

Let’s start with the next hypothetical scenario:

We start with the cardiovascular effects of Healthium.

A study finds that participants who took Healthium had a higher level of healthy cholesterol than those taking a placebo.

Their levels became similar to those of people with naturally high levels of this kind of cholesterol.

Previous research has shown that people with naturally high levels of healthy cholesterol have lower rates of heart disease.

So what makes this headline misleading:

“Healthium reduces the risk of heart disease.”

The problem with this headline is that the the research didn’t actually investigate whether Healthium reduces heart disease.

It only measured Healthium’s impact on particular kind of cholesterol level.

The fact that people with naturally high levels of that cholesterol have a lower risk of heart attacks doesn’t mean that same thing will be happening on people who elevate their cholesterol levels using Healthium.

Now that you’ve proficient the case of Healthium.

Try to solve the next particularly alluring mystery:

Relationship between eating chocolate and stress.

This fabricated study recruits ten students. Half begin consuming a daily dose of chocolate, while half do not. Because they’re classmates, they all have the same schedule.

By the end of the study, the chocolate eaters are less stressed than the chocolate-free students.

What’s wrong with this headline:

“Eating chocolate reduces stress in students.”

The conclusion based on a sample of ten students is so forced.

That’s because the fewer participants are in a random sample, the less chance that the sample will closely represent the whole target population accurately. For example, if the entire population of students consists of male and female with 1:1 ratio, the chance of drawing a sample of 10 that’s consist of 70% male and 30% female is about 12%. In a sample of 100 that would be less than a .0025% chance,

And for a sample of 1000, the chance is less than 6 x 10^-36.

Thus, with fewer participants, each individual’s outcome has a larger impact on the overall results—and can, therefore, modify trends of research result.

There are also reasons for scientists to run small studies.

With a small sample, they can evaluate whether the results are promising enough to run a more comprehensive,
expensive study or not.

And some research requires very specific participants that may be impossible to compile in large numbers.

The key is reproducibility

If it draws a conclusion from one small study, that conclusion may be suspect. but, if it’s based on many studies that have found similar results, of course, it’s more credible.

One more puzzle remains.

In this scenario, a study tests a new drug for a rare, fatal disease. In a sample of 2,000 patients, the ones who start taking the drug live longer than those who take the placebo.

This time, the problem is slightly different. What’s the thing you’d like to know before deciding the headline eligibility:

“New drug prolongs the lives of patients with a rare disease.”

Before make a statement, you should to know how much the drug prolonged the patients’ lives.

Sometimes, a study had results that, while scientifically valid, don’t have much bearing on the real world.

For example, one real-life clinical trial of a pancreatic cancer drug found an increase in ten days of life expectancy.

On the next time you see a medical headline, inspect at the science it’s reporting on.

Even when the full papers aren’t free, summaries of experimental design are still accessible and the results are freely available in abstracts, or even within the text of a news article.

It’s exciting to see scientific research to be covered in news, and it’s important to understand the studies’ findings.

Excellent!