How to deal with conflicting studies

MrRippedZilla

Retired
Joined
Mar 3, 2014
Messages
1,706
Reaction score
3,522
Points
153
I often see people say stupid things like "you can find any study to support your view on the internet". This comes with the presumption that every positive study can be countered with a negative one = all research is refutable = therefore its best to just ignore it all.
This presumption is bullshit and stems from people being frustrated with the quest to find elusive answers and preferring to just completely give up instead. To use creatine monohydrate as an example, for every study that you find showing it doesn't help performance I can find 2-3 that say it can: "Of the approximately 300 studies that have evaluated the potential ergogenic value of creatine supplementation, about 70% of these studies report statistically significant results..."

To quote myself:
.... I often here people dismiss science because every study has an opposing study but this is down to your inability to critically review the data and spot the differences - something we are here to help with.
That is what this thread is aiming to do. Help you interpret research more effectively and avoid the 1:1 evasion line :)


Weight of the evidence

The first thing to keep in mind is that not all evidence is of equal value/weight and we have a grading system in place when it comes to science that is worth highlighting:

Grade A - randomized controlled trials (RCTs) with a large body of data
- This is considered the "gold standard" of evidence and the only type of research capable of showing cause & effect.
I want to point out one major flaw that even this "gold standard" evidence has and that is a lack of external validity. What this means is that since RCTs can control all the variables in question they may be impractical in the real world because, well, you cannot control all the variables in question - a big strength for research may be a weakness when it comes to applicability. Just something to keep in mind.

Grade B - randomized controlled trials with a limited body of data

- Similar to A except with a smaller number of trials that may lack the sample size, consistency of results, etc.

Grade C - non-randomized trials/uncontrolled/observational data

- All the data in this category is correlational in nature and can only be considered hypothesis-generating at best. This means that it is capable of giving us ideas to look into, with actual RCTs, but that's it - practically, it is useless information for most people beyond providing additional support to the results of RCTs.

Grade D - panel consensus judgement/opinions of experts
- This area only comes into play when we don't have enough data from the better grades or if we need help putting the existing data together. This is a more dynamic grade (it may become more valuable depending on the data being looked at - meta analysis of RCTs vs observational studies for example) and isn't necessarily one with the weakest "weight" attached to it.

Grade E - Anecdotal data
- This consists of opinions with no scientific evidence to support it whatsoever.

So in total, the order is A-B-C-E with D being a floating variable.
Keep in mind that Grade A claims are rarely newsworthy because, well, they rarely give us any "new" information. If you rely on the media for your information then it's almost always going to be Grade C (observational data) because that can give us catchy, meaningless, headlines that take full advantage of the fact that the general public are not aware of the uncontrolled variables, inability to establish a cause, only capable of showing a correlation, etc.


Research design elements

Alongside the grading system that needs to be taken into account when weighing up the strength of the evidence, we also need to consider different design elements of the research that may lead to conflicting data. This is the stuff you need to be paying attention to when reading any paper.

Subject characteristics & species

The characteristics of the people involved in a study can influence how relevant the results will be to a specific population, especially for this (fitness) community.

The first thing you should ignore is rodent studies. We are not rodents, they are not us, we are not becoming them, they are not becoming us and yes - I am well aware of the genetic similarities, I would hazard a guess that I am more aware than most people who constantly attempt to shove that line ("we are 99% genetically similar") in my face.
For example, here we have a hyped rat study that was used, and is still frequently referenced, as proof that high fructose corn syrup (HFCS) is more dangerous than sucrose. The problem? Well, the pathway for converting carbs to fats (de novo lipogenesis) is 10x more active in rodents than humans (this ALONE = ignore)...they were fed the human equivalent of 3000cals per day from HFCS alone...no sucrose fed control in the long term phases of the experiment = impossible to conclude (as the authors did) HFCS as more lipogenic than sucrose. Do I really need to keep going here?
Rodent studies are interesting for researchers to generate an hypothesis than can be tested out in human RCTs. For everyone reading this, they should = ignore.

Age, body composition, health and training status are 4 key characteristics to consider before deciding on whether or not something is applicable to you. What works for the young may not for the old, for the lean may not work for the obese (*cough fasted cardio cough*), for the healthy may not work for the ill (diabetics, etc) and what works for the beginner may not work for the advanced trainee.

Form, dose, duration & assessment tools

The chemical form of a compound can impact results.
Creatine is an easy one to use as an example here since monohydrate has plenty of data supporting it while other forms like ethyl ester do not (pretty much every "hyped" new version of creatine loses to creatine mono). Ephedrine HCL vs Ephedra is another obvious example the former being better than the latter).

The delivery vehicle of nutrients (how they enter your body) can change the outcomes.This is highlighted when looking at the mixed data (1, 2) on isolated antioxidants vs overwhelmingly positive data on fruit & veg (3, 4).

Dosing thresholds need to be considered. For example, going too low or high may impact the effectiveness of caffeine (2-6mg/kg is the sweet spot) and carbs/protein when added to intra-wo solutions.

The duration of a study is a common reason for conflicting study outcomes with acute effects not translating to the long term.
Plenty of examples of this with the most notable related to protein intake per meal and overall nutrient timing. For protein intake, the acute data shows smaller, more frequent, dosing (30g per meal-ish) to be the way to go but this never plays out in the long term. For nutrient timing again, the acute shows a benefit (mostly due to the overnight fasted state of the participants) that the long term simply doesn't.

When it comes to assessment tools, I'm talking about how researchers choose to measure certain outcomes.
BIA is a well known inaccurate form of body comp assessment and can lead to misleading results. An obvious example is a hunger strike study with the subjects consuming only water, vitamins & electrolytes for 43 days. They lost 14.5kg (about 32lbs, going from 80.9 to 66.4kg) and the BIA calculated that this meant going from 52.2 > 19.7%bf (total fat loss 29.1kg) with a lean mass gain of 14.6kg. This is impossible in the face of zero energy intake and the authors were smart enough to admit that it was probably an instrumental error.

Results & data interpretation

What the results are saying and what the authors in their abstracts/conclusions are saying can be 2 very different things. Some people don't want to believe this since they assume the researcher is clearly the expert but it's true I'm afraid.
I've highlighted plenty of examples of this throughout this forum but to add to it, take a look at a well known HIIT study claiming that HIIT burns 9x more fat than LISS ("...the decrease in the sum of six subcutaneous skinfolds induced by the HIIT program was ninefold greater than by the ET program".). This never happened. It was an assumption based on an energy-expenditure matched comparison that was never actually done.

Commercial interests and sponsors

It should come as no surprise to anyone that industry funding tends to bias conclusions in favor of the sponsors' products. This is demonstrated in an interesting study looking at research involving juice, milk and soft drinks and the impact of the funding source. They found that, regardless of the study type (observational, review, RCT, etc), the funding source had a significant impact on the conclusions with 37% of RCTs with no industry funding reporting unfavorable conclusion vs 0% for RCTs with industry funding.
A lot of strategy goes into this stuff too. There is a tendency for negative-result supplement studies to be published in conventional medicine journals in order to impact sales. This strategy works pretty well since an investigation found that the studies that received the most citations from the media involved vitamin E, the sales of which went down 33% and continued to drop the following year.

As tempting as it is to automatically dismiss all forms of industry funded research, doing so would be a form of bias itself. Plus not all industry funded studies are biased - here we see 3g of fish oil having no impact on weight loss over placebo despite the sponsor testing its own fish oil product.


Conclusion

I understand that people can be confused by the conflicting research and lack of uniformity from researchers. We all want quick, simple, answers. I get that. I've discussed the main reasons for this apparent conflict and, hopefully, shown you readers that it is still possible to examine the data objectively instead of dismissing it all as a impenetrable mess.
A little bit of patience with a sprinkle of attentiveness goes a long way here.
 
Last edited:

snake

Veteran
Joined
Jan 29, 2014
Messages
12,307
Reaction score
19,752
Points
383
Long read but I love the examples of the different studies.
 

PillarofBalance

Elite
SI Founding Member
Joined
Feb 6, 2012
Messages
20,402
Reaction score
18,204
Points
0
I love the rat DNA similarity argument. So easy to counter.

"Yeah but look at us."
 

TRUSTNME

Senior Member
Joined
Jan 2, 2018
Messages
174
Reaction score
47
Points
0
I agree on this brother. I read the positive and negative and use as a guide doing my trial and errors. No two people are the same. What works for my lifting partner does not work for me.
 

New Threads

Top