Background: In nutrition, there are several examples of discordance between the results of observational studies and of randomized controlled trials (RCTs).
Objectives: To provide empirical data on how often the summary results of epidemiological studies and of RCTs are concordant, and to explore whether the probability of concordant findings is associated with quantifiable metrics of citation maps formed between studies belonging to the evidence base of the nutrient-outcome association at hand. Citation maps are an objective representation of the translational paths in each association, and may be a surrogate of the maturity of the relevant evidence base.
Methods: We searched MEDLINE to identify meta-analyses of RCTs or of epidemiological studies on the association between nutrients and health outcomes. Summary findings from both research designs that were statistically significant and in the same direction were considered qualitatively concordant. We also calculated the statistical significance of the difference in the summary effects from epidemiological studies and from RCTs (a measure of quantitative concordance). For each nutrient-outcome association we defined an “evidence base” including all publications identifiable by MEDLINE searches on the nutrient and outcome of interest and constructed citation maps of all articles in the evidence base that were cited by the epidemiological studies or the RCTs in the meta-analyses, either directly, or through one or more intermediary papers. We then quantified the size of the graphs (number of vertices and citation relationships), and their connectivity (density of citation relationships, mean hub and authority scores, and mean number of citations made or received over the included papers). We tested for associations between these metrics and the probability that the summary results from epidemiological studies and from RCTs are concordant between them.
Findings: In 23 out of 34 associations the summary findings from meta-analyses of epidemiological studies and of RCTs were in the same direction. In 6 of 23 associations, meta-analyses of epidemiological studies and of RCTs had statistically significant findings. In the remaining 11 out of 34 assocations, meta-analyses of epidemiological studies and of RCTs had summaries pointing in opposite directions. In 12 out of 34 associations the summary findings of epidemiological studies were statistically significantly different from those of RCTs, in 6 out of 12 point estimates were in the same direction, and in the other 6 in opposing directions. Despite the variation in the size and the connectivity of the citation graphs across the examined associations, we find no evidence of association between quantitative metrics of the citation graphs and the probability that the two research designs have concordant or discordant findings (using various definitions of concordance or discordance).
Conclusions: The examined quantitative characteristics of the citation maps in each association cannot predict the probability that the findings from the two designs agree or disagree. It is unclear whether there is a good way to describe the maturity of the evidence base on an association between nutrients and outcomes. At a minimum, purely bibliometric approaches are not a good way to prioritize which nutrient exposures merit further study, and for which health outcomes.