(Updated on August 28.)
We still don’t know what COVID-19 is doing to the heart or how we should be investigating it and treating it. Last month JAMA Cardiology published a German cohort study of 100 patients recently recovered from COVID-19. The paper concludes: “In this study of a cohort of German patients recently recovered from COVID-19 infection, CMR revealed cardiac involvement in 78 patients (78%) and ongoing myocardial inflammation in 60 patients (60%), independent of preexisting conditions, severity and overall course of the acute illness, and time from the original diagnosis. These findings indicate the need for ongoing investigation of the long-term cardiovascular consequences of COVID-19.”
These findings were incredibly impactful because they suggested that the virus is doing something directly to the heart and that we need to be doing a whole lot more about it. But there are important questions overhanging the study that have not been answered. And if the findings of the study are untrue or dubious, the consequences in this setting are massive, resulting in the misdirection of vital resources in a time of crisis.
A number of striking problems with the study were noted on Twitter. (Arguably Twitter is not the ideal forum for raising such concerns but the lack of response weeks later adds to the previous problems.)
One such discrepancy involves medians and interquartile ranges (IQR). In Table 1 of the paper, the first column of data is the 100 COVID-19 patients and the second column is 50 healthy controls. The median LV ejection fraction (IQR) of the COVID-19 patients is shown as 56 (54-58).
On Twitter Darrel Francis explained that if the IQR is 54 to 58, this means that half of the ejection fraction values lie between 54 and 58. As anyone who has ever dealt with ejection fractions will know, this is an implausibly narrow range. Other values in this table have the same issue of implausible tightness. One example is the blood pressures, reported as median 129 (IQR 125-133), which would mean that half the patients would have had blood pressures between 125 and 133. Medical professionals can undoubtedly identify similar unlikely medians in the data.
These ranges are also implausibly symmetrical, i.e the upper quartile is as far from the median as the lower quartile is from the median. Values such as CRP, BNP and Troponin are usually skewed such that the upper quartile is much further from the median than the lower quartile. Symmetry would only make sense if they were normally distributed (in which case one would present the mean standard deviation or confidence interval). Biologically, these variables are not normally distributed, unlike people’s weights or heights, because values more than 3 times higher than the mean are totally possible. For a normally distributed variable, values more than 3 times higher than the mean are not the norm.
Even if a mean and a confidence interval is being shown rather than median and IQR (because a confidence interval should be symmetrically above and below the mean), the distribution of values is still implausibly narrow.
Additional discrepancies were uncovered by Graham Cole on Twitter. For example, the patient described in Figure 1 is said to have a Troponin of 17.8pg/mL 78 days after diagnosis, but no such patient appears in Figure 3c which plots Troponin level against time since diagnosis.
The problems identified by Francis and Cole mean that it is impossible to rely on the study to guide policy in this time of crisis. The paper has been viewed over 500 thousand times and been discussed by 196 news outlets and mentioned in more than 12,000 tweets. Cardiac MRI may well be a great test to study the cardiac complications of COVID-19 because it can assess a broad range of pathology and is non-invasive. And no one is saying COVID-19 doesn’t affect the heart. But this study in its present form doesn’t help us in any way right now, when it is most needed. In fact, it gets in the way of progress, since we need accurate and reliable estimates to plan the appropriate research and clinical response to COVID-19. Unfortunately that’s not what this paper gives.
The problems with the paper also raise questions about the reliability of the medical publishing system in this time of crisis. Ideally, peer reviewers would identify areas of concern prior to publication. Failing that, when questions have been raised journals should immediately alert readers to these concerns and they should insist that authors respond to concerns and rectify errors quickly. Mistakes do happen but part of the job of a scientist is to rectify these in a timely manner. Unfortunately, at a time when there is no more urgent need for reliable and accurate data, the authors, reviewers, and editors, of this paper have been sadly lacking.
(Alex Nowbar is a research fellow working in the same department at Imperial College in London as Darrel Francis and Graham Cole.)
UPDATE (August 28):
JAMA Cardiology published a correction on 25th August. The authors were “pleased to confirm that reanalysis of the data has not led to a change in the main conclusions of the study”. They state “The errors and corrections affect the Abstract, Methods and Results sections, Tables, and Figures”. And there was rigorous statistical review. But still, perhaps we could have more confidence in the corrected paper if the origin of the errors was explained. That remains a mystery.
It’s also worth pointing out that headline number of 78% of COVID-19 patients’ hearts being affected is misleading on its own but features prominent in the “findings” section of the paper, the “abstract”, and, of course, many subsequent media reports. The reason it is misleading is because it implies that if you don’t get COVID-19 your heart is totally healthy and it’s COVID-19 that causes a problem when actually, things that make you more susceptible to getting COVID might also mean you didn’t have a healthy heart to start with. This is the kind of thing that peer review is intended to weed out (although there is no control over what authors can then say to the media, and what the media can then say to the public); peer review forms often ask whether the results justify the conclusions. The number that matters is how much more frequently the hearts of COVID-19 patients were affected compared to risk-factor-matched control patients. Does COVID-19 affect the heart independent of pre-existing conditions? This number is less scary because although more of the COVID-19 patients had abnormal cardiac MRI features, these were also present to some degree in the risk-factor-matched controls. Why isn’t that analysis in the abstract? The paper’s comparison with healthy controls is interesting but not meaningful because their hearts were healthier than the COVID-19 patients’ hearts had been before they got the disease.
While the authors should be congratulated for correcting errors and the editors for the speed and rigour of their review, if this paper didn’t warrant a retraction, and de novo submission with formal peer review, what does? To an outsider, it might seem that publishing a paper in a journal is like becoming a citizen of a country; it can’t be taken away from you.