Tale of two trials. 29.06.20
Welcome to the Plague Pit – issue number 37.
I’ve written before about COVID-19 research. I tried to use specific papers to outline briefly some general concepts like literature searches, impact factors, the peer-review process, publication bias, research fraud, systematic vs narrative reviews and meta-analysis (https://plaguepit.com/publications/).
Last week, I also had a quick go at prospective vs retrospective cohort studies, multivariate regression and observational vs randomised controlled trials (https://plaguepit.com/risk/).
I had hoped that would just about cover things.
As time moves on, though, the doctors and scientists who were so busy looking after desperately ill COVID-19 patients are getting a chance to look at their data. Major journals are putting manuscripts about large studies through the peer-review process. Randomised controlled trials – which take so long to design, approve and conduct – are starting to generate results.
I’d like to focus on two studies that have really made the news.
You put your paper in
On May 22nd, the Lancet published a study of nearly 100,000 COVID-19 patients from 671 hospitals in six continents. The authors compared deaths among patients who had received early treatment with chloroquine (CQ) or hydroxychloroquine (HCQ) with deaths in patients who had not. They also compared the rate of ventricular arrhythmia (VA) – a type of heart palpitation that is often fatal and may be linked to CQ/HCQ treatment.
Data were provided by The Surgical Outcomes Collaborative from an ongoing registry of information. The trial was therefore retrospective and observational – an examination of what had already happened. It was not a randomised controlled trial (RCT)– a study designed and approved before enrolling patients. In an RCT, participants would have been randomly allocated to receive either (1) HCQ/CQ treatment or (2) no HCQ/CQ treatment (i.e. a ‘control group’)
The results, then, require cautious interpretation – even before considering the controversy that followed. The authors claimed patients in the study who received HCQ or CQ were at least 1/3 more likely to die than those who did not. One of the problems with this sort of study is that it can demonstrate association but rarely causation
This is why. Aside from getting HCQ/CQ or not, treatment and control groups may have been different in some other way. The authors found that COVID-19 non-survivors, for example, were more likely to be smokers and diabetic. What if the treatment group just happened to have more diabetics and smokers in it than the control group? Then HCQ/CQ treatment would be statistically associated with a higher death rate, but would not be the cause.
Researchers get round this sort of problem with a variety of complicated statistical techniques. These can measure the risk of death that comes with each factor, independent of the others.
Here’s the graph from the Lancet paper. The big red letters are part of the word ‘RETRACTED’, stamped over the manuscript by the publishers. These were added after everyone realised there was something not right and persuaded the authors they should withdraw the article.
You can see that the HR (Hazard Ratio) for death in black participants – independent of any other factor – was 1.344, compared to the reference white patients. In other words, black COVID-19 patients were 34.4% more likely to die than white ones. The range in brackets, to the right of ‘1.344’, is the 95% confidence interval (95%CI). It means – if you were to repeat the study very many times, on different groups of 100,000 patients – that the figure for the increased risk of death in black over white patients would fall between 27.6% and 41.5%, 95 times out of a hundred.
This interval is relatively small. The larger the study, the smaller the 95%CI – and the more certain you can be that the result you got in your sample (100,000 patients in this case) is close to the ‘real’ result in the entire population.
Where the 95%CI includes 1.0, the result is non-significant, regardless of the HR itself. In other words, there is no increased or decreased risk. In the HCQ/CQ study, this appears to be true for ‘immunosuppressed condition’, which is surprising.
When the entire 95%CI falls below 1.0 for a given variable – say gender – it means that the risk is lower than in the reference group. According to the HCQ/CQ study, for example, female COVID-19 patients have a risk of death 82.5% of males
Even with all the clever statistics, though, the authors cannot take account of factors which they did not record. Factors, for example, that might explain the higher death rate in HCQ/CQ patients but have nothing to do with the drugs themselves. In this trial, it’s quite possible that doctors only gave HCQ/CQ to the most desperate patients -we don’t know. More of these HCQ/CQ patients would have died because they were sicker anyway, not because the drugs were killing them.
You take your paper out
So why did the authors have to withdraw the paper? On June 3rd, the Lancet contained an unusual Expression of Concern about the paper they had published, only two weeks before. This may have been in response to an open letter, sent to the authors and the journal’s editor and signed by more than a hundred academics.
https://www.documentcloud.org/documents/6933411-Open-Letter-the-Statistical-Analysis-and-Data.html
The letter contains multiple concerns about the obscure origins and plausibility of the data in the original study. The Lancet followed their EoC, by publishing – on June 13th – a formal retraction of the paper by the authors.
(https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7274621/)
The retraction is signed by three of the four original authors. The fourth, Dr Sapan Desai, is the founder of the Surgisphere Corporation, owners of the Surgical Outcomes Collaborative, who provided the data for the study. He was not prepared to release the full dataset to his co-authors for independent review when questions were raised after publication.
The principle that original research data can be provided to journal editors for independent analysis is a cornerstone in the credibility of academic literature. As Desai’s co-authors could ‘no longer vouch for the veracity of the primary data sources’, they had no option but to withdraw the paper.
Sine the original article was published, it has also been suggested that one of the authors had a significant conflict of interest. He is said to have had connections to a company involved with remdesivir, a potential competitor of HCQ/CQ in the treatment of COVID-19.
This is not the end of the world in itself. Medical researchers submitting manuscripts for publication often have links to drug companies or receive funds from them. Reputable journals, though, require disclosure of this information. They publish it alongside the relevant paper when there might be concerns about the researcher’s objectivity. On this occasion, the Lancet didn’t.
On 17th June, the World Health Organization announced that they would stop the HCQ arm of the international Solidarity Trial, to find an effective COVID-19 treatment. The talk on the wards was that this decision to stop investigating HCQ was influenced by the original Lancet paper. The WHO website tells a different story, as follows: http://(https://www.who.int/news-room/q-a-detail/q-a-hydroxychloroquine-and-covid-19)
‘The trial’s Executive Group and principal investigators made the decision based on evidence from the Solidarity trial, UK’s Recovery trial and a Cochrane review of other evidence on hydroxychloroquine.‘
In an interesting coda to this sorry tale, the New England Journal of Medicine have recently published an Expression of Concern about another study based on Surgisphere data from the Lancet’s rogue authors, this time on COVID-19 and angiotensin converting enzyme inhibitors, a class of drug used in the treatment of high blood pressure and heart failure
https://www.nejm.org/doi/full/10.1056/NEJMe2020822
A happier tale
The other big COVID-19 research news is dexamethasone. Dexamethasone is a steroid used to treat or prevent all kinds of unpleasant conditions. Most of my patients under general anaesthesia get a dose because it is very good at preventing postoperative sickness. No-one know why.
The dexamethasone COVID-19 story could not be more different from the HCQ/CQ one above. It all appears to be absolutely by the book.
The national RECOVERY clinical trial, run by Oxford University, is aiming to identify treatments that may be beneficial for people hospitalised with suspected or confirmed COVID-19. It’s an RCT so it automatically fulfils the NHS Health Research Authority’s definition of research:
http://www.hra-decisiontools.org.uk/research/
As it’s ‘research’, the study needs to be approved by a Research Ethics Committee (REC) before patients are recruited. The HCQ/CQ study only involved a retrospective analysis of a research database – so according to the HRA criteria, ethical approval was not needed. A shame, in hindsight.
A hospital researcher contemplating a new study will first have to demonstrate to the local research department that she has funds and any other support necessary to undertake the work. Without this, the hospital or university are unlikely to agree to be the sponsor – taking liability for any financial or other mishap.
Once this is agreed, the researcher will seek ethical approval. Lots and lots of paperwork is submitted to the REC, including the trial protocol and the patient information and consent forms. The researcher may also be asked to attend an REC meeting in person, to answer queries. This is pretty daunting – as I know. RECs usually include a lawyer and at least one patient representative as well as several senior academics.
The researcher will also have to register the trial protocol with one or more national databases before starting to collect data. Reputable research journals have made this a condition of publication for many years.
Registration is important because it publicly commits the researcher to the original study method from the outset. He can’t change the experiment half way through, or look at the data and decide he’ll switch the analysis he originally planned, just to get a ‘positive’ result. Here’s the protocol for the RECOVERY randomised controlled trial, registered on 2/4/20
http://www.isrctn.com/ISRCTN50189673
It is has not been possible for the investigators to ‘blind’ the doctors caring to the treatment they administer to each patient (some of the treatments are injections, some are tablets). Otherwise, it looks blameless.
STOP!!!!!
On 16th June, the RECOVERY investigators issued a statement, part of which appears below in italics. It’s pretty spectacular. The dexamethasone part of the trial was stopped early. That’s because it would have been unethical to continue witholding dexamethasone from any oxygen-dependent COVID-19 patients, given that ventilated patients treated with the drug are 1/3 less likely to die than untreated ones
“On 8 June, recruitment to the dexamethasone arm was halted since, in the view of the trial Steering Committee, sufficient patients had been enrolled to establish whether or not the drug had a meaningful benefit.
A total of 2104 patients were randomised to receive dexamethasone 6 mg once per day (either by mouth or by intravenous injection) for ten days and were compared with 4321 patients randomised to usual care alone. Among the patients who received usual care alone, 28-day mortality was highest in those who required ventilation (41%), intermediate in those patients who required oxygen only (25%), and lowest among those who did not require any respiratory intervention (13%).
Dexamethasone reduced deaths by one-third in ventilated patients (rate ratio 0.65 [95% confidence interval 0.48 to 0.88]; p=0.0003) and by one fifth in other patients receiving oxygen only (0.80 [0.67 to 0.96]; p=0.0021). There was no benefit among those patients who did not require respiratory support (1.22 [0.86 to 1.75]; p=0.14).
Based on these results, 1 death would be prevented by treatment of around 8 ventilated patients or around 25 patients requiring oxygen alone.”
As I write this, the RECOVERY results have not yet been published in a peer-reviewed journal. That will come soon. The paper will contain details allowing readers to judge whether the apparent effect of dexamethasone is a ‘real thing’ or whether it is some particular quirk of the study patients or trial design.
I can confirm, though, that my intensivist colleagues are now intending to use dexamethasone to treat seriously ill COVID-19 patients.
Barbershop corner
In my department, we are still talking a lot about NOT catching COVID-19. One of the crucial elements of our PPE is the respirator mask.
I have mentioned before that colleagues with funny faces or exotic facial hair struggle to find masks providing an adequate seal on the face. Adequate, that is, to prevent those pesky virions climbing over your goatee, up your nose and down your windpipe.
Time to get the razor out, workmates?
The National Religion
As the NHS gets ready for the second wave, in her 72nd year, I’ve started to get a bit emotional about the old girl.
This was not helped by an interview this morning on the Today Programme with Michael Rosen, the writer. He spent nearly seven weeks in intensive care with COVID-19. It’s a great interview – about ten minutes, starting 1:49:00 into the programme:
https://www.bbc.co.uk/sounds/play/m000kgnm
As it turns out, Michael Rosen wrote a poem some years ago for the 60th anniversary of the NHS, called ‘These are the hands’.
https://www.scottishpoetrylibrary.org.uk/poem/these-are-hands/
It’s lovely, but it’s not my favourite. That’s ‘Here at Homerton Hospital’, sung by Beans on Toast.
https://twitter.com/beanstoast/status/1115690029760819200?lang=en-gb
‘….There’s a paramedic from Portugal
A security guard from Senegal
A pharmacist from the Isle of Man
And a paediatrician from Pakistan…..’
In case you think Mr Toast is overdoing it, here’s the winning entry from last year’s Day Surgery Unit photography competition at my hospital.
Most of the DSU staff in the picture still work there. In the last year, newcomers from a dozen more nations have added stickers on the glass.
With diversity in mind, if you want a real feel for the NHS in the 21st century, I recommend you have a browse on this website:
The NHS at 70 project started in 2018 and is a collaboration between the University of Manchester (UoM) and the British Library Oral History Department. It is funded by the Heritage Lottery Fund. It aims to capture the stories of NHS staff and patients in recorded interviews, and store them permanently in a freely accessible public archive at the British Library for use by researchers and historians. The NHS at 70 team have already spoken to more than 800 interviewees.
I joined up as a volunteer interviewer a few months ago. In the space of a fortnight, I recorded Archie the Jamaican porter, Samuel the Ugandan domestic supervisor (former game warden and English teacher in South Korea, children’s author) and Nick, who had a famous baby named after him.
Back in September……
The Good Soldier