Every doctor makes mistakes. But, says physician Brian Goldman, medicine"s culture of denial (and shame) keeps doctors from ever talking about those mistakes, or using them to learn and improve. Telling stories from his own long practice, he calls on doctors to start talking about being wrong.
Check this useful video podcast done by Dr Andy Neill. Andy is an Emergency Medicine doctor from Dublin.
Check Andy"s blog for more useful information
Total marks: 23
Time allowed: 90 mins
Paper: A Randomized Trial of Nebulized 3% Hypertonic Saline With Epinephrine in the Treatment of Acute Bronchiolitis in the Emergency Department
1. Provide a summary / abstract for the paper. (Up to 5 marks)
Objective: To determine whether nebulised 3% hypertonic saline with epinephrine is more effective than nebulised 0.9% saline with epinephrine in the treatment of bronchiolitis in the emergency department.
Design: Randomised double blind controlled trial Setting: Single centre urban paediatric emergency department in Canada. Participants: Infants younger than 12 months with mild to moderate bronchiolitis.
Interventions: Patients were randomised to receive epinephrine in either hypertonic or normal saline.
Outcome measures: The primary outcome measure was the change in respiratory distress, as measured by the Respiratory Assessment Change Score (RACS) from baseline to 120 minutes. Change in oxygen saturation was also determined. Secondary outcome measures included rates of hospital admission and unbooked return to the ED following discharge.
Results: 46 patients were enrolled. The two groups had similar baseline characteristics. RACS from baseline to 120 minutes demonstrated no improvement in respiratory distress in the hypertonic saline group when compared to the normal saline group. The change in oxygen saturations in the hypertonic group was also no different to that of the normal saline group. Rates of admission and unplanned return to the ED were similar between the two groups.
Conclusion: In this study hypertonic saline with epinephrine did not improve clinical outcome in acute bronchiolitis when compared to normal saline with epinephrine.
2. Give 3 strengths and 3 weaknesses of the study design? (Up to 3 marks)
Strengths:
Weaknesses:
There are some things you can put into either column! These basically relate to whether you like pragmatic trials (most ED ones should be I think) or explanatory trials. Examples here include:
3. What is block randomisation? What are the benefits and pitfalls of this method? (Up to 3 marks)
Block randomisation is a technique used to ensure that at any particular time in a study, the number of patients in each group is roughly similar.
In this study the authors used blocks of 4. Each block will have 2 of each of the treatments (although their position within the block of 4 will be randomly allocated). The blocks themselves are then randomised. In this way one group can never have more than 2 patients more than any of the others at any time.
The biggest pitfall with this method is that it might be possible for researchers or the ED medical or nursing staff to guess what the next treatment is going to be. E.g. if the research assistant is on patient 7, and knows that the last 2 patients had got better more quickly than usual (if she had a pre conceived idea than hypertonic saline was marvellous!), she might assume that the next patient was going to get normal saline. This might affect decisions about whether to include the patient in the trial in the first place.
In this particular example this doesn’t seem to have been a problem because the treatments look the same and their effects were similar. However it can be a problem when the person initiating the trial treatment is not blinded (eg NIV versus standard facemask) or the treatments can be distinguished in other ways (e.g. a trial of iv Pabrinex versus iv “brown” saline for alcoholics when the smell might give it away!)
4. What alternative methods of treatment allocation are there? (Up to 1 mark)
Single patient randomisation
Quasi randomisation – e.g. days of the week, hour of the day etc..
5. What do you understand by the term “intention-to-treat”? What are the advantages of this? What is the opposite approach and what advantages does this have? (Up to 4 marks)
Intention to treat includes all randomised patients in the groups to which they were randomly assigned, regardless of their adherence with the entry criteria, regardless of the treatment they actually received, and regardless of subsequent withdrawal from treatment or deviation from the protocol.
Once again this is very “real world”. It’s no good if a treatment is excellent for those who actually take it but a lot of people can’t tolerate it and so withdraw from a trial. Intention to treat analysis will ensure that the real world effects of a treatment are reported by ensuring that the results from “drop outs” are included in the analysis.
The alternative approach is sometimes referred to as “per protocol” or “explanatory” and means that the analysis is done only for those patients who actually took the allocated treatments. This might result in a better understanding of the actual biological effects of the treatment but isn’t so “real world”.
6. The authors used Fishers Exact Test for analysis of some of their data. What type of data can be analysed in this way and when is this used (Up to 2 marks)
The data is categorical data and usually dichotomous. That is, there are two treatment groups and the outcome measure is binary (i.e a 2 x 2 table can be generated).
Fishers test is used when the “expected” number in any one of the 4 quadrants is low (typically less than 5). Chi squared tests should only be used when all the expected numbers are higher than this.
7. The authors state that a change in RACS Score of anything less than 3 would not be clinically important. Why is it important to decide on the minimally clinically important effect and how does this effect power and sample size. (Up to 3 marks)
There is no point in designing a study to detect a difference in effect which isn’t clinically important.
Decisions on what is considered clinically important can be made using “hunch”, previous published studies, expert opinion or patient expectations but should be justified in the paper.
Detecting a smaller difference will require a larger sample size or a reduction in the power of the study. Most studies use 80% power as the minimum acceptable.
8. The following table is taken from the paper.
What do you understand by the figure 0.74 (-1.45 to 2.93) in the top right section of the table? (Up to 3 marks)
However, the 95% confidence intervals for this figure cross zero with a range of -1.45 to 2.93 indicating that the range of “plausible results” lies between hypertonic saline being better and normal saline being best. i.e there is no difference between the two statistically.
9. What are your conclusions overall? Is this paper going to influence your practice? (Up to 2 Marks)
A bigger study with across the day patient selection might be useful
It won’t change my practice regarding hypertonic saline but I may have to read about nebulised adrenaline in this condition!
Guys check out the following statistics related post from UKMi. I found some of the articles very useful.
UK Medicines Information is an NHS pharmacy based service. Its aim is to support the safe, effective and efficient use of medicines by the provision of evidence-based information and advice on their therapeutic use.
Dealing with small samples
The t distribution
Using the t test to compare:
Source: UKMi
Total marks: 23
Time allowed: 90mins
Paper: High-sensitivity versus conventional troponin in the emergency department for the diagnosis of acute myocardial infarction
1. Provide a summary / abstract for the paper. (Up to 5 marks)
This should include all or some of the following points:
Background: Recently, newer assays for cardiac troponin (cTn) have been developed which are able to detect changes in concentration of the biomarker at or below the 99th percentile for a normal population.
Objective: The objective of this study was to compare the diagnostic performance of a new high-sensitivity troponin T (HsTnT) assay to that of conventional cTnI for the diagnosis of acute myocardial infarction (AMI) according to pretest probability (PTP).
Design: A prospective observational study of consecutive patients who presented to emergency departments in France with chest pain suggestive of AMI.
Setting: Three French Emergency Departments.
Participants: Adult patients presenting with chest pain suggestive of acute myocardial infarction with onset within the previous 6 hours.
Outcome measures: Levels of HsTnT were measured at presentation, blinded to the emergency physicians, who were asked to estimate the empirical pretest probability (PTP) of AMI.
The discharge diagnosis (of AMI or not) was adjudicated by two independent experts (both ED physicians) on the basis of all available data up to 30 days post presentation.
Results: A total of 317 patients were included, comprising 149 (47%) who were considered to have low PTP, 109 (34%) who were considered to have moderate PTP and 59 (19%) who were considered to have high PTP.
AMI was confirmed in 45 patients (14%), 22 (9%) of whom were considered to have low to moderate PTP and 23 (39%) of whom were considered to have high PTP (P < 0.001).
In the low to moderate PTP group, HsTnT levels ≥ 0.014 μg/L identified AMI with a higher sensitivity than cTnI (91%, (95% CI 79 to 100), vs. 77% (95%
CI 60 to 95); P = 0.001), but the negative predictive value was not different (99% (95% CI 98 to 100) vs. 98% (95% CI 96 to 100)).
There was no difference in area under the receiver operating characteristic (ROC) curve between HsTnT and cTnI (0.93 (95% CI 0.90 to 0.98) vs. 0.94 (95% CI 0.88 to 0.97), respectively).
Conclusion: In patients with low to moderate PTP of AMI, HsTnT is slightly more useful than cTnI. Our results confirm that the use of HsTnT has a higher sensitivity than conventional cTnI.
2. Give three strengths and three weaknesses of the study design (up to 3 marks)
Strengths:
Weaknesses:
3. Explain why this study may have been subject to “incorporation” or “work up” bias. Suggest two ways in which this could have been avoided in this study (up to 4 marks)
It appears to me that the result of the conventional troponin test (cTnI) was used in some cases to determine the need for admission (or not). Since all patients had a conventional troponin test, those with a low pre-test probability of AMI and a negative cTnI were probably discharged. This seems to amount to 39% of the patients (61% were admitted).
Thus the cTnI result helps to determine whether or not further investigations are carried out. It is not too great a leap to assume that the results of cTnI and those of HsTnT are highly correlated.
Ideally the diagnosis of the reference standard (which in this case is based on a review of notes and subsequent investigations in hospital) should be made entirely independently of the interpretation of the diagnostic test under evaluation.
The test under evaluation is the troponin. However, some of the investigations in hospital were only arranged if the troponin was positive. Other patients were discharged. Hence some of the patients may have been “deprived” of the opportunity to be diagnosed with AMI
Incorporation bias thus results in an over estimation of the diagnostic accuracy of a test.
Further explanation of incorporation bias / work-up bias can be found here: http://www.cjem-online.ca/v10/n2/p174
Ways to reduce the influence of incorporation bias in this study include:
Changing the primary outcome measure to, for example, death by 30 days and obtaining the answer for all patients (but probably fewer deaths and hence a bigger study would be needed)
Admitting all patients and performing a standardised set of investigations regardless of the troponin result.
4. The following is an excerpt from the methods section:
“We followed most of the recommendations concerning the reporting of diagnostic studies set forth by the Standards for Reporting of Diagnostic Accuracy initiative”
Give 4 elements of the STARD guidelines which should be reported in a diagnostic study such as this. (Up to 4 marks)
STARD checklist for reporting of studies of diagnostic accuracy (version January 2003)
| Section and Topic | Item# | On page # | |
| TITLE/ABSTRACT/KEYWORDS | 1 | Identify the article as a study of diagnostic accuracy (recommend MeSH heading "sensitivity and specificity"). | |
| INTRODUCTION | 2 | State the research questions or study aims, such as estimating diagnostic accuracy or comparing accuracy between tests or across participant groups. | |
| METHODS | |||
| Participants | 3 | The study population: The inclusion and exclusion criteria, setting and locations where data were collected. | |
| 4 | Participant recruitment: Was recruitment based on presenting symptoms, results from previous tests, or the fact that the participants had received the index tests or the reference standard? | ||
| 5 | Participant sampling: Was the study population a consecutive series of participants defined by the selection criteria in item 3 and 4? If not, specify how participants were further selected. | ||
| 6 | Data collection: Was data collection planned before the index test and reference standard were performed (prospective study) or after (retrospective study)? | ||
| Test methods | 7 | The reference standard and its rationale. | |
| 8 | Technical specifications of material and methods involved including how and when measurements were taken, and/or cite references for index tests and reference standard. | ||
| 9 | Definition of and rationale for the units, cut-offs and/or categories of the results of the index tests and the reference standard. | ||
| 10 | The number, training and expertise of the persons executing and reading the index tests and the reference standard. | ||
| 11 | Whether or not the readers of the index tests and reference standard were blind (masked) to the results of the other test and describe any other clinical information available to the readers. | ||
| Statistical methods | 12 | Methods for calculating or comparing measures of diagnostic accuracy, and the statistical methods used to quantify uncertainty (e.g. 95% confidence intervals). | |
| 13 | Methods for calculating test reproducibility, if done. | ||
| RESULTS | |||
| Participants | 14 | When study was performed, including beginning and end dates of recruitment. | |
| 15 | Clinical and demographic characteristics of the study population (at least information on age, gender, spectrum of presenting symptoms). | ||
| 16 | The number of participants satisfying the criteria for inclusion who did or did not undergo the index tests and/or the reference standard; describe why participants failed to undergo either test (a flow diagram is strongly recommended). | ||
| Test results | 17 | Time-interval between the index tests and the reference standard, and any treatment administered in between. | |
| 18 | Distribution of severity of disease (define criteria) in those with the target condition; other diagnoses in participants without the target condition. | ||
| 19 | A cross tabulation of the results of the index tests (including indeterminate and missing results) by the results of the reference standard; for continuous results, the distribution of the test results by the results of the reference standard. | ||
| 20 | Any adverse events from performing the index tests or the reference standard. | ||
| Estimates | 21 | Estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals). | |
| 22 | How indeterminate results, missing data and outliers of the index tests were handled. | ||
| 23 | Estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if done. | ||
| 24 | Estimates of test reproducibility, if done. | ||
| DISCUSSION | 25 | Discuss the clinical applicability of the study findings. |
5. The following figure is taken form the results section of the paper:
ROC curves can be generated by using different cut off values to represent “positive” and “negative” for tests with continuous data points (e.g. quantitative D-Dimer, troponin I). Several points are chosen and then the sensitivity and specificity of the test for each cut off point is calculated. The ROC curve is simply a plot of the results with 1 – specificity on the x-axis and sensitivity of the y-axis (as above).
The ROC curve gives an assessment of the overall performance of the test at different cut off points for positive and negative. The larger the area under the curve then the better the test. Good tests will have curves which tend towards the top left of the graph.
A perfect test will have an AUC of 1.0 and a useless test will have an AUC of 0.5 (a straight diagonal line at 45 degrees to the origin, representing a 50:50 chance or a test no better than tossing a coin).
You will note that the 95% confidence interval for the AUC of HsTnT ranges from 0.881 to 0.971. i.e. the test appears to have good utility in the diagnosis of AMI.
6. The following is a portion of one of the results tables:
a Write one sentence explaining the results in each of the 4 columns pertaining to all patients with a positive cTnI. (4 marks)
71% of patients who are diagnosed with AMI will have a positive cTnI at presentation in the ED. Thus a negative test is not very good at ruling out likely development of AMI (SnOUT). It is not a very sensitive test when taken this early.
97% of patients who are not diagnosed with AMI will have a negative cTnI at presentation in the ED. cTnI is quite a specific test i.e. a positive test tends to rule AMI in (SpIN).
Only 78% of patients with a positive cTnI at presentation will be diagnosed with AMI (the PPV).
95% of patients with a negative test will not be diagnosed with AMI (the NPV).
The 95% confidence intervals around these levels indicate the range of plausible results (i.e. the range within which we are 95% certain that the real result lies).
b Briefly explain how the prevalence of the target condition in the population influences the sensitivity of a test and its negative predictive value (2 marks)
The sensitivity of a test is unaffected by the prevalence of the disease as it only relates to patients who actually have the condition. Similarly the specificity of a test is unrelated to the prevalence of the disease as it only relates to those without the condition.
The negative predictive and positive predictive value, however, are both influenced by the prevalence of the disease in the population. If the prevalence of a condition in a population is small then the reported NPV will be higher than when using the same test in populations with a greater prevalence of the disease. This is one reason why it is important to ensure that the study patients are similar to your own before implementing the results of a study into your practice.
Put another way, if there are hardly any patients with the condition in any case then useless tests such as e.g. tossing a coin, might have quite a good NPV (because there aren’t that many cases to miss anyway). If however, half the patients had the disease tossing a tail wouldn’t always be associated with those without the disease and the NPV will fall.
Conversely, PPV for the same test is higher in populations with a high prevalence of disease and lower if the disease is uncommon.
7. The following is another section from the same results table:
Write down the formulae for calculating the positive and negative likelihood ratios.
What do the positive and negative likelihood ratio results given in the top row mean? (Up to 3 marks)
Positive likelihood ratio: LR+= sensitivity/1-specificity
Negative likelihood ratio: LR-=1-sensitivity/specificity
LR + = 0.71 / 0.03 Approximately 21.5
LR- = 0.29 / 0.97 Approximately 0.32
LR+ above 10 means that a positive test will significantly increase the pre test probability enough to make the test worth doing. Figures like 21.5 mean that a positive test (cTnI) significantly increases the chance of predicting AMI (and are thus helpful in decision making).
LR – less than 0.1 means that a negative test will significantly decrease the pre test probability enough to make the test worth doing. Thus, the figure of 0.32 means that a negative test (cTnI) is not at all helpful in excluding subsequent AMI. (Mainly because it is done at arrival in ED and not the usual 12 hours).
Fagans nomogram is used to convert pre test to post test probabilities based on LRs.
Fagans nomogram
8. What are your conclusions overall? Is this paper going to influence your practice? (Up to 2 Marks)
The HsTnT does appear to have a better sensitivity (93%) than the traditional troponin when taken on arrival in ED. This may be helpful in improving time to discharge following attendance with chest pain and may reduce admissions.
However, the 95% CIs for the sensitivity are 89 – 100, and the possible sensitivities are thus too low to allow it to be confidently used to rule out MI based on this study.
Further studies are required.
Note: The answers are not done by me, they were given to me when I prepared my exam. If anyone have any questions, just drop a line below and we all can discuss.
