Effectiveness studies
Whenever you look at any research, on whatever topic and of any study design, there are three things you need to consider:
- is the research trustworthy or is it likely to be prone to bias? (validity);
- if it is valid, what does it show? (results);
- what do these results mean for decision making for this particular patient or population in the particular context where the decision is being made? (relevance).
Validity
Please view the video below for an introductory talk about what makes a good study to test the effectiveness of a treatment or intervention. This talk focuses on validity, that is to say what the authors need to have done (and reported that they have done) in order to minimise potential biases.
Results
It is only worth thinking about what the findings of a study mean if the study design and methods are valid. There is a unit module on "Making sense of results" but there is a brief summary below if you want to undertake the critical appraisal before looking at this unit module. Results are presented in many different ways. In RCTs, cohort studies and case-control studies, two groups are compared and the results are often expressed as a relative risk (for example, dividing the outcome in the intervention group by the outcome in the control group). If the outcome is measured as the odds of an event occurring (for example, being cured) in a group (those with the event / those without the event), then the relative risk is known as the odds ratio (OR). If it is the frequency with which an event occurs (those with the event / the total number in that group) then the relative risk is known as the risk ratio (RR). When the there is no difference between the groups the OR and the RR are 1. A relative risk (OR or RR) of more than 1 means that the outcome occurred more in the intervention group than the control group (if it is a desired outcome, such as stopping smoking, then the intervention worked; if the outcome is not desired, for example death, then the control group performed better). Similarly, if the OR or RR is less than 1, then the outcome occurred less frequently in the intervention group. Results are usually more helpful when they are presented as risk differences. In this case you subtract the proportion of events in the control group from that in the intervention group. The risk difference can also be presented as the number needed to treat (NNT). This is the number of people to whom the treatment would have to be given - rather than the control - to produce one extra outcome of interest. There will always be some uncertainty about the true result because trials are only a sample of possible results. The confidence interval (CI) gives the range of where the truth might lie, given the findings of a study, for a given degree of certainty (usually 95% certainty). P-values report the probability of seeing a result such as the one obtained if there were no real effect. P-values can range from 0 (absolutely impossible) to 1 (absolutely certain). A p-value of less than 0.05 means that a result such as the one seen would occur by chance on less than 1 in 20 occasions. In this circumstance a result is described as statistically significant. This does not mean that it is necessarily important.
Relevance
It is important to consider whether the study is applicable to the decision being made for a particular patient or population. Any important differences between the participants in the trial and the patient or population in question that might change the effectiveness of an intervention must be identified. It is also important to think about whether the researchers considered all the important outcomes. It is no use establishing that patients had less pain but neglecting to observe that they could be dying more often simply because this outcome was not measured. Many interventions and processes that are used in everyday clinical practice have potential benefits and adverse consequences and it is important that these are weighed against each other judiciously. For example, if one patient has a major bleed for every five patients prevented from having a stroke when patients are given anticoagulants, then this intervention may be beneficial. However, if five patients have a major bleed for every stroke prevented, then the intervention may not be worthwhile. In both cases the treatment prevents strokes, but in the latter the likelihood of harm outweighs the benefit. Costs are usually not reported in a trial but if a treatment is very expensive and only gives a small health gain it may not be a good use of resources. Usually an economic evaluation is necessary to provide information on cost-effectiveness, but sometimes a ‘back-of-the-envelope' calculation can be performed. If the cost of treating one patient and the NNT can be established, these values can be multiplied to give a rough idea of the likely order of cost for producing one unit of benefit.