home about blog

Randomised trials and the alternatives, some thoughts

Last week I was in Uganda discussing the evaluation of a large-scale livelihoods programme. Understandably, there were concerns about using a randomised trial. What interested me was that the concerns were inspired by problems with previous regression-discontinuity and stepped-wedge trial designs. These designs are used when a randomised design is infeasible, often because they are thought to be more acceptable to research participants. However I think they have a greater risk of causing conflict with participants. Let me explain.

First: regression discontinuity. For this design to work, the treatment (say, a cash transfer) should be allocated to individuals or places on the basis of a cut-off on a numerical scale; perhaps only people who are earning under a threshold receive the treatment.

People who had used this design, or variants, had found that the research participants who had not received the treatment were annoyed, and resented being part of the research. Would participants not feel the same way if they were in the control arm of a randomised trial?

Perhaps not, because in a randomised trial, the investigators do not control who gets the treatment and who does not. With the regression discontinuity design, the cut-off may be fair and honestly determined (e.g. no fiddling with the numbers), but who decided what the scale should be? Or the cut off? Perhaps the programme didn’t take the number of children into account, or long-term health conditions in the family. Some research participants may feel that allocating on the basis of income alone was not fair, and could have been different (had they been listened to, perhaps).

Second, stepped-wedge trials. A stepped-wedge trial is where individuals or groups are randomly allocated to start getting the intervention at different times. In the course of discussing the design, one of our group mentioned how they had conducted a stepped-wedge trial and had angry participants. The angry participants were those allocated to receive the intervention in the later ‘steps’ of the trial; they would say they should be in the earlier steps, instead.

One of the primary arguments for randomising is that there are more people eligible for the intervention than there are resources to reach them. This is nearly always true, even if the project is restricted to specific parts of a country, or even particular countries, when it would potentially benefit many others. In a stepped-wedge trial, however, the project team have revealed their hand: they do have the resources to reach all of the participants – that’s in the design – but they are submitting to a roll-out scheme that means some people get it much later. It is actually possible that the people who are in the control arm for most of the trial could have received it earlier (this problem extends to wait-list trials also). In a standard RCT design, twice as many people are given a 50% chance of receiving the intervention than could have been offered a 100% chance. Some people will miss out, but it is fair.

In conclusion: RCTs have the advantage that the allocation is out of human control (so that humans cannot be blamed), and submits honestly to the problem of insufficient resources to reach everyone who might benefit. Neither regression discontinuity or stepped-wedge trials share these advantages. Regression discontinuity risks annoying participants when the cut-off is on a scale or at a level that does not seem reasonable. Stepped-wedge trials reveals that there is sufficient resource to reach everyone in the study, but determine that some people will get it later (or not at all if individuals become ineligible, or die).

Written on January 28, 2020