In October the Medical Section met to discuss statistical considerations in feasibility and pilot studies.
Attendees formed into three groups to discuss the following three questions:
1) How should a pilot study be analysed?
It was noted that to decide how a pilot should be analysed, we must first decide upon the study’s objectives, and that usually these will not be the same as those of the main study. Some of a pilot study’s outcomes are qualitative and observational in nature. The group felt that while outcome criteria for a pilot study could be specified, investigators should not be held rigidly to them. The group highlighted the potential for pilot studies to provide useful insights into aspects which would affect the main study, including missing data and the suitability and acceptability of questions to be asked of patients.
The group discussed whether data from patients in a pilot study ought to be included in the main study analysis, and felt the right answer was no in almost all circumstances. The point was made that it is often hard to get pilot studies published but that their results often contain information which may be invaluable to other researchers, for example in relation to recruitment rates. One route to addressing the problem of publication would be the formation of online repositories for pilot studies and their results.
2) When should internal pilot studies be used?
The group turned the question around and asked what is the motivation for doing a pilot study externally? There was some discussion of the use of external pilots as proof of concept studies in order to give some evidence or indication that the intervention is effective. The group discussed some of the issues surrounding recruitment to pilots and, in particular, the difficulties that may arise in trying to explain the study design and objectives to prospective participants. Lastly, there was some discussion concerning whether pilot studies could be broken down into stages, such that early stages would be external to the main study, but that the later stage could be internal.
3) Is the distinction between pilot and feasibility studies useful, and how could feasibility criteria be built into the study design?
The group generally felt that it was appropriate to think of feasibility as a concept within which a number of different types of study might take place. It was noted that there appears to be quite a strong influence of definitions given by bodies such as the National Institute for Health Research on how investigators describe their studies. Again it was felt that it was important that the results of pilot and feasibility studies be made available, either through a new journal or an online repository.
The group felt that having some criteria against which feasibility would be judged was a good idea. However, it was felt that hard criteria thresholds would be too rigid and that a better approach might be to define ranges which would dictate what action, if any, would follow in the main study.