Dropouts and missing data can distort experimental results when researchers assume they are random or exclude affected participants. Focusing on psychology and parapsychology, this article explains how psi studies can reduce bias through binomial analysis, conservative imputation, preregistration and complete open-data reporting.
- Excluding every participant with missing data can bias results, especially when dropout or errors are related to performance, feedback or other features of the experiment.
- For many psi experiments involving random events, analysing all completed trials with a binomial test can avoid biases created by participant-level exclusions.
- When binomial analysis is unsuitable, conservative imputation, preregistered handling rules and open publication of excluded data can reveal whether missingness compromises the study.
Nature of the Problem
The substantial possibility that experiments are biased by participants who drop out or have other missing data has often been underappreciated in psychology and parapsychology. As summarized by Gomila and Clark, ‘Standard methods used by psychology researchers to handle missingness rely on unrealistic assumptions, invalidate random assignment procedures, and bias estimates of effect sizes.’1Gomila & Clark (2022), 143.
Dropouts and other missing data cannot be assumed to be random and independent of the study procedures and participant characteristics. Missing data must be treated as a possible non-random confounding factor that may bias the study outcome. ‘It is difficult to think of any psychology studies in which researchers may safely assume that missingness was produced in a way that does not bias estimates of the population ATF [average treatment effect]’.2Gomila & Clark (2022), 147. This applies to parapsychology as well as to regular psychology.
Excluding all data for any participant who has any missing data is a common practice in parapsychology as well as psychology, but this can directly cause bias. For example, when participants receive feedback or otherwise obtain hints about their performance on an experimental task, those who are not doing well may be more likely to dropout, make mistakes, or otherwise generate missing data. If all their data are removed, the remaining data will be biased.
The potential for bias can be seen from a hypothetical example of an extreme case. Consider an online ESP experiment in which participants do three trials with the probability of a hit being .5 on each trial. Participants who obtain hits on all three trials receive a reward. Participants with any misses do not receive the reward. The experimental plan is to exclude all data for any participant who does not complete all three trials. For this hypothetical example, all participants who get a miss on the first or second trial stop participating because they realize they cannot get the reward. Thus, every participant who completes three trials has a hit on the first two trials.
If there is no ESP effect, the expected hit rate for the participants who complete all three trials is (1 + 1 +.5) / 3 = .83 rather than .5. This extreme case is not likely to occur in practice, but it demonstrates the point that the handling of missing data can bias the outcome. When the effects being investigated are small, as in most psi experiments, the potential for subtle biases from missing data is particularly of concern.
Any data analysis that excludes all data for a participant with missing data requires careful evaluation of possible bias. A common response to the concerns about bias with this practice is to evaluate the psi effect or other effect of interest in the data excluded. This may be useful in identifying the characteristics of the participants with missing data, but it does not directly evaluate how the excluded data affect the analyses and conclusions.
Bias related to feedback about performance on the experimental task is one example of the more general problem that missing data cannot be assumed to be random and safely ignored. Empirical evaluation of the potential for confounding is needed.
The Simple Option for Psi Research: Binomial Analysis
For most behavioural research, handling missing data in a way that does not create bias is a significant problem. However, psi research is fortunate in often being able to avoid that problem.
For a typical psi experiment of guessing or influencing random events, a simple binomial analysis of all the completed trials pooled can evaluate whether psi occurred and this is arguably the optimal analysis. This analysis includes the completed trails for participants with missing data.
Missing data cannot bias this analysis for overall psi if the missing data is due to the participants not properly completing a trial before the outcome of the trial is known. However, if there is a way for participants (or researchers) to convert a trial to missing after learning the trial outcome, the study has inappropriate potential for bias. This includes excluding all data for a participant with any missing data. All trials with an outcome should be included in this analysis for overall psi.
Researchers sometimes object to this binomial analysis with the trial as the unit of analysis, arguing that repeated trials by a participant are not independent. However, that does not recognize the unique properties of psi research. Under the null hypothesis that psi does not occur, the trials are unambiguously independent due to the random events. A binomial analysis of all completed trials is precisely correct and appropriate for testing the null hypothesis that psi does not occur. This point and related topics were discussed thoroughly in the Consensus Design process3Kekecs et al. (2022). for the Transparent Psi Project.4Kekecs et al. (2023).
If the null hypothesis is rejected and thus there is evidence that psi occurred, the assumption that the multiple trials for a participant are independent no longer applies. Different statistical methods are needed at that point. But the initial evaluation of whether psi occurred is not biased by the handling of missing data.
Researchers with statistical training in psychology usually assume that the same statistical analysis will be used to test the null hypothesis and to model the effect if the null hypothesis is rejected. That assumption is not necessary and may not be optimal for some psi research.
This analysis strategy is applicable for typical psi experiments that involve guessing or influencing random events, but not when the analysis is based on predicting differences in human responses such as reaction times or physiological measures. In these cases, multiple trials for a participant cannot be assumed to be random or independent under the null hypothesis of no effect. A different strategy is needed for handling missing data. This is also usually the situation for psychological research that is not investigating psi.
The More General Option: Use Conservative Assumptions
For behavioural research in general, the strategy of creating hypothetical data for the missing data is usually considered the best approach. The missing data are replaced with the hypothetical or substitute data and the planned analysis is conducted. This is applicable in psi research when the outcome measures are not based on random events, if evidence of psi has been found in the experiment, or psi is assumed to occur in the experiment. Various complicated technical methods of ‘imputation’ of missing data values have been proposed, as well as simpler methods.5Gomila & Clark (2022).
The best strategy for research on controversial topics like psi is to create hypothetical data with the conservative assumption that the effect being investigated did not occur for the missing data.6Kennedy (2016). The hypothetical data are created consistent with the null hypothesis. Data can also be created with different assumptions about the missing data as a sensitivity analysis.
If the statistical analysis with the conservative hypothetical data provides evidence for the effect, that indicates that the missing data did not bias the study conclusions. However, if the analysis with conservative hypothetical data does not provide evidence for the effect and an analysis that uses another imputation strategy or excludes participants with any missing data does provide evidence for the effect, that indicates the missing data compromise the study. In this case, the study design and conduct allowed too much missing data. This limitation cannot be confidently resolved by analysis of these data. A new study with less missing data is needed for confident conclusions.
Study Preregistration
For confirmatory research, all plans and decisions for handling missing data should be specified in the study preregistration. If this cannot be done because of uncertainty about missing data, the research is still at the exploratory stage.
For exploratory research, detailed plans for handling missing data are not necessary. Obtaining information about the types and amount of missing data for a study and developing the plans for handling the missing data are common goals for exploratory research. If research preregistered as confirmatory has nontrivial missing data that was not anticipated in the preregistration, that is an indication that confirmatory research may have been premature and more exploratory research would have been useful.
Open Data
The study data that are made publicly available as open data practices should include any data that was excluded from the analysis. Potential bias due to the handling of missing data and other data exclusions cannot be evaluated if excluded data is not made available.
James E Kennedy
Note: The author retains copyright for this article.
Works Cited
Gomila, R., & Clark, C.S. (2022). Missing data in experiments: Challenges and solutions. [Abstract.] Psychological Methods 27/2, 143-55.
Kekecs, Z., Aczel, B., Palfi, B., Szaszi, B., Zrubka, M., Szecsi, P., & Kovac, M. (2022). Responses to comments submitted during Consensus Design – Round 1. [Full text.] Web page.
Kekecs, Z., Palfi, B., Szaszi, B., Szecsi, P., Zrubka, M., & Kovacs, M., et al. (2023). Raising the value of research studies in psychological science by increasing the credibility of research reports: The Transparent Psi project. [Full text.] Royal Society Open Science 10/2, art. 191375.
Kennedy, J.E. (2016). Is the methodological revolution in psychology over or just beginning? [Full text.] Journal of Parapsychology 80/2, 156-68.