JB Rhine’s methodological writings anticipated several reforms later associated with psychology’s replication crisis, including preregistration, open data and a sharper distinction between exploratory and confirmatory research. The article also contrasts Rhine’s early pioneering meta-analysis with his later scepticism about retrospective meta-analysis as a source of confirmatory evidence.
- Rhine argued decades before the replication crisis that confirmatory studies should be registered with colleagues and records kept available for review.
- Reforms in psychology after 2012 — preregistration, open data and stricter exploratory/confirmatory distinctions — closely resemble his 1974 proposals.
- Rhine’s later critique of retrospective meta-analysis anticipated recent concerns that such analyses are exploratory, not confirmatory evidence.
Contents
Introduction
In 1974, when he was nearly eighty years old, JB Rhine wrote his ideas about the conduct of experimental research. These writings received little attention from parapsychologists or other scientists at the time. Some 38 years later, psychology began experiencing a ‘replication crisis’ that dramatically altered how research was performed and evaluated. The updated research methodology that has emerged from the replication crisis has implemented the ideas articulated by Rhine in 1974 to a remarkable degree.
Rhine’s views about meta-analysis were even more notable. A guiding principle for psychological and parapsychological research for the past few decades has been that meta-analysis (combining a group of similar studies together) provides the strongest evidence for an effect. Rhine and colleagues are widely recognized as conducting the first psychology-related meta-analysis in 1940.1Pratt et al. 1940/1966). However, in 1976 Rhine argued that attempting to combine all available studies to draw conclusions about whether an effect occurs is a post hoc analysis that has little value. This was a significant change from his earlier writings. Arguments similar to Rhine’s late-life rejection of the usual optimistic views about meta-analysis began emerging in psychological writings about 40 years later.
The evolution of methodology in psychological and parapsychological research appears to be tracking Rhine’s thinking, but with a lag of a few decades.
J.B. Rhine on Research Methods, 1974
Rhine (1974) described good research methodology as:
He [a researcher] should be free to do his own preliminary exploration within the field more or less as he prefers; but when he has performed a promising pilot experiment and wants to set up a confirmatory project, he then needs to go on record with his group and to try openly to share his project with one or more of his colleagues. The research should go through one round of experimental confirmation after another with the center’s review system keeping the complete records. … This … is necessary in order to avoid risk of omissions and improper selection of reportable data. With frequent reviews at staff meetings, and (as the work grows) at suitable conventions, … this sharing of progress (as well as failures) can … keep the data record straight, complete, and always ready for review and reexamination.2Rhine (1974), 118; emphasis in the original.
The key points here are: (a) the distinction between exploratory and confirmatory research; (b) the need to go on record with others about the research plans for confirmatory research; and (c) making the data available to others.
Rhine also recognized the need to change the research culture to have greater emphasis on confirmatory research:
The greatest difficulty will be in obtaining the cooperation necessary for the large-scale repetition needed. As it is, there is far too little “will to repeat” in this field at present; most researchers want to be innovators, since it looks more creative. But those who want the field to be taken seriously beyond its own small group will in time see the basic need for this reinforcement of security through repeating each other’s experiments.3Rhine (1974), 117.
The main topic of Rhine’s 1974 paper was researcher fraud. He considered these methodological recommendations as valuable for reducing the potential for researcher fraud, as well as for reducing other methodological deficiencies. Rhine also advocated that researchers work together with blinding and other methods to prevent fraud by one researcher acting alone. His comment that a researcher planning confirmatory research should ‘try openly to share his project with one or more of his colleagues’ refers to implementing such fraud prevention measures.
The Replication Crisis, 2012
Key factors motivating the replication crisis in psychology were Daryl Bem’s studies of precognition using standard psychological research methods a recent case of extensive researcher fraud in psychological research and the recognition that researcher flexibility or degrees of freedom allowed substantially biased research that could not be recognized from the publications.4(Pashler & Wagenmakers 2012). The papers on ‘Replicability in Psychological Science: A Crisis of Confidence?’ in Perspectives on Psychological Science in 2012 represent a clear demarcation that the replication crisis had begun in full force.5Pashler & Wagenmakers (2012). These papers described common sources of bias and proposed practices that could address the biases.
The most insidious source of bias appeared to be researcher flexibility. Researchers typically had flexibility to make methodological decisions after looking at the data. These post hoc decisions included altering the analyses and hypotheses. Simulations revealed that the degree of bias that could be obtained with this flexibility was stunning and consistent with effects commonly reported in psychology.6Simmons et al. (2011).
The underlying problem was the lack of formal confirmatory research and the failure to distinguish between exploratory and confirmatory research.7Wagenmakers et al. 2012. The common flexibility in psychological research was appropriate and expected for exploratory research. The missing step was that this flexible research needs to be followed with formal confirmatory research that does not have the flexibility.
The practices that have been developed to address the replication crisis basically track Rhine’s ideas of 38 years earlier. These include: (a) distinguishing between exploratory and confirmatory research; (b) going on record by preregistering the plans for a study; (c) open science practices, and particularly making data available for checking and analysis; and (d) changes in the research culture to place greater emphasis on confirmatory research rather than on reporting novel findings, and publishing confirmatory studies that fail to support the researchers’ hypotheses. With the advent of the internet, study preregistration and open data are handled online rather than by research institutions as proposed by Rhine in the 1970s.
In parapsychology, the KPU (Koester Parapsychology Unit) Study Registry, which began operation in 2012, specializes in preregistering psi research,8Watt & Kennedy (2015). and Psi Open Data is an open-access repository for psi research data.
As of 2026, practices for preventing researcher fraud have received little attention in response to the replication crisis, even though fraud was a major factor motivating the replication crisis.
JB Rhine’s Meta-Analysis, 1940
The first comprehensive psychology-related meta-analysis is generally recognized to be the one conducted at Rhine’s Parapsychology Laboratory at Duke University and published in 1940.9Pratt et al. (1940/1966). Rhine and colleagues used statistical methods to combine results from all ESP studies that they could find after substantial efforts searching. They also evaluated various possible moderating variables and methodological problems. Rhine’s group was far ahead of the times. Thirty-five years later in the mid-1970s, the term meta-analysis was proposed and formal methods began to be discussed for the type of evaluation Rhine and colleagues had pioneered.10Glass (1976).
JB Rhine on Meta-Analyses, 1976
Although Rhine presumably considered meta-analysis useful when he conducted the first one in 1940, by the mid-1970s he had rejected the usefulness of retrospectively combining all studies for evaluating evidence of an effect. One key reason was:
no such generalizable statistical application to independent experiments is possible. The statistics are applicable only to the usage designated in advance, for the particular purpose in a given set of conditions.11Rhine (1976), 67.
Rhine recognized that strong or confirmatory evidence requires that the planned statistical analysis be designated in advance (preregistered). That type of pre-specification is not possible for a retrospective meta-analysis of all studies, which is intrinsically post hoc. Between 1940 and the mid-1970s, Rhine appears to have developed a better understanding of the differences between confirmatory and post hoc statistical analyses, and between exploratory and confirmatory research. Rhine emphasized confirmatory research and appeared to believe that parapsychology would eventually achieve the goal of reliable confirmatory effects. He considered the field to be at an early stage of developing the needed control.
Meta-Analysis after the Replication Crisis
The replication-crisis lessons about researcher flexibility and post-hoc analyses apply to meta-analyses as well as to individual studies. A typical retrospective meta-analysis is a form of post hoc analysis with much flexibility for the analysts to make methodological decisions that bias the outcome.12Ferguson (2014); van Elk et al. (2015); Watt & Kennedy (2017). This potential bias is in addition to the flexibility and bias in the studies included in the meta-analysis.
The limitations of typical retrospective meta-analysis are increasingly recognized.13Kennedy et al. (2026); Stanley et al. (2022). Like other forms of post hoc analysis, retrospective meta-analysis is appropriately considered a form of exploratory research and not confirmatory evidence.
Conclusions
JB Rhine’s methodological writings in the 1970s were widely rejected or overlooked, including by parapsychologists. The lack of distinction between exploratory and confirmatory research did not change noticeably, nor did the enthusiasm for retrospective meta-analysis. There is no indication that Rhine’s writings contributed to the rationale for the replication crisis or to the proposed solutions.
But the fact remains that several of Rhine’s methodological insights from the mid-1970s are widely recognized as standard good practices. These include distinguishing between exploratory and confirmatory research, study preregistration, open data, and changing the research culture to give greater emphasis to confirmation.
Some other insights in Rhine’s writings have not yet been embraced. Recognition of the limitations of retrospective meta-analysis appears to be increasing, but is not yet widely established in parapsychology or psychology. And, practical methods for preventing researcher fraud such as Rhine proposed remain largely outside the thought process of most parapsychologists and academic scientists in general. These ideas appear likely to become increasingly accepted like Rhine’s other insights about research methodology.
Works Cited
Ferguson, C.J. (2014). Comment: Why meta-analyses rarely resolve ideological debates. Emotion Review 6/3, 251-52. [Web page. Abstract.]
Glass, G.V. (1976). Primary, secondary and meta-analysis of research. Educational Researcher 5/10, 3-8. [ResearchGate. Full text.]
Kennedy, J.E., Wiseman, R., & Watt, C. (2026). The emerging disenchantment with retrospective meta-analysis. Nature Human Behaviour. Advance online publication. [Web page. Abstract.]
Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? [Editorial]. Perspectives on Psychological Science 7/6, 528-30. [Web page. Download PDF.].
Pratt, J.G., Rhine, J.B., Smith, B.M., Stuart, C., & Greenwood, J.A. (1966). Extra-Sensory Perception after Sixty Years. Boston, Massachusetts, USA: Bruce Humphries. (Original work published 1940.) [Internet Archive. Download PDF.]
Rhine, J.B. (1974). Comments: Security versus deception in parapsychology. Journal of Parapsychology 38/1, 99-121.
Rhine, J.B. (1976). Comments: “Publication policy on chance results: Round two.” Journal of Parapsychology 40/1, 64-68.
Simmons, J.P., Nelson, L.D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22/11, 1359-66. [Web page. Full text.]
Stanley, T.D., Doucouliagos, H., & Ioannidis, J.P.A. (2022). Retrospective median power, false positive meta-analysis and large-scale replication. Research Synthesis Methods 13/1, 88-108. [ResearchGate. Full text.]
van Elk, M., Matzke, D., Gronau, Q.F., Guan, M., Vandekerckhove, J., & Wagenmakers, E.J. (2015). Meta-analyses are no substitute for registered replications: A skeptical perspective on religious priming. Frontiers in Psychology 6, art. 1365. [Web page. Full text.]
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H.L.J., & Kievit, R. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science 7/6, 632-38. [Web page. Full text.]
Watt, C.A., & Kennedy, J.E. (2015). Lessons from the first two years of operating a study registry. Frontiers in Psychology 6, Article 173. [Web page. Full text.]
Watt, C.A., & Kennedy, J.E. (2017). Options for prospective meta-analysis and introduction of registration-based prospective meta-analysis. Frontiers in Psychology 7, art. 2030. [Web page. Full text.]
