Researcher fraud is difficult to estimate, detect and prove, especially in parapsychology, where data anomalies may be interpreted as psi effects. This article argues that preventive research practices – including secure raw-data archives, audit trails, restricted access, quality control, blinding, preregistration, data availability and audits – offer the strongest protection.
- Undetected researcher fraud cannot be estimated objectively, and replication and peer review are weak safeguards.
- In parapsychology, post hoc fraud detection is unusually difficult because apparent data anomalies may be attributed to psi.
- Preventive data-management practices can make fraud by one researcher acting alone difficult or impossible.
Contents
Researcher Fraud in Science
Researcher fraud in science has been a controversial topic because the amount of undetected scientific fraud is unknown and cannot be estimated by objective methods. That researchers have sometimes committed fraud is well established, but the extent of undetected fraud cannot be estimated from cases of detected fraud and thus remains a matter of speculation and personal opinion. The historical assumption that replication and peer review will deter and detect researcher fraud is now known to be largely invalid.1Broad & Wade (1982); Stroebe, Postmes, & Spears (2012). No characteristics have been identified to predict fraudulent researchers.2Gross (2016). Most academic research has been conducted with the assumption that the possibility of fraud can be ignored, which can make fraud easy and tempting with little chance of detection.3Kennedy (2024).
Broad and Wade noted that most cases of detected fraud in science had ‘egregious arrogance or carelessness’ by the researcher committing fraud.4Broad & Wade (1982), 86. That has been true for cases of detected fraud in parapsychology.5Kennedy (2017). More careful fraudulent researchers may go undetected. Those who have compiled and reviewed cases of researcher fraud and devoted effort to exposing fraud typically believe that the amount of undetected fraud is much greater than the amount of detected fraud. But they also acknowledge that this is more personal speculation than an evidence-based conclusion.
As would be expected for a controversial area of research like parapsychology, researcher fraud has been a point of disagreement, with opinions ranging from optimism that little undetected fraud has occurred to the pessimistic view that researcher fraud has been a significant factor in the claims for psi. Here too, the differing opinions are speculations that cannot be resolved by available empirical evidence.
Research conducted with documented measures to prevent researcher fraud can provide evidence that fraud did not occur. This shifts the debate from personal speculations to evidence. Scientists should have evidence that undetected researcher fraud is rare, not just idealistic assumptions and hopes. Even the optimists recognise that measures to address researcher fraud are appropriate in parapsychology.6Kennedy (2017).
Implementing measures to prevent researcher fraud is particularly important in parapsychology because fraud is more difficult to establish than in other areas of science. The usual methods for investigating fraud often cannot be applied in psi research.7Kennedy (2017). Researcher fraud is typically investigated by looking for patterns or signs of fraud in the data. However, for psi research such patterns can often be claimed to be a psi effect. Other areas of science do not have psi as a basically irrefutable alternative explanation for signs of researcher fraud – and claiming that apparent signs of fraud were due to psi would be met with immediate rejection and ridicule.
In addition, looking for signs of fraud in data is post hoc analysis, with all the limitations intrinsic to post hoc analysis.8Kennedy (2024). When post hoc analyses are used to make claims about fraud that may ruin a scientist’s career, lawsuits can be expected.9Kennedy (2017, 2024). Those contemplating making claims of fraud based on post hoc analysis should carefully consider the evidence and risks, and whether legal advice might be a good investment.
A sting operation that documents fraud as it occurs is the most effective strategy for detecting fraud in parapsychology.10Kennedy (2017). This directly establishes fraud without the need for post hoc statistical analysis or surmises about what happened. The co-workers who exposed Levy’s fraud in parapsychology conducted a sting operation because they considered it the only effective option, given that Levy would claim that evidence of fraud in the data was a result of psi.11Kennedy (2017). They covertly established a recording system that documented the fraud as it occurred during an actual experiment.
The sting operation provided definitive evidence of researcher fraud and required a few days to plan and execute. By comparison, the Soal case in parapsychology was based on post hoc evaluation of research records, and the investigation that provided convincing evidence of fraud extended for over 60 years. Kennedy noted that some cases of suspected researcher fraud before the Levy exposé did not have a sting operation and were unable to resolve whether or not fraud occurred.12Kennedy (2017). The later accusations about Carl Sargent were also made without convincing evidence about what actually happened and are another case in which a consensus has not been obtained about whether fraud did or did not occur, due in part to Sargent’s unconvincing responses.
However, sting operations to detect researcher fraud have been very rare in science and are seldom a practical strategy in research laboratories.13Kennedy (2017, 2024). Research practices that prevent fraud are more feasible and less disruptive.
The long history of experimenter effects or experimenter differences in obtaining effects in psi research is another factor that makes researcher fraud prevention important for psi experiments. Differences among experimenters would be an expected symptom if researcher fraud occurred, as those with pessimistic views about fraud have pointed out. However, inconsistent results and differences among experimenters have too many possible explanations to be a good indication of researcher fraud – which is why replication is not effective for deterring or detecting researcher fraud. Given the controversial nature of parapsychology, research methods that reasonably eliminate researcher fraud from the list of possible explanations for experimenter differences are highly desirable.
JB Rhine advocated that confirmatory research in parapsychology should have measures to prevent researcher fraud. His long-held standard was that two or more researchers could work together in a way that made it impossible for one researcher acting alone to commit undetectable fraud. A similar standard has been applied for clinical trials regulated by the US Food and Drug Administration (FDA).14Kennedy (2024). The methodological practices to achieve this standard are well established and more comprehensive than Rhine’s discussion. These practices appear to be generally unknown to academic researchers, including parapsychologists. The practices have been described and adapted for academic research in an article by Kennedy.15Kennedy (2024). Key points are summarised below. Additional information and suggestions can be found in the original article.
Preventing Researcher Fraud
The Basic Goal
The basic goal is to make researcher fraud by one person acting alone impossible or very difficult. Most cases of fraud begin when a person is alone with the data and making some changes would provide the desired results with little chance of detection. Opportunities like this that make fraud easy and tempting should be eliminated. Experimental procedures that prevent fraud are increasingly important because AI can make detecting fraud from signs in the data virtually impossible.
These practices prevent unintentional errors as well as intentional errors (fraud). Research that is susceptible to researcher fraud is usually also susceptible to unintentional errors. In clinical trials these methods are generally considered as much or more valuable for preventing unintentional errors as preventing intentional errors. A healthy research environment will consider these practices as expected standard procedures and indicators of good research methods.
Archive Raw Data
A copy of the original raw data should be securely preserved in a way that prevents changes to the data or carefully tracks any changes. The original raw data should always be available or reliably recoverable. One way to achieve this is by having a copy of the data in two locations with no person having access to both copies. Any changes to one copy can be detected by comparing the data with the other copy. An online data repository is often optimal.
Audit Trail
All changes to the data should be tracked from the raw data to the final analysis data. The audit trail should include who made a change, when, and why. The data values both before and after a change should be available. This is a common-sense fundamental step for data integrity.
Restricted Access
Restrict access to the data collection process and data. Only the few people who have a direct need to change the data should be able to make changes to the data. No researcher, including primary investigators, should have access to the data that would allow changes without tracking or detection. Here too, copies of the data in two locations can achieve this.
Quality Control
All key steps in the research process should be checked and double-checked to verify that unexpected alterations of the data did not occur. This can be as simple as having a person sometimes or always observe the actions of another researcher during key research activities. Laboratory logs or notebooks can be used and compared with the recorded data. Independent verification or validation of any data-collection software is another important quality-control step that can detect unintentional and intentional programming errors.
Blinding and Preregistration
To the extent possible, decisions about data corrections and exclusions for confirmatory research should be made blind to the intervention condition and/or outcome variable. The best practice is to make all data decisions and changes while blind and lock the database (remove access to change the data) before unblinding the data. Developing and validating the data-processing and analysis programs before the study begins and including the programming in the preregistration for the study is the optimal form of blinding and research integrity. This should be possible if researchers have conducted adequate exploratory research to prespecify fully the planned data-processing and analysis steps. If the researchers need to see the data before the programming can be developed, the research usually remains at the exploratory stage.
Data Availability
When possible, the final data used for analysis should be made available, as should copies of the raw data and audit trail to verify that only preregistered data changes were made for confirmatory research.
Research Audits
A good research audit will directly address intentional and unintentional errors as well as all the points above. Documentation of all the measures to prevent errors is expected. Few academic researchers have experience with a good research audit. An example of a thorough research audit was provided by Kennedy for the Transparent Psi Project.16Kennedy (2023a, 2023b). The study included various measures to prevent intentional and unintentional errors, including preregistering the data-processing and analysis programs, restricted access to the data, archiving the raw data and formal validation of the data-collection system (which discovered a significant programming error). A good practice is to conduct research as if it will be audited, even if an audit is unlikely.
James E Kennedy
Works Cited
Broad, W., & Wade, N. (1982). Betrayers of the Truth: Fraud and Deceit in the Halls of Science. New York: Simon & Schuster.
Gross, C. (2016). Scientific misconduct. [Abstract.] Annual Review of Psychology 67, 693-711.
Kennedy, J.E. (2017). Experimenter fraud: What are appropriate methodological standards? [Full text.] Journal of Parapsychology 81, 63-72.
Kennedy, J.E. (2023a). Research audit for the Transparent Psi Project (TPP). [Full text.] Preprint. PsyArXiv.
Kennedy, J.E. (2023b). Lessons and recommendations from a research audit for the Transparent Psi Project (TPP). [Full text.] Preprint. PsyArXiv.
Kennedy, J.E. (2024). Addressing researcher fraud: retrospective, real-time, and preventive strategies – including legal points and data management that prevents fraud. [Full text.] Frontiers in Research Metrics and Analytics 9, art. 1397649.
Stroebe, W., Postmes, T. & Spears, R. (2012). Scientific misconduct and the myth of self-correction in science. [Full paper.] Perspectives on Psychological Science 7, 670-88.