The Problem With p < 0.05 in Peptide Research
Few numbers carry more weight in preclinical science than 0.05. A result that clears this threshold is routinely described as "significant," while one that does not is quietly set aside. In peptide pharmacology—where studies frequently rely on small animal cohorts, in vitro binding assays, and a single published experiment—this binary treatment of the p-value creates a systematic distortion of the evidence base.
Understanding why requires returning to first principles. A p-value does not measure the probability that a hypothesis is true. It measures the probability of observing data at least as extreme as those collected, assuming the null hypothesis is correct [1]. That distinction is not semantic. It means a p-value of 0.03 on a peptide receptor binding assay does not confirm that the peptide is biologically active; it indicates only that the observed result would occur roughly 3% of the time if there were no real effect. Whether that effect is real, meaningful, or reproducible is an entirely separate question.
What a p-Value Actually Measures
The frequentist framework underlying the p-value was developed to serve as one input among several in a broader inferential process—not as a standalone verdict [1]. In practice, the research community has collapsed this nuanced tool into a pass/fail gate.
For peptide researchers, the consequences are concrete. Consider an in vitro assay comparing the IC₅₀ values of two synthetic peptide analogues across six replicates. A t-test may return p = 0.04, which clears the conventional threshold. But if the difference in IC₅₀ is 8 nM against a baseline of 500 nM, the proportional effect is modest. The statistical test has detected a difference; it has said nothing about whether that difference is pharmacologically consequential.
The American Statistical Association's formal statement on p-values explicitly warns against using the threshold as a binary decision rule and emphasises that scientific conclusions should not be based on whether a p-value crosses a fixed cutoff [1]. That guidance, published in 2016, has yet to be uniformly absorbed into preclinical peptide literature.
Small Sample Sizes and the Inflation of False Positives
Peptide pharmacology studies conducted in rodent models commonly employ cohort sizes of five to ten animals per group. This is not unusual given the cost and regulatory requirements associated with animal research, but it carries a statistical consequence that is frequently underappreciated: underpowered studies do not merely fail to detect real effects—they also produce inflated estimates of the effects they do detect [2].
This phenomenon, sometimes called the "winner's curse," operates as follows. When a study has low statistical power—meaning a low probability of detecting a true effect even if one exists—the subset of results that happen to reach significance will, by selection, tend to overestimate the true effect size. A peptide that genuinely improves a biomarker by 12% in a rodent model may appear to produce a 35% improvement in an underpowered study that achieves significance purely through sampling variation [2].
Button and colleagues demonstrated this dynamic rigorously in neuroscience research, showing that median statistical power across a large sample of studies was approximately 20%, meaning that four out of five true effects would go undetected—and that the effects which were detected were systematically exaggerated [2]. Peptide pharmacology, which shares many of the same experimental conventions as neuroscience, faces analogous vulnerabilities.
Statistical power is a function of three variables: the significance threshold, the sample size, and the true effect size. Researchers who do not conduct a formal power calculation before beginning an experiment cannot know whether their design is capable of reliably detecting the effects they are looking for. When power calculations are absent from a published peptide study—which remains common—readers should treat the reported effect sizes with corresponding caution.
The Multiple Comparisons Problem and p-Hacking
Peptide dose-response experiments often test several concentrations, multiple time points, and several outcome measures simultaneously. Each additional statistical test performed on the same dataset increases the probability that at least one result will cross the p < 0.05 threshold by chance alone. With twenty independent tests, the probability of at least one false positive under the null hypothesis approaches 64%.
This is the multiple comparisons problem, and it is compounded by what researchers have termed "p-hacking"—the practice of adjusting analytical choices (which comparisons to run, which data points to exclude, when to stop collecting data) until a significant result emerges [6]. Simmons and colleagues demonstrated experimentally that researchers exercising undisclosed flexibility in data collection and analysis could present almost any result as statistically significant without technically falsifying data [6]. The researchers described these as "researcher degrees of freedom"—legitimate-seeming choices that collectively function as a significance-manufacturing mechanism.
In the context of peptide binding studies, researcher degrees of freedom might include: selecting which concentration range to analyse after viewing the data, excluding outlier replicates without pre-specified criteria, or reporting only the time point at which a peptide showed the strongest effect. None of these choices is necessarily dishonest in isolation; collectively, they undermine the inferential value of the resulting p-value.
Publication Bias and the File-Drawer Problem
The distortions introduced by underpowered studies and flexible analysis are amplified by a structural feature of academic publishing: journals have historically shown a strong preference for statistically significant results. Studies that find no effect tend to remain unpublished—filed away rather than submitted, or rejected after review [3].
Ioannidis formalised the implications of this dynamic in a widely cited analysis, demonstrating mathematically that under realistic assumptions about the proportion of true hypotheses being tested, the statistical power of studies, and the prevalence of bias, the majority of published research findings in certain fields are likely to be false positives [3]. The argument is not that researchers are dishonest; it is that the incentive structures and publication norms of science systematically select for surprising, significant results over accurate ones.
For peptide research specifically, this means that the published literature on any given compound is likely to overrepresent positive findings. A peptide that has been tested in five independent laboratories, with three null results and two positive ones, may appear in the literature only through the two positive studies. A researcher reviewing that literature has no way of knowing about the unpublished null results without access to trial registries or direct communication with investigators.
Effect Sizes and Confidence Intervals as Superior Metrics
If p-values are insufficient as primary evidence, what should replace them? The statistical reform literature converges on two answers: effect sizes and confidence intervals [4, 5].
An effect size quantifies the magnitude of a difference or relationship in standardised units, independent of sample size. Cohen's d, for example, expresses the difference between two group means in units of standard deviation. A d of 0.2 is conventionally described as small, 0.5 as medium, and 0.8 as large—though these conventions should be interpreted relative to the specific research context rather than applied mechanically. In a peptide receptor binding study, reporting that one analogue produced a d of 0.3 improvement in binding affinity conveys far more information than reporting p = 0.04, because it allows the reader to assess whether the magnitude of the difference is pharmacologically plausible.
Confidence intervals extend this logic by providing a range of plausible values for the true effect, rather than a single point estimate. A 95% confidence interval does not mean there is a 95% probability the true effect lies within the interval—a common misreading—but rather that the procedure used to construct the interval will contain the true value 95% of the time across repeated experiments [4]. Narrow intervals indicate precision; wide intervals, which are common in small-sample peptide studies, indicate that the data are consistent with a broad range of possible true effects, including negligible ones.
Amrhein and colleagues, writing in Nature, argued that the scientific community should abandon the language of statistical significance altogether and instead report and interpret effect sizes with their confidence intervals as continuous measures of evidence [5]. Whether or not one accepts that position in full, the underlying principle—that a number's distance from an arbitrary threshold is less informative than its magnitude and precision—applies directly to how peptide preclinical data should be read.
Translational Red Flags: When Significance Does Not Translate
The gap between statistical significance and biological relevance becomes most consequential at the translational stage. A peptide that produces a statistically significant 15% improvement in a rodent metabolic marker, detected in a study of eight animals, carries several layers of uncertainty before any human relevance can be inferred.
First, the effect size may be inflated by the winner's curse dynamics described above. Second, rodent models of human physiology are imperfect proxies; effect sizes observed in animals frequently do not replicate in human trials even when the underlying biology is sound. Third, a 15% improvement in a surrogate marker may not correspond to a clinically meaningful change in the outcome of actual interest.
Researchers and readers evaluating peptide preclinical papers should ask a series of specific questions. Was the study pre-registered, with hypotheses and analysis plans specified before data collection? Was a power calculation reported, and does the achieved sample size match the calculated requirement? Were all outcomes and statistical tests reported, or only those that reached significance? Are effect sizes and confidence intervals provided alongside p-values? These questions do not require statistical expertise to ask, but they substantially improve the quality of inference that can be drawn from a paper.
Bayesian Approaches as a Complementary Framework
The frequentist framework—within which p-values and confidence intervals operate—is not the only statistical language available to preclinical researchers. The Bayesian framework offers an alternative that some argue is better suited to the iterative, cumulative nature of scientific evidence.
In Bayesian statistics, prior knowledge about the plausibility of an effect is formally incorporated into the analysis. The result is not a p-value but a posterior probability—a direct statement about how likely the hypothesis is to be true given the data and the prior. For peptide research, where substantial prior knowledge about receptor pharmacology, structural activity relationships, and related compounds often exists, the Bayesian approach allows that knowledge to be used explicitly rather than discarded.
The Bayes factor is a related metric that quantifies how much a dataset should update belief in one hypothesis relative to another. A Bayes factor of 10 in favour of an effect means the data are ten times more likely under the hypothesis that an effect exists than under the null. This is a more direct answer to the question researchers actually want to ask than the p-value provides.
Bayesian methods are gaining traction in clinical trial design and are beginning to appear in peptide pharmacology literature, though they remain less common than frequentist approaches. Their adoption requires familiarity with prior specification, which introduces its own set of judgement calls—but those judgements are made explicit rather than hidden, which is itself a methodological advantage.
A Framework for Critical Appraisal
Reading peptide preclinical literature critically does not require dismissing studies that use conventional statistical methods. It requires asking whether those methods have been applied transparently and whether the reported results support the conclusions drawn.
A well-conducted preclinical peptide study will specify its hypotheses before data collection, report a power calculation justifying its sample size, correct for multiple comparisons where relevant, report effect sizes and confidence intervals alongside p-values, and disclose all outcomes measured regardless of whether they reached significance. Studies meeting these criteria are more likely to produce findings that replicate and that accurately represent the magnitude of any true effect.
Studies that report only significant results, omit power calculations, use sample sizes of five or six animals without justification, and present p-values without effect sizes should be read with heightened scrutiny—not because their authors have acted improperly, but because the design features that guard against false positives are absent.
The p-value is not a flawed tool in itself. Used as one element of a transparent, pre-specified analysis with adequate statistical power, it contributes useful information. The problem lies in treating it as a sufficient condition for scientific credibility. In peptide preclinical research, where the path from animal model to human application is long and uncertain, the standards of evidence applied at the earliest stages matter considerably.