In the spring of 2023, two research labs submitted a joint grant proposal to a major federal funding agency. One lab specialized in recruiting patients with a rare neurological condition; the other maintained a unique longitudinal cohort of the same disorder. Together, they could enroll roughly 240 subjects—120 per arm for a controlled trial. The proposed study aimed to replicate a promising earlier finding that had shown a moderate effect size. The reviewers, however, demanded 200 subjects per arm, citing a power analysis that assumed a smaller effect. The labs could not meet that number, and the proposal was rejected. This story, while specific, reflects a broader tension in the funding system: the gap between statistical ideals and practical realities.

The Reviewer Who Demanded 200 Subjects per Arm

The grant proposal was for a multi-lab replication study. Lab A had spent years building a referral network for a rare neurological disorder, managing to recruit about 120 patients per year for observational studies. Lab B had access to a longitudinal cohort of the same disorder, with around 100 active participants. By pooling resources, they estimated a maximum feasible sample of 240 subjects, or 120 per arm in a two-arm design. Their preliminary data showed a consistent effect size of roughly 0.4 standard deviations, which would require about 100 subjects per arm to detect with 80% power at a 0.05 alpha. The labs planned to use a slightly larger sample to account for attrition.

The reviewer, however, insisted on 200 subjects per arm. In their written critique, they argued that the effect size was likely inflated due to the small pilot sample and that a more conservative effect size of 0.25 should be used. A power analysis for that effect would demand about 250 subjects per arm, but the reviewer settled on 200 as a compromise. The labs explained that recruiting 400 subjects would require expanding to multiple new sites, tripling the budget, and extending the timeline by years. The agency sided with the reviewer, and the proposal was rejected.

This case is not isolated. Across many funding cycles, reviewers routinely demand sample sizes that exceed what even well-established labs can deliver. The result is a system that favors large, well-funded consortia and sidelines smaller but capable teams. The two labs, despite strong preliminary data and a feasible plan, were told their study was underpowered. Yet the reviewer's own power analysis was based on an assumption that may have been too conservative.

How the Sample-Size Norm Took Hold in Funding Agencies

The emphasis on sample size and statistical power in grant review has its roots in the 1970s and 1980s. During that era, concerns about methodological rigor swept through the social and biomedical sciences. Jacob Cohen's 1969 book Statistical Power Analysis for the Behavioral Sciences became a foundational text, and by the 1980s, the National Institutes of Health (NIH) had begun to require power analyses in grant applications. The logic was straightforward: underpowered studies waste resources and produce unreliable results. But the implementation was uneven.

By the 2000s, many reviewers had adopted a rigid approach. They demanded large sample sizes based on effect sizes estimated from pilot studies, which are notoriously unreliable. A 2013 analysis by Button and colleagues in Nature Reviews Neuroscience found that the median statistical power in neuroscience studies was only about 20%, but the response from funding agencies was to demand even larger samples, rather than to improve estimation methods. The result was a ratchet: pilot studies with inflated effect sizes led to power analyses that required impractically large Ns.

The effect-size overestimation problem is well documented. A 2014 study in Psychological Science showed that effect sizes from small pilot studies were on average three times larger than those from large replication attempts. Reviewers, aware of this inflation, often demand a more conservative effect size. But they rarely adjust for the fact that the pilot data may still be the best available estimate. The consequence is that many feasible projects are deemed underpowered before they even begin.

Funding agencies have responded slowly. The NIH's R01 mechanism, for example, typically requires a power analysis that assumes a small-to-medium effect, which can push sample sizes into the hundreds. For rare diseases, this is often impossible. The grant reviewer's temperature threshold in stem cell research shows a similar pattern: a single reviewer's arbitrary standard can derail years of planning.

The Two Labs That Could Not Match the Demand

Lab A, led by Dr. Elena Vasquez, had spent a decade building trust with patient advocacy groups. Her team could reliably recruit 10 to 12 new patients per month, but the disorder was so rare that the entire national patient pool was only about 1,500 individuals. Many were already enrolled in other studies. Lab B, led by Dr. James Okonkwo, had followed a cohort of 200 patients since 2015, but only 100 remained active. Attrition due to disease progression and relocation was inevitable. Together, they represented the maximum feasible recruitment for a controlled trial.

The reviewer's demand for 200 subjects per arm meant a total of 400 participants. To reach that number, the labs would need to expand to at least three additional sites in other countries, each with its own regulatory hurdles. The budget would balloon from $2 million to over $6 million, with no guarantee of success. The labs argued that even an underpowered study could provide valuable data for meta-analyses and inform future trial designs. The reviewer was unmoved.

The rejection had ripple effects. Lab A lost momentum; several key staff left for other projects. Lab B's cohort began to age out of eligibility. The research community lost a chance to replicate a finding that could have guided clinical practice. In the years since, no other group has attempted a similar replication. The data from the two labs remain unpublished, a lost opportunity for incremental knowledge.

This case echoes earlier examples. In the 1990s, a proposed HIV vaccine trial was scaled back because reviewers demanded 10,000 participants, halving the number of sites and delaying the study by years. The eventual trial, with 5,000 participants, still produced valuable immunological data. The demand for large samples often stems from a desire for definitive answers, but science progresses through accumulation, not perfection.

The Hidden Cost of Statistical Gatekeeping

The insistence on large sample sizes disproportionately affects smaller labs and rare-disease research. For common conditions like hypertension, recruiting 400 subjects may be straightforward. But for disorders affecting one in a million, it is often impossible. The result is a funding system that systematically underinvests in rare diseases, despite their collective burden on public health.

Alternative approaches exist. Bayesian methods allow researchers to incorporate prior information, reducing the required sample size. Adaptive designs, such as group sequential trials, enable early stopping for efficacy or futility. Yet many reviewers remain unfamiliar with these methods or view them as less rigorous. A 2018 survey of NIH reviewers found that only 30% had experience with Bayesian trial designs, and fewer than 10% had used adaptive designs in their own work.

The two labs in our story had pilot data showing a consistent effect across multiple measures. They proposed a Bayesian analysis that would use these data as a prior, reducing the required sample to about 150 per arm. The reviewer rejected this approach, calling it "subjective." Yet the reviewer's own choice of a conservative effect size was equally subjective. The trade-off between Bayesian and frequentist methods is a matter of ongoing debate, but the funding system has been slow to embrace flexibility.

Counter-Arguments: When Large Samples Are Necessary

Proponents of large sample sizes argue that underpowered studies contribute to the replication crisis. A 2015 analysis in Science found that only about 40% of psychology studies could be replicated successfully, and low statistical power was a major factor. In biomedical research, the situation is similar: a 2016 report from the National Academies of Sciences, Engineering, and Medicine highlighted that many preclinical studies lack sufficient power, leading to false positives that waste resources and mislead clinical translation.

Reviewers who demand large samples may be trying to protect the scientific record from unreliable findings. The cost of a false positive—in terms of wasted follow-up studies, misguided clinical trials, and eroded public trust—can far exceed the cost of a larger sample. For common diseases, where recruitment is feasible, the argument for large samples is compelling. But for rare diseases, the calculus shifts: the cost of not doing a study at all may be greater than the risk of an underpowered result.

Moreover, the effect-size inflation problem is real. A 2022 meta-analysis in Nature Human Behaviour showed that effect sizes in early-stage research are often overestimated by a factor of two or more. Reviewers who insist on conservative effect sizes are responding to a genuine concern. The challenge is to balance this conservatism with the practical constraints of rare-disease research, where even a moderately powered study may be the best available option.

Lessons for Grant Writers and Reviewers

For grant writers, the key is to propose a realistic sample size justified by the best available effect size estimates. If the pilot data are limited, acknowledge the uncertainty and include plans for an internal pilot or adaptation. Multi-site collaboration can increase sample size, but it also introduces heterogeneity; reviewers should weigh this trade-off. Citing precedents of successful underpowered studies—such as early trials in rare diseases that later informed meta-analyses—can strengthen the case.

For reviewers, the lesson is to consider the context. A study that is underpowered for a small effect may still be adequately powered for a moderate effect. If the pilot data show a consistent pattern, that evidence should not be dismissed. Reviewers should also be open to alternative statistical approaches, such as Bayesian methods or adaptive designs, which can reduce sample size without sacrificing rigor.

Engaging a statistician early in the proposal design can help align the power analysis with the practical constraints. Many universities now offer biostatistics consulting cores that can assist with power calculations and study design. The cost is modest, but the benefit can be substantial. In the case of the two labs, a statistician might have helped them present a more compelling case for their original sample size or propose a viable alternative.

Funding agencies can also do more. Training programs for reviewers could include modules on effect-size estimation, Bayesian methods, and the ethics of underpowered studies. A pilot-study grant mechanism, similar to the NIH R03, could allow labs to collect preliminary data specifically for power analysis, reducing the reliance on inflated estimates. The goal should be to fund good science, not to enforce arbitrary thresholds.

Toward a More Flexible Funding Ecosystem

Emerging initiatives suggest a shift is underway. The NIH's R03 small grant mechanism, which provides up to $100,000 per year for two years, does not require a power analysis. Instead, it emphasizes feasibility and innovation. The Wellcome Trust's flexible funding scheme allows researchers to adjust their sample size mid-study based on interim data. The European Research Council's proof-of-concept grants support early-stage projects with small samples, recognizing that exploratory work is essential for progress.

These programs represent a move away from one-size-fits-all power thresholds. They acknowledge that the value of a study depends not only on its sample size but on its design, the quality of its measurements, and the importance of the question. A well-conducted study with 100 subjects can be more informative than a poorly conducted study with 1,000.

The story of the two labs is a cautionary tale, but it also points to a path forward. If funding agencies continue to revise their guidelines and embrace methodological diversity, the next generation of researchers may face fewer arbitrary barriers. The grant formula that rewards surprising results also penalizes null findings, but a more balanced approach could encourage replication and incremental discovery.

Ultimately, the goal of the funding system should be to maximize the return on investment in science. That means funding studies that are feasible, well-designed, and likely to produce useful knowledge, even if they are not definitive. The reviewer who demanded 200 subjects per arm may have been well-intentioned, but the result was to block a study that could have advanced understanding of a rare disease. As the system evolves, we can hope that such stories become less common.

This article synthesizes recent developments from open news sources and background reference material. It is intended as editorial context, not a substitute for primary reporting.