Data is drawn from two identical normal distributions, . For each sample size , ranging from 5 to , a t-test is performed on the first <math>n<math> samples from each distribution, and the resulting p-value is plotted. The red dashed line indicates the commonly used significance level of 0.05.

If the data collection or analysis were to stop at a point where the p-value happened to fall below the significance level, a spurious statistically significant difference could be reported.

Illustration based on

Wagenmakers, Eric-Jan. "A practical solution to the pervasive problems of p values." Psychonomic bulletin & review 14.5 (2007): 779-804.

```python import numpy as np import matplotlib.pyplot as plt from scipy import stats

1. Set random seed for reproducibility

np.random.seed(42)

1. Function to perform t-test and return p-value

def perform_t_test(sample1, sample2):

   _, p_value = stats.ttest_ind(sample1, sample2)
   return p_value

1. Initialize parameters

max_samples = 10**4 start_samples = 5 p_values = [] sample_sizes = range(start_samples, max_samples + 1)

1. Generate data and perform t-tests

population1 = stats.norm(loc=0, scale=10) population2 = stats.norm(loc=0, scale=10)

samples1 = population1.rvs(max_samples) samples2 = population2.rvs(max_samples)

for n in sample_sizes:

   p_value = perform_t_test(samples1[:n], samples2[:n])
   p_values.append(p_value)

1. Create the plot

plt.figure(figsize=(12, 6)) plt.semilogx(sample_sizes, p_values, 'b-') plt.axhline(y=0.05, color='r', linestyle='--', label='p = 0.05') plt.xlabel('Sample Size (log scale)') plt.ylabel('p-value') plt.title('Variability of p-value as Sample Size Increases') plt.grid(True, which="both", ls="-", alpha=0.2) plt.legend() plt.ylim(0, 1) plt.tight_layout() plt.savefig('p-hacking.svg') plt.show()