Programmatic p-hacking from scratch with Python
Below is a short script to demonstrate the 'process of p-hacking'.
From the Data Science from Scratch book.
import random
from typing import List
First we define a usual experiment consisting of 1000 binomial trials with 0.5 probability.
def run_experiment(trials) -> List[bool]:
return [random.random() < 0.5 for _ in range(trials)]
experiment = run_experiment(1000)
print("Proportion of heads:", sum(experiment) / len(experiment))
print("First 10 elements:", experiment[:10])
Then we examine whether the outcome an experiment is beyond the 95% confidence levels around p = 0.5, that is, the hypothesis of having a fair coin.
def reject_fairness(experiment: List[bool]) -> bool:
num_heads = sum(experiment)
return num_heads < 469 or num_heads > 531
reject_fairness(experiment)
We run 1000 independent experiments with the exact same parameters.
random.seed(42)
experiments = [run_experiment(1000) for _ in range(1000)]
Now we can simply pick those experiments which fall outside the confidence level.
number_of_unfair = sum([reject_fairness(experiment) for experiment in experiments])
print("Number of experiments 'showing' that the coin if unfair:", number_of_unfair)
print("\nProbabilities:")
print("\t".join([str(sum(experiment) / len(experiment)) for experiment in experiments if reject_fairness(experiment)]))