A statistical test will not be able to detect a true difference if the sample size is too small compared with the magnitude of the difference.
Since data are sampled at random, there is always a risk of reaching a wrong conclusion, and things can go wrong in two ways - Dalgaard (2008)
Null hypothesis testing
Null hypothesis testing
Null hypothesis testing
Null hypothesis testing
Null hypothesis testing
Null hypothesis testing
Null hypothesis testing
Type i & type ii errors
Type I error (i.e., false positive)
The null hypothesis is correct, but the test rejects it.
\[\alpha = Pr(Type\;I\;error)\]
Type II error (i.e., false negative)
The null hypothesis is wrong, but the test fails to reject it.
\[\beta = Pr(Type\;II\;error)\]
Power
The test’s ability to reject a false null hypothesis.
\[Power = 1 - \beta\]
Type i & type ii errors
The type I error rate is set by the scientist
The type II error rate, and hence the power of the test, depends on many factors
In the context of a linear model, these are:
Magnitude of the slope coefficients (\(\beta\))
Standard deviation (or variance) of population (\(\sigma\))
The sample size (\(n\))
The Type I error rate (\(\alpha\))
Magnitude of the difference
Magnitude of the difference
Standard deviation
Standard deviation
Sample size
Sample size
Type i error rate
\(\alpha = 0.05\)
type i error rate
\(\alpha = 0.001\)
Factors affecting power
In general, power increases when:
The difference in means/magnitude of slope increases
The standard deviation of the population decreases
The sample size increases
The Type I error rate increases
Question: Which of these, as researchers, do we have control over?
Example in R
\(\mu_1 = 90\), \(\mu_2 = 100\)
\(\sigma = 5\)
Example in R
\(\mu_1 = 90\), \(\mu_2 = 100\)
\(\sigma = 5\)
\(n = 5\)
power.t.test(n =5, delta =10, sd =5, sig.level =0.05, power =NULL)
Two-sample t test power calculation
n = 5
delta = 10
sd = 5
sig.level = 0.05
power = 0.7905
alternative = two.sided
NOTE: n is number in *each* group
Example in R
\(\mu_1 = 90\), \(\mu_2 = 100\)
\(\sigma = 5\)
\(n = 15\)
power.t.test(n =15, delta =10, sd =5, sig.level =0.05, power =NULL)
Two-sample t test power calculation
n = 15
delta = 10
sd = 5
sig.level = 0.05
power = 0.9996
alternative = two.sided
NOTE: n is number in *each* group
Example in R
\(\mu_1 = 90\), \(\mu_2 = 100\)
\(\sigma = 5\)
\(\alpha = 0.001\)
power.t.test(n =15, delta =10, sd =5, sig.level =0.001, power =NULL)
Two-sample t test power calculation
n = 15
delta = 10
sd = 5
sig.level = 0.001
power = 0.9501
alternative = two.sided
NOTE: n is number in *each* group
Example in R
\(\mu_1 = 94\), \(\mu_2 = 97\)
\(\sigma = 5\)
\(n = 15\)
power.t.test(n =15, delta =3, sd =5, sig.level =0.001, power =NULL)
Two-sample t test power calculation
n = 15
delta = 3
sd = 5
sig.level = 0.001
power = 0.03597
alternative = two.sided
NOTE: n is number in *each* group
Example in R
\(\mu_1 = 94\), \(\mu_2 = 97\)
\(\sigma = 5\)
\(n = 100\)
power.t.test(n =100, delta =3, sd =5, sig.level =0.001, power =NULL)
Two-sample t test power calculation
n = 100
delta = 3
sd = 5
sig.level = 0.001
power = 0.8143
alternative = two.sided
NOTE: n is number in *each* group
When should I do a power analysis?
Retrospective (conducted after experiment)
If you failed to reject the null, then your power was low
But you can’t use this as an excuse!
Only useful as a way of planning a subsequent experiment
Prospective (Done before the experiment)
Used to determine sample size or power, given \(\beta\) and \(\sigma\)
How can \(\beta\) and/or \(\sigma\) be known ahead of time?
Requires prior knowledge, perhaps from a pilot study
Requires clear-headed thinking about what consitutes a biologically signiffcant difference
Prospective is always better than retrospective!
What level of power should I aim for?
We want power to be as close to 1 as possible
Sometimes it may be prohibitively expensive to obtain a sample size large enough to achieve power close to 1
In practice, we are usually satisfied with power > 0.8
Summary
Power analysis let’s you determine the necessary sample size (or power) for testing an effect size of interest
Power is influenced by the magnitude of the effect, the standard deviation of the population, the Type I error rate, and the sample size
Retrospective power analysis isn’t useful unless you are planning a subsequent experiment
R has several functions for conducting power analysis, but only for simple tests
More complicated power analysis can be performed using simulation (not covered in this course)
