LECTURE 2: Introduction to linear models

Outline

1) Basic structure of a linear model

2) Parameter interpretation

3) Assumptions

Statistics = Information + Uncertainty

In the last lecture, we learned that statistics is what allows us to make inferences about a population in the face of uncertainty

Statistical inference requires models

What is a model?

What is a model?

An abstraction of reality used to describe the relationship between two or more variables

Many types (conceptual, graphical, mathematical)

In this class, we will deal with statistical models¹

• Mathematical representation of our hypothesis

• Explicitly measure and account for uncertainty

• By necessity, models will be simplifications of reality (“all models are wrong…”)

• Do not have to be complex

But i don’t want to be a modeler!

• Inference requires models

• Models link observations to processes

• Models are tools that allow us understand processes that we cannot directly observe based on quantities that we can observe

A simple example

Suppose we are interested in the hypothesis that water availability limits acorn production by oak trees

• Prediction: Acorn production increases following years with higher rainfall

A simple model

\[\LARGE y = a + bx\]

It may not be obvious, but this is essentially the only model we will use this semester²

A simple model

\[\LARGE y = a + bx\]

If we want to use this as a statistical model, what’s missing?

Stochasticity

A simple model

\[\LARGE y = a + bx\]

If we want to use this as a statistical model, what’s missing?

Stochasticity

The linear model

Statistics cookbook

The linear model

\[ response = deterministic\; part+stochastic\; part\]

\[\underbrace{\large E[y_i] = \beta_0 + \beta_1 \times x_i}_{Deterministic}\]

\[\underbrace{\large y_i \sim normal(E[y_i], \sigma)}_{Stochastic}\]

The linear model

Sometimes you will see linear models written as:

\[\Large y_i = \beta_0 + \beta_1 x_i + \epsilon_i\]

\[\Large \epsilon_i \sim normal(0, \sigma)\]

with the \(\large \epsilon_i\) terms referred to as residuals or residual error

• Mathematically, the two formulations are identical

• We will use both formulations

A note on notation

Throughout the semester, we will try to be consistent with mathematical notation, e.g.

• \(y\) = response/dependent variable
• \(x\) = predictor/independent variable
• \(\mu\) = population mean
• \(\sigma\) = population standard deviation
• \(\beta\) = model parameter (i.e., intercept/slope)

To the extent possible, these follow conventions used in many textbooks/papers (but there is a lot of variation between authors)

Remember that these are just symbols - you could replace them with other symbols (e.g., emoji) and it would not change their interpretation!

Parameter interpretation

\(\large \beta_0\) = intercept

• The expected value of the response variable, \(\large E[y_i]\), when all predictors \(=0\)

\(\large \beta_n\) = slope

• The expected change in the response variable for a one unit change in the associated predictor variable \(\large x_n\)

Mathematically, the interpretation of the intercept and slope(s) is always the same

Ecologically, the interpretation will depend on:

• the structure of the predictor variables (continuous vs. categorical)
• the units of the predictor variables (e.g., scaled vs unscaled)
• the inclusion of interaction terms

We will discuss each of these scenarios in detail as the semester progresses

The linear model

A “simple” example

\[\underbrace{\large E[y_i] = -2 + 0.5 \times x_i}_{Deterministic}\]

\[\underbrace{\large y_i \sim normal(E[y_i], \sigma=0.25)}_{Stochastic}\]

The linear model

A more complex model

\[\large y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2} + ... + \beta_px_{ip} + \epsilon_i\]

• Each \(\beta\) coefficient is the effect of a specific predictor variables \(x\)
• Predictor variables may be continuous, binary, factors, or a combination
• We will cover more complex models (and interpretation) later

Is this a linear model?

\[\large y = 20 + 0.5x - 0.3x^2\]

Residuals

One concept we will talk about a lot is residuals, i.e., \(\epsilon_i\)

• Residuals are the difference between the observed values \(y_i\) and the predicted values \(E[y_i]\) — how much variation in \(y\) is explained by \(x\)?

Residuals

One concept we will talk about a lot is residuals, i.e., \(\epsilon_i\)

• Residuals are the difference between the observed values \(y_i\) and the predicted values \(E[y_i]\) — how much variation in \(y\) is explained by \(x\)?

• Parameters are generally estimated by finding values that minimize residual error (e.g., ordinary least squares)
• Useful for assessing whether data violate model assumptions

Assumptions

Assumptions

EVERY model has assumptions

• Assumptions are necessary to simplify real world to workable model
• If your data violate the assumptions of your model, inferences may be invalid
• Always know (and test) the assumptions of your model³

Linear model assumptions

\[\Large y_i = \beta_0 + \beta_1 x_i + \epsilon_i\]

\[\Large \epsilon_i \sim normal(0, \sigma)\]

1. Linearity: The relationship between \(x\) and \(y\) is linear
2. Normality: The residuals are normally distributed⁴
3. Homogeneity: The residuals have a constant variance at every level of \(x\)
4. Independence: The residuals are independent (i.e., uncorrelated with each other)

Assumptions

\[\large y_i = \beta_0 + \beta_1x_i + \epsilon_i\] \[\large \epsilon_i \sim normal(0, \sigma)\]

Assumptions - linearity

\[\large y_i = \color{#D47500}{\beta_0 + \beta_1x_i} + \epsilon_i\] \[\large \epsilon_i \sim normal(0, \sigma)\]

Assumptions - normality

\[\large y_i = \color{#D47500}{\beta_0 + \beta_1x_i} + \epsilon_i\] \[\large \epsilon_i \sim \color{#3CB521}{normal}(0, \sigma)\]

Assumptions - homogeneity

\[\large y_i = \color{#D47500}{\beta_0 + \beta_1x_i} + \epsilon_i\] \[\large \epsilon_i \sim \color{#3CB521}{normal}(0, \color{#CD0200}\sigma)\]

Assumptions - normality

Remember that the normality assumption applies to the residuals not the data.

In the data below, the histogram of the response variable y shows that the data is clearly not normally distributed

Assumptions - normality

Remember that the normality assumption applies to the residuals not the data.

In the data below, the histogram of the response variable y shows that the data is clearly not normally distributed

But a histogram of the residuals (green lines) shows that they are normal (or at least close)

Linear models

Very flexible

• Predictor(s) can take different forms (binary, continuous, factor)
• Can contain many predictors
• Can model non-linear relationships

Link different “tests” (e.g., t-tests, ANOVA, ANCOVA, linear regression)

Can be used for different statistical goals

• Estimating unknown parameters
• Testing hypotheses
• Describing stochastic systems
• Making predictions that account for uncertainty

Statistical inference

Linear models allow us to quantify relationships between variables

Generally, models are fit to samples

• The intercept and slope values correspond to the observed response and predictor values

• If we repeated our study, the sampled values would change and so would the intercept/slope values

We’re generally interested in the population, not the sample

• The intercept and slope values fitted to a particular sample are considered estimates of the true population values

Inference about populations requires quantifying how well our estimated values represent the true population values (which we can never know)

• This will be the topic of the next lecture (and a theme we will revisit throughout the semester)

Looking ahead

Next time: Principles of statistical inference

Reading: Feiberg chp. 1.6-1.8

Footnotes

1. Though graphical models will play an important role later in the semester

2. With some minor variations, mainly in \(x\)

3. You know what happens when you assume…

4. Note that these assumptions apply to the residuals, not the data!