Stock against days: straight line and log fit
R Output: the straight line

    
Residuals against fitted values

    
Residuals in row order

    
Normal QQ plot

    
500 repeated samples: what is true

    
The fix

    
Worked question

      
A logistics officer wants to fit stock on hand after \(x\) days as \(y = \beta_0 e^{\beta_1 x}\). Can lm() fit it? What would you tell them to run, and what does \(\beta_1\) mean afterward?

      
      
Not as written: \(\beta_1\) is used as a power. Take logs of both sides, \(\log y = \log\beta_0 + \beta_1 x + \log\varepsilon\), and the \(\beta\)'s add. Run lm(log(y) ~ x). Then \(\beta_1\) is a rate: each day multiplies the stock by \(e^{\beta_1}\), about \(100\beta_1\) percent per day.

    

  

  
  
◆ The Blueprint

    

      
What it says

      
\(y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \varepsilon\)

      
Each \(\beta\) multiplies one term and the terms add. The \(x\)'s may be squared or logged; the \(\beta\)'s enter as plain multipliers.

      
What it buys you
Least squares solves a set of linear equations for the \(\beta\)'s.

      
What fails
With a \(\beta\) used as a power, or a \(\beta\) you divide by, there are no linear equations to solve, and lm() fits a different model from the one you wrote.

      
Where to look
Read the equation. There is no test. On this tab the straight line's residual plot bends as the curve steepens, and RESET starts to reject; that is assumption 2 catching a model that broke assumption 1.

      
The fix
Take logs of both sides when that makes the \(\beta\)'s add; the slope then means a percentage change. Otherwise the model needs nonlinear least squares, beyond this course.

      
In Wooldridge
MLR.1, Linear in Parameters. The first of the five assumptions behind the Gauss-Markov theorem (Theorem 3.4).

      
Using the dial
The dial is \(\beta_1\), the growth rate per day. At the low end \(e^{\beta_1 x}\) is close to a line over 60 days and both fits forecast day 60 well. Turn it up and watch the straight line's day-60 forecast fall away from the truth while the log model stays on it.

    







  
  

    
Data and fitted line

    
R Output

    
Residuals against fitted values

    
Residuals in row order

    
Normal QQ plot

    
500 repeated samples: what is true

    
The fix

    
Worked question

      
A model of fare on miles fails RESET. Name the two different things that could have caused it, and say which one RESET can actually see.

      
      
A curve (fare rises faster, or slower, than a straight line in miles), or a left-out variable such as the market the ride was in, whose effect leaks into the miles slope. RESET sees the first directly. It only catches the second when the left-out variable happens to bend the fitted relationship; otherwise nothing in the residuals shows it.

    

  

  
  
◆ The Blueprint

    

      
What it says

      
\(\mathrm{E}[\varepsilon \mid x] = 0\)

      
At every \(x\) the errors average zero: no curve the line missed, no left-out variable that moves with \(x\).

      
What it buys you
An unbiased slope. Over repeated samples it averages to the truth.

      
What fails
The slope is the wrong number, and every standard error, \(p\)-value, and interval built on it is wrong. More data makes you surer of the wrong number. The truth panel's sample-size buttons show it: at \(n = 1280\) the interval is narrow and misses the 40-pound slope almost every time.

      
Where to look
Residuals against fitted values; a bend means a missed curve. For a left-out variable, look at the columns the data has that the model leaves out, and at how the data were collected.

      
The test
resettest(model), Ramsey's RESET, adds the squared fitted values and tests whether they help. \(H_0\): the squared fitted values add nothing. \(F\) on 1 and \(n-p-2\) degrees of freedom. A left-out variable has no test.

      
The fix
Add the curve (\(x^2\), \(\log x\)) or the missing variable, and refit. The fix card fits load squared and prints the minutes per pound at 40 and at 70.

      
In Wooldridge
MLR.4, Zero Conditional Mean. The assumption that carries unbiasedness (Theorem 3.1). Wooldridge's discussion of omitted-variable bias (Section 3.3) is the left-out-variable case.

      
Using the dials
Dial 1 is \(c\) in \(\text{time} = 95 + 0.10(\text{load}-35) + c(\text{load}-35)^2 + \varepsilon\). Its middle notch is the memo's data, \(c = 0.028\). At zero the line is the right shape. Dial 2 is the correlation between load and the soldier's unrecorded fitness: at zero loads were handed out at random; turned up, the fitter soldiers carry more. The truth is still a line, and \(\mathrm{E}[\varepsilon \mid \text{load}]\) is no longer zero, so the slope is biased with nothing in the residuals to show it. RESET rejects at its false-alarm rate. The fix card shows what happens when fitness is recorded and added. Every notch reuses the same 80 draws, so only the planted thing changes.

    







  
  

    
Distance against flight hours

    
R Output

    
Fuel against flight hours

    
Residuals against fitted values

    
Residuals in row order

    
Normal QQ plot

    
500 repeated samples: what is true

    
The fix

    
Worked question

      
Meridian Rides: fare ~ miles + minutes + surge on 200 rides. Miles and minutes have correlation 0.989. VIF: miles 46.7, minutes 46.7, surge 1.0. The miles slope is \(-1.24\) (SE 0.73); drop minutes and it becomes 1.83 (SE 0.11). Which slopes in the full model can you read, and which can you not? What does the sign of the miles slope mean?

      
      
Surge can be read: its VIF is 1.0, so nothing in the model copies it. Miles and minutes cannot: the data has almost no rides where one moves without the other, so the model splits one real effect between two slopes arbitrarily. The negative miles slope says nothing about fares.

    

  

  
  
◆ The Blueprint

    

      
What it says

      
no constants with \(x_j = a_0 + \sum_{k \ne j} a_k x_k\)

      
No predictor is a combination of the others.

      
What it buys you
Each predictor gets its own slope, the effect of \(x_1\) with \(x_2\) held fixed.

      
What fails
Perfect: no unique solution, and lm() drops a predictor. Near: the slopes exist, but their standard errors balloon and their signs can flip. Predictions stay fine. The truth panel shows the slope's average staying at 1400 at every notch while its spread explodes: this failure is about precision, and the interval stays honest.

      
Where to look
With two or more predictors, plot them against each other. Points on a straight line are the warning sign.

      
The test
vif(model) from car gives one number per predictor: \(\text{VIF}_j = 1/(1-R^2_j)\), where \(R^2_j\) is from regressing \(x_j\) on the other predictors. Above 5 is worth a look; above 10 is a problem. There is no \(p\)-value.

      
The fix
Drop one of the pair or combine them. Ridge and lasso keep both and shrink the slopes; that is the next lesson.

      
In Wooldridge
MLR.3, No Perfect Collinearity. Wooldridge's variance formula (Section 3.4) has \(1 - R^2_j\) in the denominator; the VIF is that term inverted.

      
Using the dial
The dial is the scatter of distance around \(380 \times\) hours, in nautical miles. Wide scatter means the two predictors move separately. The second-to-last notch is the planner's data, \(r = 0.996\), VIF 136.

    







  
  

    
Data and fitted line

    
R Output

    
Residuals against fitted values

    
Residuals in row order

    
Normal QQ plot

    
500 repeated samples: what is true

    
The fix

    
Worked question

      
Breusch-Pagan regresses the squared residuals on \(x\). Why not the residuals themselves?

      
      
The residuals average zero by construction, so regressing them on \(x\) finds a flat line no matter what. Their size is the spread, and the squared residual at \(x_i\) estimates the error variance there. If that variance changes with \(x\), the squared residuals trend with \(x\).

    

  

  
  
◆ The Blueprint

    

      
What it says

      
\(\mathrm{Var}(\varepsilon \mid x) = \sigma^2\)

      
The spread of the errors is the same at every \(x\).

      
What it buys you
Correct standard errors. Least squares uses one spread for the whole data set.

      
What fails
The slope is still unbiased, but its standard error, \(p\)-value, and every interval are wrong at any sample size: too wide where the data is tight, too narrow where it is loose. The truth panel splits prediction-interval coverage by mileage so you can watch the high-mileage trucks fall out of the window.

      
Where to look
Residuals against fitted values. A fan, wider or narrower as the fitted value grows.

      
The test
bptest(model), Breusch-Pagan, regresses the squared residuals on the predictors; \(H_0\) is that the spread does not depend on them. The statistic is \(n\) times that regression's \(R^2\), against a \(\chi^2\) with one degree of freedom per predictor.

      
The fix
Robust (sandwich) standard errors keep the slope and replace its standard error: coeftest(model, vcov = vcovHC(model)). Logging the response often evens the spread.

      
In Wooldridge
MLR.5, Homoskedasticity. The last of the five Gauss-Markov assumptions; Chapter 8 is heteroskedasticity, with the robust standard errors in Section 8.2 and the Breusch-Pagan test in Section 8.3.

      
Using the dial
The dial is \(f\) in the error's standard deviation \(60 + f \times \text{miles}\). At zero every truck misses the line by about the same amount. The middle notch is the memo's data, \(f = 0.022\).

    







  
  

    
Data and fitted line

    
R Output

    
Residuals against fitted values

    
Residuals in collection order

    
Normal QQ plot

    
500 repeated samples: what is true

    
The fix

    
Worked question

      
Your data is 200 soldiers chosen at random from the brigade roster, one row each. Should you run Durbin-Watson? What does independence mean for this data?

      
      
No. The row order is the order of the roster draw, which means nothing, so a streak or the lack of one says nothing. Independence here is about how the sample was drawn: random selection makes the errors independent, and no residual plot can check that. A sample of 200 soldiers from one platoon would not be independent, and no test would show it.

    

  

  
  
◆ The Blueprint

    

      
What it says

      
\(\mathrm{Cov}(\varepsilon_i, \varepsilon_j) = 0\) for \(i \ne j\)

      
One observation's error says nothing about the next one's.

      
What it buys you
Correct standard errors. Each observation is a fresh piece of information, so \(n\) observations count as \(n\).

      
What fails
The slope is still unbiased, but correlated observations carry less information than the same number of independent ones. The standard error comes out too small, and the \(p\)-values overstate the evidence. The truth panel shows how often the printed \(p\) lands below 0.001 against how often the honest one clears 0.05.

      
Where to look
Residuals in the order collected. Long runs above and then below zero. If the row order means nothing, the plot means nothing.

      
The test
dwtest(model), Durbin-Watson, tests \(H_0\) that consecutive residuals are uncorrelated. DW runs from 0 to 4: near 2 means none, well below 2 means each residual resembles the last.

      
The fix
Newey-West standard errors, coeftest(model, vcov = NeweyWest(model)). Or add yesterday's \(y\) as a predictor, so the model carries the day-to-day dependence.

      
In Wooldridge
For a cross section this is MLR.2, Random Sampling: independence comes from how the rows were drawn. For rows in time order it is TS.5, No Serial Correlation, in Chapter 10, with the Durbin-Watson test and Newey-West standard errors in Chapter 12.

      
Using the dial
The dial is \(\varphi\), the share of yesterday's miss that carries into today. The innovations behind the memo's data were recovered and rescaled so every notch has the same draws and the same day-to-day spread; only the carry-over changes. The memo's data is \(\varphi = 0.8\).

    







  
  

    
Data and fitted line

    
R Output

    
Residuals against fitted values

    
Residuals in row order

    
Normal QQ plot

    
500 repeated samples: what is true

    
The fix

    
Worked question

      
\(n = 200\) and Shapiro-Wilk rejects. Which of these still works: the \(p\)-value on the slope, or a 95% prediction interval for one new observation? Why?

      
      
The \(p\)-value on the slope works: at \(n = 200\) the slope is close to normal whatever the errors look like. The prediction interval does not: it is built from one error, and that error has the skewed shape the QQ plot showed, so a 95% interval from the normal formula covers some other percentage.

    

  

  
  
◆ The Blueprint

    

      
What it says

      
\(\varepsilon \sim N(0, \sigma^2)\)

      
Most misses small, big misses rare, above and below the line equally likely.

      
What it buys you
Exact \(t\) and \(F\) tests and intervals at any sample size.

      
What fails
At large \(n\) the slope's \(t\)-test and confidence interval still work, because the slope averages many errors and averages come out normal. The prediction interval for one new observation is wrong, because it depends on the shape of a single error. The truth panel scores both.

      
Where to look
A QQ plot of the residuals. Points peeling off the line at one end are a heavy or skewed tail on that side.

      
The test
shapiro.test(resid(model)), \(H_0\) that the residuals are normal. At large \(n\) it rejects small departures that do not matter, so read it with the QQ plot.

      
The fix
Log the response when the tail is on the high side. Bootstrap the prediction interval. The fix card fits the log model and prints both windows for a complexity-5 ticket.

      
In Wooldridge
MLR.6, Normality. It sits on top of the Gauss-Markov five and gives the classical linear model assumptions (Section 4.1). Wooldridge's Section 5.2 is the large-sample argument for why the slope's tests survive without it.

      
Using the dial
The error is \(9(a z + b(E - 1))\) with \(z\) standard normal, \(E\) standard exponential, and \(a^2 + b^2 = 1\), so the spread is the same at every notch and only the shape changes. The dial is \(b\). At the far right the error is a shifted exponential, the memo's data. Skewness is \(2b^3\).

    







  
  

    
Data and fitted line

    
R Output

    
Residuals against fitted values

    
Residuals in collection order

    
Normal QQ plot

    
500 repeated samples: what is true

  

  
  
◆ The Blueprint

    

      
The Anvil
Every case here was computed once, in R, with a seed: 81 combinations of the four dials, each fitted, tested, and then repeated 500 times with the design held fixed. The page swaps between them. Nothing is generated live.

      
The dials

      
Curve adds \(c(x-5)^2\) to the truth. The single slope is then 2 on average but wrong at both ends. Zero conditional mean.

      
Fan scales the error's spread by \(1 + f\,x/5\). The slope is fine; the standard error and the prediction window are wrong. Equal variance.

      
Carry-over makes each row's error \(\varphi\) times the last row's plus a fresh piece. The printed standard error is too small. No autocorrelation.

      
Skew mixes an exponential into the error, \(a z + b(E-1)\) with \(a^2 + b^2 = 1\). The slope's interval holds; the prediction window fails on the high side. Normal errors.

      
What to try

      
Turn one dial to strong and read which row of the truth panel moves. Then turn a second one and see whether the tests can still tell you which is which. Fan plus skew is the pair that confuses Shapiro-Wilk. Carry-over is the one the residual-versus-fitted plot never shows.

      
Wooldridge's theorem
Under MLR.1 to MLR.5 the least squares estimators are the best linear unbiased estimators, BLUE (Theorem 3.4). The Anvil with every dial off is that case. Each dial removes one of the five, or MLR.6, and the truth panel shows what was lost: the U in unbiased, the standard error that made it best, or the exact window that normality bought.

      
Credits

      
Jeffrey M. Wooldridge, Introductory Econometrics: A Modern Approach, Chapters 3, 4, 8, 10, and 12, for the assumption list, the BLUE argument, and the plain reading of what each assumption buys. The stories, the questions, and the data are from the MA376 regression diagnostics lesson.

      
Other Forge tools: OLS Basics, Regression & Regularization, Bias-Variance.