Model Evaluation

Goodness of Fit for Logistic and Poisson

Last week: Model Evaluation for MLR

Does the model fit the data well? Do we need more/less variables in the model?

• Goodness of fit

• \(F\)-test to compare our model against an intercept-only model

• a more general F-test to compare nested models

Most metrics used to evaluate LR measure how far \(y\) is from \(\hat{y}\) Can we do the same for other logistic and Poisson models?

Today: Model Evaluation beynod MLR

• Residuals

• Deviance

• Chi-square test

Prediction: Logistic Regression

An estimated logistic model can be used to predict an outcome

• the predicted outcome depends on the form of the model
  • log-odds (default, since this is a linear model)
  • probabilities
  • odds
• we can predict the outcome of:
  • observations in the same sample used to estimate the model, called fitted values
  • new observations, usually refered as predicted values

Prediction: Poisson Regression

An estimated Poisson model can be used to predict an outcome

• the predicted outcome depends on the form of the model
  • log-mean (default, since this is a linear model)
  • mean
• we can predict the outcome of:
  • observations in the same sample used to estimate the model, called fitted values
  • new observations, usually refered as predicted values

Residuals for Logistic regression

Scroll down to see full content

The raw residuals are, as usual, the differences between the observed and the fitted values.

\[\begin{aligned} r_i &= \text{obs}_i - \text{fit}_i \\ &= y_i - \hat{p}_i \end{aligned}\]

However, there are 2 problems with these residuals:

• the variance of the observations is not constant! for a binary response:

\[Var(Y_i) = p_i(1-p_i)\]

technically, we should write the conditional variance and probability

Thus, residuals of different observations are not comparable and should be adjusted!!

• the \(y_i\) take only 2 values \(y_i = 0\) and \(y_i = 1\), so the raw residuals have values \(-\hat{p}_i\) and \(1- \hat{p}_i\), respectively

Scroll down to see full content

Many variations of the raw residuals have been proposed for Logistic regression (and GLMs in general) that take the variance of the observations into account.

• Pearson residuals: divides the raw residuals by the standard deviation of the response

\[r_i = \frac{y_i - \hat{p}_i}{\sqrt{\hat{p}_i(1-\hat{p}_i)}}\]

• deviance residuals

• standardized residuals: for both Pearson and deviance residual

We won’t get to the details of these but it is important to know which ones you are calculating.

Residuals for Poisson regression

The Poisson regression has the same problems as the Logistic Regression:

• responses are discrete (at least more than 2 values now)

• the variance is not constant

\[\text{resid}_{\text{raw}}= y_i - \hat{\lambda}_i, \quad\quad \text{resid}_{\text{pearson}}= \frac{y_i - \hat{\lambda}_i}{\sqrt{\hat{\lambda}_i}}\]

Deviance

Scroll down to see full content

Beyond MLR, a quantity called the deviance is used to measure the difference between the log-likelihoods the estimated model and that of a perfect model, known as the saturated model, which fits the data perfectly.

Image from Notes for Predictive Modeling

Deviance: interpretation

The deviance generalizes the Residual Sum of Squares (RSS) of the linear model.

If the errors of a LR are normal, the deviance is proportional to the RSS.

The lower the deviance the better the fit!

Deviance to test

Image from Notes for Predictive Modeling

The deviance can also be used to compare nested models (similar to the \(F\)-test in LR).

The test that compares deviances has a \(\chi^2\) distribution.

Don’t worry, R computes this test for you using anova()!!