7FFB?;:H;=H;II?ED7D7BOI?

I
J^_hZ;Z_j_ed
FTP SITE
NOW
AVAILABLE

DEHC7DH$:H7F;H
>7HHOIC?J>
M_b[oI[h_[i_dFheXWX_b_joWdZIjWj_ij_Yi

Applied Regression Analysis

WILEY SERIES IN PROBABILITY AND STATISTICS

TEXTS AND REFERENCES SECTION
Established by WALTER A. SHEWHART and SAMUEL S. WILKS
Editors: Vic Barnett, Ralph A. Bradley, Noel A. C. Cress ie, Nicholas I. Fisher,
lain M. Johnstone, 1. B. Kadane, David G. Kendall, David W. Scott,
Bernard W. Silverman, Adrian F. M. Smith, Joze! L. Teugels, Geoffrey S. Watson;
1. Stuart Hunter, Emeritus
A complete list of the titles in this series appears at the end of this volume.

Applied Regression Analysis

THIRD EDITION

Norman R. Draper
Harry Smith

A Wiley-Interscience Publication
JOHN WILEY & SONS, INC.
New York . Chichester . Weinheim

. Brisbane

Singapore

. Toronto

This book is printed on acid-free paper. @)

Copyright 1998 by John Wiley & Sons, Inc. All rights reserved.
Published simultaneously in Canada.
No part of this publication may be rcproduced, stored in a retrieval system or transmilled
in any form or by any means, electronic, mechanical, photocopying, recording, scanning
or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States
Copyright Act, without either the prior written permission of the Publisher, or
authorization through payment of the appropriate per-copy fee to the Copyright
Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax
(978) 750-4470. Requests to the Publisher for permission should be addressed to the
Permissions Department, John Wiley & Sons, Inc., III River Street, Hoboken, NJ 07030,
(201) 748-6011 , fax (201) 748-6008.
Library of Congress Cataloging-in-Publication Data:

Draper, Norman Richard .

Applied regression analysis / N.R. Draper, H. Smith. - 3rd ed.
p. cm. - (Wiley series in probability and statistics. Texts
and references section)
,. A Wiley-Interscience publication."
Includes bibliographical references (p. - ) and index.
ISBN 0-471-17082-8 (acid-free paper)
II. Title.
I. Regression analysis. I. Smith, Harry. 1923Ill. Series.
QA278.2.D7 1998
97-17969
SI9.S'36-dc21
CIP
Printed in the United States of America.

18

Contents

Preface
About the Software

o Basic Prerequisite Knowledge

xiii
xvii

0.1 Distributions: Normal, t, and F, 1

0.2 Confidence Intervals (or Bands) and t- Tests, 4
0.3 Elements of Matrix Algebra, 6
1 Fitting a Straight Line by Least Squares

15

1.0
1.1
1.2
1.3

Introduction: The Need for Statistical Analysis, 15

Straight Line Relationship Between Two Variables, 18
Linear Regression: Fitting a Straight Line by Least Squares, 20
The Analysis of Variance, 28
1.4 Confidence Intervals and Tests for f3() and f31, 34
1.5 F- Test for Significance of Regression, 38
1.6 The Correlation Between X and Y, 40
1.7 Summary of the Straight Line Fit Computations, 44
1.8 Historical Remarks, 45
Appendix lA Steam Plant Data, 46
Exercises are in "Exercises for Chapters 1-3", 96
2 Checking the Straight Line Fit

47

2.1
2.2
2.3
2.4
2.5

Lack of Fit and Pure Error, 47

Testing Homogeneity of Pure Error, 56
Examining Residuals: The Basic Plots, 59
Non-normality Checks on Residuals, 61
Checks for Time Effects, Nonconstant Variance, Need for
Transformation, and Curvature, 62
2.6 Other Residuals Plots, 67
v

vi

CONTENTS

2.7 Durbin-Watson Test. 69

2.8 Reference Books for Analysis of Residuals, 70
Appendix 2A Normal Plots, 70
Appendix 2B MINITAB Instructions, 76
Exercises are in "Exercises for Chapters 1-3", 96
3 Fitting Straight Lines: Special Topics

79

3.0
3.1
3.2
3.3

Summary and Preliminaries, 79

Standard Error of Y, 80
Inverse Regression (Straight Line Case), 83
Some Practical Design of Experiment Implications of Regression,
86
3.4 Straight Line Regression When Both Variables Are Subject to Error,
89
Exercises for Chapters 1-3, 96

4 Regression in Matrix Terms: Straight Line Case

115

4.1 Fitting a Straight Line in Matrix Terms, 115

4.2 Singularity: What Happens in Regression to Make X'X Singular?
An Example, 125
4.3 The Analysis of Variance in Matrix Terms, 127
4.4 The Variances and Covariance of b o and b 1 from the Matrix
Calculation, 128
4.5 Variance of Y Using the Matrix Development, 130
4.6 Summary of Matrix Approach to Fitting a Straight Line
(Nonsingular Case), 130
4.7 The General Regression Situation, 131
Exercises for Chapter 4, 132
5 The General Regression Situation

135

5.1 General Linear Regression, 135

5.2 Least Squares Properties, 137
5.3 Least Squares Properties When E ~ N(O, Ia2), 140
5.4 Confidence Intervals Versus Regions, 142
5.5 More on Confidence Intervals Versus Regions, 143
Appendix SA Selected Useful Matrix Results, 147
Exercises are in "Exercises for Chapters 5 and 6", 169
6 Extra Sums of Squares and Tests for Several Parameters
Being Zero

6.1 The "Extra Sum of Squares" Principle, 149

6.2 Two Predictor Variables: Example, 154
6.3 Sum of Squares of a Set of Linear Functions of Y's, 162

149

CONTENTS

vii

Appendix 6A Orthogonal Columns in the X Matrix, 165

Appendix 6B Two Predictors: Sequential Sums of Squares, 167
Exercises for Chapters 5 and 6, 169
7 Serial Correlation in the Residuals and the Durbin-Watson Test

179

7.1 Serial Correlation in Residuals, 179

7.2 The Durbin-Watson Test for a Certain Type of Serial
Correlation, 181
7.3 Examining Runs in the Time Sequence Plot of Residuals:
Runs Test, 192
Exercises for Chapter 7, 198
8 More on Checking Fitted Models

205

8.1 The Hat Matrix H and the Various Types of Residuals, 205
8.2 Added Variable Plot and Partial Residuals, 209
8.3 Detection of Influential Observations: Cook's Statistics, 210
8.4 Other Statistics Measuring Influence, 214
8.5 Reference Books for Analysis of Residuals, 214
Exercises for Chapter 8, 215
9 Multiple Regression: Special Topics

217

9.1
9.2
9.3
9.4
9.5
9.6
9.7

Testing a General Linear Hypothesis, 217

Generalized Least Squares and Weighted Least Squares, 221
An Example of Weighted Least Squares, 224
A Numerical Example of Weighted Least Squares, 226
Restricted Least Squares, 229
Inverse Regression (Multiple Predictor Case), 229
Planar Regression When All the Variables Are Subject
to Error, 231
Appendix 9A Lagrange's Undetermined Multipliers, 231
Exercises for Chapter 9, 233

10 Bias in Regression Estimates, and Expected Values of Mean

Squares and Sums of Squares

235

10.1 Bias in Regression Estimates, 235

10.2 The Effect of Bias on the Least Squares Analysis of Variance, 238
10.3 Finding the Expected Values of Mean Squares, 239
10.4 Expected Value of Extra Sum of Squares, 240
Exercises for Chapter 10, 241
11 On Worthwhile Regressions, Big F's, and R2

11.1 Is My Regression a Useful One?, 243

11.2 A Conversation About R2, 245

243

viii

CON TEN T S

Appendix 11 A How Significant Should My Regression Be?, 247

Exercises for Chapter 11, 250
12 Models Containing Functions of the Predictors, Including
Polynomial Models

12.1 More Complicated Model Functions, 251

12.2 Worked Examples of Second-Order Surface Fitting for k
k = 2 Predictor Variables, 254
12.3 Retaining Terms in Polynomial Models, 266
Exercises for Chapter 12. 272

251
=

3 and

13 Transformation of the Response Variable

277

13.1 Introduction and Preliminary Remarks, 277

13.2 Power Family of Transformations on the Response: Box~Cox
Method. 280
13.3 A Second Method for Estimation A, 286
13.4 Response Transformations: Other Interesting and Sometimes
Useful Plots, 289
13.5 Other Types of Response Transformations, 290
13.6 Response Transformations Chosen to Stabilize Variance, 291
Exercises for Chapter 13, 294
14 "Dummy" Variables

299

14.1 Dummy Variables to Separate Blocks of Data with Different

Intercepts, Same Model, 299
14.2 Interaction Terms Involving Dummy Variables, 307
14.3 Dummy Variables for Segmented Models, 311
Exercises for Chapter 14, 317
15 Selecting the "Best" Regression Equation

327

15.0 Introduction, 327

15.1 All Possible Regressions and "Best Subset" Regression, 329
15.2 Stepwise Regression, 335
15.3 Backward Elimination, 339
15.4 Significance Levels for Selection Procedures, 342
15.5 Variations and Summary, 343
15.6 Selection Procedures Applied to the Steam Data, 345
Appendix 15A Hald Data. Correlation Matrix, and All 15 Possible
Regressions. 348
Exercises for Chapter 15, 355
16 III-Conditioning in Regression Data

16.1 Introduction, 369

16.2 Centering Regression Data, 371

369

CONTENTS

ix

16.3 Centering and Scaling Regression Data, 373

16.4 Measuring Multicollinearity, 375
16.5 Belsley's Suggestion for Detecting Multicollinearity, 376
Appendix 16A Transforming X Matrices to Obtain Orthogonal
Columns, 382
Exercises for Chapter 16, 385
17 Ridge Regression

387

17.1
17.2
17.3
17.4

Introduction, 387
Basic Form of Ridge Regression, 387
Ridge Regression of the RaId Data, 389
In What Circumstances Is Ridge Regression Absolutely the
Correct Way to Proceed?, 391
17.5 The Phoney Data Viewpoint, 394
17.6 Concluding Remarks, 395
Appendix 17 A Ridge Estimates in Terms of Least Squares Estimates,
396
Appendix 17B Mean Square Error Argument, 396
Appendix 17C Canonical Form of Ridge Regression, 397
Exercises for Chapter 17, 400
18 Generalized Linear Models (GUM)

401

18.1 Introduction, 401

18.2 The Exponential Family of Distributions, 402
18.3 Fitting Generalized Linear Models (GUM), 404
18.4 Performing the Calculations: An Example, 406
18.5 Further Reading, 408
Exercises for Chapter 18, 408
19 Mixture Ingredients as Predictor Variables

409

19.1 Mixture Experiments: Experimental Spaces, 409

19.2 Models for Mixture Experiments, 412
19.3 Mixture Experiments in Restricted Regions, 416
19.4 Example 1, 418
19.5 Example 2, 419
Appendix 19A Transforming k Mixture Variables to k - 1 Working
Variables, 422
Exercises for Chapter 19, 425
20 The Geometry of Least Squares

20.1
20.2
20.3
20.4

The Basic Geometry, 427

Pythagoras and Analysis of Variance, 429
Analysis of Variance and F-Test for Overall Regression, 432
The Singular X'X Case: An Example, 433

427

CONTENTS

20.5 Orthogonalizing in the General Regression Case, 435

20.6 Range Space and Null Space of a Matrix M, 437
20.7 The Algebra and Geometry of Pure Error, 439
Appendix 20A Generalized Inverses M-, 441
Exercises for Chapter 20, 444
21 More Geometry of Least Squares

447

21.1
21.2
21.3
21.4
21.5
2l.6

The Geometry of a Null Hypothesis: A Simple Example, 447

General Case HI): A{3 = c: The Projection Algebra, 448
Geometric Illustrations, 449
The F-Test for H(), Geometrically, 450
The Geometry of R2, 452
Change in R2 for Models Nested Via A{3 = 0, Not Involving /30,
452
21.7 Multiple Regression with Two Predictor Variables as a Sequence
of Straight Line Regressions, 454
Exercises for Chapter 21, 459
22 Orthogonal Polynomials and Summary Data

461

22.1 Introduction, 461

22.2 Orthogonal Polynomials, 461
22.3 Regression Analysis of Summary Data, 467
Exercises for Chapter 22, 469
23 Multiple Regression Applied to Analysis of Variance Problems

23.1
23.2
23.3

Introduction, 473
The One-Way Classification: Standard Analysis and an Example,
474
Regression Treatment of the One-Way Classification Example,
477

23.4

Regression Treatment of the One-Way Classification Using the

Original Model, 481
23.5 Regression Treatment of the One-Way Classification:
Independent Normal Equations, 486
23.6 The Two-Way Classification with Equal Numbers of
Observations in the Cells: An Example, 488
23.7 Regression Treatment of the Two-Way Classification Example,
489
23.8 The Two-Way Classification with Equal Numbers of
Observations in the Cells, 493
23.9 Regression Treatment of the Two-Way Classification with Equal
Numbers of Observations in the Cells, 494
23.10 Example: The Two-Way Classification, 498

473

CONTENTS

xi

23.11 Recapitulation and Comments, 499

Exercises for Chapter 23, 500
24 An Introduction to Nonlinear Estimation

505

24.1 Least Squares for Nonlinear Models, 505

24.2 Estimating the Parameters of a Nonlinear System, 508
24.3 An Example, 518
24.4 A Note on Reparameterization of the Model, 529
24.5 The Geometry of Linear Least Squares, 530
24.6 The Geometry of Nonlinear Least Squares, 539
24.7 Nonlinear Growth Models, 543
24.8 Nonlinear Models: Other Work, 550
24.9 References, 553
Exercises for Chapter 24, 553
25 Robust Regression

567

25.1 Least Absolute Deviations Regression (LI Regression), 567

25.2 M-Estimators, 567
25.3 Steel Employment Example, 573
25.4 Trees Example, 575
25.5 Least Median of Squares (LMS) Regression, 577
25.6 Robust Regression with Ranked Residuals (rreg), 577
25.7 Other Methods, 580
25.8 Comments and Opinions, 580
25.9 References, 581
Exercises for Chapter 25, 584
26 Resampling Procedures (Bootstrapping)

585

26.1 Resampling Procedures for Regression Models, 585

26.2 Example: Straight Line Fit, 586
26.3 Example: Planar Fit, Three Predictors, 588
26.4 Reference Books, 588
Appendix 26A Sample MINIT AB Programs to Bootstrap Residuals
for a Specific Example, 589
Appendix 26B Sample MINITAB Programs to Bootstrap Pairs for a
Specific Example, 590
Additional Comments, 591
Exercises for Chapter 26, 591
Bibliography

593

True/False Questions

605

Answers to Exercises

609

xii

CON TEN T S

Tables

684

Normal Distribution, 684

Percentage Points of the t-Distribution, 686
Percentage Points of the X2-Distribution, 687
Percentage Points of the F-Distribution, 688
Index of Authors Associated with Exercises

695

Index

697

Preface to the Third Edition

The second edition had 10 chapters; this edition has 26. On the whole (but not
entirely) we have chosen to use smaller chapters, and so distinguish more between
different types of material. The tabulation below shows the major relationships between second edition and third edition sections and chapters.
Material dropped consists mainly of second edition Sections 6.8 to 6.13 and 6.15,
Sections 7.1 to 7.6, and Chapter 8. New to this edition are Chapters 16 on multicollinearity, 18 on generalized linear models, 19 on mixture ingredients, 20 and 21 on the
geometry of least squares, 25 on robust regression, and 26 on resampling procedures.
Small revisions have been made even in sections where the text is basically unchanged.
Less prominence has been given to printouts, which nowadays can easily be generated
due to the excellent software available, and to references and bibliography, which are
now freely available (either in book or computer form) via the annual updates in
Current Index to Statistics. References are mostly given in brief either in situ or close
by, at the end of a section or chapter. Full references are in a bibliography but some
references are also given in full in sections or within the text or in exercises, whenever
this was felt to be the appropriate thing to do. There is no precise rule for doing this,
merely the authors' predilection. Exercises have been grouped as seemed appropriate.
They are intended as an expansion to the text and so most exercises have full or
partial solutions; there are a very few exceptions. One hundred and one true/false
questions have also been provided; all of these are in "true" form to prevent readers
remembering erroneous material. Instructors can reword them to create "false" questions easily enough. Sections 24.5 and 24.6 have some duplication with work in Chapter
20, but we decided not to eliminate this because the sections contain some differences
and have different emphases. Other smaller duplications occur; in general, we feel
that duplication is a good feature, and so we do not avoid it.
Our viewpoint in putting this book together is that it is desirable for students of
regression to work through the straight line fit case using a pocket calculator and then
to proceed quickly to analyzing larger models on the computer. We are aware that
many instructors like to get on to the computer right away. Our personal experience
is that this can be unwise and, over the years, we have met many students who enrolled
for our courses saying "I know how to put a regression on the computer but I don't
understand what I am doing." We have tried to keep such participants constantly
in mind.
xiii