How to Forecast using Regression Analysis
Regression is the study of relationships among variables,
a principal purpose of which is to predict, or estimate the value of one
variable from known or assumed values of other variables related to it.
Variables of Interest: To make predictions
or estimates we must identify the effective predictors of the variable
of interest: which variables are important indicators and can be measured
at the least cost, which carry only a little information, and which are
Predicting the Future: Predicting a
change over time or extrapolating from present conditions to future conditions
is not the function of regression analysis. To make estimates of the future,
use time series analysis.
Experiment: Begin with a hypothesis
about how several variables might be related to another variable and the
form of the relationship.
Types of Analysis
Simple Linear Regression: A regression
using only one predictor is called a simple regression.
Multiple Regression: Where there are
two or more predictors, multiple regression analysis is employed.
Data: Since it is usually unrealistic
to obtain information on an entire population, a sample which is a subset
of the population is usually selected. The sample may be either randomly
selected for a researcher may chose the x-values based on the capability
of the equipment utilized in the experiment or the experiment design.
Where the x-values are preselected, usually only limited inferences can
be drawn depending upon the particular values chosen. When both x and
y are randomly drawn, inferences can generally be drawn over the range
of values in the sample.
Scatter Diagram: A graphical representation
of the pairs of data called a scatter diagram can be drawn to gain an
overall view of the problem. Is there an apparent relationship? Direct?
Inverse? If the points lie within a band described by parallel lines we
can say there is a linear relationship between the pair of x and y values.
If the rate of change is generally not constant, then the relationship
The Model: If we have determined there
is a linear relationship between t and y we want a linear equation stating
y as a function of x in the form Y = a + bt + e where a is the intercept,
b is the slope and e is the error term accounting for variables that affect
y but are not included as predictors, and/or otherwise unpredictable and
Least Squares Method: To predict the
mean y-value for a given t-value, we need a line which passes through
the mean value of both t and y and which minimizes the sum of the distance
between each of the points and the predictive line. Such an approach should
result in a line which we can call a "best fit" to the sample
data. The least squares method achieves this result by calculating the
minimum average squared deviations between the sample y points and the
estimated line. A procedure is sued for finding the values of a and b
which reduces to the solution of simultaneous linear equations. Shortcut
formulas have been developed as an alternative to the solution of simultaneous
Solution Methods: Techniques of Matrix
Algebra can be manually employed to solve simultaneous linear equations.
When performing manual computations, this technique is especially useful
when there are more than two equations in two unknowns.
Several well-known computer packages are widely available and can be utilized
to relieve the user of the computational problem, all of which can be
used to solve both linear and polynomial equations: the BMD packages (Biomedical
Computer Programs) from UCLA; SPSS (Statistical Package for the Social
Sciences) developed by the University of Chicago; and SAS (Statistical
Analysis System). Another package that is also available is IMSL, the
International Mathematical and Statistical Libraries, which contains a
great variety of standard mathematical and statistical calculations. All
of these software packages use matrix algebra to solve simultaneous equations.
Use and Interpretation of the Regression Equation:
The equation developed can be used to predict an average value over the
range of the sample data. The forecast is good for short to medium ranges.
Measuring Error in Estimations: The
scatter or variability about the mean value can be measured by calculating
the variance, the average squared deviation of the values around the mean.
The standard error of estimate is derived from this value by taking the
square root. This value is interpreted as the average amount that actual
values differ from the estimated mean.
Confidence Intervals: Interval estimates
can be calculated to obtain a measure of the confidence we have in our
estimates that a relationship exists. These calculations are made using
t-distribution tables. From these calculations we can derive confidence
bands, a pair of non-parallel lines narrowest at the mean values which
express our confidence in varying degrees of the band of values surrounding
the regression equation.
Assessment: How confident can we be
that a relationship actually exists? The strength of that relationship
can be assessed by statistical tests of that hypothesis such as the null
hypothesis which are established using t-distribution, R-squared, and
F-distribution tables. These calculations give rise to the standard error
of the regression coefficient, an estimate of the amount that the regression
coefficient b will vary from sample to sample of the same size from the
same population. An Analysis of Variance (ANOVA) table can be generated
which summarizes the different components of variation.
When you want to compare models of different size (different
numbers of independent variables and/or different sample sizes) you must
use the Adjusted R-Squared, because the usual R-Squared tends to grow
with the number of independent variables.
The Standard Error of Estimate (i.e. square root of error mean square)
is a good indicator of the "quality" of a prediction model since
it "adjusts" the Error Sum of Squares (EMS) for the number of
predictors in the model as follow:
EMS = Error Sum of Squares/(N - Number of Linearly Independent
If one keeps adding useless predictors to a model, the
EMS will become less and less stable. R-squared is also influenced by
the range of your dependent value so if two models have the same residual
mean square but one model has a much narrower range of values for the
dependent variable that model will have a higher R-squared. This explains
the fact that both models will do as well for prediction purposes.
A considerable portion of the output of the computer
programs previously mentioned are devoted to a description of the tests
of significance of the regression.
Back to Statistical
Forecasting Home Page