Polynomial least squares


In mathematical statistics, polynomial least squares comprises a broad range of statistical methods for estimating an underlying polynomial that describes observations. These methods include polynomial regression, curve fitting, linear regression, least squares, ordinary least squares, simple linear regression, linear least squares, approximation theory and method of moments. Polynomial least squares has applications in radar trackers, estimation theory, signal processing, statistics, and econometrics.
Two common applications of polynomial least squares methods are generating a low-degree polynomial that approximates a complicated function and estimating an assumed underlying polynomial from corrupted observations. The former is commonly used in statistics and econometrics to fit a scatter plot with a first degree polynomial. The latter is commonly used in target tracking in the form of Kalman filtering, which is effectively a recursive implementation of polynomial least squares. Estimating an assumed underlying deterministic polynomial can be used in econometrics as well. In effect, both applications produce average curves as generalizations of the common average of a set of numbers, which is equivalent to zero degree polynomial least squares.
In the above applications, the term "approximate" is used when no statistical measurement or observation errors are assumed, as when fitting a scatter plot. The term "estimate", derived from statistical estimation theory, is used when assuming that measurements or observations of a polynomial are corrupted.

Polynomial least squares estimate of a deterministic first degree polynomial corrupted with observation errors

Assume the deterministic first degree polynomial equation ' with unknown coefficients ' and ' is written as

This is corrupted with an additive stochastic process described as an error, resulting in
Given observations from a sample, where the subscript ' is the observation index, the problem is to apply polynomial least squares to estimate , and to determine its variance along with its expected value.

Definitions and assumptions

The term linearity in mathematics may be considered to take two forms that are sometimes confusing: a linear system or transformation and a linear equation. The term "function" is often used to describe both a system and an equation, which may lead to confusion. A linear system is defined by
where and are constants, and where and are variables. In a linear system, where is the linear expectation operator. A linear equation is a straight line as is the first degree polynomial described above.
The error is modeled as a zero mean stochastic process, sample points of which are random variables that are uncorrelated and assumed to have identical probability distributions, but not necessarily Gaussian, treated as inputs to polynomial least squares. Stochastic processes and random variables are described only by probability distributions.
Polynomial least squares is modeled as a linear signal processing system which processes statistical inputs deterministically, the output being the linearly processed empirically determined statistical estimate, variance, and expected value.
Polynomial least squares processing produces deterministic moments, which may be considered as moments of sample statistics, but not of statistical moments.

Polynomial least squares and the orthogonality principle

Approximating a function with a polynomial
where hat denotes the estimate and is the polynomial degree, can be performed by applying the orthogonality principle. The sum of squared residuals can be written as
According to the orthogonality principle, this is at its minimum when the residual vector is orthogonal to the estimate, that is
This can be described as the orthogonal projection of the data values onto a solution in the form of the polynomial. For N > J, orthogonal projection yields the standard overdetermined system of equations used to compute the coefficients in the polynomial approximation. The minimum sum of squared residuals is then
The advantage of using orthogonal projection is that can be determined for use in the polynomial least squares processed statistical variance of the estimate.

The empirically determined polynomial least squares output of a first degree polynomial corrupted with observation errors

To fully determine the output of polynomial least squares, a weighting function describing the processing must first be structured and then the statistical moments can be computed.

The weighting function describing the linear polynomial least squares "system"

The weighting function can be formulated from polynomial least squares to estimate the unknown ' as follows:
where N is the number of samples, are random variables as samples of the stochastic , and the first degree polynomial data weights are
which represent the linear polynomial least squares "system" and describe its processing. The Greek letter is the independent variable ' when estimating the dependent variable after data fitting has been performed. The overbar defines the deterministic centroid of as processed by polynomial least squares – i.e., it defines the deterministic first order moment, which may be considered a sample average, but does not here approximate a first order statistical moment:

Empirically determined statistical moments

Applying yields
where
and
As linear functions of the random variables, both coefficient estimates and are random variables. In the absence of the errors, and, as they should to meet that boundary condition.
Because the statistical expectation operator E is a linear function and the sampled stochastic process errors are zero mean, the expected value of the estimate is the first order statistical moment as follows:

The statistical variance in is given by the second order statistical central moment as follows:

because
where is the statistical variance of random variables ; i.e., for i = n and for
Carrying out the multiplications and summations in yields

Measuring or approximating the statistical variance of the random errors

In a hardware system, such as a tracking radar, the measurement noise variance can be determined from measurements when there is no target return – i.e., by just taking measurements of the noise alone.
However, if polynomial least squares is used when the variance is not measurable, it can be estimated with observations in from orthogonal projection as follows:

As a result, to the first order approximation from the estimates and as functions of sampled and
which goes to zero in the absence of the errors, as it should to meet that boundary condition.
As a result, the samples are considered to be the input to the linear polynomial least squares "system" which transforms the samples into the empirically determined statistical estimate , the expected value , and the variance.

Properties of polynomial least squares modeled as a linear "system"

The empirical statistical variance is a function of, N and . Setting the derivative of with respect to equal to zero shows the minimum to occur at ; i.e., at the centroid of the samples. The minimum statistical variance thus becomes . This is equivalent to the statistical variance from polynomial least squares of a zero degree polynomial – i.e., of the centroid of.
The empirical statistical variance is a function of the quadratic . Moreover, the further deviates from , the larger is the variance due to the random variable errors . The independent variable can take any value on the axis. It is not limited to the data window. It can extend beyond the data window – and likely will at times depending on the application. If it is within the data window, estimation is described as interpolation. If it is outside the data window, estimation is described as extrapolation. It is both intuitive and well known that the further is extrapolation, the larger is the error.
The empirical statistical variance due to the random variable errors is inversely proportional to N. As N increases, the statistical variance decreases. This is well known and what filtering out the errors is all about. The underlying purpose of polynomial least squares is to filter out the errors to improve estimation accuracy by reducing the empirical statistical estimation variance. In reality, only two data points are required to estimate and ; albeit the more data points with zero mean statistical errors included, the smaller is the empirical statistical estimation variance as established by N samples.
There is an additional issue to be considered when the noise variance is not measurable: Independent of the polynomial least squares estimation, any new observations would be described by the variance.
Thus, the polynomial least squares statistical estimation variance and the statistical variance of any new sample in would both contribute to the uncertainty of any future observation. Both variances are clearly determined by polynomial least squares in advance.
This concept also applies to higher degree polynomials. However, the weighting function is obviously more complicated. In addition, the estimation variances increase exponentially as polynomial degrees increase linearly. However, there are ways of dealing with this as described in.

The synergy of integrating polynomial least squares with statistical estimation theory

Modeling polynomial least squares as a linear signal processing "system" creates the synergy of integrating polynomial least squares with statistical estimation theory to deterministically process samples of an assumed polynomial corrupted with a statistically described stochastic error ε. In the absence of the error ε, statistical estimation theory is irrelevant and polynomial least squares reverts to the conventional approximation of complicated functions and scatter plots.