Ordinary least squares
A statistical method that fits a line, plane or hyperplane to data by minimizing the sum of squared vertical distances between the points and the model.
- Year
- 1805
- People involved
- Carl Friedrich Gauss, Adrien-Marie Legendre
- Section
- Mathematics
History
Ordinary least squares is a way of estimating the parameters of a linear regression model from observed data. Such a model assumes that a response variable depends linearly on one or more explanatory variables, plus an error term that captures everything the model does not explain. The method chooses the parameters that make the sum of squared residuals as small as possible, that is, the sum of the squared vertical gaps between each observed value and the value the model predicts. Squaring the gaps prevents positive and negative differences from cancelling out, and it leads to a mathematical problem with a single solution when the explanatory variables are not redundant with one another. That is why the idea matters: it underpins much of applied statistics today.
Before the method existed, astronomers and surveyors already faced the problem of combining measurements that contained errors. When there were more observations than unknowns, no exact solution existed, and each researcher resolved the conflict in their own way, often with averages or rough adjustments. There was no common criterion for choosing a reasonable answer, and several methods competed.
The method was published in 1805 by the French mathematician Adrien-Marie Legendre, in an appendix on determining the orbits of comets. Legendre proposed minimizing the sum of squared errors as a practical rule, without a complete probabilistic justification. Shortly afterwards, Carl Friedrich Gauss claimed he had used the approach since 1795, which led to a priority dispute that was never fully settled. In his 1809 work on the motion of celestial bodies, Gauss linked the method to the normal distribution of errors, giving it a firmer statistical foundation. The 1795 date rests on Gauss's own claim and should be treated with caution, since it was not documented at the time.
Throughout the nineteenth century the method spread into geodesy, astronomy and statistics. In the twentieth century Ronald Fisher and others brought it into the analysis of experimental data, and regression theory developed further through the work of Yule and Pearson. The conditions for its validity became clearer: the estimator is unbiased and has the smallest variance among linear estimators when errors have constant variance and are uncorrelated, and it coincides with the maximum likelihood estimator when errors follow a normal distribution. When these conditions fail, variants such as weighted or generalized least squares are used instead.
Today ordinary least squares is used in economics and econometrics to estimate how variables influence one another, in electrical engineering for control theory and signal processing, and in nearly every field that works with data. Fitting a line to a cloud of points in a spreadsheet applies the same idea. Its simple calculation and clear interpretation explain why it remains the starting point for almost every regression analysis.
Timeline
- 1795According to Gauss's later claim, he was already using the method by this date; the claim was not documented at the time.
- 1805Legendre publishes the least squares method in an appendix on determining comet orbits.
- 1809Gauss links the method to the normal distribution of errors in his work on the motion of celestial bodies.
- 1900Yule and Pearson contribute to developing linear regression theory, a basis of modern statistical analysis.
Trivia and anecdotes
A priority dispute
Gauss claimed to have used the method since 1795 but published after Legendre, fueling a dispute over who invented it first.
Carl Friedrich Gauss1777–1855 · Mathematics
Adrien-Marie Legendre1752–1833 · Mathematics