least squares Archives

Weighted Least Squares and locally weighted linear regression

From the post on Closed Form Solution for Linear regression, we computed the parameter vector $\theta$ which minimizes the square of the error between the predicted value $h_{\theta}(x)$ and the actual output $y$ for all $j$ values in the training set. In that model all the $j$ values in the training set is given equal importance. Let us consider the case where it is known some observations are important than the other. This post attempts to the discuss the case where some observations need to be given more weights than others (also known as weighted least squares).

Continue reading “Weighted Least Squares and locally weighted linear regression”

Straight line fit using least squares estimate

Two points suffice for drawing a straight line. However we may be presented with a set of data points (more than two?) presumably forming a straight line. How can one use the available set of data points to draw a straight line?

A probable approach is to draw a straight line which hopefully minimizes the error between the observed data points and estimated straight line.

$err = \Sigma_{i=1}^N\left(y_i - \hat{y}_i\right)^2$ where $y_i$ is the observed data points and $\hat{y}_i$ is the points from estimated straight line.

Continue reading “Straight line fit using least squares estimate”