Follow @solverworld Tweet this

Suppose you forget how to do linear curve fitting and want to derive the formula from scratch. Or say you want to find the best parameters to describe some data.

First you make a model, with unknowns \(x_i\). Assume that your parameters are a and b. You might postulate a model of the form

[Note: put more realistic example in here]

\begin{align}
y = a x_1 + a^2 x_2 + b x_3 + a b x_4
\end{align}
Note that the equation does not need to be linear in the parameters a and b, only in the unknowns.

Now suppose that we have 100 data points, each with corresponding a,b,c and d’s. We can write
\begin{align}
y=Ax
\end{align}
where
\begin{align}
\begin{bmatrix}
y_1\\
y_2\\
\vdots \\
y_m
\end{bmatrix}
=
\begin{bmatrix}
a_1 & a_1^2 & b_1 & a_1 b_1 \\
a_2 & a_2^2 & b_2 & a_2 b_2 \\
\vdots & \vdots & \vdots & \vdots\\
a_m & a_m^2 & b_m & a_m b_m \\
\end{bmatrix}
\begin{bmatrix}
x_1\\
x_2\\
x_3\\
x_4
\end{bmatrix}
\end{align}

Now we ask the question. What values of \(x_i\) match our data most closely. If we say:

Problem:

Find the values of \(x_i\) that minimize the error \(e=y-Ax\)

\begin{align}
\DeclareMathOperator*{\argminA}{arg\,min}
x_{opt} = \argminA_x \| e \| = \argminA_x \left\| y – Ax \right\|
\end{align}
As we know, the solution to this problem is our good old linear solve. You can get the answer various ways:

1. In MATLAB or Octave, use linear solve A\y
2. For a closed-form expression, use \(x_{opt} = (A^T A)^{-1} A^T y\)
3. Any other way you want to solve \(y=Ax\)