A least squares fit

Suppose we'd like a least squares fit to the data
$$\left\{(2,0), (3,2), (4,3) \right\}.$$
Write down the normal equations for this fit for basis functions

1. $\{1,x\}$, and
2. $\{1,\log(x)\}$

I am not sure if I am carrying the answer to what you want but here it goes.

I took $[1,x]$ and expanded it to $a+bx$ and created the equation:

$$F(a,b) = (2b + a + 0)^2 + (3b + a -2)^2 + (4b + a -3 )^2$$

I think this is where you wanted us to stop but I continued to check to see if I set it up right.

I took the partial derivatives for the equation in respect to a and b and got

$$\frac{dF}{da}= 2( a + 2b ) + 2( a + 3b - 2 ) + 2( a + 4b -3) = 0$$
$$\frac{dF}{da}= 6a + 14b - 10 = 0$$

$$\frac{dF}{db}= 4(a + 2b) + 6(a + 3b - 2) + 8(a + 4b -3) = 0$$
$$\frac{dF}{db}= 18a + 58b - 36 = 0$$

I found $a = \frac{19}{24}$ and $b = \frac{6}{16}$ but I compared these to what I got from modifying the LS program and I did not get the same answer.

My recommendation is that you formulate this using matrices like you did on the most recent lab. That is, you should set up the matrix $A$ whose columns are the basis functions evaluated at the given $x$ data. You should then write down the system
$$A^{T}A\vec{c} = A^{T}\vec{y},$$
where $\vec{c}$ is the unknown vector of coefficients and $\vec{y}$ is the vector of $y$ data. Since it's just a $3\times3$ situation, it's pretty easy to write it down in terms of equations.

Using $$A^{T}A\vec{c} = A^{T}\vec{y},$$ where
$$A = \begin{bmatrix}
1 & 2 \\
1 & 3 \\
1 & 4 \\
\end{bmatrix}$$ and $$ \vec{c} = \begin{bmatrix}
a_{0} \\
a_{1} \\
\end{bmatrix}$$ and $$\vec{y} = \begin{bmatrix}
0 \\
2 \\
3 \\ \end{bmatrix}. $$
We have $$A^{T}\vec{y} = \begin{bmatrix}
1 & 1 & 1 \\
2 & 3 & 4 \\ \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 2 \\ 3 \end{bmatrix} = \begin{bmatrix} 5 \\ 18 \\ \end{bmatrix}
$$
and $$ \begin{bmatrix}
1 & 1 & 1 \\
2 & 3 & 4 \\
\end{bmatrix} \cdot \begin{bmatrix}
1 & 2 \\
1 & 3 \\
1 & 4 \\
\end{bmatrix} \cdot \begin{bmatrix}
a_{0} \\
a_{1} \\
\end{bmatrix} = \begin{bmatrix}
3 & 9 \\
9 & 29 \\ \end{bmatrix} * \begin{bmatrix}
a_{0} \\
a_{1} \\ \end{bmatrix}.$$
So the system can be written
$$
\begin{align}
3a_{0} + 9a_{1} &= 5 \\
9a_{0} + 29a_{1} &= 18
\end{align}
$$
Not sure how to do this for the basis $\{1,\log(x)\}$.

@djett This looks great! Though, I didn't check your arithmetic and I made a couple little typesetting adjustments.

To deal with the basis $\{1,\log(x)\}$, you simply have to adjust the second column of the matrix $A$. The point is:

The $i^{\text{th}}$ column of $A$ consists of the $x$-values plugged into the $i^{\text{th}}$ basis function!

Thus, your second column would be those $x$ values plugged into the logarithm. If you were going all the way to a computation of the fit, then you'd also have to solve the system and then use the appropriate basis functions when you write down the final solution.

we can find explicitly that for our coordinate vector $\bar{c}$, $c_{1} = \frac{-17}{6}$ and $c_{2} = \frac{3}{2}$. In other words, $f(x) = \frac{-17}{6} + \frac{3}{2}x$. w/ regards to our second basis we have the following situation:

$A = \begin{pmatrix}
1 & \log(2) \\
1 & \log(3) \\
1 & \log(4) \\
\end{pmatrix};
A^{T} = \begin{pmatrix}
1 & 1 & 1 \\
\log(2) & \log(3) & \log(4) \\
\end{pmatrix}$

Some work gives us that:

$A^{T}A\bar{c} = \begin{pmatrix}
3c_{1} + c_{2}\log(24) \\
\log(24)\left(c_{1} + c_{2}\log(24) \right) \\
\end{pmatrix}; A^{T}\bar{y} =
\begin{pmatrix}
5 \\
\log(108)
\end{pmatrix}$

Normally I'd solve this via row reduction, but with those logs it gets tricky ... Instead we can solve the equivalent system of equations and find the values of $c_{1}, c_{2}$. To get our fit we'd then have $f(x) = c_{1} + c_{2}\log(x)$.

I took exactly the same approach and got the same resulting system.

Yeah it's a real pain in the butt... I tried and gave up because I forgot my log rules. Good luck to all of us if this is on our quiz tomorrow.

I just got the same thing too! I'm now regretting my rebellious teenage years when I refused to memorize log rules in pre-calc.

@gbrock @sgerall , et. al.

This being numerical analysis I decided to get an approximation for $c_{1}, c_{2}$ via the following code:

import numpy as np
from numpy.linalg import lstsq, solve

xs = [2,3,4]
ys = [0,2,3]
B = np.vstack([np.ones(3), np.log(xs)]).transpose()
solution = solve(B.transpose().dot(B), B.transpose().dot(ys))
solution`

output: $[-2.96032841, 4.36776278]$

So we can see that $f(x) \approx -2.96032841 + 4.36776278\log(x)$

Are we able to leave the final system as you have it:
$$\begin{align} 3a_{0} + 9a_{1} &= 5 \\ 9a_{0} + 29a_{1} &= 18 \end{align} ?$$

Or would we be expected to solve it for $a_0=\frac{-17}{6}$ and $a_1=\frac{3}{2}$?

I'm fairly positive we won't have to solve the resulting system. The problem only asks for the Normal Equations ($A^TAc=A^Ty$) using each type of basis functions, so just finding the system seems sufficient to me.

That was my understanding as well!

Definitely! If Mark doesn't say "solve the system", don't solve it!

The proper way to determine how to write down the final answer to a question is (hopefully) contained in the question itself. That's why I often tell Calculus students to reread the question before finishing it.