Answer
(a) $h=9$ and $k\ne6$
(b) $h\ne9$ and $k$ can take any value
(c) $h=9$ and $k=6$
Work Step by Step
We know we can find how many solutions a system has by looking at the echelon form of the corresponding augmented matrix.
The augmented matrix of our system is
$$\left[
\begin{array}{ccc}
1 & 3 & 2\\
3 & h & k
\end{array}
\right].$$
To convert this to echelon form, we replace row 2 with -3*(row 1)+(row 2), which gives
$$\left[
\begin{array}{ccc}
1 & 3 & 2\\
0 & -9+h & -6+k
\end{array}
\right].$$
Now we can answer (a), (b), and (c).
(a) For the system to have no solutions (i.e., to be inconsistent), the last column of the augmented matrix must be a pivot column.
Therefore, we must have
$$-9+h=0$$
and $$ -6+k\neq0.$$
Hence, we must have $$h=9$$
and $$k\neq 6$$
Since any value for $k$ other than $6$ will work, we can choose $k=0.$ Then for $h=9$ and $k=0$, the second row of our matrix is
[0 0 -6], which corresponds to the equation $0=-6,$ which means the system is inconsistent.
(b) For the system to have a unique solution, it must be consistent, which means the last column of the corresponding augmented matrix cannot be a pivot column, and have no free variables.
Now, looking at the echelon form of our augmented matrix, we can see the system will not have any free variables if $-9+h \neq0$. Note that if $-9+h\neq 0$ or equivalently $h\neq9$, then the last column is not a pivot column for any value of $k.$
Since we can choose any value other than $9$ for $h$ and any value for $k$, we choose $h=0$ and $k=0.$ Then our matrix becomes
$$\left[
\begin{array}{ccc}
1 & 3 & 2\\
0 & -9 & -6
\end{array}
\right],$$
which corresponds to the system
$$x_1+3x_2=2\\ -9x_2=-6.$$
Solving for $x_1$ and $x_2$, we get $(x_1, x_2)=(0,x_2=\frac{2}{3})$ as our unique solution.
(c) For the system to have many solutions, it must be consistent and have a free variable.
Looking again at the echelon form of our matrix, we see the system will have a free variable if $-9+h=0$, or equivalently $h=9$.
Now if $h=9$, then in order for the system to be consistent (i.e., in order for the last column of our matrix not to be a pivot column), we must have $-6+k=0$, or equivalently $k=6$.
So for $h=9$ and $k=6$ our matrix becomes
$$\left[
\begin{array}{ccc}
1 & 3 & 2\\
0 & 0 & 0
\end{array}
\right],$$
which means our system is consistent and has infinitely many solutions.
