Answer
See below
Work Step by Step
The given system can be written as $x′ = Ax$,
where $A=\begin{bmatrix}
3 & -4 & -1\\
0 & -1 & -1\\
0 & -4 & 2
\end{bmatrix}$
The transformed system is $y′ = (S−1AS)y$ where $x = Sy$
To determine S, we need the eigenvalues and eigenvectors of A. The characteristic polynomial of A is
$p(\lambda)=\begin{bmatrix}
3-\lambda & -4 & -1\\
0 & -1-\lambda & -1\\
0 & -4 & 2-\lambda
\end{bmatrix}=(3-\lambda)(\lambda^2-\lambda-6)$
Hence, A is nondefective by Corollary 7.2.10. The eigenvectors are easily computed:
\lambda_1=-2: v=r(1,1,1)$
\lambda_2=3: v=s(1,0,0)\\
\lambda_3=3: v=t(0,1,-4) \\
$
Set $S = \begin{bmatrix}
1 & 1 & 0\\
1 & 0 & 1\\
1 & 0 & -4
\end{bmatrix}$
then from Theorem 7.3.4, $S−1AS = diag(-2,3,3)$
so that the system becomes:
$\begin{bmatrix}
y_1'\\
y_2'
\end{bmatrix}=\begin{bmatrix}
-2 & 0 & 0\\
0 & 3 & 0\\
0 & 0 & 3
\end{bmatrix}\begin{bmatrix}
y_1\\
y_2 \\
y_3
\end{bmatrix}$
Hence, $y_1'=-2y_1\\
y_2'=3y_2 \\
y_3'=3y_3$
Both of these equations can be integrated to obtain $y_1(t)=C_1e^{-2t}\\
y_2(t)=C_2e^{3t}\\
y_3(t)=C_3e^{3t}$
Return to the original variables, we have:
$x=Sy=\begin{bmatrix}
1 & 1 & 0\\
1 & 0 & 1\\
1 & 0 & -4
\end{bmatrix}\begin{bmatrix}
C_1e^{-2t}\\
C_2e^{3t}\\
C_3e^{3t}
\end{bmatrix} \\
\rightarrow x_1(t)=C_1e^{-2t}+C_2e^{3t}\\
x_2(t)=C_1e^{-2t}+C_3e^{3t} \\
x_3(t)=C_1e^{-2t}-4C_3e^{3t}$
