Answer
$a=\frac{4}{5}, b=-\frac{3}{5}$
Work Step by Step
Solve $\left[\begin{array}{cc}a & -b \\ b & a\end{array}\right]\left[\begin{array}{l}4 \\ 3\end{array}\right]=\left[\begin{array}{l}5 \\ 0\end{array}\right]$
$4 a-3 b=5$
$4 b+3 a=0$
Multiply first equation with 4 and second with 3 and add them
$16 a-12 b+12 b+9 a=20$
$25 a=20$
$a=\frac{20}{25}=\frac{4}{5}$
$4 a-3 b=5$
$\frac{16}{5}-3 b=5$
$-3 b=\frac{25-16}{5}=\frac{9}{5}$
$b=-\frac{3}{5}$