Answer
The solution is
$$f(x)=x^2-6.$$
Work Step by Step
Since $f(f(x))$ has to give a polynomial of 4th degree we will seek $f(x)$ in the form of a polynomial of the 2nd degree, i.e. we will demand that $f(x)=ax^2+bx+c$. Let's determine $a,$ $b$ and $c$:
$$f(f(x))=f(ax^2+bx+c)=a(ax^2+bx+c)^2+b(ax^2+bx+c)+c=a(a^2x^4+b^2x^2+c^2+2abx^3+2bcx+2acx^2)+abx^2+b^2x+bc+c=a^3x^4+2a^2bx^3+(ab^2+2a^2c+ab)x^2+(2abc+b^2)x+ac^2+bc+c=x^4-12x^2+30.$$
Equating the coefficients multiplying same powers of $x$ we get
$$a^3=1\Rightarrow a=1;$$
$$2a^2b=0\Rightarrow2b=0\Rightarrow b=0;$$
$$ab^2+2a^2c+ab=-12\Rightarrow 2c=-12\Rightarrow c=-6.$$
The rest of the equations can be used for checking. Indeed, we have
$$2abc+b^2=2\cdot1\cdot0+0^2=0;$$
$$ac^2+bc+c=1\cdot3^2+0-6=36-6=30.$$
Finally
$$f(x)=x^2-6.$$