Answer
\[y(x)=x^{-1}(x^3\ln x+5)\]
Work Step by Step
$(3x^2\ln x+x^2-y)dx-xdy=0$ ___(1)
Here,
$M(x,y)=3x^2\ln x+x^2-y\\
N(x,y)=-x$
$M_y=-1\;\;\;,\;\;\;N_x=-1$
$\Rightarrow M_y=N_x$
(1) is exact differential equation
Therefore, there exists a potential function $\phi$ such that
$\frac{\partial\phi}{\partial x}=3x^2\ln x+x^2-y$ ____(2)
$\frac{\partial\phi}{\partial y}=-x$ ____(3)
Integrating (3) with respect to $y$ holding $x$ fixed
$\phi = -xy+h(x)$ _____(4)
$h(x)$ is arbitrary function of $x$
Differentiating (4) with respect $x$ holding $y$ fixed
$\frac{\partial\phi}{\partial x}=-y+\frac{dh}{dx}$ ____(5)
From (2) and (5)
$\frac{dh}{dx}=3x^2\ln x+x^2$
Integrating,
\begin{eqnarray*}
h&=&3\int x^2\ln xdx+\int x^2dx\\
&=&3\left[\ln x\frac{x^3}{3}-\int \frac{1}{x}.\frac{x^3}{3}dx\right]+\frac{x^3}{3}\\
&=&3\left[\frac{x^3}{3}\ln x-\frac{x^3}{9}\right]+\frac{x^3}{3}\\
&=&x^3\ln x
\end{eqnarray*}
From (4)
$\phi(x,y)=-xy+x^3\ln x$
Therefore general solution of (1) is
$-xy+x^3\ln x=C$ _____(6)
$C$ is constant
Using initial condition $y(1)=5$
$-5+1\ln 1=C\Rightarrow C=-5$
From (6)
$xy-x^3\ln x=5$
$xy=5+x^3\ln x$
$\Rightarrow y(x)=x^{-1}(x^3\ln x+5)$
Hence solution of given initial value problem is $y(x)=x^{-1}(x^3\ln x+5)$.