Question: I've tried it by applying integratrion by parts, but I'm not getting the answer correct. Its answer is
$$\frac{1}{\sqrt{2}}\,\sin\left(\frac{k^2}{4} +\frac{\pi}{4}\right).$$
Please help in this.

Answer: 
$$I(\xi)=\int\_{\mathbb{R}}\sin(x^2)\cos(\xi x)\,dx = \int\_{\mathbb{R}}\sin(x^2)\,dx - \int\_{0}^{+\infty}\frac{1-\cos(\xi\sqrt{x})}{\sqrt{x}}\sin(x)\,dx $$
is a convergent integral if $\int$ is intended as an improper Riemann integral, by Dirichlet's criterion.

Moreover, $\sin(x^2)\cos(\xi x) = \frac{1}{2}\left(\sin(x^2+\xi x)+\sin(x^2-\xi x)\right)$, and:

$$ \int\_{\mathbb{R}}\sin(x^2+\xi x)\,dx = \int\_{\mathbb{R}}\sin\left(\left(x+\frac{\xi}{2}\right)^2-\frac{\xi^2}{4}\right)\,dx = \int\_\mathbb{R}\sin(x^2-\xi^2/4)\,dx$$
just depends on $\sin(\xi^2/4)$ and $\cos(\xi^2/4)$ by the sine addition formulas, since:
$$ \int\_{\mathbb{R}}\sin(x^2)\,dx = \int\_{\mathbb{R}}\cos(x^2)\,dx = \sqrt{\frac{\pi}{2}}$$
are Fresnel's integrals. Put everything together and you will get your Fourier transform defined by an improper Riemann integral.
There is not enough information in the question to allow us writing a full answer. However, we can formally derive the sine form.

If $u(x):=\sin x^2$, then one can check that u solves the $2$nd order ODE
$$xu''-u'+4x^3u=0.$$
If $\hat{u}$ is the unitary Fourier transform of $u$, then $\hat{u}$ must solve the $3$rd order ODE
$$\hat{u}'''+\frac{k^2}{4}\hat{u}'+\frac{3k}{4}\hat{u}=0$$
because of the formula (see e.g. <https://en.wikipedia.org/wiki/Fourier_transform#Applications>)
$$\widehat{x^n u}(k)=\mathrm{i}^n\frac{\mathrm{d}^n}{\mathrm{d}k^n}\hat{u}(k),\quad\quad \widehat{\left(\frac{\mathrm{d}^n}{\mathrm{d}x^n}u\right)}(k)=\mathrm{i}^nk^n\hat{u}(k).\quad\quad\quad\quad(\*)$$
Now define $v(k):=A\sin\left(\frac{k^2}{4}+K\right)$ for some real constants $A,K$. One computes :
\begin{align}
v'(k)&=\frac{Ak}{2}\cos\left(\frac{k^2}{4}+K\right),\\
v''(k)&=\frac{A}{2}\cos\left(\frac{k^2}{4}+K\right)-\frac{Ak^2}{4}\sin\left(\frac{k^2}{4}+K\right),\\
v'''(k)&=-\frac{3Ak}{4}\sin\left(\frac{k^2}{4}+K\right)-\frac{Ak^3}{8}\cos\left(\frac{k^2}{4}+K\right).\\
\end{align}
One can check that $v$ solves $(\*)$. Now, it remains to show that $v=\hat{u}$ (by invoking an argument of unicity of the solution of $(\*)$) and to find the constants $A,K$, but one needs more information.