Let $I = \int x \cos^{-1} x \, dx$.
Using integration by parts,where $\cos^{-1} x$ is the first function and $x$ is the second function:
$I = \cos^{-1} x \int x \, dx - \int \left( \frac{d}{dx} \cos^{-1} x \cdot \int x \, dx \right) dx$
$I = \cos^{-1} x \cdot \frac{x^2}{2} - \int \left( \frac{-1}{\sqrt{1-x^2}} \cdot \frac{x^2}{2} \right) dx$
$I = \frac{x^2 \cos^{-1} x}{2} + \frac{1}{2} \int \frac{x^2}{\sqrt{1-x^2}} \, dx$
To solve $\int \frac{x^2}{\sqrt{1-x^2}} \, dx$,rewrite the numerator as $-(1-x^2) + 1$:
$I = \frac{x^2 \cos^{-1} x}{2} + \frac{1}{2} \int \frac{-(1-x^2) + 1}{\sqrt{1-x^2}} \, dx$
$I = \frac{x^2 \cos^{-1} x}{2} - \frac{1}{2} \int \sqrt{1-x^2} \, dx + \frac{1}{2} \int \frac{1}{\sqrt{1-x^2}} \, dx$
Using the standard integral $\int \sqrt{a^2-x^2} \, dx = \frac{x}{2} \sqrt{a^2-x^2} + \frac{a^2}{2} \sin^{-1} \left( \frac{x}{a} \right)$:
$I = \frac{x^2 \cos^{-1} x}{2} - \frac{1}{2} \left( \frac{x}{2} \sqrt{1-x^2} + \frac{1}{2} \sin^{-1} x \right) + \frac{1}{2} \sin^{-1} x + C$
$I = \frac{x^2 \cos^{-1} x}{2} - \frac{x}{4} \sqrt{1-x^2} + \frac{1}{4} \sin^{-1} x + C$