Associativity of a composition $x ∘ y = xy+\sqrt{(x^2-1)(y^2-1)}$ For many hours I had been stucked at this problem. For the following composition 
$x ∘ y = xy+\sqrt{(x^2-1)(y^2-1)}$
I have to demonstrate that this composition is associative($(x ∘ y)∘z=x∘(y∘z)$
and that $x ∘ y = xy-\sqrt{(x^2-1)(y^2-1)}$ it's not associative.

For the second operation, as was pointed out in the comments, it should be easy to just find a counterexample.
For the first one, there's a rather easy way using hyperbolic functions. Assuming that $x,y$ are greater than or equal to $1$, we can find such nonnegative real numbers $a$ and $b$ that $x = \cosh a$ and $y = \cosh b$. Then $\sqrt{x^2-1} = \sinh a$ and $\sqrt{y^2-1} = \sinh b$, according to the well known relations for hyperbolic functions. Then $$x \circ y = \cosh a \cosh b + \sinh a \sinh b = \cosh (a+b),$$ according to another well known relation.
Now to prove associativity see that if $x = \cosh a, y = \cosh b, z = \cosh c$, then
$$
(x\circ y)\circ z = \cosh (a+b) \circ \cosh c = \cosh(a+b+c) =
\cosh a \circ \cosh (b+c) = x \circ (y \circ z).
$$
PS: I assumed the operation is defined on $[1, +\infty)$. I haven't really thought what will happen if we allow numbers less than $1$. Ideally, you should indicate where the operation is defined in the question itself.
UPDATE: Here is another way, without hyperbolic functions. Let us just calculate $(x\circ y)\circ z$ (again, assuming all the variables are $\geq 1$). First, a helpful calculation:
$$
\begin{align}
(x \circ y)^2 - 1 & = \left( xy + \sqrt{(x^2-1)(y^2-1)} \right)^2 - 1 \\
& = 2x^2y^2 - x^2 - y^2 + 2xy\sqrt{(x^2-1)(y^2-1)} \\
& = \left( x\sqrt{y^2-1} + y\sqrt{x^2-1} \right)^2.
\end{align}
$$
Now, moving on to $(x\circ y)\circ z$:
$$
\begin{align}
(x\circ y)\circ z & = (x\circ y)z + \sqrt{((x \circ y)^2-1)(z^2-1)} \\
& = \left(xy + \sqrt{(x^2-1)(y^2-1)}\right)\cdot z + \left( x\sqrt{y^2-1} + y\sqrt{x^2-1} \right)\sqrt{z^2-1} \\
& = xyz + x\sqrt{(y^2-1)(z^2-1)} + y\sqrt{(x^2-1)(z^2-1)} + z\sqrt{(x^2-1)(y^2-1)}.
\end{align}
$$
Now, if you look at $x \circ (y \circ z)$, since the operation is clearly symmetric, it is equal to $(y \circ z) \circ x$. And this one you can get if you substitute $x, y$ and $z$ for each other cyclically in the large formula above. The large formula above is symmetric, so it will remain the same. Therefore $(x\circ y)\circ z = x \circ (y \circ z)$.
Or you can just repeat all the calculations for $x \circ (y \circ z)$ again, makes no matter.
One last word: I do advise to understand the solution with hyperbolic functions too. Relations with hyperbolic functions are even easier to grasp than trigonometric ones: it's just the exponent function wearing a mask. Also, it wasn't just a lucky grouping of terms that led me to the equality $$(x \circ y)^2 - 1 = \left( x\sqrt{y^2-1} + y\sqrt{x^2-1} \right)^2.$$ It was this relation for hyperbolic functions: $\sinh (a + b) = \sinh a \cosh b + \sinh b \cosh a$. So, to sum up: hyperbolic functions are important if you want to really understand the problem. It's almost guaranteed that that is how the author of the problem came up with it.
