Question: Does there exist a matrix $A$ in $\Bbb{R}^{3\times 3}$ that has only $0$ as eigenvalue? 

As far as I think, an upper diagonal matrix with only zeros on the diagonal is such an example. 

And if I'm correct, is such a matrix diagonalizable?

Answer: 
Anyway, in your case, $y\ge 0$, so you should transform all this to
$$|x|<3\;\textit{ or }\; |x|\ge 5\iff -3<x<3\;\textit{ or }\; x\ge 5\;\textit{ or }\; x\le -5$$
or, as a set, the union of three intervals:
$$(-\infty,-5]\cup(-3,3)\cup[5,+\infty). $$
For $|x|<3$, the LHS is negative and the inequation certainly holds. Now assuming $|x|>3$, we can solve the equation

$$\frac1{|x|-3}=\frac12,$$ which gives $|x|=5$. The inequation holds for $|x|\ge5$. Your resolution was right.

So the solution set is

$$(-3,3)\cup(-\infty,-5]\cup[5,\infty).$$