There are two problems with your derivation. The first one is in the calculation of $[\vec S,\vec J^2]$. In the final step, you are implicitly using $\vec S\times \vec S$ which is false. 359
A close attention to the anticommutation rules gives: $$ \vec S\times\vec S = i\hbar\vec S $$ so this leads to: $$ [\vec J{}^2,\vec S] = 2i\hbar((\vec S\times \vec J)-i\hbar \vec S) $$
You also have a second problem in the triple cross product formula. If you look closely, you are implicitly assuming that the three operators commute when you apply it.
Instead, you'd get: $$ ((\vec S\times \vec J)\times \vec J)k = S_lJ_kJ_l-S_kJ_lJ_l\ = S_lJ_lJ_k+S_l i\hbar\epsilon{klm}J_m-S_kJ_lJ_l $$ so in vectorial notation: $$ (\vec S\times \vec J)\times \vec J = (\vec S\cdot \vec J)\vec J+i\hbar(\vec S\times \vec J)-\vec S\vec J{}^2 $$ as you can see, you forgot the extra cross product term and the ordering of your second term was a bit cavalier. You can further symmetrize the third, final term: $$ \vec S\vec J{}^2 = \frac{1}{2}(\vec S\vec J{}^2+\vec J{}^2\vec S-[\vec J{}^2,\vec S])\ =\frac{1}{2}(\vec S\vec J{}^2+\vec J{}^2\vec S)-\frac{1}{2} [\vec J{}^2,\vec S] $$ to get: $$ (\vec S\times \vec J)\times \vec J = (\vec S\cdot \vec J)\vec J+i\hbar(\vec S\times \vec J)-\frac{1}{2}(\vec S\vec J{}^2+\vec J{}^2\vec S)+\frac{1}{2} [\vec J{}^2,\vec S] $$
Incorporating these modifications, you'd rather get: $$ [\vec J{}^2,[\vec J{}^2,\vec S]] = 2i\hbar[\vec J{}^2,(\vec S\times \vec J)-i\hbar \vec S)]\ = 2i\hbar([\vec J{}^2,\vec S]\times \vec J+\vec S\times[\vec J{}^2,\vec J]-i\hbar[\vec J{}^2,\vec S])\ = -4\hbar^2(((\vec S\times \vec J)-i\hbar \vec S)\times \vec J-\frac{1}{2}[\vec J{}^2,\vec S])\ = -4\hbar^2((\vec S\cdot \vec J)\vec J+i\hbar(\vec S\times \vec J)-\frac{1}{2}(\vec S\vec J{}^2+\vec J{}^2\vec S)+\frac{1}{2} [\vec J{}^2,\vec S]-i\hbar(\vec S\times \vec J)-\frac{1}{2}[\vec J{}^2,\vec S])\ = -4\hbar^2((\vec S\cdot \vec J)\vec J-\frac{1}{2}(\vec S\vec J{}^2+\vec J{}^2\vec S)) $$ which is the advertised form, with $\alpha = -4\hbar^2$ (which has the correct dimensions).
Hope this helps and tell me if something's not clear. 532