Deriving the irreps of the d-orbitals under C3 and S6 operations in octahedral symmetry Contextual background: I am trying to reproduce the symmetries in the graph on this page (MO diagram for $\ce{ML6}$ complex). I have reproduced the metal s and p orbital representations as well as the linear combination of representations for the $\ce{L6}$ fragment. All that remain are the metal d orbitals.

So I know the five d-orbitals collectively reduce to the $\mathrm{e_g}$ + $\mathrm{t_{2g}}$ representation. I put my five d-orbitals at the origin and went through the operations for the reducible representation. Everything else adds up, except I can't find that $-1$ in the $C_3$ and $S_6$ operations.

Character table

$$\small\begin{array}{c|cccccccccc|cc}\hline
O_\mathrm{h} &amp; E &amp; 8C_3 &amp; 6C_2 &amp; 6C_4 &amp; \begin{aligned}3C_2 \\ \scriptsize=C_4^2\end{aligned} &amp; i &amp; 6S_4 &amp; 8S_6 &amp; 3\sigma_\mathrm{h} &amp; 6\sigma_\mathrm{d} &amp; &amp; \\ \hline
\mathrm{A_{1g}} &amp; 1  &amp; 1  &amp; 1  &amp; 1  &amp; 1  &amp; 1  &amp; 1  &amp; 1  &amp; 1  &amp; 1  &amp; &amp; x^2+y^2+z^2 \\
\mathrm{A_{2g}} &amp; 1  &amp; 1  &amp; -1 &amp; -1 &amp; 1  &amp; 1  &amp; -1 &amp; 1  &amp; 1  &amp; -1 &amp; &amp; \\
\mathrm{E_g}    &amp; 2  &amp; -1 &amp; 0  &amp; 0  &amp; 2  &amp; 2  &amp; 0  &amp; -1 &amp; 2  &amp; 0  &amp; &amp; \begin{aligned}(2z^2-x^2-y^2,\\ x^2-y^2)\,\,\,\,\,\, \end{aligned} \\
\mathrm{T_{1g}} &amp; 3  &amp; 0  &amp; -1 &amp; 1  &amp; -1 &amp; 3  &amp; 1  &amp; 0  &amp; -1 &amp; -1 &amp; (R_x,R_y,R_z) &amp; \\
\mathrm{T_{2g}} &amp; 3  &amp; 0  &amp; 1  &amp; -1 &amp; -1 &amp; 3  &amp; -1 &amp; 0  &amp; -1 &amp; 1  &amp; &amp; (xy,xz,yz) \\
\mathrm{A_{1u}} &amp; 1  &amp; 1  &amp; 1  &amp; 1  &amp; 1  &amp; -1 &amp; -1 &amp; -1 &amp; -1 &amp; -1  &amp; &amp; \\
\mathrm{A_{2u}} &amp; 1  &amp; 1  &amp; -1 &amp; -1 &amp; 1  &amp; -1 &amp; 1  &amp; -1 &amp; -1 &amp; 1  &amp; &amp; \\
\mathrm{E_u}    &amp; 2  &amp; -1 &amp; 0  &amp; 0  &amp; 2  &amp; -2 &amp; 0  &amp; 1  &amp; -2 &amp; 0  &amp; &amp; \\
\mathrm{T_{1u}} &amp; 3  &amp; 0  &amp; -1 &amp; 1  &amp; -1 &amp; -3 &amp; -1 &amp; 0  &amp; 1  &amp; 1  &amp; (x,y,z) &amp; \\
\mathrm{T_{2u}} &amp; 3  &amp; 0  &amp; 1  &amp; -1 &amp; -1 &amp; -3 &amp; 1  &amp; 0  &amp; 1  &amp; -1 &amp; &amp; \\ \hline
\end{array}$$

Work

$$\begin{array}{c|cccccccccc} \hline
O_\mathrm{h} &amp; E &amp; 8C_3 &amp; 6C_2 &amp; 6C_4 &amp; 3C_2 &amp; i &amp; 6S_4 &amp; 8S_6 &amp; 3\sigma_\mathrm{h} &amp; 6\sigma_\mathrm{d} &amp; &amp; \\ \hline
\mathrm{E_g}    &amp; 2  &amp; -1 &amp; 0  &amp; 0  &amp; 2  &amp; 2  &amp; 0  &amp; -1 &amp; 2  &amp; 0 \\
\mathrm{T_{2g}} &amp; 3  &amp; 0  &amp; 1  &amp; -1 &amp; -1 &amp; 3  &amp; -1 &amp; 0  &amp; -1 &amp; 1  \\ \hline
\Gamma_{\text{d-orbitals}} &amp; 5 &amp; ? &amp; 1 &amp; -1 &amp; 1 &amp; 5 &amp; -1 &amp; ? &amp; 1 &amp; 1 \\ \hline
\end{array}$$

As can be seen, everything matches up but the $C_3$ and $S_6$ operations. I tried using Molecule Viewer and rotating the d-orbitals, but I still don't see the $-1$. They all look like zero to me.

I would expect the $\mathrm d_{x^2-y^2}$ orbital to correspond to $E_g$ from the character table, but I only see it rotate into a "$\mathrm d_{y^2-z^2}$" orbital, with the orbitals along the y and z axes instead of x and y axes. The $\mathrm d_{z^2}$ orbital rotates into a "$\mathrm d_{x^2}$" orbital, lying along the x axis instead of the z axis.

Anyone know which d-orbital rotates into $-1$ for a $C_3$ operation? I assume a $-1$ for $S_6$ is the same orbital, but if not then does anyone which that one is as well?

Thank you
The $C_3$ operation rotates $x \rightarrow y$, $y \rightarrow z$ and $z \rightarrow x$.
Hence,
$$\phi_1=2z^2-(x^2+y^2)$$ 
rotates into 
$$-(1/2)\phi_1 +(3/2)\phi_2$$ and
$$\phi_2=x^2-y^2$$
rotates into 
$$-(1/2)\phi_1 -(1/2)\phi_2$$
so, in the basis
 $( \phi_1, \phi_2 )$ the matrix that represents $C_3$ is

$$\left [\begin{matrix} -1/2 &amp; -1/2  \\ 3/2 &amp; -1/2  \end{matrix}\right]$$

Denoting this 2 by 2 matrix by M what we have is
$C_3(\phi_1  \phi_2) = (\phi_1 \phi_2)M$.
The trace of this matrix is $-1$. 

This is what you find in the character table.

Since $C_3^3=I$ (identity) its eigenvalues are the cubic roots of unity:
$1, \lambda$ and $ \lambda^*$
where $\lambda = \exp(2\pi i/3)=-1/2+i\sqrt 3/2$,
and $\lambda^*$ is the complex conjugate of $\lambda $.

It is easy to find eigenvectors with the eigenvalue 1 [e.g., an s orbital or x+y+z].

You can find a linear combination of $\phi_1$ and $\phi_2$ that will be an eigenvector of $C_3$ with the eigenvalue $\lambda$ (transform into itself multiplied by $\lambda$)
and another linear combination that will have eigenvalue $\lambda^*$. Now, the matrix representing $C_3$ will be diagonal with $ \lambda$ and $\lambda^*$ along the diagonal. The trace remains $-1$.
Something similar should work for $S_6$.