Q:

Center-of-mass frame for massless particles

Given a scattering event where a photon and electron go in and a photon and electron come out, what is the center of mass frame?
I'd say, since the photon has no mass, it's the rest frame of the incoming electron.

A:

By the Newtonian definition, the photon wouldn't count due to its zero mass, but this is a relativistic collision, so you need a relativistic definition of the center of mass. Relativistically, the c.m. frame is the one in which the total momentum four-vector of the system is purely timelike. No, this does not coincide with the electron's frame.
What you're describing is Compton scattering. In the c.m. frame, both the electron and the photon are scattered at 180 degrees with an energy the same as that with which they came in.
To determine the c.m. frame, add the momentum four-vectors and normalize the result. This is the velocity four-vector of the c.m. observer.
There are cases where this definition fails. For example, if the system consists of a single ray of light, then the momentum four-vector is lightlike, so you can't normalize it; there is no frame of reference in which it's at rest. But, e.g., if your system consists of two rays of light moving in different directions, then there is a c.m. frame.

A:

The reference frame of the center of mass is, by definition,  the one where the total $3$-momentum vanishes. It exists almost always also for massless particles as I go to discuss.
The total $4$-momentum $P$ of a system of $N$ free particles is the sum of their $4$-momenta of the particles, i.e.,
$$P = \sum_{a=1}^N P_{(a)}\:,$$
where each $P_{(a)}$ is a non-vanishing, future-oriented, causal $4$-vector (i.e., it is  either timelike or lightlike). 
By standard properties of sums of causal vectors for $N\geq 2$, one realizes that $P$  is causal as well, and it cannot be lightlike unless all  $4$-momenta of the system $P_{(a)}$ are parallel and lightlike. If at least one particle is not massless, like in the case you consider, $P$ turns out to be timelike and thus the center of mass is well defined. 
Indeed, when the total $4$ momentum $P$ is timelike and future directed (up to $4$-translations and spatial rotations which do not change kinematic) there is a future oriented Minkowskian reference frame (*) whose temporal axis $\partial_t$ verifies $P = k \partial_t$ for some $k>0$. 
In that reference frame, $P$ has no components perpendicular to $\partial_t$. In other words, the $3$-momentum vanishes. That is the center of mass reference frame of the physical system.  
The fact that even a system of massless particles has a center of mass (excluding the case pointed out above) should not be surprising because the (invariant) mass of the particles does not matter here. Differently form the classical case, it is the energy which matters and it is strictly positive also for photons. Let us illustrate this fact. In the classical picture, one has to look for a reference frame where
$$\vec{0} =\sum_{a=1}^N  m_a \vec{v}_a\:.$$
Instead, in the relativistic realm, instead, the relevant equation is
$$\vec{0} = \sum_{a=1}^N \vec{P}_a = \sum_{a=1}^N  E_a\vec{v}_a $$  
where the $4$-momentum is decomposed into its energy (temporal) component and its $3$-momentum (spatial) components as
$$P_a = (E_{(a)}, \vec{P}_{(a)})$$  (I assume $c=1$) 
and consequently the $3$-velocity is $$\vec{v}_{(a)}= \frac{\vec{P}_{(a)}}{E_{(a)}}\:.$$
You see that, also if some of the particles are photons, we have $E_{(a)}>0$, because $E_{(a)}^2 - \vec{P}^2_{(a)}=0$ and $P_{(a)}\neq 0$ in that case. 

(*) This is consequence of the fact that, up to dilations, the orthochronous Lorentz group $O(3,1)_+$ acts transitively in the interior of the future lightcone.

