yes for this simple model there is no gating mechanism yet. The output from each expert layer is concatenated at the end. My concern is that currently this model is very slow to train. Adding the gating mechanism will even make it slower. The training speed is far slower than a MLP model with similar number of parameters, which is kind of wierd since the number of FLOPS should be roughly the same.