Suppose I want a layer to consist of 10 different activation functions. Right now the only way to do this seems to basically involve the creation of multiple layers and concatenating them. This obviously introduces some computational inefficiency.
Is there some clever hack to apply point-wise nonlinearities based on masks or something?
The way I envision this possibly happening is by doing the linear tranformation as usual  but then stacking multiple nonlinearities on top such that each nonlinearity layer ignores all but its own portion of the layer. Not sure how to approach that.