I’ve been experimenting with a transformer-like architecture where one input vector x is passed through h different linear layers, one for each head, and let’s call them y_1, y_2, … y_h. I want to pass the results for each head (y_1,y_2,…y_h) into an additional linear layer independently of each other, something like y_1->lin_1->y’1, y_2->lin_2->y’_2,…
The first step, namely converting x to y_1, y_2, …y_h is simple. I can just use a linear layer of hidden_size as input size, and h*hidden_size as output size, and divide evenly the output into each head.
But the second step is hard to implement with GPU parallelization in mind. Currently, I’m iterating through the number of heads and doing each of the linear layers one by one. The pseudocode is:
# lins, y, y_prime are lists
for head in h:
    y_prime[head] = lins[head](y[head])

Is there a way supported by pytorch that can calculate the results for all linear layers simultaneously? Thanks.