Would it work to define a single linear layer that has output of size 1 and disable the bias using self.linear = torch.nn.Linear(1*1*80, 1, bias=False)? Then you perform the linear FC layer only once and add the biases manually? You can define the bias tensors b1 and b2 as nn.Parameter and add the bias to the output of the final FC layer to get the two different outputs. Therefore, the weights of the FC layer are shared, and biases are defined separately and added independently to each output unit.