Hello Mishoo!



 mishooax:

class DNN(nn.Module):
       ...
        # output layer
        output_layer_mean = nn.Linear(hidden_layer_sizes[-1], 1)
        output_layer_sigma = nn.Linear(hidden_layer_sizes[-1], 1)
        # make sure that sigma > 0
        output_layer = torch.cat(output_layer_mean, F.softplus(output_layer_sigma))  <--- NO GOOD
        self.linear_layers_list.append(output_layer)
...
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        for layer in self.linear_layers_list:
            x = layer(x)
        return x

should i call F.softplus in forward instead of __init__()?


Yes.  As you see, you can’t apply softplus() to a Linear.  You need
to apply it to the output of the Linear, which is a tensor.
I would not append output_layer (nor output_layer_mean nor
output_layer_sigma) to linear_layers_list.  Something like this:
        output_layer_mean = nn.Linear(hidden_layer_sizes[-1], 1)
        output_layer_sigma = nn.Linear(hidden_layer_sizes[-1], 1)
        # do this stuff in forward ...
        # # make sure that sigma > 0
        # output_layer = torch.cat(output_layer_mean, F.softplus(output_layer_sigma))  <--- NO GOOD
        # self.linear_layers_list.append(output_layer)
...
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        for layer in self.linear_layers_list:
            x = layer(x)
        # haven't yet applied output_layer_mean nor output_layer_sigma 
        x = torch.cat((output_layer_mean(x), F.softplus(output_layer_sigma(x))), dim = -1)
        return x

Best.
K. Frank