I manage to fix it. Honestly, I rewrote the code and now it is working. It looks the same, except that now it works. I really appreciate your reply.
I have one or two more questions.
Additionally to Q-learning, and Deep Q-learning, I am planning to use metaheuristics (e.g., differential evolution, genetic algorithms, etc.) instead of gradient descent to train the network.
As you probably know, in this context, each metaheuristic’ solution is represented by a d dimensional array, where d is the number of parameters of the ANN. Given that I have a few questions:

How can I transform all the parameters of the ANN to two one-hot vectors (one vector for weights and one for biases)? Is there any more elegant way than:

    def get_parameters(self):
        one_hot = [] 
        for key in self.state_dict():
            for tensor in self.state_dict()[key]:
                if not tensor.size():
                    one_hot.append(float(tensor))
                else:
                    for value in tensor:
                        one_hot.append(float(value))
        return one_hot


How can I define a straight forward function that sets the ANN parameters given two one-hot vectors (weights, biases)?
Are there any bounds with respect to the weights or biases? Should I constraint it between a specific range? What do you suggest?

This is more related to your experience in the field.

I wonder when it is recommended (or mandatory) to use metaheuristics instead of gradient descent? I believe that it is related to the cases where you don’t have an input, output, and target. If you dont have a target you cannot calculate the loss, and then you cannot perform gradient descent. Is it right?
Do you have any example of problems where the model can only be trained by the usage of metaheuristics?

The idea of this project was for me to learn exactly that, the trade-offs between the usage of each strategy. With deep Q-learning, I am trying to approximate the Q-values. However, I can easily formulate it as an optimization problem where the metaheuristic is used to optimize the weights of the ANN and the objective function is to maximize the score (or it can be a multi-objective function such as maximize score and minimize moves, etc.).
For me, it looks like there is much more freedom in the metaheuristic case. Maybe it is because I am more used to formulate problems like that. Do you have any experience with that?