Thank you, Thomas. I realize I made a silly mistake on the optimizer that does not set the learning rate for time_factors:
    self.optimizer = torch.optim.SGD([
        {"params": self.model.user_factors.weight},
        {"params": self.model.item_factors.weight},
        {"params": self.model.user_biases.weight, "lr": config.bias_learning_rate},
        {"params": self.model.time_biases.weight, "lr": config.bias_learning_rate},
        {"params": self.model.item_biases.weight, "lr": config.bias_learning_rate}
    ], lr=config.learning_rate, weight_decay=config.weight_decay)

Thank you for your help.