I haven’t checked the logic of your code. But your code to work, you have to change this line
scaling_mat = torch.eye(3, device=frame.device).repeat(batch_size, 1, 1)

to
scaling_mat = torch.eye(256, device=frame.device).repeat(batch_size, 1, 1)