For some reason, the size of inputs in model(*inputs) is getting dependent on the batch size.
For a batch size of 16, doing model(*inputs) gives me Expected 3 inputs, got 16. I must have  made a mistake.