I worked around the problem by skipping the training & validation for those batches which were not of the shape (50,20). Below is a simple if condition that helped me to solve the problem.

if( (inputs.shape[0],inputs.shape[1]) != (batch_size,seq_length)):
print(‘Validation - Input Shape Issue:’,inputs.shape)
continue