I came across this post beacuse i had a problem in taking the output of the padded input with bidirectional lstm and your case is the same.
I might not have have understood your problem completely but here’s what i think,  You don’t have to pass the ‘ht’ as it is because it is a collection of hidden states over the timesteps.What you could do is select first and last timestep vectors and remove padding from both of them, concatenate and then pass it to the FC.
Please correct if i’m wrong and also reply if you found another way around.