Solved by tom in post #4 
              
              
                CNN -> Linear: 
No. The batch size should never be flattened. 
You keep  (10, 100) and feed it into a (100, 1) linear (or whatever) to get a 10, 1 - one prediction per batch item. 
You could concatenate the (10, 100) max with the mean (over the seq_len=5) to get a (10, 200). 
CNN -> RNN -> Linear 
H…