I think that BERT uses the special [CLS] token to get the entire sentence representation of the sequence in one token.  Perhaps you can try something similar and then take this vector as your input.  For BERT, they generally put this token in the first position.