If you are using the BertModel class of pytorch-bert, be sure to read in detail the doc on the initialization and outputs which is here 26. I am not sure you are using it correctly (the output is a tuple and the first element is either a list with all the encoded layers or only the last layer depending on the initialization of the class, read the doc!)