I’ve not seen a pre-trained BERT with sequence length 2048.
Training expense goes up as the square of the sequence length, so len 2048 would be very costly to produce.  I think you would need a TPU to do fine-tuning on it, even if it exists.
See this page https://github.com/google-research/bert 13