In my case the documents were short emails, most of which could fit in the 512 token limit of USE - I did not try fancy things like summarization / chunking, but the latter would be my first thing to try for a long document 
Regarding the annotations, they might help, but you’d have to think carefully about how you plan to combine them with the embeddings before applying UMAP.
Perhaps a “quick and dirty” approach would be to experiment with is concatenating the hidden states from multiple layers to see if that improves your document representation (assuming you’re just taking the last hidden state).
Alternatively, you could try composing different UMAP models for different embeddings (see e.g. here 3 for a discussion), but I’ve never tried that so cannot vouch for its utility.