Looks like it was a silly incompatibility. With the GPU version of the same notebook (hyperparameters changed slightly), I was able to export the model to SavedModel. Does not look like a bug in transformers. Might be stemming from some incompatible Op within TensorFlow itself.