Deep Learning, Artificial Intelligence, Bert, Ehr, NLP.
dimension is 300, dimension of position-wise feed-forward networks is 300, 2 hidden layers with 4 attention heads for each layer. Specially, authors alternated the pre-training with 5 epochs and fine-tuning procedure with 5 epochs for 15 times to stabilize the training procedure. (This training