This model was contributed by jdemouth. The original code can be found here. That repository contains a multi-GPU and multi-node implementation of the
Megatron Language models. In particular, it contains a hybrid model parallel approach using "tensor parallel" and
"pipeline parallel" techniques.
MegatronBertConfig
[[autodoc]] MegatronBertConfig
MegatronBertModel
[[autodoc]] MegatronBertModel
    - forward
MegatronBertForMaskedLM
[[autodoc]] MegatronBertForMaskedLM
    - forward
MegatronBertForCausalLM
[[autodoc]] MegatronBertForCausalLM
    - forward
MegatronBertForNextSentencePrediction
[[autodoc]] MegatronBertForNextSentencePrediction
    - forward
MegatronBertForPreTraining
[[autodoc]] MegatronBertForPreTraining
    - forward
MegatronBertForSequenceClassification
[[autodoc]] MegatronBertForSequenceClassification
