OpusNMT/MarianMT Machine Translation Models
### Model description
1,026 Language Pair Models, downloadable [here](http://opus.nlpl.eu/Opus-MT/)
Trained with Marian C++ [library](https://github.com/marian-nmt/marian)

### Open source status

* [ x] the model implementation is available: (give details)
* [ x] the model weights are available: (give details)
* [ x] who are the authors: TODO, find gh-usernames of authors!


### Proposed API:
```python
model_name = 'marian/en-fr'
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianModel.from_pretrained(model_name)
src_text = "how are you today?"
tgt_text = "comment allez vous aujourd'hui?"
# Training API
full_inputs: dict = tokenizer.prepare_batch_for_translation(src_text, tgt_text=tgt_text)
loss, logits, *other_outputs = model(
        full_inputs['input_ids'],
        full_inputs['attention_mask'],

        full_inputs['decoder_input_ids'], # this argument is mandatory for the forward pass
)
# Inference/generate API
src_inputs: dict = tokenizer.prepare_batch_for_translation(src_text)
generated_fr_ids = model.generate(src_inputs['input_ids'], src_inputs['attention_mask'])
french_text: List[str] = processor.decode_batch(generated_fr_ids)
```

### Implementation Details
`MarianTokenizer` Signatures

(Not calling it Tokenizer to avoid confusion, but I don't feel strongly.)
- All models require `MosesSentenceSplitter` and `MosesPunctuationNormalizer` preprocessing
    -  There are some additional perl scripts we will not port for pre/post-processing
- 81 of the models require BPE, 960 require SentencePiece.
- We can decide which is which


```python
class MarianTokenizer:
    def __init__(self, vocab_file, source_bpe, target_bpe, source_spm, target_spm):
        # decide whether to use BPE/SPM based on which files are present in S3.
        self.source_lang, self.target_lang #inferred from paths/config
        #self.source_spm =

    @property
    def uses_sentencepiece(self) -> bool:
        return self.source_spm is not None

    def from_pretrained(self, *args, **kwargs):
        # needs to be overwritten or modified to not fail if certain files not present.

    def prepare_batch_for_translation(src_text:str, tgt_text=None, return_tensors='pt', max_length=512, pad_to_max_length=True) -> Dict[str, tensor/list]:
		return {}


    def decode_batch(self, target_lang_ids: List[List[int]]) -> List[str]:


    def decode(self, target_lang_id: List[int]) -> str:
```

#### Edits
- renamed `MarianProcessor` -> `MarianTokenizer` for consistency
So no way to get output logits?<|||||>The `forward` method returns logits, like other models with language modeling heads. Is that what you meant?