How to apply Torchtext convenience classes to prepare data for a Transformer?
Hello,
Reading the tutorial [Language Translation with torchText](https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html) I wondered how someone could use those convenience classes (`Field, BucketIterator`) to train/fine-tune a `Transformer`.

For instance, I'm currently working with a large dataset distributed in jsonl files which looks like:
```python
{ "query": "this is a query 1", "doc": "relevant document regarding query 1" },
{ "query": "this is a query 2", "doc": "relevant document regarding query 2" },
               ...
 
```

Now, to forward this data into a transformer like Bert, it is necessary to convert this dataset into the format:

```python3
(
#queries
 {
    'input_ids': tensor([
        [  101,  2023,  2003,  1037, 23032,  1015,   102,     0],
        [  101,  2023,  2003,  1037, 23032,  1016,   102,     0]]), 
    'attention_mask': tensor([
        [1, 1, 1, 1, 1, 1, 1, 0],
        [1, 1, 1, 1, 1, 1, 1, 0]])
 }, 

 #docs
 {
    'input_ids': tensor([
        [ 101, 2023, 2003, 2028, 7882, 6254, 4953,  102],
        [ 101, 2023, 2003, 2028, 7882, 6254, 4953,  102]]), 
    'attention_mask': 'input_ids': tensor([
        [1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1]])
}
```
So, what would be a clear and efficient approach to apply those convenience classes to tokenize a text dataset to fit it in the required format of a transformer?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
