Stack Exchange
Replacing list of lists of string by function objects to speed up ML training process
from datasets import load_dataset #Huggingface
from transformers import BertTokenizer #Huggingface:

def tokenized_dataset(dataset):
    """ Method that tokenizes each document in the train, test and validation dataset

    Args:
        dataset (DatasetDict): dataset that will be tokenized (train, test, validation)
    
    Returns:
        dict: dataset once tokenized
    """

    tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

    encode = lambda document: tokenizer(document, return_tensors='pt', padding=True, truncation=True)
    train_articles = [encode(document) for document in dataset["train"]["article"]]
    test_articles = [encode(document) for document in dataset["test"]["article"]]
    val_articles = [encode(document) for document in dataset["val"]["article"]]
    train_abstracts = [encode(document) for document in dataset["train"]["abstract"]]
    test_abstracts = [encode(document) for document in dataset["test"]["abstract"]]
    val_abstracts = [encode(document) for document in dataset["val"]["abstract"]]

    return {"train": (train_articles, train_abstracts),
            "test": (test_articles, test_abstracts),
            "val": (val_articles, val_abstracts)}

if __name__ == "__main__":
    dataset = load_data("./train/", "./test/", "./val/", "./.cache_dir")
    tokenized_data = tokenized_dataset(dataset)

I would like to modify the function tokenized_dataset because it creates a very heavy dictionary. The dataset created by that function will be reused for a ML training. However, dragging that dictionary during the training will slow down the training a lot.
Be aware that document is similar to
[['eleven politicians from 7 parties made comments in letter to a newspaper .',
  "said dpp alison saunders had ` damaged public confidence ' in justice .",
  'ms saunders ruled lord janner unfit to stand trial over child abuse claims .',
  'the cps has pursued at least 19 suspected paedophiles with dementia .'],
 ['an increasing number of surveys claim to reveal what makes us happiest .',
  'but are these generic lists really of any use to us ?',
  'janet street-porter makes her own list - of things making her unhappy !'],
 ["author of ` into the wild ' spoke to five rape victims in missoula , montana .",
  "` missoula : rape and the justice system in a college town ' was released april 21 .",
  "three of five victims profiled in the book sat down with abc 's nightline wednesday night .",
  'kelsey belnap , allison huguet and hillary mclaughlin said they had been raped by university of montana football '
  'players .',
  "huguet and mclaughlin 's attacker , beau donaldson , pleaded guilty to rape in 2012 and was sentenced to 10 years .",
  'belnap claimed four players gang-raped her in 2010 , but prosecutors never charged them citing lack of probable '
  'cause .',
  'mr krakauer wrote book after realizing close friend was a rape victim .'],
 ['tesco announced a record annual loss of £ 6.38 billion yesterday .',
  'drop in sales , one-off costs and pensions blamed for financial loss .',
  'supermarket giant now under pressure to close 200 stores nationwide .',
  'here , retail industry veterans , plus mail writers , identify what went wrong .'],
 ...,
 ['snp leader said alex salmond did not field questions over his family .',
  "said she was not ` moaning ' but also attacked criticism of women 's looks .",
  'she made the remarks in latest programme profiling the main party leaders .',
  'ms sturgeon also revealed her tv habits and recent image makeover .',
  'she said she relaxed by eating steak and chips on a saturday night .']]

So in the dictionary, the keys are just strings, but the values are all list of lists of strings. Instead of having value = list of list of strings, I think it is better to create sort of a list of object function instead of having list of lists of strings. It will make the dictionary much lighter. How can I do that?
EDIT
I preprocess a dataset (article and abstract) before feeding a Neural Network model (NLP). First, I load the dataset with datasets.load_dataset() from different repositories, i.e. ./train/, ./test/ and ./val/. Once done, I tokenize each sentence of the articles and abstracts
