Thanks for the reply! When running it in the command line within the .venv the same error occurs… Should I try to run it outside the virtual env as well?
EDIT: It returns the same error if I run it in cli outside the virtual env:
>>> tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True)
>>> tokenizer
PreTrainedTokenizerFast(name_or_path='roberta-base', vocab_size=50265, model_max_len=512, is_fast=True, padding_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'mask_token': AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False)})

>>> datasets = datasets.map(
...     lambda x: tokenizer(x['text'], return_special_tokens_mask=True),
...     batched=True,
...     batch_size=10,
...     num_proc=2,#psutil.cpu_count(),
...     remove_columns=['text'],
... )
Token indices sequence length is longer than the specified maximum sequence length for this model (4078 > 512). Running this sequence through the model will result in indexing errors
#1:   0%|                                                                                                                | 0/50 [00:00<?, ?ba/s]
#0:   0%|                                                                                                                | 0/50 [00:00<?, ?ba/s]
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Python\lib\site-packages\multiprocess\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Python\lib\site-packages\datasets\arrow_dataset.py", line 203, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "C:\Python\lib\site-packages\datasets\fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "C:\Python\lib\site-packages\datasets\arrow_dataset.py", line 1695, in _map_single
    batch = apply_function_on_filtered_inputs(
  File "C:\Python\lib\site-packages\datasets\arrow_dataset.py", line 1608, in apply_function_on_filtered_inputs
    function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "<stdin>", line 2, in <lambda>
NameError: name 'tokenizer' is not defined
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python\lib\site-packages\datasets\dataset_dict.py", line 432, in map
    {
  File "C:\Python\lib\site-packages\datasets\dataset_dict.py", line 433, in <dictcomp>
    k: dataset.map(
  File "C:\Python\lib\site-packages\datasets\arrow_dataset.py", line 1485, in map
    transformed_shards = [r.get() for r in results]
  File "C:\Python\lib\site-packages\datasets\arrow_dataset.py", line 1485, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "C:\Python\lib\site-packages\multiprocess\pool.py", line 771, in get
    raise self._value
NameError: name 'tokenizer' is not defined