I am using the Fake news dataset that is used in this google colab notebook. 2 with the goal of adapting this example. For full reproducability, I uploaded the exact files I am using for training and testing in a github repository here 1.
However it appeared that some of the classes and methods were deprecated so I was trying to re-do it using the notebook as a guide: IMDb Classification with Trainer.ipynb
I am getting error after running this train_dataset = ds_train.map(tokenize) where you will find tokenize defined below along with the rest of the code. I copied and pasted the error message after the code. (see full error in comment)
In case anyone has further advice or comments I also added the rest of the code I am planning to run, which you will find after the error message.
Thank you for viewing this post and I appreciate any help you can offer.
from nlp import Dataset
import pandas as pd
from torch import tensor
from transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfig, EvalPrediction
import torch

# read csv in pandas
df_train = pd.read_csv("~/Downloads/fakenewstrain.csv")
df_test = pd.read_csv("~/Downloads/fakenewstest.csv")

# convert pandas df (only columns 'titletext' and 'label') to nlp Dataset
ds_train = Dataset.from_pandas(df_train[['titletext','label']])
ds_test = Dataset.from_pandas(df_test[['titletext','label']])

# set up configuration, tokenizer and model
config = AutoConfig.from_pretrained('bert-base-uncased')    
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
model = AutoModelForSequenceClassification.from_config(config)

# function to tokenize a line of text using tokenizer
def tokenize(batch):
    return tokenizer(batch['titletext'], 
                     max_length = 64, 
                     truncation = True,
                     padding = True, 
                     return_tensors = "pt")

# loop through Dataset using Dataset map function for tokenization
train_dataset = ds_train.map(tokenize)
test_dataset = ds_test.map(tokenize)

Here is the error I am getting:
Full error is in the comment.
ArrowInvalid: Could not convert tensor([[  101,  8499,  4642,  1106,  5307,  1614,  1166,  1114, 27157,  2101,           1656,  1733,   119,  4613,   117,  2631,   113, 13597,   114,  5554,           8499,   112,   188,  1207, 13715,   176, 12328,  1500,  3215,  1786,           1656,  1733,  1120,   170,   185, 14695,  8037,  1303,  1113,  9170,           1115,  1103, 26961,  1524,   118,  6057,  1110,  1231,  7867, 27885,           1103,  1226,   107,  1115,  1119,   112,   188,  1151,  1773,   107,           1105,  1110,  2407,   102]]) with type Tensor: did not recognize Python value type when inferring an Arrow data type
Here is the rest of my code (feel free to ignore as not relevant to exact question) I just figured I would add it in case anyone had any helpful comments:
I have actually been confused about how the labels are specified. From what I see, they are only referenced to when using set_format however it looks like columns is actually just a list of column names, and I did not see anywhere in the documentation that implied that Trainer specifically looks for certain columns.
# loop through Dataset using Dataset map function for tokenization
train_dataset = ds_train.map(tokenize)
test_dataset = ds_test.map(tokenize)

# Set format of Dataset, and specify columns to use 
# (columns are "input_ids", "attention mask", "token_type_ids" and "label")
# Do I need attention mask since im not doing two sentences? Do I need token type ids?
train_dataset.set_format('torch', columns=['input_ids', 'token_type_ids', 'label'])
test_dataset.set_format('torch', columns=['input_ids', 'token_type_ids', 'label'])

def compute_metrics(p: EvalPrediction) -> dict():
    preds = np.argmax(p.predictions, axis=1)
    return glue_compute_metrics(data_args.task_name, preds, p.label_ids)

training_args = transformers.TrainingArguments(
    output_dir="./Downloads/tmp/",
    overwrite_output_dir=True,
    do_train=True,
    do_eval=True,
    per_gpu_train_batch_size=16,
    per_gpu_eval_batch_size=64,
    num_train_epochs=1,
    logging_steps=500,
    logging_first_step=True,
    save_steps=1000,
    evaluate_during_training=True,
)

trainer = transformers.Trainer(model = model, 
                  args = training_args, 
                  train_dataset = text_train, 
                  eval_dataset = text_test,
                  compute_metrics = compute_metrics)

trainer.train()