Yes, that’s already a good idea. You can indeed make 1 dataset with 2 columns, text and label.
Next, there are several options:

Either you create your dataset as a csv file, and then you turn it into a HuggingFace Dataset object, as follows:

from datasets import load_dataset
dataset = load_dataset('csv', data_files='my_file.csv')

… or if you have multiple files:
dataset = load_dataset('csv', data_files=['my_file_1.csv', 'my_file_2.csv', 'my_file_3.csv'])

… or if you already want to determine which ones are for training, which ones for testing:
dataset = load_dataset('csv', data_files={'train': ['my_train_file_1.csv', 'my_train_file_2.csv'] 'test': 'my_test_file.csv'})

The benefit of HuggingFace Datasets is that it allows you to quickly tokenize the entire dataset and prepare it for the model, using the .map(function) functionality.

Alternatively, you can implement a classic PyTorch dataset with the getitem method. Each dataset item should then return the input_ids, attention_mask and label.