Thanks @philschmid. I tested your code with encoding='utf-8' but it did not change the content of my json file with strange letters instead of letters with accents.
However, I just found the complementary code in this post 1 that solves my problem: ensure_ascii=False as an argument of json.dump().
Here the code I use now (taken from notebook lab2_batch_transform.ipynb and modified with the 2 cited arguments):
with open(dataset_csv_file, "r+") as infile, open(dataset_jsonl_file, "w+", encoding='utf-8') as outfile:
    reader = csv.DictReader(infile)
    for row in reader:
        json.dump(row, outfile, ensure_ascii=False)
        outfile.write('\n')