Hi! Could you please copy and paste the entire stack trace?

Is there some way to do intermediate Arrow-file writes to lower memory overhead?

You can control RAM usage in dataset scripts with the DEFAULT_WRITER_BATCH_SIZE attribute of GeneratorBasedBuilder (as we do in this script).

The files themselves are a large set of videos. I load in each video using pims and extract a set of frames as PIL Images. These are then stored in dataset rows with a few other bits of info e.g. class-labels and filenames.

In datasets 1.18.0, we added support for nested decoding of the Image feature which is ideal for your use-case. To use it, just define the features dict as:
features = Features({
    "frames": Sequence(Image(), length=10),
    "meta": ...
}) 

and yield data as:
yield idx, {
    "frames": [pil_img_frame1, pil_img_frame2, ...],
    "meta": ...,
}