Order does matter apparently ! Find the arrow_dataset.py file (it can be found in Lib\site-packages\datasets-1.8.1.dev0-py3.9.egg\datasets); look up line 3181 of the file (my version of the package is transformers==4.9.0dev0) :
if axis == 0 and not all([dset.features.type == dsets[0].features.type for dset in dsets]):
        raise ValueError("Features must match for all datasets")
elif axis == 1 and not all([dset.num_rows == dsets[0].num_rows for dset in dsets]):
        raise ValueError("Number of rows must match for all datasets")

dset.features.type is sensitive to the order of the columns; it gives a StructType, as below, which is clearly order-sensitive:
pawsqqp['train'].column_names
output : ['idx', 'question1', 'question2', 'label']
pawsqqp['train'].features.type
output: StructType(struct<idx: int32, question1: string, question2: string, label: int64>)

So I simply changed that first line in the arrow_dataset.py file to
if axis == 0 and not all([dset.features == dsets[0].features for dset in dsets]):

and now it no longer cares about the order of the columns, and I can confirm that it’s working well in the examples I had. But I don’t recommend applying this change without a double-check! Make sure this modification works well for your datasets too.