I’m trying to filter a dataset based on the ids in a list. This approach is too slow. The dataset is an Arrow dataset.
responses = load_dataset('peixian/rtGender', 'responses', split = 'train')
# post_id_test_list contains list of ids
responses_test = responses.filter(lambda x: x['post_id'] in post_id_test_list)