Indeed this is slower that expected.
May I suggest to run ds_all_utts.set_format("numpy", columns=["embedding"]) first and re-run your benchmark to see if it speeds up things ?
Arrow has a NumPy integration that allows to read data with zero-copy, so this should increase the speed.