Maybe the smaller datasets produce smaller embedding file (processed.embed) and since this is fully loaded in the GPU cache then it leaves a smaller amount free when it is bigger (80 Mb with smaller datasets, 200Mb with bigger)