First life in samsara In Buddhism, we believe in rebirth and karma. But was there a time when it was our first time in any of the 31 Planes of Existence? 

If that is the case, how would we be reborn anywhere since we did not have any karma to pay off?

If that is not the case, how were we just infinitely here?
If i may kindly correct the beginning sentence of your question, i.e. &quot;In Buddhism, we believe in rebirth and karma.&quot;
We actually try to verify these things for ourselves by studying reality through insight meditation, thereby gaining insights into how reality functions. We practice to gain experiental knowledge instead of believing things to be in a certain way.
Through insight meditation one will come to see that phenomena arise dependent on causes and conditions. One sees that phenomena can spawn new phenomena.
Looking for a first cause is problematic. Why is that?
A first cause breaks causality. One can then ask &quot;What caused the first cause? Or What caused the cause that caused the first cause? And so on ad infinitum&quot;.
This question of origin also belongs to The Four Imponderables, namely the 4th one &quot;Speculation about [the origin, etc., of] the cosmos is an imponderable that is not to be speculated about&quot;.
The Buddha explained that is it not conducive to ones practice to think about these things. That is because one cannot understand it unless one is a Buddha. If one is not a Buddha thinking about them will only leave the mind agitated and distressed which will not be beneficial for ones practice.

&quot;Therefore, o monks, do not brood over [any of these views] Such brooding, O monks, is senseless, has nothing to do with genuine pure conduct (s. ādibrahmacariyaka-sīla), does not lead to aversion, detachment, extinction, nor to peace, to full comprehension, enlightenment and Nibbāna.&quot;
-- SN 56: Saccasamyutta.