The difference between Colab and your local setup would come from all used processing units, i.e.:

used CPU
data storage and bandwidth to access it
used GPU
different libraries

etc.
You could profile the code to check, which part of the training takes most of the time.