The actual model workload is tiny, so that changing the size of this single layer might not be directly visible in the end2end training time.
You would have to properly profile the code parts to determine how much time is actually spent on the training itself.
The current runtime might be determined largely by the data loading and processing time or even the dispatching or kernel launch overheads.