update: I just found it out as I was going over my message that (if i % 40 == 39) is not satisfied in the final batch so the average_cost=0 is not applied at the end of each epoch leading the cost of the first batch to be larger than usual. Adding i % 40 == 39 after each epoch resolves this issue.