Based on your code snippet it seems you are directly appending all intermediate features without detaching them, so I would assume to see an increase in memory usage in each iteration.
If that’s the case I would also expect to see a slowdown as you would need to allocate memory during the training. Could you check this behavior?