I’m not sure what “Note 3” means and how you are releasing the memory, but in case you are clearing the cache this would hit the performance.
Note 5 makes it quite hard to give you a valid answer. Would it be possible to get a model with the input shapes showing the performance difference?