The neutrons will never return once a black hole has formed and outgrown the neutron star. A black hole would exist within a neutron star until added mass increases the radius of the Event Horizon to engulf the neutron star matter. Once that happens things progress very slowly.
Hawking Radiation has still not been proved conclusively. Even if it's true there is still a delay ahead. Black holes with mass about two thirds of our Moon's upwards would be colder than the Cosmic Microwave Background and would therefore absorb energy from it faster than they would emit Hawking Radiation.
Black holes get colder as their mass increases so the CMB would be almost used up before evaporation would start and it's a hard guess as to how long that would take. Smaller black holes would evaporate more quickly for two reasons.
1) is that the space curvature is more extreme at the Event Horizon so Hawking Radiation would occur more quickly due to a higher number of particle-pairs created.
2) is that smaller Black Holes are predicted to have a higher temperature implying higher radiative power.
If a neutron star has a high enough angular momentum it can delay a Black Hole from forming because the centrifugal force would combat gravitational force but this would be a rare occurrence because neutron stars typically only lose mass via thermal radiation. 43
Therefore they don't slow down by much, maybe by a second every few millenia if we're overestimating.
By gaining enough mass from its surroundings via accretion (consuming nearby planets, dust and debris etc) the object may collapse further. A neutron star's rotation can also be slowed by several mechanisms.
One would be in the emission of pulses of electromagnetic radiation if it's a pulsar type.
Another would be an "anti-glitch", which is an observed case of a reversed "glitch" - when a portion of the crust cracks and matter is tossed off which speeds up rotation. Anti-glitch causes are still under investigation.
A third would be if accrued matter is moving in the opposite direction to the star's rotation. This is called retrograde accretion (see this paper https://arxiv.org/abs/1704.06364v1) and is thought to be conclusive because slowing-down and speeding up rates are very similar in observation.
It seems for now that whatever ends up inside a Black Hole is destroyed. Evidence points to the fact that matter would choose any form rather than allow Black Hole physics to happen. It just seems to be the inevitable fate of large stars. 43