text: Title: Plotting model with second spectrum then removing it does not work
  Body: Something goes wrong with the navigator when adding and then removing a second navigator.
  
  Example:
  
  ```python
  import numpy as np
  import hyperspy.api as hs
  
  data = np.random.random((10, 20, 500))
  s = hs.signals.Signal1D(data)
  m = s.create_model()
  m.plot()
  ```
  
  - Move the red navigator
  - Press the `E` key on your keyboard
  - Move the new, blue, navigator
  - Press the `E` key to remove the second navigator
  
  The red navigator can no longer be moved, and an error is raised: 
  
  ```python
  matplotlib/axes/_base.py", line 3570, in _validate_converted_limits
      raise ValueError("Axis limits cannot be NaN or Inf")
  ValueError: Axis limits cannot be NaN or Inf
  ```
  
  Movie example of this:
  
  [model_plot_bug_second_navigator.webm](https://github.com/hyperspy/hyperspy/assets/1690979/649d311f-870b-48bc-aea0-348a4403b082)
  
  ## Python environment:
   - HyperSpy version: current `RELEASE_next_patch`
   - Python version: 3.11
  
  