text: Title: Prefect Workers crash when server returns 500s
  Body: ### Bug summary
  
  We run a pretty big self-hosted installation of Prefect 2.x (20k flow runs/day, 200k tasks) and noticed that when self-hosted API server becomes overloaded, it starts returning HTTP 500s.
  
  That's OK by itself, but this makes Workers (we use Kubernetes) exit unexpectedly, then get restarted by K8s (issue 1).
  Specifically we see the following problematic stack traces after which it exits:
   * _submit_run -> _check_flow_run -> read_deployment
   * cancel_run -> _get_configuration -> read_deployment
  
  Restarting also seems OK by itself, however we noticed that if a different flow was marked PENDING, but no K8s Job was scheduled yet when the worker exited, it'll be stuck in PENDING forever (issue 2). Here's the relevant code:
  
  https://github.com/PrefectHQ/prefect/blob/c4ac23189af1d27f8260452df302be8daee792b6/src/prefect/workers/base.py#L972-L977
  
  Issue 2 seems relatively hard to fully resolve, as it's impossible to atomically mark flow as pending and submit a job to K8s. Maybe we can do something by storing the state locally, but that will not work if the pod is restarted on a different node.
  
  Issue 1 looks more straightforward though. There's already a try/except around these places, but it only catches some exceptions, not all of them. Hopefully it'd be easy to resolve.
  
  ### Version info
  
  ```Text
  Version:             2.20.14
  API version:         0.8.4
  Python version:      3.11.7
  Git commit:          fb919c67
  Built:               Mon, Nov 18, 2024 4:41 PM
  OS/Arch:             linux/x86_64
  Profile:             default
  Server type:         ephemeral
  Server:
    Database:          sqlite
    SQLite version:    3.40.1
  ```
  
  ### Additional context
  
  _No response_