How macOS detects apps that stopped responding? Sometimes apps stop responding to user events probably due to bugs or unexpected edge cases. But it's different from a crash because an app may stop responding for some time and then "start responding" again, for example Photoshop may get stuck when opening huge images because processing might take some time, but eventually it loads the image and continues working fine.



And I am wondering, on the technical level, how the operating system detects such cases to offer the user an option to force quit the app.
GUI programs on macOS (and most other graphical operating systems) work by having a main loop that empties a so called "message queue". Some times this is called a "message pump".

Messages are put in the message queue by the operating system's windowing subsystem to tell the application about things that happened - for example "the mouse was clicked at (x,y)" or "the window was resized to size (width,height)". 

The application then pulls these messages from the queue, processes them, and then waits for new messsages.

If the application for some reason isn't pulling these messages from the queue for a period of time, macOS will say that the application is not responding. This is all it is.

Many GUI application split their execution in multiple "threads" - which can be thought of as a program streams executing at the same time. The main thread is responsible for the main message loop, whilst other threads can be responsible for network communication, disk I/O, background calculations, etc. However any changes to the graphical user interfaces that is managed by the operating system must go through the main thread.

If for some reason the main thread is busy doing something else than GUI work, then it could be marked as non-responding. This could be because the program is stuck in an infinite loop, or is in a deadlock waiting for some resource or similar. The program will never recover from this state, but the operating system can't know this.

Another reason for the main thread to be busy could be that it is saving a large file, waiting for communication on the network, or doing a large calculation. After that is done, the program will start pulling messages from the queue again, and will be responding again. Usually programmers do a lot of work to ensure that nothing like that happens on the main thread to ensure that the program is responsive at all times.