@VisualMelon's comments nailed it, but to elaborate a bit in answer form:
Silberschatz, Galvin and Gagner are talking about how an operating system kernel can manage processes: it could set a strict resource limit on CPU time for a process and kill the process (not just report that it's unresponsive) if it exceeds that limit.
People typically don't do this, because arbitrarily killing processes is usually a bad idea. However, it is supported by, e.g., Linux's
ulimit -t command.
The Windows error message saying that a program appears to be frozen is operating at a much higher level than the low-level CPU time limit that Silberschatz, Galvin, and Gagner are discussing. The Windows GUI operates by messages: messages are regularly sent to running applications, to handle everything from notifying the application of mouse and keyboard events to requesting that an application redraw its screen contents after an update. At its core, then, a Windows GUI application typically has a
while loop that reads messages from the message queue and processes each one.
Problems arise if the application takes too long to process a message (because it's blocked waiting for file or network I/O, or because it started a lengthy CPU task, or because it entered an endless loop). When that happens, the application isn't receiving and processing new messages, so it can't respond to input or even redraw its screen contents. That's a bad user experience, so to make it less bad, Windows will detect when applications aren't checking their message queues and pop up the message you mentioned, asking if you want to kill the application.
Of course, Windows doesn't actually know if the application has truly locked up (e.g., endless loop or deadlock) or if it's just doing something longer than expected and will resume processing messages shortly. (Windows can't know - that's the halting problem.) That's why applications will sometimes appear to recover after Windows says they're frozen.
Because of this problem, well-written Windows applications will move any potentially lengthy tasks in background threads so that they can continue to pump the message queue and thus remain responsive, but making sure that this is done everywhere it's needed is easier said than done.