Q:

Was the PDP-11 coroutine instruction actually used?

In a typical description of the coroutine mechanism it usually mentioned that the PDP-11 instruction set provided a way to effect the coroutine switch by a single instruction, namely
JSR PC,@(SP)+. 
Was that instruction actually used in standard system software (an OS, a language runtime library, etc.)?

A:

Well, I used it.
Back in the late 1980s, I led a small team who produced a signalling message distributor for British Rail (as it was then).  This was on the PDP-11 in a mixture of C and Assembler.
We used the coroutine switch instruction in the communications handler part of the application.
It shaved the odd millisecond off the system's response to incoming messages on any of a dozen modem lines.  
Don't ask me exactly how it was used.  My memory isn't that good and the system was scrapped in the early 2000s.

A:

Here is a link to an octal debugger written in PDP-11 Assembly. It's using this instruction in the I/O routines. 
Apparently, the RSTS APIs use a global data structure called XRB to do I/O. And this application keeps a backup of XRB which is called O.XRB. Now, O.IO is a routine which, when called, does two things:

Copies XRB into O.XRB, and writes some constants into XRB in preparation for a syscall
Leaves on the top of the stack the address to a routine which restores XRB from the backup, O.XRB.

This is why O.IO contains the coroutine instruction (it exits the subroutine and leaves PC + 2 on top of stack), and is also why every call to O.IO is followed by the coroutine instruction (it calls the subroutine to restore XRB and makes that subroutine return to the caller).

A:

We used coroutines in a system where we were implementing a new human interface. The original code was a disaster, no - strike that, was not well layered so that new interfaces were reasonably accommodated. So our approach was to redirect the original I/O register addresses to a memory block. Then the new interface code would read from real registers and spoon feed the shared memory to the old code.
We implemented this with two functions KBD and DISP in place of the original code's idle loop and used coroutine calls to go between them. There were probably 20 or so places inside KBD where input sequences were being accumulated and I/O waits were in progress. Instead of waiting in the KBD code the code would call DISP. DISP looks at the shared memory and when something changes it writes to the new display. While DISP was doing that it may have had to wait and would go back to KBD with a cocall. So neither KBD nor DISP knew exactly where in the peer's process the cocall would go each could call the other with the exact $PC after the last cocall.
KBD might go line this:
while true {
  while no_input { cocall }
  read input
  dispatch on input data
  do-x:
    while no_input { cocall }
    read extra input
    write to memory block
    while old_code_reads_memory_block { cocall }
  do-y:
    write to memory block
    while old_code_reads_memory_block { cocall }
}

Then DISP does similarly:
while true {
    while memory_block_is_unchanged {cocall}
    figure out what display objects to write
    write a:
      place data in register A
      while waiting_for_ack {cocall}
      signal to memory block that data is accepted
      while waiting for memory block ack {cocall}
    write b:
      <more of the same>
}

In the PDP-11 you couldn't set this up with normal JSR and RTS calls. There was a startup gimmick where KBD would push the address of DISP on the stack. Then as KBD wanted to defer processing it would execute the JSR PC,@(SP)+ and go off into DISP. So the two merrily bounce back and forth each using their respective $PC registers as a context pointer for the other to return to.
There were, of course, some down sides to this approach but it was pretty slick.

