It is a good idea as long as 
a. An openly standardized direct-to-PC bus is used
b. An openly standardized interface to co-processor is used (not utilizing blobs).
c. Possibly a transparent to user mixing of CPU and special instructions.