How can I create a backup of my iMac when it won't start and recovery mode fails At this point I'm just trying to get a copy of my hard drive before I wipe it. 

I have an iMac running El Capitan and I am unable to login to my user account (spinning wheel of death).

I've tried to start up in Recovery Mode and backup files using Disk Utility. 
After waiting several hours I got the error message "The operation couldn't be completed. (com.apple.diskutililty error 3.)"

This was the process I followed.

When I try to re-install a new copy of El Capitan I get the error "Unable to install because this disk has S.M.A.R.T errors"

What else can I try?

How can I create a backup of my hard drive?
Have you tried starting your iMac in Target Disk Mode? 

To do this, switch off your iMac and then boot it up again making sure to hold the T key down until you see an icon appear on the screen.

If that works, then you may be able to use this mode to access your Hard Drive from another Mac (or even a Windows PC). However, if using a Windows computer it has to have the right connections and also be able to read a Mac's drive, usually by having extra drivers such as MacDrive, TransMac, MacDisk, or HFSExplorer installed.

Speaking of connections, you didn't specify what model of iMac you have. Typically you would use an appropriate cable connecting the two computers via your FireWire or Thunderbolt port. USB is not supported.

Summary 

In a nutshell, if you can get your iMac to start in Target Disk Mode, you can connect it to another computer (almost always another Mac) and your iMac's hard drive will then appear as an external drive on the other computer. If so, copy whatever files you can from it to another drive.

I should warn you though that it sounds as if your hard drive has failed, and doing this may not work. Also, even if it does, I wouldn't try reformatting it to continue using it. You really should replace the drive or use an external drive as your bootable drive.

Also, if you can boot from an external drive, another option is to just do that and try copying across your data that way (i.e. From the internal drive onto another drive). This avoids the need to try using Target Disk Mode.