How can I rebind Caps lock to delete/backspace? I tried using DoubleCommand, but it doesn't let you hold down caps lock, and pressing it causes caps lock light to be enabled/disabled. Ideally I want caps lock to never be enabled and for the key to behave exactly as though it were the delete key.

Another problem that DoubleCommand has is that every second press just toggles the light, and doesn't delete a character.
You may want to try Seil (formerly PCKeyboardHack).
I use it to bind Caps Lock to Right-Ctrl (not plain Ctrl), so that I can bind Caps Lock to Meta in Emacs (while Ctrl remains Ctrl, and other keys are kept to the usual behavior).