Question: On my MacPro there is one issue (and guys from Apple think this is feature...) that annoys me very much: there is a time delay between CapsLock switching. I used to do that fast. If I go to preferences and reassign caps lock to any other button and then back it works OK then, so it is not a hardware problem.

Is there any way to fix that ?

Answer: 
I was running the latest firmware on an Apple 2007 Aluminum Wireless Keyboard, and MacBook Pro Keyboard, but the issue still persisted. However, oddly this fixed it:

1. Open **System Preferences**
2. Click **Keyboard**
3. Click **Modifier Keys...**
4. Select your keyboard from the **Select keyboard:** list.
5. Set the **Caps Lock Key:** to **No Action**.
6. Press **OK**.
7. Ensure your Caps Lock no longer works on your keyboard.
8. Click **Modifier Keys...** again.
9. Set the **Caps Lock Key:** to **Caps Lock**.
10. Press **OK**.

Now, the Caps Lock key works instantly for that keyboard. Repeat as-needed for each keyboard you own.
Originally the answer by Martin Steven worked for me. This was when I used OSX 10.9.

After upgrading to OSX Yosemite (10.10) it stopped working. I found another, essentially the same, solution documented here

<http://sleepycow.org/2014/07/removing-the-caps-lock-delay-on-a-macbook/>

It uses updated version of the apps (with different names) that work with 10.10.

As requested - quoting the relevant sections from the link

1. Install [Seil](https://pqrs.org/osx/karabiner/seil.html.en) and [Karabiner](https://pqrs.org/osx/karabiner/).
2. Go to System Preferences > Keyboard > Modifier Keys and change “Caps Lock” to “No Action.”
3. Using Seil, check “Change Caps Lock Key” and change the key code to 110. (Application Key)
4. Go to Karabiner, and in the search box, enter “application key to capslock.” Under For PC Users > Change PC Application Key, check off “Application Key to CapsLock.”