What video resolutions does late-2015 iMac 5K support on external monitors? I've got a Core i5 3.2GHz iMac with AMD R9 M390 2Gb. I decided to connect my old HP LP3065 monitor, but to my surprise it would only show either 1280x800 or 1024x640, though it supposed to be up to 2560x1600.

Could the problem be with graphics card, cable, or old monitor?
You haven't detailed how you have actually connected the monitor and the iMac, but it sounds like the problem is the cabling/adapters. 

The monitor is from a time where you couldn't achieve the 2560x1600 resolution without using dual-link DVI. Note that this is different from just "DVI" as a non-dual link DVI cable will not allow you to transfer such a resolution.

You'll to use an adapter especially for this purpose. For example Apple's mini-DisplayPort to dual-link DVI adapter:

https://www.apple.com/dk/shop/product/MB571Z/A/mini-displayport-til-dual-link-dvi-mellemstik