Is it possible to enable 1.5x scaling using HiDPI on OS X? I use a Mac Mini on a 46" 1080p TV and using native resolution the UI is too small, and using HiDPI the UI is too big.

Is there a way to hack the OS X to enable 150% scaling instead of 100% and 200% ??
No, this is not possible as it would result in a significantly distorted image with loss of image data. There is no way to accurately display one dot over 1.5 x 1.5 pixels.