using "old" non-retina screen resolution chooser I have a 4k display, but my laptop seems unable to performantly use it (videos jitter, etc) is there a way to treat the display as 1080p display and let the display upscale it? Of course, Preferences lets me change the "resolution", but it's not actually changing the resolution, and more just changing the size of the UI. 

It's actually sort of the opposite of this question:

MacBook Pro Retina and 4k monitor (Samsung U28D590D) now showing &quot;scaling options&quot; in display
I found the screen resolution options in OS X very poor.  I use a third party tool called SwitchResX.  This is a pay app, however I believe that there are free third party apps which allow similar kinds of control... but I haven't used them.

How are you connecting to the display?  Only DisplayPort can drive a 4k screen, so you would probably need a mini-DisplayPort to DisplayPort cable.  Amazon have several.  The current HDMI standard does not have the bandwidth to transfer enough data fast enough to support 60Hz, which would be a smoother experience.  It only allows for 30Hz.

The display has a native resolution (3840x2160).  Every resolution that is not the native is scaled (due to the display having fixed pixels).  Only resolutions that divide exactly into the native resolution would be sharp.  

TL;DR you can't change the resolution of a monitor, only how the display is drawn on it.  Get a miniDP to DP cable.