Why bother with the desktop environment at all in that case? You could just install a window manager (I like Openbox but you may prefer a tiling wm) and xfce4-panel, which works pretty well standalone. This has been my setup for years now :).