Can we run OS X apps inside X11 (XQuartz)? I'm wondering if that is possible, so that I can use some X11 window manager for everything.
No. Native OS X apps don't call down to X11 protocol and no one (Apple or other) has implemented any sort of shim/translation layer/conversion tool to port over the API.

It would surely involve slowdown, reduced acceleration, possibly loss of fidelity and loss of functionality.