looking for something more modern than X11 "streaming" with local Vagrant Linux VMs I have a similar question like Alternatives for XQuartz?, because XQuartz is discontinued since 2016 and X11 feels in general obsolete.

I use vagrant to spin up Linux VMs and connect with ssh to the VM. It's very fast way to test Linux stuff, but it's commandline only.

I wish to test also stuff like XFCE and LXDE, but using the Virtualbox Console is slow and the usability bad.

What are the different ways of using GUI apps and full desktop environments?
The site alternativeto.net lists Wayland as an XQuartz alternative, but there seems no macOS client.
X11 in general is not really "obsolete". 

You compare it to Wayland, but Wayland has only very recently become something that can be used in a standard install of Linux for example. It still has rough edges and use cases that are not implemented at all - so many Linux distributions has X11 as the default, or if they default to Wayland they have an option to use X11 instead.

For connecting to Linux VMs with desktop environments like XFCE and LXDE, I would still recommend X11 as the best option. 

You can use XQuartz, or you could use for example the xorg-server package from MacPorts, which has a more recent version of basically the same software as XQuartz).

Perhaps a better alternative would be use the X2Go (free) or NoMachine NX (commercial) software for accessing your X11 system from macOS. 

Another option would be to use VNC. Many VNC clients exists for Mac, and Linux has software like x11vnc that exposes X11 software (such as XFCE and LXDE environments) over VNC. The VNC protocol works substantially different than the X protocol, and I wouldn't recommend it if you're using this on a more permanent basis, as you probably will see some degradation of graphics quality, and you would loose the ability to have individual X11 apps appear as individual windows in macOS.

Considering Wayland on macOS. You would actually not need a client as you write, it is the server you're after. However, as you're trying to access a Linux VM over a network (even though it is locally on the same machine) - this is not natively supported by Wayland. The standard Wayland setup requires the clients (individual applications) and the server (what actually handles the composition of all the applications onto the displays) to be on the same computer. There are some, very new, attempts at accessing Wayland remotely over the network - but currently they actually just work like a VNC server would. I.e. there's no advantage to using Wayland here.