Q:

Why is there no "ephemeral non-DH EC" key exchange in TLS?

In RSA key exchange, the client picks a premaster secret and encrypts it with the RSA public key in the server certificate, and sends it to the server. This offers no forward secrecy.
In ECDHE key exchange, the client and server both generate ephemeral EC key pairs, and use DH key exchange to negotiate the premaster secret. The server public ephemeral public key is signed using the server authentication certificate. This gives forward secrecy.
In the latter, two ephemeral key pairs have to be generated, by the server and the client. How about the following mix of both key exchange methods that does not need the client to generate an EC key pair:

Server generates ephemeral EC key pair, signs the public key using the server authentication certificate.
Server sends the ephemeral EC key pair public key to the client.
Client picks a premaster secret, encrypts it using the server ephemeral EC public key and sends it back.

What are the inevitable downsides that make ECDHE superior? I don't see a glaring security hole. Maybe difficulties to use an EC keypair for encryption?

A:

"Maybe difficulties to use an EC keypair for encryption?"; yes exactly, as there are no direct EC encryption primitives that can be used. ElGamal EC encryption is possible, but it requires all kinds of tricks to get it working. There is the mapping of the message to an EC point, for instance. There is ECIES, but that really comes down to performing ECDHE in the first place, with the additional disadvantage of needing a cipher.
Besides that, for RSA, key pair generation is one of the slowest algorithms that you may see. Not only is it slow, but it is also very unpredictable as it is unknown when the two required primes are found. With EC, it is very easy:

create a random number between 1 and N (exclusive);
perform point multiplication of that number with the base point G to get to the public point.

So basically it is as fast as the DH operation itself. There is no long waiting time for primes to be generated at all. So because the key generation itself does not pose much of a problem, there is no pressing need to change either.

One important idea of TLS 1.3 was bringing down the number of round trips. It even guesses the algorithms present at the other end for the initial key agreement. It can then immediately send the ephemeral public key of the client to the server, which responds with the server's ephemeral public key to setup the channel. That way the encrypted channel can (and will) be setup after only two messages. With encryption the client (who has to initiate the connection) first asks for the public key, receives it and then encrypts with it.
That is, ECDH speeds things up if the client doesn't use session resumption, a separate trick to speed things up.

Another smaller difference is that the random key in ECDH is now dependent on the random data of two parties. The various clients may have different strengths of random number generator, and encryption of a secret has the disadvantage that the secret itself is not fully random, e.g. when a RNG is used that is not cryptographically secure. However, with ECDH is does mean that a completely failing RNG will completely show the established shared secret as the ephemeral private key may become known.

Note too that simply bringing the number of options down was an important initiative in TLS 1.3. Having options just for the sake of it only makes a protocol less secure, not more secure. IMHO it doesn't even make sense to have different security levels for the various algorithms used for a specific ciphersuite (not including the difference with the security level of the certificates used for authentication).

