My server can run on either Python 3.6 or Python 3.8. The problem seems to be the client.
If I run the client on Python 3.6, I get TLSv1.2 working with no error. But if I run the same client code on Python 3.8, I get NO_SHARED_CIPHER on the server and SSLV3_ALERT_HANDSHAKE_FAILURE on the client.
I suspect it has something to do with server or client certificate.
UPDATE: I created some new server and client certificates to experiment around with: ed25519, rsa, and secp256k1. From what I've seen, the problem above only shows up when the server certificate is secp256k1 and both the client and server are running Python 3.8 (and therefore with TLSv1.3). The client can use any type of certificate.
... NO_SHARED_CIPHER ...
  I suspect it has something to do with server or client certificate.

This is a problem of cipher selection and cipher selection has almost nothing to do with the certificate, at least if the the certificate type is still the same, i.e. ECC vs. RSA. And I assume that you are still using the same certificate or at least the same certificate type in the working and non-working cases.
It is more likely that your client tries to use some RC4 or 3DES based cipher which is today considered obsolete and thus is no longer supported  by a server when using a newer version of the software stack (i.e. Python+OpenSSL).
Python SSL/TLS - no shared cipher on Python 3.8 but working on Python 3.6