  The Secure Sockets Layer (SSL) and Transport Layer Security (TLS) protocols
are the foundation of network security. The certificate verification in SSL/TLS
implementations is vital and may become the weak link in the whole network
ecosystem. In previous works, some research focused on the automated testing of
certificate verification, and the main approaches rely on generating massive
certificates through randomly combining parts of seed certificates for fuzzing.
Although the generated certificates could meet the semantic constraints, the
cost is quite heavy, and the performance is limited due to the randomness. To
fill this gap, in this paper, we propose DRLGENCERT, the first framework of
applying deep reinforcement learning to the automated testing of certificate
verification in SSL/TLS implementations. DRLGENCERT accepts ordinary
certificates as input and outputs newly generated certificates which could
trigger discrepancies with high efficiency. Benefited by the deep reinforcement
learning, when generating certificates, our framework could choose the best
next action according to the result of a previous modification, instead of
simple random combinations. At the same time, we developed a set of new
techniques to support the overall design, like new feature extraction method
for X.509 certificates, fine-grained differential testing, and so forth. Also,
we implemented a prototype of DRLGENCERT and carried out a series of real-world
experiments. The results show DRLGENCERT is quite efficient, and we obtained
84,661 discrepancy-triggering certificates from 181,900 certificate seeds, say
around 46.5% effectiveness. Also, we evaluated six popular SSL/TLS
implementations, including GnuTLS, MatrixSSL, MbedTLS, NSS, OpenSSL, and
wolfSSL. DRLGENCERT successfully discovered 23 serious certificate verification
flaws, and most of them were previously unknown.

Cryptography and Security