I can't use Mail.app in El Capitan to encrypt email with a self-signed certificate I generated a self-signed root S/MIME certificate using Certificate Assistant. I made sure the resulting certificate was in my Keychain, and trusted it explicitly. In attempts to send encrypted mail to that account the encryption icon is never activated and the certificate does not show up on the TLS certificate list for the account (Mail Preferences). I've also tried creating a Certificate Authority and using that to generate a leaf S/MIME certificate, with the same results. I found detailed instructions for setting up server and client certificates at

developer.apple.com/library/ios/technotes/tn2326/_index.html

but, following those instructions, trying to adapt for S/MIME, produced the same result.

Signing mail from that account works fine.

Is there any way to do this?
First, to get the certificate to show up in the TLS section of the account, you need to import the cert into your keychain in P12 format. This is important because if you only import the cert (.crt), then that is only the public-facing portion of the identity (i.e. the public key). When you import the cert as a .p12, you are importing both the .crt and the private key associated with it. Now, keychain recognizes this as YOUR identity and will let you select it in mail as the source of encryption and signing. Also be sure that the certificate you are using has the correct authority to encrypt and sign email. You can verify this by checking for the presence of "email protection" under extended key usage and "Digital signature" and "key encipherment" under Key usage. Without these, you will likely have similar problems.

Similarly, be sure the .crt of the person you are encrypting is in your keychain, or you won't be able to encrypt TO them.