How do I enable DNSSec on OSX? (Or DNSCurve) I want to support encrypted IPSec communications, and DANE certs in DNS.  Notably, DNSSSec is a prerequisite is IPSec (secure IP addresses from DNS)

How can I configure my OS X computer to use DNSSec when available, and to refuse incorrect responses.

I understand that DNSSec is best done on a server, and have your workstation point to THAT, however, this also applies to OS X Server.
You should install unbound and have that be your dns resolver on Mac OS X.  Unbound supports DNSSEC.   Next, you need whatever apps you want to use, to be aware of and respect the results of the DNSSEC validation that unbound does.  As an example, SSH can use DNSSEC along with SSHFP to validate the fingerprint of the server that you are connecting to.  

Here is a blog post that details how to install unbound. https://spatof.org/blog/2013/8/unbound-on-osx/

As to having IPSec using DANE, I believe that is still something that it being worked out.
https://tools.ietf.org/html/draft-osterweil-dane-ipsec-02