I am interested to build a mail service that allows you to incorporate custom logic in the your mail server.
For example, user A can reply to helloworld@mysite.com once and subsequent emails from user A to helloworld@mysite.com will not go through until certain actions are taken. 
I am looking for something simple and customizable, preferably open-sourced. I am fluent in most modern languages. 
What email servers do you guys recommend for this?
Almost every mail server has some form of extensibility where you can insert logic in the mail-flow process, it's how some spam filters were implemented before they were built directly in to the servers. Personally, I use Exchange server which has a variety of points and APIs to extend it, such as SMTP Sinks.
However, this question is off-topic and shouldn't be on StackOverflow.
I suggest you build your own server - implementing a server-side version of SMTP and IMAP can be done by a single person, or use an existing library, it shouldn't take you more than a year if you put in a couple of hours each day.
Customizable mail server - what are my options?