A suspicious 'installer' was executed on my macbook. What would be the safest course of action? My brother just tried to download and install a cracked application from some fishy website. In my own Macbook.

Needless to say, the suspiciously small installer didn't install the application (even though it said it was successful), and it apparently "didn't do anything at all".

I'm not sure what are the proper steps to take at this point. What would be the most reasonable course of action for me? Would reinstalling the OS be overkill?

I'm running macOS Sierra 10.12.5. I haven't shutdown/restarted the computer in fear that some irreversible changes would execute when rebooting.
Run something like Malware Bytes (formerly Adware Medic), try and find out what the actual filename of the item installed was and search the internet.