How to test my installation script on macOS? What is the best way to test an install script on macOS? 

I have written an installation script so that others can easily install the program qme-ng which I forked and am working on.

However testing the script is very hard. I can manually uninstall all the dependencies, Boost and gmp, each time and see whether my script can properly reinstall them but it is very costly. Furthermore I'm worried about the possibility of dependency removal that isn't complete which can complicate the issue.
Your best bet is to create a macOS VM for testing. Install VirtualBox (it's free), create a macOS virtual machine, and then create a snapshot before testing your script. Once you've finished testing your script you can restore the VM to its previous, unmodified state using the snapshot feature. Hope this helps!