Is there an equivalent of Linux's kernel memmap on OS X? According to memtest86, there is a persistent single bit error in my MacBook Pro's 16GB of DDR RAM. 

Since the error is in the second half of the physical address space, I can work around the problem by running sudo nvram boot-args="maxmem=8192". However, this is a bit of a waste of the other half - a working 8GB minus one bit.

Is there a way, perhaps with an Open Firmware command or configuration, or a special boot-args option (e.g. nvphash?), that will allow me to mark as unusable just the small section that contains the bit error?
You can achieve what you want with the system call mmap(2) passing as argument for the protection value PROT_NONE which means pages can't be accessed.

But I would suggest to buy a new memory chip, because the error you detected is the signature of bad quality and your erroneous bit won't stay alone for too long (within the same chip).