M2 Ultra can run 128 streams of Llama 2 7B in parallel
For applications that aren't latency sensitive my company has found that Apple hardware inference is far less expensive than the competition when calculating for electricity usage. I wish they would make a cloud offering.