WebLLM: Llama2 in the Browser
I ran Llama 2 70B in my browser (Chrome Canary on a 64GB MacBook M2) using this. It took a long time to start running, but then:<p>prefill: 0.9654 tokens/sec, decoding: 3.2589 tokens/sec<p>Honestly amazed that this is even possible. I haven't even run Llama 2 70B on my laptop NOT using a web browser yet.