We desperately need GPU for model inference. CPU can't replace GPU.
text
We desperately need GPU for model inference. CPU can't replace GPU.
new_line
new_line
I will start with the basics. GPU is designed to serve predictable workloads with many parallel units (pixels, tensors, tokens). So a GPU allocates as much transistor budget as possible to build thousands of compute units (Cuda cores in NVidia or execution units in Apple Silicon), each capable of running a thread.
text
I will start with the basics. GPU is designed to serve predictable workloads with many parallel units (pixels, tensors, tokens). So a GPU allocates as much transistor budget as possible to build thousands of compute units (Cuda cores in NVidia or execution units in Apple Silicon), each capable of running a thread.
new_line
new_line
But CPU is designed to handle all kinds of workloads. CPU cores are much larger (hence a lot fewer) with branch prediction and other complex things. In addition, more and more transistors are allocated to build larger cache (~50% now) to house the unpredictable, devouring the compute budget.
text
But CPU is designed to handle all kinds of workloads. CPU cores are much larger (hence a lot fewer) with branch prediction and other complex things. In addition, more and more transistors are allocated to build larger cache (~50% now) to house the unpredictable, devouring the compute budget.
new_line
new_line
Generalists can't beat specialists.
text
Generalists can't beat specialists.
We desperately need GPU for model inference. CPU can't replace GPU.

I will start with the basics. GPU is designed to serve predictable workloads with many parallel units (pixels, tensors, tokens). So a GPU allocates as much transistor budget as possible to build thousands of compute units (Cuda cores in NVidia or execution units in Apple Silicon), each capable of running a thread.

But CPU is designed to handle all kinds of workloads. CPU cores are much larger (hence a lot fewer) with branch prediction and other complex things. In addition, more and more transistors are allocated to build larger cache (~50% now) to house the unpredictable, devouring the compute budget.

Generalists can't beat specialists.
🚀
YaTharThShaRma999 John6666 alkautsardika darkzbaron GWPECK MuzammilMax
😎
d8rt8v
🧠
xcyclon
👍
MuhammetTaskin
64f97e59d493d8b0d2d6e48c
https://cdn-avatars.huggingface.co/v1/production/uploads/64f97e59d493d8b0d2d6e48c/uXfU2T19uYGljs4e-vkXj.png
Neurlang Project
neurlang
user
663a18c876e6d5b98f3cc94d
/avatars/faa2a47dbdd3bce6d47e0fbed6170627.svg
Alan Tseng
agentlans
user
6787f6033d342e4cc8d8c6cc
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/BapzHmY4xylemybaIjAEn.jpeg
Joseph Robert Turcotte
Fishtiks
user
669dbd709a4bf63e08f1ddc2
https://cdn-avatars.huggingface.co/v1/production/uploads/669dbd709a4bf63e08f1ddc2/aV10ZJPPzH5LbnHFZNqc7.png
Yi Cui
onekq
user