Hardware and Software Platform Inference
It is now a common business practice to buy access to large language model
(LLM) inference rather than self-host, because of significant upfront hardware
infrastructure and energy costs. However, as a buyer, there is no mechanism to
verify the authenticity of the advertised service including the serving
hardware platform, e.g. that it is actually being served using an NVIDIA H100.
Furthermore, there are reports suggesting that model providers may deliver
models that differ slightly from the advertised ones, often to make them run on
less expensive hardware. That way, a client pays premium for a capable model
access on more expensive hardware, yet ends up being served by a (potentially
less capable) cheaper model on cheaper hardware. In this paper we introduce
\textbf{hardware and software platform inference (HSPI)} -- a method
for identifying the underlying  architecture and software stack of a
(black-box) machine learning model solely based on its input-output behavior.
Our method leverages the inherent differences of various  architectures
and compilers to distinguish between different  types and software
stacks. By analyzing the numerical patterns in the model's outputs, we propose
a classification framework capable of accurately identifying the  used
for model inference as well as the underlying software configuration. Our
findings demonstrate the feasibility of inferring  type from black-box
models. We evaluate HSPI against models served on different real hardware and
find that in a white-box setting we can distinguish between different s
with between 83.9% and 100% accuracy. Even in a black-box setting we are
able to achieve results that are up to three times higher than random guess
accuracy.
67349ba576c7f079f723a657
Cheng Zhang
67349ba576c7f079f723a658
Hanna Foerster
admin_assigned
2024-11-18T08:06:09.812Z
666c1544de375f3114d590c0
/avatars/de6c4d3395160603b0bf68c9bca398df.svg
Hanna Foerster
user
aprilflower
67349ba576c7f079f723a659
Robert D. Mullins
admin_assigned
2024-11-18T08:06:17.645Z
66349b8dc53115681c534d07
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Suu9bZJecfCcJccjRsrqH.png
Robert Mullins
user
rcmullins
67349ba576c7f079f723a65a
Yiren Zhao
admin_assigned
2024-11-18T08:06:23.792Z
6654464df0c8c891ca528618
/avatars/26aa1f673da92e3632576954bcee1b28.svg
Yiren Zhao
user
YirenZH
67349ba576c7f079f723a65b
Ilia Shumailov
admin_assigned
2024-11-18T08:07:12.854Z
6475c2794766357252e69e9f
/avatars/757ed789423113369868e972f21ce559.svg
i
user
iliashum
Hardware and Software Platform Inference
It is now a common business practice to buy access to large language model
(LLM) inference rather than self-host, because of significant upfront hardware
infrastructure and energy costs. However, as a buyer, there is no mechanism to
verify the authenticity of the advertised service including the serving
hardware platform, e.g. that it is actually being served using an NVIDIA H100.
Furthermore, there are reports suggesting that model providers may deliver
models that differ slightly from the advertised ones, often to make them run on
less expensive hardware. That way, a client pays premium for a capable model
access on more expensive hardware, yet ends up being served by a (potentially
less capable) cheaper model on cheaper hardware. In this paper we introduce
\textbf{hardware and software platform inference (HSPI)} -- a method
for identifying the underlying  architecture and software stack of a
(black-box) machine learning model solely based on its input-output behavior.
Our method leverages the inherent differences of various  architectures
and compilers to distinguish between different  types and software
stacks. By analyzing the numerical patterns in the model's outputs, we propose
a classification framework capable of accurately identifying the  used
for model inference as well as the underlying software configuration. Our
findings demonstrate the feasibility of inferring  type from black-box
models. We evaluate HSPI against models served on different real hardware and
find that in a white-box setting we can distinguish between different s
with between 83.9% and 100% accuracy. Even in a black-box setting we are
able to achieve results that are up to three times higher than random guess
accuracy.
A method, HSPI, is introduced to identify the GPU architecture and software stack of a machine learning model based on its input-output behavior, providing a means to verify the authenticity of LLM inference services.
GPU LLM input-output behavior numerical patterns classification framework white-box setting black-box setting