Hi @mfuntowicz
Thanks again for the great article.

Indeed, everything in the blog was run in AWS (c5.metal) instance(s).
The way I’m currently using it:

git clone https://github.com/huggingface/tune
cd tune
pip install -r requirements.txt
export PYTHONPATH=src
python src/main.py --multirun backend=pytorch batch=1 sequence_length=128,256,512



So you ran these command lines directly from the c5.metal instance? You didn’t need to install before on it a docker image with OS and pip/python packages?
With this overall framework you can simulate different hardware parameters on the latency/throughput outputs. But, generally transformers are encaspulated within a docker image providing an API before deployment on cloud services. How could this benchmark could simulate real latency/throughput of the docker image deployed?

Matthieu