kile0:

Hi,
I’m fairly new to pytorch so this will probably seem like a silly question, but here we go: I’m curious about the expected throughput of inference on CPUs while using various modes of pytorch. Using one of the pretrained models I benchmarked it on an 8-core ryzen machine with the below script but I’m seeing times that seem rather slow (around ~2.4 seconds for a batch size of 16, plus or minus a bit depending on eager/trace/script mode). Could someone tell me what I’m doing wrong? Much appreciated


I think the times are ok for a ResNet50 running in cpu.
edit: I run your code in a amd fx 8150, and this is the result
{'eager_mode': [3.1236065919983957,
                2.8740777429993614,
                3.215031881998584,
                3.2408665209986793,
                3.0817273770007887,
                3.182126468000206,
                3.217468533999636,
                3.0495879129994137,
                3.2823124830028974,
                2.98734099399735,
                3.04414951400031,
                3.1191899989971716,
                3.2561472420020436,
                2.989001477999409,
                2.8834749839988945,
                3.2728907790005906,
                2.7947310479976295,
                2.7229248879993975,
                3.1714364620020206,
                3.187919829000748,
                2.7909267199975147,
                2.7637479750010243,
                3.2621708949991444,
                3.2539125790026446,
                2.758795009998721,
                3.099266881999938,
                3.2731154520006385,
                3.321376304000296,
                3.2308814560019528,
                3.0906896560009045],
 'script_mode': [5.663022714001272,
                 3.1545001340018644,
                 3.20035366799857,
                 3.2082225320009456,
                 3.0376815849995182,
                 3.23846422600036,
                 3.1248215429986885,
                 3.2207794620007917,
                 3.1486177900005714,
                 2.992629713000497,
                 3.273916879999888,
                 3.2852682630000345,
                 2.965044250999199,
                 3.24503594899943,
                 2.7061276380009076,
                 3.1414647249985137,
                 3.2591017490012746,
                 3.071839774002001,
                 3.269749651000893,
                 3.2680227549972187,
                 2.9813807400023506,
                 3.1570755319989985,
                 3.2570094219991006,
                 3.2274487190006766,
                 3.2254650680006307,
                 3.1624261510005454,
                 3.307589273001213,
                 3.277479943997605,
                 3.0594900569994934,
                 3.168268078003166],
 'trace_mode': [3.331946746002359,
                3.454950250998081,
                3.384993150997616,
                3.2832865930031403,
                3.296919559001253,
                3.3446977690000494,
                3.334649086002173,
                3.267488012003014,
                3.3727113850000023,
                3.276030206998257,
                3.2170279210004082,
                3.447721204996924,
                3.373897843001032,
                3.3251263899983314,
                3.363704613002483,
                3.4962456700013718,
                3.459224797999923,
                3.4048159969970584,
                3.380363905002014,
                3.399266141001135,
                3.4834778490003373,
                3.366177444000641,
                3.4348102459989605,
                3.3493297409986553,
                3.402436749998742,
                3.3380109899990202,
                3.4338676759980444,
                3.3019120470016787,
                3.317763601000479,
                3.3335056399992027]}