YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
dinov2_vits14
ONNX Model
Check this PR.
Run
Run triton container.
make triton
docker logs dinov2_vits14_triton
=============================
== Triton Inference Server ==
=============================
NVIDIA Release 23.04 (build 58408265)
Triton Server Version 2.33.0
Copyright (c) 2018-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
WARNING: CUDA Minor Version Compatibility mode ENABLED.
Using driver version 525.105.17 which has support for CUDA 12.0. This container
was built with CUDA 12.1 and will be run in Minor Version Compatibility mode.
CUDA Forward Compatibility is preferred over Minor Version Compatibility for use
with this container but was unavailable:
[[Forward compatibility was attempted on non supported HW (CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE) cuInit()=804]]
See https://docs.nvidia.com/deploy/cuda-compatibility/ for details.
I0625 08:05:36.712010 1 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f6c46000000' with size 268435456
I0625 08:05:36.712625 1 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I0625 08:05:36.717785 1 model_lifecycle.cc:459] loading: dinov2_vits14:1
I0625 08:05:36.723707 1 onnxruntime.cc:2504] TRITONBACKEND_Initialize: onnxruntime
I0625 08:05:36.723725 1 onnxruntime.cc:2514] Triton TRITONBACKEND API version: 1.12
I0625 08:05:36.723731 1 onnxruntime.cc:2520] 'onnxruntime' TRITONBACKEND API version: 1.12
I0625 08:05:36.723735 1 onnxruntime.cc:2550] backend configuration:
{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","default-max-batch-size":"4"}}
I0625 08:05:36.770311 1 onnxruntime.cc:2608] TRITONBACKEND_ModelInitialize: dinov2_vits14 (version 1)
I0625 08:05:36.770781 1 onnxruntime.cc:666] skipping model configuration auto-complete for 'dinov2_vits14': inputs and outputs already specified
I0625 08:05:36.771205 1 onnxruntime.cc:2651] TRITONBACKEND_ModelInstanceInitialize: dinov2_vits14_0 (GPU device 0)
2023-06-25 08:05:37.157976034 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 465, index: 122, mask: {125, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.158142138 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 466, index: 123, mask: {62, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.158159030 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 467, index: 124, mask: {126, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.158174259 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 468, index: 125, mask: {63, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.165944431 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 344, index: 1, mask: {1, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.158230084 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 469, index: 126, mask: {127, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.169979079 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 347, index: 4, mask: {66, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.169927531 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 345, index: 2, mask: {65, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.169954703 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 346, index: 3, mask: {2, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.173982388 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 350, index: 7, mask: {4, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.173929448 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 348, index: 5, mask: {3, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.173954065 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 349, index: 6, mask: {67, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.181926759 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 351, index: 8, mask: {68, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.185932583 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 352, index: 9, mask: {5, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.189924821 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 353, index: 10, mask: {69, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.193940975 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 464, index: 121, mask: {61, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.194020786 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 357, index: 14, mask: {71, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.193940915 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 354, index: 11, mask: {6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.193968147 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 355, index: 12, mask: {70, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.193992072 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 356, index: 13, mask: {7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.197974211 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 360, index: 17, mask: {9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.197928554 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 358, index: 15, mask: {8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.197950686 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 359, index: 16, mask: {72, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.201924259 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 361, index: 18, mask: {73, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.205931957 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 362, index: 19, mask: {10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.209926179 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 363, index: 20, mask: {74, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.213927705 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 364, index: 21, mask: {11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.217799496 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 365, index: 22, mask: {75, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.217849460 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 366, index: 23, mask: {12, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.221966294 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 367, index: 24, mask: {76, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.221966304 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 463, index: 120, mask: {124, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.225931100 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 462, index: 119, mask: {60, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.225933645 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 368, index: 25, mask: {13, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.229929350 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 369, index: 26, mask: {77, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.233930445 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 370, index: 27, mask: {14, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.233930525 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 461, index: 118, mask: {123, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.237930518 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 371, index: 28, mask: {78, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.241927085 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 372, index: 29, mask: {15, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.245926977 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 373, index: 30, mask: {79, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.249931199 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 374, index: 31, mask: {16, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.253927515 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 375, index: 32, mask: {80, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.257925694 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 376, index: 33, mask: {17, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.261929715 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 377, index: 34, mask: {81, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.265966397 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 378, index: 35, mask: {18, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.269926725 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 379, index: 36, mask: {82, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.273931337 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 380, index: 37, mask: {19, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.281941021 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 381, index: 38, mask: {83, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.282017776 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 398, index: 55, mask: {28, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.282038465 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 382, index: 39, mask: {20, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.282090914 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 383, index: 40, mask: {84, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.286235010 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 385, index: 42, mask: {85, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.285955121 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 401, index: 58, mask: {93, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.282070957 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 399, index: 56, mask: {92, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.286082321 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 384, index: 41, mask: {21, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.285929422 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 400, index: 57, mask: {29, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.293926803 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 405, index: 62, mask: {95, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.289931018 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 402, index: 59, mask: {30, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.289956767 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 403, index: 60, mask: {94, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.301929004 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 388, index: 45, mask: {23, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.289975973 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 404, index: 61, mask: {31, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.294054945 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 406, index: 63, mask: {32, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.294078880 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 407, index: 64, mask: {96, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.314023441 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 409, index: 66, mask: {97, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.289931068 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 386, index: 43, mask: {22, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.318030297 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 411, index: 68, mask: {98, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.289956797 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 387, index: 44, mask: {86, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.301929014 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 408, index: 65, mask: {33, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.314096058 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 410, index: 67, mask: {34, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.334030890 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 414, index: 71, mask: {36, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.305931271 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 389, index: 46, mask: {87, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.321929038 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 390, index: 47, mask: {24, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.321948134 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 391, index: 48, mask: {88, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.321965006 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 392, index: 49, mask: {25, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.321981437 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 393, index: 50, mask: {89, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.321996396 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 394, index: 51, mask: {26, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322012065 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 395, index: 52, mask: {90, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322026713 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 396, index: 53, mask: {27, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322049907 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 397, index: 54, mask: {91, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322065276 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 460, index: 117, mask: {59, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322080735 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 425, index: 82, mask: {105, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322096315 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 426, index: 83, mask: {42, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322112155 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 427, index: 84, mask: {106, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322127053 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 428, index: 85, mask: {43, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322143324 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 429, index: 86, mask: {107, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322157170 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 430, index: 87, mask: {44, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322173340 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 431, index: 88, mask: {108, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322188569 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 432, index: 89, mask: {45, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322205311 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 433, index: 90, mask: {109, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322219938 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 434, index: 91, mask: {46, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322235177 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 435, index: 92, mask: {110, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322249955 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 436, index: 93, mask: {47, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322267158 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 437, index: 94, mask: {111, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322281345 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 438, index: 95, mask: {48, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322296904 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 439, index: 96, mask: {112, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322312113 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 440, index: 97, mask: {49, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322329005 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 441, index: 98, mask: {113, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322343652 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 442, index: 99, mask: {50, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322359492 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 443, index: 100, mask: {114, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322377907 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 444, index: 101, mask: {51, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322393366 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 445, index: 102, mask: {115, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322408725 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 446, index: 103, mask: {52, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322423233 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 447, index: 104, mask: {116, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322437289 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 448, index: 105, mask: {53, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322453440 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 449, index: 106, mask: {117, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322467697 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 450, index: 107, mask: {54, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322483076 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 451, index: 108, mask: {118, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322496812 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 452, index: 109, mask: {55, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.445929743 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 417, index: 74, mask: {101, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322511880 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 453, index: 110, mask: {119, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322525526 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 454, index: 111, mask: {56, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322541977 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 455, index: 112, mask: {120, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.454013818 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 422, index: 79, mask: {40, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322555663 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 456, index: 113, mask: {57, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.457932126 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 423, index: 80, mask: {104, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322571683 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 457, index: 114, mask: {121, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.322585920 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 458, index: 115, mask: {58, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.318158029 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 412, index: 69, mask: {35, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.334163851 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 415, index: 72, mask: {100, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.341919085 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 416, index: 73, mask: {37, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.323408365 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 413, index: 70, mask: {99, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.453923387 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 418, index: 75, mask: {38, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.453947493 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 419, index: 76, mask: {102, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.453965727 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 420, index: 77, mask: {39, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.453991656 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 421, index: 78, mask: {103, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.458087059 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 424, index: 81, mask: {41, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:37.585007204 [E:onnxruntime:log, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 459, index: 116, mask: {122, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2023-06-25 08:05:38.570069572 [W:onnxruntime:, session_state.cc:1136 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-06-25 08:05:38.570088387 [W:onnxruntime:, session_state.cc:1138 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
I0625 08:05:39.975559 1 model_lifecycle.cc:694] successfully loaded 'dinov2_vits14' version 1
I0625 08:05:39.975625 1 server.cc:583]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+
I0625 08:05:39.975662 1 server.cc:610]
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","default-max-batch-size":"4"}} |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
I0625 08:05:39.975683 1 server.cc:653]
+---------------+---------+--------+
| Model | Version | Status |
+---------------+---------+--------+
| dinov2_vits14 | 1 | READY |
+---------------+---------+--------+
I0625 08:05:39.991510 1 metrics.cc:808] Collecting metrics for GPU 0: NVIDIA GeForce RTX 3090
I0625 08:05:39.992145 1 metrics.cc:701] Collecting CPU metrics
I0625 08:05:39.992360 1 tritonserver.cc:2387]
+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.33.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data parameters statistics trace logging |
| model_repository_path[0] | /models |
| model_control_mode | MODE_NONE |
| strict_model_config | 0 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
| cache_enabled | 0 |
+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I0625 08:05:39.993603 1 grpc_server.cc:2450] Started GRPCInferenceService at 0.0.0.0:8001
I0625 08:05:39.993771 1 http_server.cc:3555] Started HTTPService at 0.0.0.0:8000
I0625 08:05:40.034678 1 http_server.cc:185] Started Metrics Service at 0.0.0.0:8002
Perf analyzer dinov2_vits14
make perf
docker run --gpus all --rm -it --net host nvcr.io/nvidia/tritonserver:23.04-py3-sdk perf_analyzer -m dinov2_vits14 --percentile=95 -i grpc -u 0.0.0.0:8001 --concurrency-range 16:16 --shape input:3,280,280
=================================
== Triton Inference Server SDK ==
=================================
NVIDIA Release 23.04 (build 58408269)
Copyright (c) 2018-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
WARNING: CUDA Minor Version Compatibility mode ENABLED.
Using driver version 525.105.17 which has support for CUDA 12.0. This container
was built with CUDA 12.1 and will be run in Minor Version Compatibility mode.
CUDA Forward Compatibility is preferred over Minor Version Compatibility for use
with this container but was unavailable:
[[Forward compatibility was attempted on non supported HW (CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE) cuInit()=804]]
See https://docs.nvidia.com/deploy/cuda-compatibility/ for details.
*** Measurement Settings ***
Batch size: 1
Service Kind: Triton
Using "time_windows" mode for stabilization
Measurement window: 5000 msec
Latency limit: 0 msec
Concurrency limit: 16 concurrent requests
Using synchronous calls for inference
Stabilizing using p95 latency
Request concurrency: 16
Client:
Request count: 9403
Throughput: 522.33 infer/sec
p50 latency: 30482 usec
p90 latency: 32100 usec
p95 latency: 32564 usec
p99 latency: 34203 usec
Avg gRPC time: 30589 usec ((un)marshal request/response 93 usec + response wait 30496 usec)
Server:
Inference count: 9403
Execution count: 1177
Successful request count: 9403
Avg request latency: 24295 usec (overhead 220 usec + queue 9042 usec + compute input 1511 usec + compute infer 13485 usec + compute output 37 usec)
Inferences/Second vs. Client p95 Batch Latency
Concurrency: 16, throughput: 522.33 infer/sec, latency 32564 usec