I convert the entire model to FP16 with code
network->to(torch::kCUDA).to(torch::kHalf),
and also the input data (kfloat to kHalf)
input = input.to(torch::kCUDA).to(torch::kHalf).
Then I package data preprocessing, forward() and following analysis processing into a predict function, and use std::chrono to measure the elapsed time of this predict function.