Which library are you using?
The lib should have some information, how the feature importance is calculated.
You are posting in the PyTorch forum, which might not be the best place to ask about other toolkits.