The problem here was, I wasn’t importing the pytorch library before the extension. ie;
import torch
import test_ext
import unittest

class TestLib(unittest.TestCase):
    def test_matrix_mal(self):
        mm = test_ext.MatrixMultiplier(3, 4)
        mm.get()

import torch has to be done before importing the custom binding.
I was expecting to bind it without the dependency from pytorch, and use only the dynamic libraries that are linked with the my extension and standard python dependencies. Seems I was wrong here, pytorch has to be loaded into python interpreter.