The default memory order of Eigen is ColMajor which each column is stored sequentially. But libtorch use row major so that data stores row by row. You need to use
Matrix<double, -1, -1, RowMajor>
to define a row major matrix.