A new paper about Auto-differentiating linear algebra 11 was published just a few days ago. As i skimmed it, there seems to be an implementation of symmetric eigen, which could be useful for your approach.
I think it won’t take long until someone starts to implement it for PyTorch.