Yes but I meant how the validation error is distributed over classes.
Before debugging your custom model have you tried to reproduce approximate results with any well known model on this dataset?

  

      paperswithcode.com
  

  
    

Papers with Code - NTU RGB+D Benchmark (Skeleton Based Action Recognition) 25

  The current state-of-the-art on NTU RGB+D is PoseC3D (w. HRNet 2D skeleton). See a full comparison of 85 papers with code.