Solved by ptrblck in post #2 
              
              
                Could you post the code of dct.dct_3d or check, if you might be dividing by zero or any other operation which could cause invalid outputs?