Probably the organization of the channels in your code is problematic. You can refer to the following example for a proper usage:

  
      

      keras.io
  

  
    

Keras documentation: Image Super-Resolution using an Efficient Sub-Pixel CNN 25