Okay, what i did now is before putting my Images Batch into the Network i change
inputs = inputs.view(-1, N_N_PIXELS,N_PIXELS)
to
inputs = inputs.view(-1, N_CHANNELS*N_PIXELS,N_PIXELS)
which just appends all channels. Thanks again for your help @chetan_patil.