Learning Sensor Multiplexing Design through Back-propagation
Recent progress on many imaging and vision tasks has been driven by the use of deep feed-forward neural networks, which are trained by propagating gradients of a loss defined on the final output, back through the network up to the first layer that operates directly on the image. We propose back-propagating one step further-to learn camera sensor designs jointly with networks that carry out inference on the images they capture. In this paper, we specifically consider the design and inference problems in a typical color camera-where the sensor is able to measure only one color channel at each pixel location, and computational inference is required to reconstruct a full color image. We learn the camera sensor's color multiplexing pattern by encoding it as layer whose learnable weights determine which color channel, from among a fixed set, will be measured at each location. These weights are jointly trained with those of a reconstruction network that operates on the corresponding sensor measurements to produce a full color image. Our network achieves significant improvements in accuracy over the traditional Bayer pattern used in most color cameras. It automatically learns to employ a sparse color measurement approach similar to that of a recent design, and moreover, improves upon that design by learning an optimal layout for these measurements.
Learning Sensor Multiplexing Design through Back-propagation  This paper proposes to learn jointly i) the color filter pattern of RGB camera sensors and ii) the color reconstruction of a final output image, using a neural network framework. Contributions have potential high impact in camera designing, since the concept of automatically learning the color filter pattern moves the industry beyond the arbitrary Bayer filter, and the jointly learned color reconstruction offers better image quality in terms of psnr. Experiments are following previous literature and evaluate noise distributions in various camera sensor designs.   
Paper is nicely written, methodology is easy to follow, and validation shows the benefit of the proposed approach. Comments below:  One main question that could arise in learning automatically a color filter pattern, is the dependence on the dataset. Here benchmarking was performed with the Gehler-Shi dataset. Would the use of a different heterogeneous dataset produce a completely different sensor pattern?  If the pattern changes significantly between training sets, would it be relevant to question the robustness of a chosen pattern?   On the same note, how could this impact camera designs, ie., could a universal pattern exist, or would there be multiple patterns, each designed for particular situations (could patterns designed for outdoor, indoor, urban or nature settings, produce a much better image quality than using a universal pattern?)  In the experiments, the pattern was chosen to be learned with one particular noise level. Would learning in different noise levels produce drastically different patterns? If so, why one would choose one particular pattern to another? Would there be a trend in generated sensors across different noise levels that could be learned?   A temperature factor is used when learning the initial sensor layer. What would happen if this factor \alpha_t is sensibly slower/faster than the chosen warming rate?  Does the rate of increase of this factor condition the number of effective possible iterations during training? How is \gamma chosen?  In the reconstruction network, what is the motivation for learning the color combinations and their gating values in two separate paths?  What is the meaning of K, why is K chosen to be 24 in the experiments?  Is the log-domain used to ensure positiveness of the color channels?  In the second path, for learning the gating values \lambda, why the number of layers is chosen to be 128?   Perhaps a naive question: why choosing nips over a vision/photography conference?