summaries: The paper proposed to adopt differentiable network architecture search (DARTS) for the co-design of the sensor (a lensless camera) and the deep model for visual recognition tasks, so as to maximize the accuracy and minimize the energy consumption. The key idea is to include the sensor configuration, in this case the phase mask of a lensless camera modeled as 2D convolutions, as additional parameters in architecture search.  The proposed method was evaluated on simulated data for a number of vision tasks (image classification, face recognition and head pose estimation), as well as using fabricated masks on a real world camera. The results demonstrated significantly increase recognition performance given the same energy level.

summaries: This paper presents a method called SACoD to develop a more efficient CNN-powered Phlatcam. The proposed method optimizes both the PhlatCam sensor and the backend CNN model simultaneously.  That is, the coded mask in Phlatcam and neural network weights are regarded as learnable parameters. The coded mask (the optical layer) can be considered as a special convolution layer. As a result, it achieves energy saving, model compressing as well as good accuracy. Extensive experiments and ablation studies are presented to show the effectiveness of the method.

summaries: SACoD presents a novel attempt to integrate the computational capabilities of a lensless imaging system, PhlatCam, with the search for the optimal convolutional neural network design for a given task. SACoD provides a framework which enables joint optimization of sensor and CNN resulting in IoT devices that achieve higher task accuracy’s with limited resource budgets of a typical IoT system. The authors present a new an optical layer design that enables above described features. Detailed experiments comparing SACoD sensor + CNN with other baseline models covering past papers, demonstrate the superiority of SACoD’s accuracy/efficiency curve over that of separately optimizing CNN arch or sensor/CNN joint-optimizations that do not vary network architecture. Additionally, ablation studies and results from measurements from actual phase masks fabricated help breakdown the accuracy/efficiency benefits of SACoD while analyzing the noise limitations of mask fabrication process.

abstractText: There has been a booming demand for integrating Convolutional Neural Networks (CNNs) powered functionalities into Internet-of-Thing (IoT) devices to enable ubiquitous intelligent “IoT cameras”. However, more extensive applications of such IoT systems are still limited by two challenges. First, some applications, especially medicineand wearable-related ones, impose stringent requirements on the camera form factor. Second, powerful CNNs often require considerable storage and energy cost, whereas IoT devices often suffer from limited resources. PhlatCam, with its form factor potentially reduced by orders of magnitude, has emerged as a promising solution to the first aforementioned challenge, while the second one remains a bottleneck. Existing compression techniques, which can potentially tackle the second challenge, are far from realizing the full potential in storage and energy reduction, because they mostly focus on the CNN algorithm itself. To this end, this work proposes SACoD, a Sensor Algorithm Co-Design framework to develop more efficient CNN-powered PhlatCam. In particular, the mask coded in the PhlatCam sensor and the backend CNN model are jointly optimized in terms of both model parameters and architectures via differential neural architecture search. Extensive experiments including both simulation and physical measurement on manufactured masks show that the proposed SACoD framework achieves aggressive model compression and energy savings while maintaining or even boosting the task accuracy, when benchmarking over two state-of-the-art (SOTA) designs with six datasets on four different tasks. We also evaluate the performance of SACoD on the actual PhlatCam imaging system with visualizations and experiment results. All the codes will be released publicly upon acceptance.

authors:
Jesse K Adams
Vivek Boominathan
Benjamin W Avants
Daniel G Vercosa
Fan Ye
Richard G Baraniuk
Jacob T Robinson
Ashok Veeraraghavan

title: Single-frame 3d fluorescence microscopy with ultraminiature lensless flatscope

venue: Science advances,

authors:
Nick Antipa
Grace Kuo
Reinhard Heckel
Ben Mildenhall
Emrah Bostan
Ren Ng
Laura Waller

title: Diffusercam: lensless single-exposure 3d imaging

venue: Optica,

authors:
Salman Asif
Ali Ayremlou
Aswin Sankaranarayanan
Ashok Veeraraghavan
Richard Baraniuk

title: Flatcam: Thin, bare-sensor cameras using coded aperture and computation

authors:
Ron Banner
Itay Hubara
Elad Hoffer
Daniel Soudry

title: Scalable methods for 8-bit training of neural networks

venue: In Advances in neural information processing systems,

authors:
V. Boominathan
J. Adams
J. Robinson
A. Veeraraghavan

title: Phlatcam: Designed phase-mask based thin lensless camera

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
Andrew Brock
Theodore Lim
James M Ritchie
Nick Weston

title: Smash: one-shot model architecture search through hypernetworks

venue: arXiv preprint arXiv:1708.05344,

authors:
Han Cai
Tianyao Chen
Weinan Zhang
Yong Yu
Jun Wang

title: Efficient architecture search by network transformation

venue: In Thirty-Second AAAI conference on artificial intelligence,

authors:
Han Cai
Ligeng Zhu
Song Han

title: Proxylessnas: Direct neural architecture search on target task and hardware

venue: arXiv preprint arXiv:1812.00332,

authors:
Ezio Caroli
J. Stephen
G. Cocco
L. Natalucci
A. Spizzichino

title: Coded aperture imaging in x- and gamma-ray astronomy

venue: Space Science Reviews, 45:349–403,

title: Learning sensor multiplexing design through back-propagation

venue: In Advances in Neural Information Processing Systems,

authors:
Julie Chang
Gordon Wetzstein

title: Deep optics for monocular depth estimation and 3d object detection

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Huaijin G Chen
Suren Jayasuriya
Jiyue Yang
Judy Stephen
Sriram Sivaramakrishnan
Ashok Veeraraghavan
Alyosha Molnar

title: Asp vision: Optically computing the first layer of convolutional neural networks using angle sensitive pixels

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Liang-Chieh Chen
George Papandreou
Florian Schroff
Hartwig Adam

title: Rethinking atrous convolution for semantic image segmentation

venue: arXiv preprint arXiv:1706.05587,

authors:
Wuyang Chen
Xinyu Gong
Xianming Liu
Qian Zhang
Yuan Li
Zhangyang Wang

title: Fasterseg: Searching for faster real-time semantic segmentation

authors:
Marius Cordts
Mohamed Omran
Sebastian Ramos
Timo Rehfeld
Markus Enzweiler
Rodrigo Benenson
Uwe Franke
Stefan Roth
Bernt Schiele

title: The cityscapes dataset for semantic urban scene understanding

venue: In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

title: Scatter-Hole Cameras for X-Rays and Gamma Rays

authors:
Xinyu Gong
Shiyu Chang
Yifan Jiang
Zhangyang Wang

title: Autogan: Neural architecture search for generative adversarial networks

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Nicolas Gourier
James Crowley

title: Estimating face orientation from robust detection of salient facial structures

venue: FG Net Workshop on Visual Observation of Deictic Gestures,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: Gans trained by a two time-scale update rule converge to a local nash equilibrium

authors:
Geoffrey Hinton
Oriol Vinyals
Jeff Dean

title: Distilling the knowledge in a neural network

venue: arXiv preprint arXiv:1503.02531,

authors:
Andrew G Howard
Menglong Zhu
Bo Chen
Dmitry Kalenichenko
Weijun Wang
Tobias Weyand
Marco Andreetto
Hartwig Adam

title: Mobilenets: Efficient convolutional neural networks for mobile vision applications

venue: arXiv preprint arXiv:1704.04861,

authors:
Justin Johnson
Alexandre Alahi
Li Fei-Fei

title: Perceptual losses for real-time style transfer and super-resolution

venue: In European conference on computer vision,

authors:
Xing Lin
Yair Rivenson
Nezih T. Yardimci
Muhammed Veli
Yi Luo
Mona Jarrahi
Aydogan Ozcan

title: All-optical machine learning using diffractive deep neural networks

venue: doi: 10.1126/science.aat8084. URL https://science. sciencemag.org/content/361/6406/1004

authors:
Hanxiao Liu
Karen Simonyan
Oriol Vinyals
Chrisantha Fernando
Koray Kavukcuoglu

title: Hierarchical representations for efficient architecture search

venue: arXiv preprint arXiv:1711.00436,

authors:
Hanxiao Liu
Karen Simonyan
Yiming Yang

title: Darts: Differentiable architecture search

venue: arXiv preprint arXiv:1806.09055,

authors:
Christopher A Metzler
Hayato Ikoma
Yifan Peng
Gordon Wetzstein

title: Deep optics for singleshot high-dynamic-range imaging

venue: arXiv preprint arXiv:1908.00620,

authors:
Patrick Mochel
Mike Murphy

title: sysfs - The filesystem for exporting kernel objects

authors:
Hieu Pham
Melody Y Guan
Barret Zoph
Quoc V Le
Jeff Dean

title: Efficient neural architecture search via parameter sharing

venue: arXiv preprint arXiv:1802.03268,

authors:
Mark Sandler
Andrew Howard
Menglong Zhu
Andrey Zhmoginov
Liang-Chieh Chen

title: Mobilenetv2: Inverted residuals and linear bottlenecks, 2018

authors:
Takeshi Shimano
Yusuke Nakamura
Kazuyuki Tajima
Mayu Sao
Taku Hoshizawa

title: Lensless light-field imaging with fresnel zone aperture: quasi-coherent coding

venue: Appl. Opt.,

authors:
Han Shu
Yunhe Wang
Xu Jia
Kai Han
Hanting Chen
Chunjing Xu
Qi Tian
Chang Xu

title: Co-evolutionary compression for unpaired image translation

authors:
Vincent Sitzmann
Steven Diamond
Yifan Peng
Xiong Dun
Stephen Boyd
Wolfgang Heidrich
Felix Heide
Gordon Wetzstein

title: End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging

venue: ACM Transactions on Graphics (TOG),

authors:
Xavier Suau
Nicholas Apostoloff

title: Filter distillation for network compression

venue: IEEE Winter Conference on Applications of Computer Vision (WACV),

authors:
Qilin Sun
Jian Zhang
Xiong Dun
Bernard Ghanem
Yifan Peng
Wolfgang Heidrich

title: Endto-end learned, optically coded super-resolution spad camera

authors:
J. Tan
L. Niu
J.K. Adams
V. Boominathan
J.T. Robinson
R.G. Baraniuk
A. Veeraraghavan

title: Face detection and verification using lensless cameras

venue: IEEE Trans. Comput. Imag.,

authors:
Yue Wang
Ziyu Jiang
Xiaohan Chen
Pengfei Xu
Yang Zhao
Yingyan Lin
Zhangyang Wang

title: E2-train: Training state-of-the-art cnns with over 80% energy savings

authors:
Bichen Wu
Xiaoliang Dai
Peizhao Zhang
Yanghan Wang
Fei Sun
Yiming Wu
Yuandong Tian
Peter Vajda
Yangqing Jia
Kurt Keutzer

title: Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search

authors:
Yicheng Wu
Vivek Boominathan
Huaijin Chen
Aswin Sankaranarayanan
Ashok Veeraraghavan

title: Phasecam3d—learning phase masks for passive single view depth estimation

venue: IEEE International Conference on Computational Photography (ICCP),

authors:
Sirui Xie
Hehui Zheng
Chunxiao Liu
Liang Lin

title: Snas: stochastic neural architecture search

venue: arXiv preprint arXiv:1812.09926,

authors:
Jun-Yan Zhu
Taesung Park
Phillip Isola
Alexei A Efros

title: Unpaired image-to-image translation using cycle-consistent adversarial networks

venue: In Computer Vision (ICCV),

authors:
Barret Zoph
Quoc V. Le

title: Neural architecture search with reinforcement learning, 2017

heading: 1 INTRODUCTION

text:
Recent CNN breakthroughs trigger a growing demand for intelligent IoT devices, such as wearables and biology devices (e.g., swallowed endoscopes). However, two major challenges are hampering more extensive applications of CNN-powered IoT devices. First, some applications, especially medicine- and biology-related ones, impose strict requirements on the form factor, especially the thickness, which are often too stringent for existing lens-based imaging systems. Second, powerful CNNs require considerable hardware costs, whereas IoT devices only have limited resources.
For the first challenge, lensless imaging systems (Asif et al., 2015; Shimano et al., 2018; Adams et al., 2017; Antipa et al., 2018; Boominathan et al., 2020) have emerged as a promising rescue. For example, PhlatCam (Boominathan et al., 2020) replaces the focal lenses with a set of phase masks, which encodes the incoming light instead of directly focusing it. The encoded information can be either computationally decoded to reconstruct the images or processed specifically for different applications. Such lensless imaging systems can be made much smaller and thinner, because the phase masks are smaller than the focal lens, and they can be placed much closer to the sensors and fabricated with much lower costs. For the second challenge, many recent works focus on designing CNNs with improved hardware efficiency, i.e., by applying generic neural architecture search (NAS) to find efficient CNNs.
As such, a naive way to address the two aforementioned challenges simultaneously is to introduce lensless cameras as the signal acquisition frontend and then apply NAS to optimize the backend CNN. However, such approaches would result in disjoint optimization that can be far from optimal. A generic NAS would treat the camera as given, and only optimize the CNN. Likewise, existing
phase mask designs for lensless cameras treat the CNNs as given, and only optimize the masks. Such disjoint optimization fails to (1) take advantage of the masks’ potential computational capacity, with which the NAS optimization can be fundamentally improved, and (2) perform end2end optimization.
It is shown in (Boominathan et al., 2020) that, under some assumptions, the phase masks in PhlatCam essentially perform 2D convolutions on the incoming lights, and the convolution kernel is encoded in the masks. Moreover, unlike other convolutional layers, the phase masks’ convolutions are almost free – they do not consume additional energy, computation power, or storage, regardless of what value each mask takes. Therefore, we aim to incorporate the phase mask design into NAS to enable an end2end optimization of the sensing-processing pipeline, while exempting a portion of the pipeline from the efficiency penalties. Such co-designs are expected to achieve better tradeoffs between accuracy and efficiency.
To this end, we propose a Sensor Algorithm Co-Design (SACoD) framework to enable more energyefficient CNN-powered IoT devices. While SACoD is general and can be applied to different sensing and intelligent processing systems, it is developed and evaluated in the context of PhlatCam (Boominathan et al., 2020) based imaging systems. Our main contributions are:
• We propose SACoD, a novel co-design framework that jointly optimizes the sensor and neural networks to enable more energy-efficient CNN-powered IoT devices. To our best knowledge, SACoD is the first to propose sensor algorithm co-design for CNN inferences.
• We develop an effective design of the optical layer to (1) exploit its potential computation capability and (2) enable co-search of the optical layer and backend algorithm. We then characterize the trade-off between accuracy and the required area of the corresponding imaging systems to demonstrate its effectiveness under practical size constraints.
• Extensive experiments and ablation studies validate that the proposed SACoD consistently achieves reduced hardware costs/area while offering a comparable or even better task accuracy, when evaluated over two SOTA lensless imaging systems on four tasks and six datasets. And part of the experiments are further evaluated with fabricated masks to validate SACoD’s effectiveness in the physical measurement besides simulation.

heading: 2 RELATED WORKS

text:
Neural architecture search. Recently NAS has attracted increasing attention. It eliminates the handcrafting process and automatically designs neural architectures. Existing NAS techniques can be divided into three categories, evolution-based, reinforcement-learning (RL)-based, and one-shot NAS. As the computational overheads of evolution- or RL-based approaches can be unacceptably high, many techniques (Brock et al., 2017; Cai et al., 2018a; Liu et al., 2017; 2018; Pham et al., 2018; Xie et al., 2018) have been proposed to reduce the searching cost, among which differentiable architecture search (DARTS) has gained intensive interests. While being conceptually general, SACoD in this paper adopts the DARTS method, where a super-network is optimized during search and the strongest sub-network is preserved and then retrained.
Lensless imaging systems. To eliminate the size or thickness burden caused by the lens, various lensless imaging systems have been developed. While lensless imaging systems have been widely used for capturing X-ray and gamma-ray (Dicke, 1968; Caroli et al., 1987), it is still in an exploring stage for visible spectrum uses (Asif et al., 2015; Shimano et al., 2018; Antipa et al., 2018; Boom-
inathan et al., 2020). In general, lensless imaging systems capture the scene either directly on the sensor or after being modulated by a mask element.
In this paper, we focus on a specific lensless imaging system based on phase masks called PhlatCam (Boominathan et al., 2020), which is a general-purpose framework to create phase masks that can achieve desired sharp pointspread-functions (PSFs). A phase mask modulates the phase of incident lights, and allows most of the light to pass through, providing a high signal-to-noise ratio. Hence, they are desirable for low light scenarios and photon-
limited imaging. Fig. 2 shows a fabricated phase mask, which is essentially a transparent material with different thicknesses at different locations. Based on this lensless imaging system, we develop our SACoD framework, aiming to enable more energy-efficient CNN-powered IoT devices.
Sensor-algorithm co-training. There have recently been some attempts that try to jointly optimize the sensor parameters and the neural network backend. For lens-based image systems, novel lens designs are introduced and trained concurrently with the neural network backend to jointly optimize for image reconstruction (Sitzmann et al., 2018), depth estimation (Chang & Wetzstein, 2019), and high-dynamic-range imaging (Metzler et al., 2019). Similar approaches have been applied to other imaging systems as well, including cameras with color multiplexing (Chakrabarti, 2016), PhaseCam3D (Wu et al., 2019b), and Single Photon Avalanche Photodiodes cameras (Sun et al., 2020). Yet these methods still consider the neural network architecture as fixed, and do not explore the potential of applying the co-design principle to the neural architecture design.

heading: 3 THE PROPOSED SACOD FRAMEWORK

text:
This section presents the SACoD framework. We will first outline the framework and explain why the optical layer can be considered as a convolutional layer, and then introduce how we implement SACoD’s optimization algorithm. Finally, we describe our developed optical layer design.
Framework setup. The SACoD framework shown in Fig. 1 consists of two modules, an optical sensing frontend and a neural network backend. The coded mask of PhlatCam in the sensor and the backend are jointly optimized using a SOTA differential NAS algorithm (Liu et al., 2018), where the coded masks, together with the neural network weights, are regarded as network parameters.
Specifically, the first module, i.e. the optical sensing frontend, is denoted as O(⋅;m), where m = m(x, y) denotes the phase mask values. The optical layer is based on the PhlatCam system (Boominathan et al., 2020). It receives the light signal from the object in front of the camera, processes the signal using the phase masks, and generates the sensor output. The second module, i.e. the neural network backend, is denoted as F (⋅;w,α), where w represents weights of the neural network, and α parameterizes the architecture. The neural network backend receives the sensor signal and produces an output for the intended applications.
Formally, denote the light signal as I(x, y) ∈ RH×W×3, where x and y are coordinate indices and H and W represent the height and width of the range of light the camera can receive. The light signal contains RGB channel, and hence the last dimension is 3. Denote the signal received at sensor as Z(x, y) ∈ RH
′×W ′×N , where H ′ and W ′ represent the height and width, and N represent the number of channels. Denote Y as the final output of the neural network backend. Then we have
Z = O(I;m), Y = F (Z;w,α). (1)
The following subsections will introduce the form of O(⋅;m) and how to determine m, w, and α. The optical sensing frontend. Assuming that the light signal I(x, y) comes from an object whose distance to the camera is d, and that the depth of the object is relatively small, it can be shown (Boominathan et al., 2020) that O(⋅;m) takes the following convolutional form:
Z(x, y) = O(I;m) = p(x, y;m, d) ∗ I(x, y), (2) where ∗ denotes 2D convolution, p(x, y;m, z) is called the point spread function of the phase mask, which is determined by the phase mask m(x, y) and the distance d. Once we optimize the PSF, the
phase masks are designed for the PSF and a chosen d. The fabricated mask then produces the PSF at the given d. For the fabricated system shown in 4.5, d is set to be 2 mm for making our system much thinner than conventional cameras (thickness ranges between 7-20 mm). The mask is fixed at distance d to the sensor during operation, and thus the convolution property will continue to hold. According to Eq. equation 2, the optical layer can be regarded as a special convolutional layer. Note that one phase mask can only perform a single-channel convolution with a positive kernel. It takes two phase masks to implement a single-channel convolution with a real-valued kernel, where one implements the positive part of the kernel and the other implements the negative part. Therefore, in order to construct a convolutional layer with three channels and real-valued kernels, we need six masks in the imaging system. Also, the input light has three color channels (R, G, and B), and each phase mask operates on all the color channels. Therefore, a three-channel convolution will produce a total of nine feature maps (FMs).
We propose three different designs for using the rendered FMs, as shown in Fig. 3. Specifically, the design in Fig. 3 (a) accumulates the FMs across the same color and outputs a 3-channel FM, which is still in an RGB shape; the design in Fig. 3 (b) accumulates the FMs from one mask across three colors and outputs a 2-channel FM; and the design in Fig. 3 (c) simply concatenates all the FMs from different colors. These three different designs extract different amounts or types of information from the scene which are then passed to the following neural network.
Experiments under various settings show that SACoD based on Fig. 3 (a) achieves higher accuracies than those based on Fig. 3 (b), and similar accuracies as the designs based on Fig. 3 (c). We conjecture the reason is that design (a) applies independent transformations on the RGB channels to maintain the original channel-wise discriminative information; and maintaining all the information as design (c) does not contribute to higher accuracy than design (a), since the accumulated information across the same color has provided sufficient information for the following processing under the constrained number of masks. So in our experiments, we adopt the optical layer design (a).
The SACoD formulation and algorithm. This subsection introduces the formulation and optimization of SACoD which aims to simultaneously optimize the phase mask m, and the neural network’s architecture α, and the neural network’s weights w. Formally, SACoD aims to solve:
min α Lval (m∗(α),w∗(α),α) + λLe(α), (3) m
∗(α),w∗(α) = argmin {m,w} Ltr(m,w,α). (4)
Ltr and Lval are task-specific performance losses evaluated on the training and validation set, respectively, Le is the efficiency loss, e.g. model size, computational cost, or energy consumption, and λ is the tuning parameter trading-off the accuracy and efficiency. Following the same parameterization scheme in DARTS (Liu et al., 2018), α denotes the weights of different candidate operations.
There are two major modifications compared to the original DARTS framework. The first difference is that the efficiency loss Le, measured by the sum of each layer’s FLOPs weighted by the network parameter α, is introduced. More importantly, the second and major difference is that the phase maskm is optimized jointly in the framework. It is worth pointing out that although mathematically similar, m∗ and w∗ have different degrees of dependencies on α. w∗ is directly impacted by α because α governs which subset of the w is ultimately used. m∗ is only indirectly influenced by α. Therefore, incorporating m will significantly improve the tradeoff between performance and model complexity. Besides, SACoD is naturally compatible with other NAS methods. We adopt differential NAS for the fast generation of the optical mask and network. When using other NAS methods, e.g., reinforcement learning-based NAS (Zoph & Le, 2017), we still observe similar system performance (within 0.3% accuracy on CIFAR-100), but the search time increases to 8 GPU-days from 0.5 GPU-days.
The whole co-design process can be divided into two stages, a searching stage and a training stage. In the searching stage, we apply the alternate gradient descent of Eqs. equation 3 and equation 4 to search for the optimal network architecture α∗. In the training stage, the optimal mask and weights are determined by optimizing Eq. equation 4 conditioning on the optimal network architecture α∗.

heading: 4 EXPERIMENTS

text:
This section presents evaluation results of SACoD applied to PhlatCam lensless imaging systems. We first describe the experiment settings, and then benchmark SACoD over SOTA lensless
imaging systems on standard classification tasks, IoT applications, and other vision tasks in Sections 4.2, 4.3, 4.4, respectively. Finally, we provide ablation studies of SACoD in Section 4.6.

heading: 4.1 EXPERIMENT SETUP

text:
Optical layer constraints. As mentioned, the optical layer first performs convolutional operations on the input scene optically, the outputs of which are then processed by the following neural network. The physical device construction imposes design constraints on the optical layer design. Specifically, since the phase mask is placed closer to the sensor, the optically achievable kernel size cannot be arbitrarily small. Here, we adopt kernel sizes that are not smaller than 7x7. Additionally, since all the designed masks are sharing the same sensor area, the number of designed masks cannot be large due to the limited sensor area. Here, we constrain the number of masks to be no more than 6.
Algorithm setting. Datasets: we evaluate SACoD on a total of four vision tasks with six datasets: two classification datasets CIFAR-10/100, two IoT datasets including FlatCam Face (Tan et al., 2019) and Head Pose Detection (Gourier & Crowley, 2004), one segmentation dataset Cityscapes (Cordts et al., 2016), and one unsupervised image translation dataset horse2zebra (Zhu et al., 2017). The same and standard data augmentation methods (e.g., random crop and normalization) are adopted for both SACoD and the baselines.
Baselines: we benchmark SACoD over two SOTA lensless imaging systems:
• Gabor-mask System: we fix the optical layer to be the Gabor-mask (Chen et al., 2016) and search for networks using the same NAS method as SACoD.
• Co-train System: we fix the network to be a SOTA IoT CNN MobileNetV2 (Sandler et al., 2018) and jointly train the optical layer and the backend network.
Efficiency metrics: we consider both FLOPs (Floating Point Operations) and energy cost based on real-device measurements as the efficiency metrics. Specifically, we adopt an NVIDIA JETSON TX2 (NVIDIA Inc.), a popular IoT GPU, as the target platform, which is connected to a laptop with the real-time energy cost being obtained via the sysfs (Patrick Mochel and Mike Murphy.) of the embedded INA3221 (Texas Instruments Inc.) power rails monitor.

heading: 4.2 SACOD OVER SOTA IMAGING SYSTEMS ON CLASSIFICATION TASKS

text:
Search and training setting. To search neural networks on CIFAR-10/100 for both the SACoD and Gabor-mask systems, we quantize all the operations to 8-bit using a SOTA quantization training method (Banner et al., 2018), which is a common practice considering the constrained sources on IoT devices. We adopt the search settings in (Wu et al., 2019a) with minor changes discussed in appendix. Specifically, we search for 50 epochs with 64 batch size, and update the supernet weights
on half of the training dataset using an SGD optimizer with a 0.9 momentum, and an initial learning rate of 0.025 with the cosine decay, and update network architecture parameters on the other half of the training dataset using an Adam optimizer with 0.9 momentum, and a fixed learning rate of 3e-4. For training the derived network architectures from scratch, we adopt an SGD optimizer with 0.9 momentum, and an initial learning rate of 0.01 with cosine decay for 600 epochs with 96 batch size.
To benchmark SACoD over SOTA imaging systems, we fix the number of masks to be six among all the settings, and then study their accuracy under different FLOPs and energy costs. We control the FLOPs of the SACoD and Gabor-mask systems by controlling λ in Eq. equation 3 and that of the Co-train system by changing the width multiplier (Howard et al., 2017).
Results analysis. Fig. 4 shows the trade-off between the accuracy and required hardware costs in terms of both FLOPs and energy cost for the SACoD and the two baseline lensless imaging systems on CIFAR-10/100. We can observe that SACoD consistently requires reduced FLOPs or energy cost while achieving a comparable or higher accuracy over the baselines. On CIFAR-10, SACoD achieves 44.1% and 70.9% reduction in FLOPs, and 27.6% and 48.4% reduction in energy, while offering a +0.01% and +1.58% higher accuracy, compared with the Co-train and Gabor-mask baselines, respectively; On CIFAR-100, SACoD reduces the FLOPs by 62.9% and 64.1%, and energy cost by 49.7% and 50.9%, while achieving a +0.71% and +7.92% higher accuracy, compared to the Co-train and Gabor-mask baselines, respectively. This set of experiments validate that the end-to-end optimization engine in SACoD indeed can lead to superior performance in both task performance and hardware efficiency.
Considering that the form factor or area is another influential design factor in lensless IoT imaging systems, we evaluate SACoD over the baselines in terms of the trade-off between accuracy and area by controlling the number of masks in the optical layer, and summarize the results in Fig. 5. We can see that the proposed SACoD achieves the best accuracy-area tradeoffs among all the designs under the same number of masks (and thus area) and the same model size. In particular, SACoD achieves 60.0% and 80.0% reduction in area while offering a 0.01% and 1.05% higher accuracy, compared with the Co-train and Gabor-mask baselines, respectively.

heading: 4.3 SACOD OVER SOTA IMAGING SYSTEMS ON IOT APPLICATIONS

text: Here we benchmark SACoD over the SOTA baselines on two IoT applications (including FlatCam Face recognition (Tan et al., 2019) and Head-pose task (Gourier & Crowley, 2004)) to evaluate its effectiveness on real-world IoT tasks. In this set of experiments, we further constrain FLOPs of the derived neural networks to see if SACoD is still applicable to extremely energy-constrained scenarios. As shown in Fig. 6, we can see that again SACoD consistently outperforms the baselines under all settings in terms of accuracy-cost tradeoffs. Specifically, compared with the Co-train baseline, SACoD achieves 59.5% and 57.1% reduction in FLOPs, 32.9% and 30.1% reduction in energy cost with a +0.11% and +0.07% higher accuracy, on the Flatcam Face and Head-pose tasks, respectively. Meanwhile, compared with the Gabor-mask baseline, SACoD shows better scalability to more energy-constrained scenarios. In particular, when the FLOPs or energy constraint is extremely low, SACoD achieves an 8.75% and 5.85% higher accuracy, under the same FLOPs/energy cost on the Flatcam Face and Head-pose tasks. These results show the effectiveness of SACoD extends to read-world IoT applications and the superior scalability of SACoD over SOTA IoT imaging systems.

heading: 4.4 SACOD OVER SOTA IMAGING SYSTEMS ON OTHER VISION TASKS

text:
Considering the diverse applications of IoT devices, we also evaluate SACoD on other vision tasks including one segmentation task and two unpaired image-to-image translation tasks, which require a more challenging tradeoff on CNN-powered intelligent IoT devices. We show the quantitative comparison, visualization, and detailed experiment settings in the appendix. Note that segmentation tasks are commonly evaluated in terms of both FID (Heusel et al., 2017) and image visualization as
Figure 7: Physical Masks
Red Green Blue
Positive Components
Negative Components
Optimized Experimental Optimized Experimental Optimized Experimental
Figure 8: PSF comparison between optimized and experimental ones
the former cannot always capture the image quality. From the results, SACoD requires the smallest FLOPs under all the six cases, and it shows that SACoD provides notably the best visualization effect among all the methods. In particular, compared to the Gabor-mask baseline, SACoD achieves 12.0% ∼ 50.1% reduction in FLOPs and +1.26 ∼ +20.27 improvement in FID, while providing notably better visualization effect; compared to the Co-train baseline, SACoD reduces the FLOPs by 92.56% ∼ 93.4% and offers notably better visualization effect, while achieving better FID on the zebra2horse dataset. Note that although the Co-train baseline achieves better FID than SACoD on the horse2zebra dataset, its visualization images (see Fig. 12 the appendix) suffer greatly from color shifts, distortion and chase-board effect, similar to those of the Gabor-mask baseline.

heading: 4.5 EVALUATION ON PHYSICAL FABRICATED MASKS

text: To evaluate the performance of SACoD on the actual PhlatCam imaging system besides simulation, we fabricate the actual masks with the PSF from the optimized optical layer. We first visualize the fabricated masks and compare them with the optimized PSF. Fig. 7 shows 6 fabricated masks setting on CIFAR-100 with a microscope image, in which the top row is positive masks and bottom ones are negative, and columns from left to right represent RGB-channels respectively. Then we compare the experimental PSF generated by the fabricated masks with the optimized PSF in Fig. 8. From the comparison, we find the experimental PSF basically keeps the original shape, although the brightness of some pixels is changed. We further compare the achieved accuracy based one simulated and fabricated masks on CIFAR-10. As shown in Table 1, the accuracy gap is 4.41% which is resulting from possible alignment issues and other experimental error sources in the mask fabrication process and has been shown by other optical computing systems, i.e., (Lin et al., 2018).

heading: 4.6 ABLATION STUDIES ON SACOD

text:
Influence of mask flexibility. To better understand SACoD’s superior performance, we perform ablation experiments where the optical mask is fixed at either or both of the search and training stages. The experiment setting is the same as described in Section 4.2, in which the models are all under comparable model sizes (1.0 M), and the pre-trained optical mask is obtained by co-training with MobileNetV2 (Sandler et al., 2018). We explore four cases: (mask) fixed during both the search and training stages, fixed only during the training stage, fixed only during the search stage, and unfixed
during both (i.e., SACoD). From the results in Table 2 we can see that (1) fixing the mask during both the search and training stages causes the biggest accuracy drop (more than 0.93% on CIFAR-10 and CIFAR-100 datasets); and (2) fixing the mask at only one of the stages would also significantly impact the performance. This study verifies that the success of the co-design principle can be ascribed to two causes. First, with the mask included as hyperparameters, the optimization (during fine-tuning) can reach a better minimum. Second and more importantly, the trainable mask can assist finding a better architecture because the neural network backend can entrust more computation responsibilities to the ‘costless’ optical layer and thus can be made more efficient.
Effectiveness of the optical layer. To further explore the reason behind the success of SACoD, we compare the discriminative power of the features captured by the optical layers of SACoD and the Gabor-mask baseline. Specifically, following (Suau et al., 2020), we average the optical layer’s activations over the output channels to obtain a vector and use the corresponding softmax value as the feature distribution for each input image. We then calculate the KL divergence between the feature distribution from different classes to see how discriminative the features are. Fig. 9 visualizes the average KL divergence (over 100 random selected im-
ages) between every two classes on the test dataset of CIFAR-10. We can see that the feature distribution difference of SACoD between different classes is notably and consistently larger than that of the Gabor-mask baseline, which further verifies that the optical layer of SACoD undertakes more computations to extract the discriminative information from the input, so as to save the computations for the neural network backend.
Considering the potential defection when fabricating the PhlatCam sensors, we study the robustness of the SACoD, Co-train, and Gabor-mask systems when injecting noises into the masks. To simulate the noise effect, we sample noises from a standard normal distribution and add it to the masks; after that, we finetune the backend network for restoring the accuracy until convergence. By controlling the standard deviation (σ) of the standard normal distribution, i.e., the noises’ dynamic range, we evaluate the robustness of the three lensless systems under different noise intensity and summarize the results. We can observe that (1) as the σ increases (i.e., noises’ intensity increases), the restored accuracy of all the three lensless systems decreases while the accuracy of SACoD is consistently higher than that of both the Co-train and Gabor-mask baselines; and (2) the Co-train baseline is the most sensitive to the noises injected to the masks, and suffers from up to 2.71% accuracy loss, the Gabor-mask baseline shows the least accuracy degradation (1.19%) under noises yet offering the lowest overall accuracy, and our proposed SACoD achieves a favorable tradeoff with the highest task accuracy and a medium level accuracy drops under noises. More details are provided in appendix.

heading: 5 CONCLUSION

text: We propose SACoD, a sensor algorithm co-design framework, to enable more energy-efficient CNNpowered IoT devices based on PhlatCam. A novel end-to-end co-search algorithm is presented to jointly optimize the coded mask of PhlatCam in the sensor and the neural network backend. Extensive experiments and ablation studies validate the superiority of the proposed SACoD in terms of both task performance and hardware efficiency as well as the optical layer effectiveness, when evaluated over SOTA lensless imaging systems on various tasks and datasets.

heading: A EXPERIMENT DETAILS FOR SEARCHING ON CIFAR-10/100

text: For the experiments on CIFAR-10/10, we adopt the same search space as FBNet (Wu et al., 2019a) except each group’s stride settings, i.e., a set of blocks with the same number pf output channels, for adapting to the resolution of images on CIFAR-10/100. In particular, we follow the strides settings for MobileNetV2 on CIFAR-10/100 as described in (Wang et al., 2019), which is [1, 1, 2, 2, 1, 2, 1] for all the seven groups. In addition, following (Wu et al., 2019a), we apply a gumbel softmax on each architecture parameter option’s contribution weights to the supernet, where the initial temperature is 3 and decays by 0.92 at the end of each epoch.

heading: B MORE ABLATION STUDIES FOR SACOD: ROBUSTNESS AGAINST NOISE

text:
Considering the potential defection when fabricating the PhlatCam sensors, we study the robustness of the SACoD, Co-train, and Gabormask systems when injecting noises into the masks. To simulate the noise effect, we sample noises from a standard normal distribution and add it to the masks; after that, we finetune the backend network for restoring the accuracy until convergence. By controlling the standard deviation (σ) of the standard normal distribution, i.e., the noises’ dynamic range, we evaluate the robustness of the three lensless systems under different noise intensity and summarize the results in Fig. 10. We can observe that (1) as the σ increases (i.e., noises’ intensity increases), the restored accuracy of all the three lensless systems decreases while the accuracy of SACoD is consistently higher than that of both the Cotrain and Gabor-mask baselines; and (2) the Co-
train baseline is the most sensitive to the noises injected to the masks, and suffers from up to 2.71% accuracy loss, the Gabor-mask baseline shows the least accuracy degradation (1.19%) under noises yet offering the lowest overall accuracy, and our proposed SACoD achieves a favorable tradeoff with the highest task accuracy and a medium level accuracy drops under noises.

heading: C EXPERIMENT RESULTS WHEN USING DIFFERENT MASK DESIGNS

text: As illustrated in Fig. 3 of the main content, three different mask designs can be adopted. We here provide more experiment results when using the three mask designs for SACoD. Specifically, we co-train MobileNetV2 with the three different mask designs on CIFAR-10/100 and find that the design with the masks in Fig. 3 (a) (in the main content) achieves a 0.86% and 1.78% higher accuracy compared with that based on the masks in Fig. 3 (b), while offering a comparable accuracy (0.01% and -0.02%) with a 66.67% reduced rendered feature maps, on CIFAR-10 and CIFAR-100, respectively. This set of experiments motivates us to adopt the masks’ design based on the one in Fig. 3(a) for experiments shown in the main content.

heading: D SACOD VS. LENS-BASED SYSTEMS

text:
Here we provide experiments for benchmarking SACoD over lens-based systems under the same search space (Wu et al., 2019a) and datasets (CIFAR-10/CIFAR-100). For designing the lens-based systems, we release the FLOPs constraints for the first layer of the network, i.e., removing the optical layer and its associated constraints, and search for the optimal network given the search space and datasets. We find that under a slightly reduced FLOPs (154M FLOPs vs. 158M FLOPs), SACoD achieves a 0.39% and 0.62% lower accuracy on CIFAR-10 and CIFAR-100, respectively, while reducing the thickness of the imaging systems by 10× which makes it possible to be integrated into more IoT applications. This set of experiments show that our proposed SACoD can offer similar
task performance and hardware efficiency as compared to lens-based systems, while being able to shrink the thickness of the system by one order.

heading: E MORE DETAILS AND RESULTS ON THE UNPAIRED IMAGE-TO-IMAGE TRANSLATION TASKS

text:
In this subsection, we provide more details for the experiments on the unpaired image-to-image translation tasks, results of which are summarized in Table 2 and Fig. 7 of the main content.
E.1 THE SEARCH SPACE AND METHOD
For the search space, we drew inspiration from existing works and build a sequential supernet with hardware-friendly regular connection patterns where each candidate operators are sequentially connected. For the search method, we adopt a SOTA differential neural architecture search (NAS) method (Liu et al., 2018) to search for both the operators and their widths to maximize efficiency improvements.
Search for the operators. The supernet consists of a total of nine operators from the following options:
• The Conv layer: Conv 1×1, Conv 3×3; • The residual block (i.e., 2 Conv 3×3 layers with a skip connection); • The depthwise (DW) block (Conv 1 ×1 + DW Conv 3×3 + Conv 1×1 with a skip connec-
tion). The above options cover popular and efficient building blocks in SOTA GAN generators (Zhu et al., 2017; Shu et al., 2019; Gong et al., 2019). In particular, following SOTA differential NAS search method, we use the architecture parameter αij to control the probability of choosing the j-th operator in the i-th layer, and treat its softmax values as its contributing weight to the supernet.
Search for the widths. Considering our target IoT applications, we also search for the widths (i.e., the number of output channels) of each operator for more efficient GAN models. Naively building a set of independent convolutional kernels with different widths for each operator is not practical due to the required large memory consumption. We thus build a superkernel with a maximal width, and then search for the expansion ratio φ to make use of only a subset of the superkernel’s input/output dimensions. In particular, we set φ ∈ [ 1
3 , 1 2 , 3 4 , 5 6 , 1] and use the architecture parameter γi to control
the probability of choosing each expansion ratio for the i-th layer; Meanwhile, we apply gumbel softmax to approximate differentiable sampling for φ based on γ as (Cai et al., 2018b). Therefore, during the searching process only one expansion ratio will be activated in one iteration, saving both the required memory and computational costs.
E.2 PROXY TASK
We search for an efficient generator via distillation as formulated below:
min G,α,γ
1
N
N
∑ i=1 d(G(xi, α, γ), G0(xi)) + λF (α, γ). (5)
Where d(⋅, ⋅) is a distance metric for the knowledge distillation (Hinton et al., 2015) betweenG (the searched generator) and G0 (the original generator in CycleGAN (Zhu et al., 2017)), which is the perceptual loss (Johnson et al., 2016) in our case, F is the computational budget determined by the network architecture (FLOPs in our case), and α and γ are the architecture parameters controlling the operator and width of each layer, respectively. Note that, the objective function in Eq. 5 is independent of any trained discriminators since in practice the discriminator is often discarded after the generator is trained and thus not necessarily available when compressing the generator.
E.3 MORE VISUALIZATION RESULTS
Here we show more visualization results of SACoD as compared with the two lensless baselines on the unpaired image-to-image translation tasks (horse2zebra and zebra2horse) in Fig. 12. We can
see that again the Gabor-mask baseline suffers greatly from color shift and distortion especially on the horse2zebra task, and the proposed SACoD provides better or competitive visualization effects with high-contrast textures compared with the Co-train baseline while achieving a 92.6% reduction in FLOPs.
E.4 SACOD OVER SOTA IMAGING SYSTEMS ON OTHER VISION TASKS
Segmentation tasks. Real-time segmentation on resource-constrained platforms have growing demands in many applications such as medical image segmentation, so we explore its combination with PhlatCam lensless imaging systems for scenarios with constrained imaging environments. To search for efficient segmentation networks, we adopt the search space and search method of a SOTA NAS method (Chen et al., 2019). For a fair comparison, all the searching and training settings follow the original ones in (Chen et al., 2019). Specifically, the co-train baseline adopts the DeepLabV3 (Chen et al., 2017) model with a ResNet50 (He et al., 2016) backbone. Table 3 with 2048x1024 images show that under all the mask constraints SACoD achieves the highest mean Intersection Over Union (mIOU) while requiring the smallest FLOPs. Specifically, SACoD achieves a 0.8%∼4.3% higher mIOU and 12.4%∼93.2% reduction in FLOPs over the Co-train and Gabor-mask baselines, respectively, under all the mask settings.
Source image Co-train Gabor-mask SACoD 54.17 GFLOPs 7.15 GFLOPs 3.57 GFLOPs
Unpaired image-to-image translation tasks. To apply GAN (Goodfellow et al., 2014) to lensless imaging systems under constrained imaging environments, we evaluate SACoD on unpaired imageto-image translation tasks (Zhu et al., 2017), one of the most popular applications of GAN.
Experiment settings: since no prior NAS works search for image-to-image GANs, we design a new search space (see details in the supplement) targeting extremely efficient GAN architectures. Following the settings of CycleGAN (Zhu et al., 2017), we keep the stem and header structures in the generator and search for both the operator type and channel width within the supernet which include nine sequential operators. In addition, we use the knowledge distillation (Hinton et al., 2015) between the searched architecture and the pre-trained CycleGAN generator as the loss function. More search settings are in the supplement. The Co-train baseline system consists of an original CycleGAN following the optical layer.

title: SACOD: SENSOR ALGORITHM CO-DESIGN TOWARDS EFFICIENT CNN-POWERED INTELLIGENT PHLATCAM


summaries: The authors introduce a novel method for non-negative matrix factorization for timeseries and apply it to longitudinal honey bee interaction data.  The model leverages consistency of individuals over time by forcing the factors (or rather, the residuals of the factors with respect to a global trajectory) to be linear combinations of a small set of temporal basis functions.  These temporal basis functions are functions of the bee’s age.  In other words, the factor embedding of each bee is a vector of linear combinations of 16 learned basis functions of time.  Since all bees use the same 16 temporal basis functions, given these basis functions the lifetime embedding of each bee is encapsulated by a small matrix of numbers, namely the coefficients for the temporal basis functions for each factor (and in practice only two factors were significant, so each bee’s life is embedded in 32-dimensional space).  There are a number of regularizations on the temporal basis functions and the embedding coefficients.

summaries: The authors present a matrix factorization model to jointly characterize the lifetime interactions of thousands of bees over generations. The problem is fascinating as both a technical and scientific question and the modeling framework appears novel. Although not directly addressed, the authors appear to be trying to solve a *tensor* factorization problem, not just the special case of a matrix (which is of course a 2-d tensor). It would have been interesting to see results comparing their method with a non-negative variant of, say, PARAFAC/CANDECOMP or generalizations thereof. It would have at least have been appropriate to explain why or why not existing tensor methods are not appropriate.

summaries: This paper proposes a NMF formulation ||A-FF^T||^2 where A and F are different types of information extracted from social datasets. In the honeybee example the authors highlight, A represents the spatial relationship between bees, and F encodes the age of the bees. The authors setup F to be decomposable into two types of embeddings, one which characterizes the group activity and the other which characterizes the individual activity.

abstractText: Honey bees are a popular model for complex social systems, in which global behavior emerges from the actions and interactions of thousands of individuals. While the average life of a bee is organized as a sequence of tasks roughly determined by age, there is substantial variation at the individual level. For example, young bees can become foragers early in life, depending on the colony’s needs. Using a unique dataset containing lifetime trajectories of all individuals over multiple generations in two honey bee colonies, we propose a new temporal matrix factorization model that jointly learns the average developmental path and structured variations of individuals in the social network over their entire lives. Our method yields inherently interpretable embeddings that are biologically plausible and consistent over time, which allows comparing individuals regardless of when or in which colony they lived. Our method provides a quantitative framework for understanding behavioral heterogeneity in complex social systems applicable in fields such as behavioral biology, social sciences, neuroscience, and information science.

authors:
Ehsaneddin Asgari
Mohammad R.K. Mofrad

title: Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics

venue: PLOS ONE,

authors:
Josefine Bohr Brask
Samuel Ellis
Darren P. Croft

title: Animal social networks – an introduction for complex systems scientists

authors:
Jose Camacho-Collados
Mohammad Taher Pilehvar

title: From word to sense embeddings: a survey on vector representations of meaning

venue: Journal of Artificial Intelligence Research,

authors:
Michelle M. Elekonich
Stephen P. Roberts

title: Honey bees as a model for understanding mechanisms of life history transitions

venue: Comparative Biochemistry and Physiology. Part A, Molecular & Integrative Physiology,

authors:
Andrea Frome
Greg S Corrado
Jon Shlens
Samy Bengio
Jeff Dean
Tomas Mikolov

title: DeViSE: A Deep Visual-Semantic Embedding Model

venue: Advances in Neural Information Processing Systems

authors:
Laetitia Gauvin
André Panisson
Ciro Cattuto

title: Detecting the Community Structure and Activity Patterns of Temporal Networks: A Non-Negative Tensor Factorization Approach

venue: PLoS ONE,

authors:
Tim Gernat
Vikyath D. Rao
Martin Middendorf
Harry Dankowicz
Nigel Goldenfeld
Gene E. Robinson

title: Automated monitoring of behavior reveals bursty interaction patterns and rapid spreading dynamics in honeybee social networks

venue: Proceedings of the National Academy of Sciences,

authors:
Ian J. Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative Adversarial Networks

authors:
Jacob M Graving
Daniel Chae
Hemal Naik
Liang Li
Benjamin Koger
Blair R Costelloe
Iain D Couzin

title: DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife, 8:e47994, October 2019

authors:
Pengfei Jiao
Haodong Lyu
Xiaoming Li
Wei Yu
Wenjun Wang

title: Temporal community detection based on symmetric nonnegative matrix factorization

venue: International Journal of Modern Physics B, March 2017. doi: 10.1142/S0217979217501028

title: Division of labor in honeybees: form, function, and proximate mechanisms

venue: Behavioral Ecology and Sociobiology,

authors:
Jingu Kim
Yunlong He
Haesun Park

title: Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework

venue: Journal of Global Optimization,

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A Method for Stochastic Optimization

venue: In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations,

authors:
T. Kolda
B. Bader

title: Tensor Decompositions and Applications

venue: SIAM Rev.,

authors:
Jens Krause
Richard James
Daniel W. Franks
Darren P. Croft

title: Animal Social Networks

authors:
Da Kuang
Sangwoon Yun
Haesun Park

title: SymNMF: nonnegative low-rank approximation of a similarity matrix for graph clustering

authors:
Emily L Mackevicius
Andrew H Bahle
Alex H Williams
Shijie Gu
Natalia I Denisenko
Mark S Goldman
Michale S Fee

title: Unsupervised discovery of temporal sequences in high-dimensional datasets, with applications to neuroscience. eLife, 8:e38471, February 2019

authors:
Alexander Mathis
Pranav Mamidanna
Kevin M. Cury
Taiga Abe
Venkatesh N. Murthy
Mackenzie Weygandt Mathis
Matthias Bethge

title: DeepLabCut: markerless pose estimation of userdefined body parts with deep learning

venue: Nature Neuroscience,

authors:
Danielle P. Mersch
Alessandro Crespi
Laurent Keller

title: Tracking individuals shows spatial fidelity is a key regulator of ant social organization

authors:
Morten Mørup
Lars Kai Hansen
Sidse Marie Arnfred
Lek-Heng Lim
Kristoffer Hougaard Madsen

title: Shift-invariant multilinear decomposition of neuroimaging

venue: data. NeuroImage,

authors:
Walter Nelson
Marinka Zitnik
Bo Wang
Jure Leskovec
Anna Goldenberg
Roded Sharan

title: To Embed or Not: Network Embedding as a Paradigm in Computational Biology

venue: Frontiers in Genetics,

authors:
Fabian Pedregosa
Gaël Varoquaux
Alexandre Gramfort
Vincent Michel
Bertrand Thirion
Olivier Grisel
Mathieu Blondel
Peter Prettenhofer
Ron Weiss
Vincent Dubourg
Jake Vanderplas
Alexandre Passos
David Cournapeau
Matthieu Brucher
Matthieu Perrot
Édouard Duchesnay

title: Scikit-learn: Machine Learning in Python

venue: Journal of Machine Learning Research,

authors:
Talmo D. Pereira
Diego E. Aldarondo
Lindsay Willmore
Mikhail Kislin
Samuel S.-H. Wang
Mala Murthy
Joshua W. Shaevitz

title: Fast animal pose estimation using deep neural networks

venue: Nature Methods,

authors:
Noa Pinter-Wollman
E.A. Hobson
J.E. Smith
A.J. Edelman
D. Shizuka
S. de Silva
J.S. Waters
S.D. Prager
T. Sasaki
G. Wittemyer
J. Fewell
D.B. McDonald

title: The dynamics of animal social networks: analytical, conceptual, and theoretical advances

venue: Behavioral Ecology,

title: Adaptive significance of the age polyethism schedule in honeybee colonies

authors:
Xiaohua Shi
Hongtao Lu
Yangchen He
Shan He

title: Community detection in social network with pairwisely constrained symmetric non-negative matrix factorization

venue: In 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM),

authors:
Nguyen Xuan Vinh
Julien Epps
James Bailey

title: Information theoretic measures for clusterings comparison: is a correction for chance necessary

venue: In Proceedings of the 26th Annual International Conference on Machine Learning,

authors:
Fei Wang
Tao Li
Xin Wang
Shenghuo Zhu
Chris Ding

title: Community discovery using nonnegative matrix factorization

venue: Data Mining and Knowledge Discovery,

authors:
Wei Wang
Wenchao Yu
Wei Cheng
Charu C. Aggarwal
Haifeng Chen

title: Link Prediction with Spatial and Temporal Consistency in Dynamic Networks

venue: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence,

authors:
Benjamin Wild
David M. Dormagen
Adrian Zachariae
Michael L. Smith
Kirsten S. Traynor
Dirk Brockmann
Iain D. Couzin
Tim Landgraf

title: Social networks predict the life and death of honey

venue: bees. bioRxiv,

title: Combining tensor decomposition and time warping models for multineuronal spike train analysis

venue: bioRxiv, pp

authors:
Margaret K. Wray
Heather R. Mattila
Thomas D. Seeley

title: Collective personalities in honeybee colonies are linked to colony fitness

venue: Animal Behaviour,

authors:
Hsiang-Fu Yu
Nikhil Rao
Inderjit S Dhillon

title: Temporal Regularized Matrix Factorization for High-dimensional Time Series Prediction

authors:
Wenchao Yu
Charu C. Aggarwal
Wei Wang

title: Temporally Factorized Network Modeling for Evolutionary Network Analysis

venue: In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining,

heading: 1 INTRODUCTION

text:
Animals living in large groups often coordinate their behaviors, resulting in emergent properties at the group level, from flocking birds to democratic elections. In most animal groups, the role an individual plays in this process is thought to be reflected in the way it interacts with group members. Technological advances have made it possible to track all individuals and their interactions in animal societies, ranging from social insects to primate groups (Mersch et al., 2013; Gernat et al., 2018; Mathis et al., 2018; Graving et al., 2019; Pereira et al., 2019). These datasets have unprecedented scale and complexity, but understanding these data has emerged as a new and challenging problem in itself (Pinter-Wollman et al., 2014; Krause et al., 2015; Brask et al., 2020).
A popular approach to understand high-dimensional data is to learn semantic embeddings (Frome et al., 2013; Asgari & Mofrad, 2015; Camacho-Collados & Pilehvar, 2018; Nelson et al., 2019). Such embeddings can be learned without supervision, are interpretable, and are useful for accomplishing downstream tasks. Individuals in animal societies can be described with semantic embeddings extracted from social interaction networks using matrix factorization methods. For example, in symmetric non-negative matrix factorization (SymNMF), the dot products of any two animals’ factor vectors reconstruct the interaction matrix (Wang et al., 2011; Shi et al., 2015), see Figure 1 a and b). If the embeddings allow us to predict relevant behavioral properties, they serve our understanding as semantic representations. However, in temporal settings where the interaction matrices change over time, there is no straightforward extension of this algorithm. The interaction matrices at different time points can be factorized individually, but there is no guarantee that the embeddings stay semantically consistent over time, i.e. the prediction of relevant behavioral properties will deteriorate.
In living systems, interaction dynamics are highly variable; individuals differ in when they appear in the data and how long they live. Different non-overlapping groups of individuals, e.g. from different years, may not interact with each other at all. How can we find a common semantic embedding even in these extreme cases? How do we learn embeddings that generalize to different groups and still provide insights into each individual’s functional role? If animals take on roles partially determined by a common factor, such as age, how can we learn this dependency?
Several approaches to extend NMF to temporal settings have been proposed in a variety of problem settings. Yu et al. (2016) and Mackevicius et al. (2019) propose a factorization method for time
series analysis. Gauvin et al. (2014) focus on the analysis of communities that are determined by their temporal activity patterns. Jiao et al. (2017) consider the case of communities from graphs over time and enforce temporal consistency with an additional loss term. Yu et al. (2017) and Wang et al. (2017) represent networks as a function of time. Temporal matrix factorization can be seen as a tensor decomposition problem, for which methods with many applications have been proposed in the literature, see Kolda & Bader (2009) for a review. In particular, time-shifted tensor decomposition methods have been used in multi-neuronal spike train analysis when recordings of multiple trials from a population of neurons are available (Mørup et al., 2008; Williams, 2020). However, to our knowledge, no method yet considers the case of entities following a common trajectory depending on an observable property (e.g., the age of an individual), which we show to be a powerful inductive bias.
We approach this question using honey bees, a popular model system for studying individual and collective behavior (Elekonich & Roberts, 2005). Bees allocate tasks across thousands of individuals without central control, using an age-based system: young bees care for brood, middle-aged bees perform within-nest labor, and old bees forage outside (Seeley, 1982; Johnson, 2010). Colonies are also organized spatially: brood is reared in the center, honey and pollen are stored at the periphery, and foragers offload nectar near the exit. Therefore, an individual’s role is partially reflected in its location, which allows us to evaluate whether the embeddings our method learns are semantically meaningful.
We propose jointly learning two meaningful representations of honey bee social behavior: 1) an individuality embedding that characterizes the lifetime behavior of each individual and 2) a daily representation of the individual’s functional position in the social network that can be derived from the individuality embeddings. We show that these representations can be learned in an unsupervised fashion, using only interaction matrices of the individuals over time. We analyze a dataset obtained by tracking thousands of individually marked honey bees in two colonies, at high temporal and spatial resolution over a total of 155 days, covering entire lifespans and multiple generations. We evaluate how well the embeddings capture the semantic differences of individual honey bee development by evaluating their predictiveness for different tasks and behaviorally relevant metrics.
Our method can be used to study dynamic groups in which individual units change their latent properties over time, and group composition is not fixed due to birth, death, or migration of individuals. We provide a new perspective on how to extract fundamental factors that underlie behavioral and developmental plasticity in animal groups.

heading: 2 METHODS

heading: 2.1 TEMPORAL NMF ALGORITHM

text:
SymNMF factorizes a matrix A ∈ RN×N+ such that it can be approximated by the product FF T , where F ∈ RN×M+ and M N :
F̂ = argmin F≥0 ∥∥∥A− FF T∥∥∥2 Ai,j ≈ f(i) · f(j)T f(i) = Fi,: f(i) ∈ RM+ (1) When applied to social networks, f(i) can represent the role of an entity within the social network A (Wang et al., 2011; Shi et al., 2015) - however, in temporal settings, factorizing the matrices for different times separately will result in semantically inconsistent factors F .
Here we present a novel temporal NMF algorithm (TNMF) which extends SymNMF to temporal settings in which A ∈ RT×N×N+ changes over time t. We assume that the entities i ∈ {0, 1, . . . , N} follow to some extent a common trajectory depending on an observable property (for example the age of an individual). We represent an entity at a specific point in time t using a factor vector f+(t, i) such that
Ât,i,j = f+(t, i) · f+(t, j)T Â ∈ RT×N×N+ f+(t, i) ∈ RM+ (2)
In contrast to SymNMF, we do not directly factorize At to find the optimal factors that reconstruct the matrices. Instead, we decompose the problem into learning an average trajectory of factors m(c(t, i)) and structured variations from this trajectory o(t, i) that depend on the observable property c(t, i):
f(t, i) = m(c(t, i)) + o(t, i) f+(t, i) = max(0,f(t, i)) (3)
c : NT×N → N m : N→ RM+ o : NT×N → RM
This decomposition is an inductive bias that allows the model to learn semantically consistent factors for entities, even if they do not share any data points (e.g., there is no overlap in their interaction partners), as long as the relationship between functional role and c(t, i) is stable. Note that in the simplest case c(t, i) = t, TNMF can be seen as a tensor decomposition model, i.e. the trajectory of all entities is aligned with the temporal dimension t of A. In our case, c(t, i) maps to the age of individual i at time t.
While many parameterizations for the function o(t, i) are possible, we only consider one particular case in this work: We learn a set of individuality basis functions b(c(t, i)) (shared among all entities) that define a coordinate system of possible individual variations and the individuality embeddings φ, which capture to what extent each basis function applies to an entity:
o(t, i) = K∑ k=0 φi,k · bk(c(t, i)) φ : RN×K bk : NT → R (4)
where K is the number of learned basis functions. This parameterization allows us to disentangle the forms of individual variability (individuality basis functions) and the distribution of this variability (individuality embeddings) in the data.
We implement the functions m(c(t, i)) and b(c(t, i)) with small fully connected neural networks with non-linearities and several hidden layers. The parameters θ of these functions and the entities’ embeddings φ are learned jointly using minibatch stochastic gradient descent:
θ̂, φ̂ = argmin θ,φ ∥∥∥A− Â∥∥∥2 (5) Note that non-negativity is not strictly necessary, but we only consider the non-negative case in this work for consistency with prior work (Wang et al., 2011; Shi et al., 2015). Furthermore, instead of one common property with discrete time steps, the factors could depend on multiple continuous properties, i.e. c : RT×N → RP , e.g. the day and time in a intraday analysis of social networks.

heading: 2.2 REGULARIZATION

text:
Due to the inductive bias, the model performs well without additional explicit regularization in the datasets considered in this work (see Table 2). However, we find that the model’s interpretability can be improved using additional regularization terms without significantly affecting its performance. We encourage sparsity in both the number of used factors and individuality basis functions by adding L1 penalties of the mean absolute magnitude of the factors f(t, i) and basis functions b(c(t, i)) to the objective. We encourage individuals’ lifetimes to be represented with a sparse embedding using an L1 penalty of the learned individuality embeddings φ.
We also introduce an optional adversarial loss term to encourage the model to learn embeddings that are semantically consistent over time, i.e. to only represent two entities that were present in the dataset at different times with different embeddings if this is strictly necessary to factorize the matrices A. We jointly train a discriminative network d(φi) that tries to classify the time of the first occurrence of all entities based on their individuality embeddings φ. The negative cross-entropy loss of this model is added as a regularization term to equation 5 in a training regime similar to generative adversarial networks (Goodfellow et al., 2014). See appendix A.2.1 for more details and A.4 for an ablation study of the effect the individual regularization terms have on the results of the model.

heading: 2.3 DATA

text:
2.3.1 HONEY BEE DATA
Two colonies of honey bees were continuously recorded over a total of 155 days. Each individual was manually tagged at emergence, so the date of birth is known for each bee. Locations and identities of all honey bees (N=9286) were extracted from the raw images and used to con-
struct daily aggregated temporal interaction networks based on the counts of spatial proximity events. The dataset also contains labels that can be used in proxy tasks (see section 2.5) to quantify if the learned embeddings and factors are semantically meaningful and temporally consistent. See table 1, and appendix A.1 for details. We publish the data together with this paper1.

heading: 2.3.2 SYNTHETIC DATA

text:
Although some ground truth labels exist for the honey bee datasets (section 2.5), factorizing these data is still fundamentally an unsupervised learning problem. We generated synthetic datasets to evaluate whether the model can identify groups of individuals with common latent factors that determine their interaction frequencies. We model a common lifetime trajectory and groups of individual variation of factors using smoothed Gaussian random walks. We randomly assign individuals to a group with random times of emergence and disappearance from the data and compute interaction matrices by calculating the dot product of the factors of 1024 individuals for 100 simulated days. We then measure how well the individuality embeddings φ of a fitted model correspond to the truth groups using the adjusted mutual information score (Vinh et al., 2009), and the mean squared error between the ground truth factors and the best permutation of the factors f+. We evaluate the model on 128 different random synthetic datasets with increasing Gaussian noise levels in the interaction tensor. See appendix A.1.2 for more details on the generation process.
In both datasets, we define c(t, i) as the age in days of an individual i at time t.

heading: 2.4 BASELINE MODELS

text:
Symmetric NMF: We compute the factors that optimally reconstruct the original interaction matrices using the standard symmetric NMF algorithm (Shi et al., 2015; Kuang et al., 2015), for each day separately, using the same number of factors as in the TNMF model.
Aligned symmetric NMF: We consider a simple extension of the standard SymNMF algorithm that aligns the factors to be more consistent over time. For each pair of subsequent days, we consider all combinatorial reorderings of the factors computed for the second day. For each reordering, we compute the mean L2 distance of all individuals that were alive on both days. We then select the reordering that minimizes those pairwise L2 distances and greedily continue with the next pair of days until all factors are aligned. Furthermore, we align the factors across colonies (where individuals cannot overlap) as follows: we run this algorithm for both datasets separately and align the resulting factors by first computing the mean embedding for all individuals grouped by their ages. As before, we now select from all combinatorial possibilities the reordering that minimizes the L2 distance between the embeddings obtained from both datasets. See section A.5.1 for pseudo code.
Tensor decomposition: We also compare against a constrained non-negative tensor decomposition model with symmetric factors F ∈ RN×M+ and temporal dynamics constrained to the diagonals, i.e. D ∈ RT×M×M+ and Dt = diag(dt), dt ∈ RM+ .
Ât = FDtF T (6)
F̂ , D̂ = argmin F ,D T−1 T∑ t=0 ∥∥∥At − Ât∥∥∥2 (7) Temporal NMF models: We evaluate variants of the temporal symmetric matrix factorization algorithms proposed by Jiao et al. (2017) and Yu et al. (2017).
1See Anonymous (2020) for dataset
For the tensor decomposition and temporal NMF baselines, we follow the procedure given above for the Aligned symmetric NMF to find the optimal reordering to align the factors obtained by applying models to the two datasets separately.

heading: 2.5 EVALUATION METRICS

text:
Reconstruction: We measure how well the original interaction matrices A can be reconstructed from the factors. We do not require the model to reconstruct the interaction matrices as well as possible because we only use the reconstruction as a proxy objective to learn a meaningful representation. Still, a high reconstruction loss could indicate problems with the model, such as excessive regularization.
Consistency: We measure to what extent the individuality embeddings φ change over time. For each model, we train a multinomial logistic regression model to predict the source cohort (date of birth) and calculate the area under the ROC curve (AUCcohort) using a stratified 100-fold cross-validation with scikit-learn (Pedregosa et al., 2011). The baseline models do not learn an individuality embedding; therefore we compute how well the model can predict the cohort using the mean factor representation of the individuals over their lives. We define consistency as 1− AUCcohort of this linear model. Note that a very low temporal consistency would indicate that the development of individual bees changes strongly between cohorts and colonies, which we know not to be true.
Mortality and Rhythmicity: We evaluate how well a linear regression model can predict the mortality (number of days until death) and circadian rhythmicity of the movement (R2 score of a sine with a period of 24 h fitted to the velocity over a three-day window). These metrics are strongly correlated with an individual’s behavior (e.g. foragers exhibit strong circadian rhythms because they can only forage during the daytime; foragers also have a high mortality). We follow the procedure given in Wild et al. (2020) and report the 100-fold cross-validated R2 scores for these regression tasks.
For this data, we expect the factors f+ and individuality embeddings φ to be semantically meaningful and temporally consistent if they reflect an individual’s behavioral metrics (mortality and rhythmicity) and if they do not change strongly over time (measured in the consistency metric).

heading: 3 RESULTS

text: We implemented the model using PyTorch (Paszke et al., 2019) and trained it in minibatches of 256 individuals for 200 000 iterations with the Adam optimizer (Kingma & Ba, 2015). See appendix A.2.3 for the architecture of the learned functions, a precise description of the regularization losses, and further hyperparameters. The code is publicly available 2.

heading: 3.1 SYNTHETIC DATA

text:
We factorize the interaction matrices of the 128 synthetic datasets with varying levels of Gaussian noise. We confirmed that our model converges in all datasets and evaluate whether we can distinguish
2https://anonymous.4open.science/r/b2b7e2fc-aa04-4cf8-85c7-646d8dc46400
the individuals’ ground truth group assignments. To that end, we extract the individuality embeddings φ from the models and measure how well they correspond to ground truth data using the adjusted mutual information (AMI) score. Furthermore, we measure the mean squared error between the best permutation of learned factors f+ and the ground truth factors. We find that for low levels of noise, our model can identify the truth group assignments with high accuracy, and are still significantly better than random assignments even at very high levels of noise (see figure 3). Note that for this experiment, we evaluated a model with the same hyperparameters as used in all plots in the results section (see Table 2) and a variant without explicit regularization except the L1 penalty of the learned individuality embeddings φ (λembeddings, because this regularization is required to meaningfully extract clusters), which was set to 0.1. See appendix A.1.2 for more details on the synthetic datasets.

heading: 3.2 HONEY BEES

text:
Mean lifetime model: The model learns a sparse representation of the developmental trajectory of a honey bee in the space of social interactions. Only two factors are effectively used (they exceed the threshold value of 0.01). These factors show a clear trend over the life of a bee, indicating that the model captures the temporal aspects of the honey bee division of labor (See Figure 4).
Interpretability of factors: To understand the relationship between the factors and division of labor, we calculate how the factors map to the fraction of time an individual spent on the brood area, honey storage, or dance floor (where foragers aggregate). Time spent on these different substrates is a strong indicator of an individual’s task. The factor f1, which peaks at young age (Figure 4), correlates with the proportion of time spent in the brood area, while a high f0 indicates increased time spent on the dance floor. Therefore, the model learned to map biologically relevant processes.
Individuality basis functions and individuality embeddings: Due to the regularization of the embeddings, the model learns a sparse set of individuality basis functions. As encouraged by the model, most individuals can predominantly be described by a single basis function. That means that while each honey bee can collect a unique set of experiences, most can be described with a few common individuality embeddings which are consistent across cohorts and colonies. In the context of honey bee division of labor, the basis functions are interpretable because the factors correspond to different task groups. For example, b12(c(t, i)) (accounting for≈ 10.7% of the individuals) describes workers that occupy nursing tasks much longer than most bees. As the individuality embeddings φ only scale the magnitude of the basis functions, they can be interpreted in the same way. Individual lifetime trajectories in the factor space can be computed based on the mean lifetime trajectories (m), individuality basis functions (b(c(t, i))) and individuality embeddings (φ). See figure 5 for examples
of individual lifetime trajectories from workers that most strongly corresponded to the common individuality basis functions.
Evaluation: We verify that the learned representations of the individuals are meaningful (i.e., they relate to other properties of the individuals, not just their interaction matrices) and semantically consistent over time and across datasets using the metrics described in the section Evaluation metrics. We compare variants of our model with different adversarial loss scaling factors and factor L1 regularizations, the baseline models, and the individuals’ ages. We expect a good model to be temporally consistent and semantically meaningful. All variants of our model outperform the baselines in terms of the semantic metrics Mortality and Rhythmicity, except for the Yu et al. (2017) model, which performs comparably well in the Mortality metric. The adversarial loss term further increases the Consistency metric without negatively affecting the other metrics. A very strong adversarial regularization (see row with λadv = 1 in Table 2) prevents the model from learning a good representation of the data. See Table 2 for an overview of the results. We also evaluate the tradeoff between the different metrics using a grid search over the hyperparameters (see appendix A.3).
Scalability: The functions m(c(t, i)) and b(c(t, i)) are learned neural networks with non-linearities. The objective is non-convex and we learn the model parameters using stochastic gradient descent. Optimization is therefore slower than the standard NMF algorithms that can be fitted using algorithms such as Alternating Least Squares (Kim et al., 2014). We found that the model converges faster if the reconstruction loss of the age based model m(c(t, i)) is additionally minimized with the main objective in equation 5. Due to the minibatch training regime, our method should scale well in larger datasets. Small neural networks were sufficient to learn the functions m(c(t, i)) and b(c(t, i)) in our experiments. Most of the runtime during training is spent on the matrix multiplication f+(t, i) · f+(t, j)T and the corresponding backwards pass.

heading: 4 CONCLUSION

text:
Temporal NMF factorizes temporal matrices with overlapping and even disjointed communities by learning a common embedding of the lifetime development of the individual entities. In the context of honey bees, this embedding is biologically meaningful, consistent over time, and shows that interaction patterns follow a common lifetime trajectory. Differences from the mean are described in a coordinate system of individual variability. Honey bee colonies are known to exhibit suites of correlated traits (Wray et al., 2011), but this method provides a perspective at the individual level and opens the possibility for understanding how colony-level traits respond to biotic and abiotic pressures. The basis functions are interpretable with respect to the division of labor within colonies and offer a valuable tool to understand and quantify the influence of experimental manipulations (e.g. pesticides or increased predatory pressure) on an individual, even between experimental trials on completely different sets of individuals. While we applied our method to honey bees as an exemplary system
with many individuals that exhibit an entangled, non-overlapping social structure, our method can be applied to any setting in which some interaction structure follows a general pattern over an observable (such as time) to detect structured deviations at the individual level.

title: INDIVIDUALITY IN THE HIVE - LEARNING TO EMBED LIFETIME SOCIAL BEHAVIOR OF HONEY BEES