  Deep Neural Networks (DNNs) have drawn attention because of their outstanding
performance on various tasks. However, deploying full-fledged DNNs in
resource-constrained devices (edge, mobile, IoT) is difficult due to their
large size. To overcome the issue, various approaches are considered, like
offloading part of the computation to the cloud for final inference (split
computing) or performing the inference at an intermediary layer without passing
through all layers (early exits). In this work, we propose combining both
approaches by using early exits in split computing. In our approach, we decide
up to what depth of DNNs computation to perform on the device (splitting layer)
and whether a sample can exit from this layer or need to be offloaded. The
decisions are based on a weighted combination of accuracy, computational, and
communication costs. We develop an algorithm named SplitEE to learn an optimal
policy. Since pre-trained DNNs are often deployed in new domains where the
ground truths may be unavailable and samples arrive in a streaming fashion,
SplitEE works in an online and unsupervised setup. We extensively perform
experiments on five different datasets. SplitEE achieves a significant cost
reduction ($>50\%$) with a slight drop in accuracy ($<2\%$) as compared to the
case when all samples are inferred at the final layer. The anonymized source
code is available at
\url{https://anonymous.4open.science/r/SplitEE_M-B989/README.md}.

Machine Learning;Artificial Intelligence;Computation and Language;Neural and Evolutionary Computing