WattsApp: Power-Aware Container Scheduling

Containers are becoming a popular workload deployment mechanism in modern distributed systems. However, there are limited software-based methods (hardware-based methods are expensive requiring hardware level changes) for obtaining the power consumed by containers for facilitating power-aware container scheduling, an essential activity for efficient management of distributed systems. This paper presents WattsApp, a tool underpinned by a six step software-based method for power-aware container scheduling to minimize power cap violations on a server. The proposed method relies on a neural network-based power estimation model and a power capped container scheduling technique. Experimental studies are pursued in a lab-based environment on 10 benchmarks deployed on Intel and ARM processors. The results highlight that the power estimation model has negligible overheads for data collection - nearly 90% of all data samples can be estimated with less than a 10% error, and the Mean Absolute Percentage Error (MAPE) is less than 6%. The power-aware scheduling of WattsApp is more effective than Intel's Running Power Average Limit (RAPL) based power capping for both single and multiple containers as it does not degrade the performance of all containers running on the server. The results confirm the feasibility of WattsApp.

INTRODUCTION
Container technology is a lightweight virtualization technique that has low overheads when compared to Virtual Machines (VMs) [28]. erefore, they are becoming popular for deploying workloads on clusters and clouds [30,35] and for upcoming distributed systems that use the edge of the network [33,34].
Container scheduling is an important avenue explored in the literature for distributed systems. Existing container scheduling strategies consider a number of relevant parameters, including resource demand, service level agreements and hardware/so ware requirements [19,38]. However, container scheduling like any deployment strategy need to be power-aware so that the total power consumption of a system does not exceed prede ned power cap limits.
Modern processors are equipped with power capping techniques, such as Dynamic Voltage and Frequency Scaling (DVFS) and Running Average Power Limit (RAPL) [22,37]. ese are hardwarebased and reduce the CPU frequency and voltage to lower processor power consumption. However, this degrades the entire system performance and consequently the deployed application.
It is valuable to gather the power consumption of individual containers running in a system so that they can be scheduled in a power-aware manner. However, there are limited so ware-based methods that measure container power consumption. Commercial vendors of Uninterrupted Power Supply (UPS) employ so warebased power estimation techniques based on static information, such as input voltage for di erent types of devices (for example, laptops) 1 , but ignore the workload-level granularity of power estimation. Other so ware based approaches, such as cWa s [26], cWa s++ [27] and SmartWa s [13] are either CPU architecture speci c, do not capture all components of the system that contribute to container power consumption, and are intrusive methods (further considered in Section 7). is fundamental gap is addressed in this paper by developing WattsApp, underpinned by a six step so ware-based (not hardware-based since they are expensive and require hardware level changes), hardware architecture agnostic and relatively non-intrusive power-aware container scheduling method. e aim is to estimate power consumption of containers and schedule power capped containers to stay within safe power budgets.
e research contributions of WattsApp are as follows: (i) A six-step so ware-based power-aware container scheduling method that accurately predicts power consumption of containers. e proposed method has negligible overheads (in relation to system power consumption) for collecting data required for estimating container power consumption. Additionally, nearly 90% of all data samples can be estimated using the power model with less than a 10% error. e Mean Absolute Percentage Error (MAPE) is observed to be between 1%-6%, which is relatively low. WattsApp is the rst prototype that builds power models and enforces power capping for parallel applications that execute on a cluster of containers.
(ii) e proposed power-aware method of WattsApp implements power capped scheduling for both single and multiple containers on the same server. e proposed power capping method is more bene cial than when no power cap or Intel's RAPL power cap is employed since the performance of all running containers on the system is not degraded (only containers that violate the budget are penalized). Potential approaches based on containers to achieve the power cap are to migrate the container to another server or deallocate resources of the container that violates the power cap. Experimentally, deallocating resources is a more viable approach than migration due to the inherent limitations of migrating containers. e remainder of the paper is organized as follows. Section 2 discusses the motivation for WattsApp. Section 3 proposes the power-aware container scheduling method. e underlying power model is discussed in Section 4. Section 5 presents the power capped container scheduling approach. Section 6 presents experimental studies. Section 7 discusses the related work. Section 8 concludes the paper by considering future work.

BACKGROUND
Predicting container power consumption is complex because it depends on the resource allocated to it and the workload running in the container. It is di erent when compared to the power prediction of VMs, other processes and hardware (processors, memory etc) because of the limited availability of data about the resource utilization and hardware performance counters speci c to containers.
is is because containers create multiple processes on the host operating system (OS). e number of processes varies depending on the activity that is performed within the containers. is paper observes that containers with more allocated resources consume more power for the same workload than on containers with fewer resources. However, in all cases the power consumed does not correspond to the increase in resources (CPU cores, memory). For example, the average power of an application running on twice the resources as another container, may not necessarily directly correspond to a factor of two.
is hypothesis is veri ed on 10 di erent scienti c workloads that are listed in Table 1. ese workloads are obtained from two sources. e rst is DCBench, a bench marking suite [17] from which ve MPI based workloads are chosen. e second is a collection of C/C++ based scienti c programs 2 . ese workloads are 2 h p://people.sc.fsu.edu/ ∼ jburkardt/c src/c src.html a combination of CPU-bound, I/O-bound and memory-bound scienti c workloads that execute to completion. is paper does not consider alternate classes of workloads, such as Internet-of-ings, stream processing, or sensor-based applications. e workloads considered in this paper may have di erent power consumption pa erns during execution. is is captured in the resource utilization and power data that is collected at a ne granularity and used for building the power model. is ensures that estimation can be carried out for di erent potential phases of a workload. Figure 1 highlights the average, minimum and maximum power consumed by a container with 3 CPU cores and 4GB RAM. Figure 2 provides results for the same workloads for twice the resources (6 CPU cores and 8GB RAM) It is evident that although the resources allocated are doubled the average power consumed does not necessarily double for all workloads (for example, refer to the workloads HEATED, MD, POISSON, PRIME and SGEFA).
Similar trends are obtained when the power consumption is noted for the above workloads over time (the results are exhaustive and are not presented in this paper). When more resources are added parallel workloads (applications running within a single container) execute faster, but reach their peak power consumption at di erent times. is paper does not aim to explain individual power pro les of workloads, but notes that a server that executes multiple containers could violate the power cap; speci cally, when multiple large size containers are multi-tenant on a server. ese large containers may consume high power and when they are multitenant their combined total power consumption could be higher and close to the maximum power consumption of the server.
Power cap violations are undesirable and need to be e ectively managed on servers running di erent workloads. ey occur when the total power consumed by a server exceeds a threshold de ned by the server administrators. When power cap violations occur, the server performance starts degrading since power management techniques like Dynamic Voltage and Frequency Scaling (DVFS) that are bundled with processors come in to play. DVFS reduces the server power consumption by using two power saving techniques, namely dynamic voltage scaling and dynamic frequency scaling [22]. Power saving is achieved by lowering the frequency and/or voltage of the CPU and other system resources. is reduction negatively impacts the performance of workloads executed on the server. For example, the performance of a container running on a server will drop when there is a power cap violation.
In order to avoid the above, a power aware container scheduling strategy is required. It is observed that power/energy saving bene ts of techniques like DVFS are diminishing because of the complexity in hardware technologies used for processors and memory (for example, increased memory performance and multi-core processors) [22]. erefore, a so ware-based power capping technique is desirable in addition to speci c hardware-based techniques.
is motivates the power capping technique proposed in this paper.

THE WATTSAPP METHOD
is section presents a method for so ware-based power aware scheduling of containers to minimize power cap violations on a server, which is fundamental in developing WattsApp. Power aware container scheduling is the distribution/consolidation of containers such that the total power consumed by a server does not cross a prede ned threshold (or cap) speci ed by an administrator.
A primary requirement for the WattsApp power aware container scheduling approach is obtaining information on the power consumed by an individual container. Resource utilization statistics of each container running on a server is used to calculate its power consumption. is information is used for container scheduling, such that the maximum power consumed does not violate any power restriction on an individual server.
Currently, there are no hardware methods for obtaining the power consumed by containers. Moreover, there are a few so ware methods to measure the container power consumption and these methods have concerns like they are architecture speci c, ignores essential system resources or intrusive as discussed in Section 7.
is article aims to bridge this gap. Hardware-based methods will require modi cation of the hardware (such as additional probes) resulting in more expensive processors. Hence, a so ware-based method is adopted to develop a model of container power consumption that depends on resource utilization information of the container. e model uses linear regression-based neural network to correlate container resource utilization statistics with system power consumption information to estimate the container power consumption. e model is further presented in Section 5.
e WattsApp method relies on two activities: (i) Container power prediction, which is estimating the power consumed by an individual container using so ware-based methods, and (ii) Power capped container scheduling, which is using the estimated power values to place containers equitably on a server. e importance of such a power-aware container scheduling method is that if the power cap exceeds on any server, then the performance of all containers running on the server will be degraded. To mitigate this, any power cap violation is periodically detected on each server by observing the total power consumption of the server. If the power consumed is above the threshold, then it is considered as a power cap violation. When a violation is detected, power capping is performed on the server without signi cantly a ecting the run time performance of all running containers.
Power capping in WattsApp is performed using two approaches, namely container migration and container resource (CPU cores) reduction. e method aims to nd a server on to which a container causing the server power capping violation can be migrated. If such a server is available, then the container is migrated to the identi ed server thereby avoiding any power cap violation.
If no servers are available to migrate a container, then a second approach is used by reducing the resources allocated to the container. e processor subsystem consumes nearly 85% power of the total system power [24]. erefore, to achieve power capping the CPU cores allocated to the container are rstly reduced one at a time until the power cap limit is restored. WattsApp uses technology speci c commands (such as docker update command) to change the cores allocated to a container; the change is immediately re ected. As a result the container will use fewer CPU cores, which eventually reduces the server power consumption.
When a container is migrated to a di erent server, the power consumption of the source server is again reduced below the power cap limit. A detailed discussion on both container migration and CPU core reduction is provided in Section 5. e proposed method of WattsApp shown in Figure 3 comprises six steps: namely Data Collection, Model Building, Power Estimation, Power Capped Allocation, Violation Detection and Power Capping. e rst three steps are for container power prediction, and the remaining steps are for power capped container scheduling.
Step 1 -Data Collection: e training data for supervised learning is collected to estimate the power consumption of containers. It collects system power consumption and resource usage statistics for each container from the host OS. is is correlated with the system power consumption to obtain container power consumption.
e system power consumption information is necessary as the regression techniques require labeled data for building a model. System power consumption data is collected from Wa s Up .net hardware power meter. It facilitates the power consumption sampling at a one second granularity. Real time system power consumption data can be obtained by connecting it through a USB interface. It is reported that the accuracy of this hardware meter is +/-1.5%+0.3W 3 [16]. e readings provided are also considered to be generally reliable although a high error rate is observed for readings below 1 Wa [16]. A series of power meters from Wa s Up are used in research reported in the literature [23][1][6] [3].
Step 2 -Model Building: e collected data is used to train data and build a neural network for individual containers running on the server. We build a model for each container on the server as di erent applications exhibits di erent properties. ese models will be utilized during run time for predicting the power consumption of the container. e input to the neural network is container resource utilization statistics (including the percentage of host CPU and memory the container is using, the total memory the container is using and the maximum allocated memory, the amount of data the container sent and received on its network interface, and the amount of the data read from and wri en to the block input output devices) and system power consumption.
e input data to the model is obtained in the previous step. e model (linear regressionbased) was developed using the Keras 4 deep learning library.
Step 3 -Power Estimation: Consider there are n containers (C 1 , C 2 , C 3 , · · · C n ) running on a server. e resource utilization statistics (CPU usage, memory usage, amount of block I/O and network data transfer) are collected for each container. e models developed (previous step) and the run-time statistics are used to predict the power consumed by the container.
Step 4 -Power Capped Allocation: e models are used for power capped container scheduling. It uses the predicted power consumption of the container and current total power consumption of the server before scheduling the container on the server. e power consumption information of the containers are obtained by using the power estimation model that uses the power pro le of each container for power capped allocation.
Step 5 -Violation Detection: To perform power capped container scheduling a power cap violation has to be detected. A er initial scheduling of a container, this step is executed at a ve minute interval to check the server for any power cap violation. If there is a violation, then the nal step enforces the power capping limit on the server.
Step 6 -Power Capping: is nal step adopts two techniques to enforce power capping. e rst is referred to as migrationanother server that can accommodate the container causing the power cap violation on a current server is identi ed; migration should not violate the power cap of the recipient server. If such a server is available, then the power cap violating container is migrated to the identi ed server. If no such server is identi ed, then a second technique, referred to as resource deallocation, is performed in which the number of CPU cores allocated to the container is reduced until the power cap limit is reached. is rst prototype of WattsApp only considers a single container causing power cap violations. However, if multiple containers cause power 3 h ps://www.vernier.com/ les/manuals/wu-pro.pdf 4 h ps://keras.io/ cap violation, then a priority based container selection approach is required, which is not considered in this paper. e underlying approaches of the rst three steps are presented in Section 4. e last three steps are further discussed in Section 5.

WATTSAPP CONTAINER POWER ESTIMATION
is section describes the Data Collection (Step 1), Model Building (Step 2), and Power Estimation (Step 3) steps of WattsApp.
e Data Collection step gathers (i) the system power consumption data, and (ii) resource utilization data of running containers.
is is a black box technique as the data is collected from the host operating system and no pro ling data is obtained from within the container. e other approaches (referred to as white box) collect pro ling information inside the container and should be avoided to maintain the integrity of the containers [14].
Power consumption data is collected from the Wa s Up .net power meter. e data obtained contains the time stamp and power consumption (in Wa s). e resource utilization data of the container is collected using the docker stats command, which provides the following data: (i) Id of the container and the name of the container, (ii) Percentage of host CPU and memory the container is using, (iii) Total memory the container is using and the maximum allo ed memory, (iv) e amount of data the container has sent and received on its network interface, (v) e amount of the data read from and wri en to the block input/output devices, and (vi) e number of processes/threads created by container. e data collection time stamp is also added to the output of docker stats. Both the power consumption data and the resource utilization data of the container is concatenated with respect to the time stamp. Multiple CPU cores allocated to the container are taken care of by the host CPU usage (for example, it ranges from 0 to 100% for one core, up to 200% for two cores and so on. Resource utilization and system power consumption data are obtained once per second during the execution of the benchmarks. e sequence of steps is presented in Algorithm 1. e inputs to the model are the container resource utilization (including percentage of host CPU and memory the container is using, total memory the container is using, maximum allo ed memory, amount of data the container has sent and received on its network interface, amount of the data read from and wri en to the block input output devices), and system power consumption. e output is container power consumption.
In the Power Estimation step, the power consumption of containers is rstly modeled by WattsApp. e power consumption of a server (P ser er ) comprises static power and dynamic power. Static power (P st at ic ) is de ned as the power consumption of system when there is no active container. is power is measured by using the Wa s Up .net power meter. If there is only one running container then the total power consumption is sum of idle power consumption and dynamic power consumption of the server. e dynamic power consumption (P d namic ) is de ned as the power consumption of the running container.
If there are n active containers on the server, then the dynamic power consumption of the server is the aggregate power consumption of all the containers.
where P cont ainer is the power consumption of the container. e dynamic power consumption of the system is considered as the sum of the power consumed by the CPU, the memory (RAM), the disk and the network.
where a, b, c, and d are constants, n is the number of running containers in a server, l represents the number of CPU cores allocated to a container, U cpu is the CPU utilization factor, U ram is the RAM utilization factor, U disk is the disk utilization factor, and U net is the network utilization factor.
e case when a single workload is executed across a cluster of containers is considered by WattsApp. In this case, the workload's power consumption will be the aggregate power consumption of all the containers of the cluster.
where Container k is element of C, the set of containers in the cluster C = (Container 1 , Container 2 , · · · Container n ).

WATTSAPP POWER CAPPED CONTAINER SCHEDULING
is section presents the use of the estimated power values at run time for a proposed power capped container scheduling method of

Algorithm 2: Power Capped Scheduling of Containers
Input :S, C i j , md WattsApp. e approach adopted is to initially allocate containers using a best t or rst t strategy, and subsequently when there is a power cap violation on the server, migrate the container elsewhere or reduce the allocated CPU cores of the running containers until the power capping is not violated. Power Capped Allocation (Step 4), Violation Detection (Step 5), and Power Capping (Step 6) proposed in Section 3 is considered in this section. Power Capped Allocation uses the estimated power consumption of containers to schedule containers by calculating the total power consumption of the candidate server a er adding the estimated power consumption of the container ready for deployment. If the power consumption of the server is anticipated to be below the power cap limit, then the container will be placed on the candidate server. is is repeated for all containers that are ready for placement. Algorithm 2 highlights this and Table 2 presents the notation used in this algorithm and the other algorithms (Algorithm 3 and Algorithm 4) presented in this section. It is assumed that there are n servers, and each server may have up to m containers.
Algorithm 2 executes for all containers ready for placement (line 1). e ag variable is initialized to false (line 2); this variable will be used to identify the case when no suitable server for power capped placement. Each server is checked one by one whether it can accommodate the container under consideration (line 3). e container power consumption is added to the candidate server power (line 4) to check if it can accommodate the container (line 5). If the candidate server can deploy the container, then it is allocated to the server (line 6). When container placement is successful, the ag is updated to true (line 7). If it is not possible to allocate on a given server, then the remaining servers are processed. When no suitable server is found for power capped placement (line 12), the container is allocated to the server with the lowest power consumption (line 13 and 14). A er this, the Algorithm 3 will work to detect the possibility of power cap violation, and if required, the power cap is applied using Algorithm 4. Available memory in the server s i C i j List of containers deployed in server S i for j = 1, 2, · · · , m C List of all the containers ready to be placed ACr i List of CPU core allocation to each container AM i List of memory allocation to each container md Power consumption Model PC i j Power consumption of container c i j PS i Calculated total power consumption of server s i , ps i ε PS c j , pc j e candidate container and its power consumption causing the server power increasing beyond the power cap threshold cap Power cap A process to determine any power cap violation is executed on the servers, ve minutes a er initially scheduling containers (pro ling data is collected for the rst ve minutes). is process uses the power prediction model to estimate the power consumption of each running container on the server. If the power consumption of any server (sum of power consumption of all the running containers) is beyond the power cap, then there is a power cap violation caused by the newly placed container). is container will be considered as the candidate for migration or CPU core reduction.
Power capping is achieved in two ways. e rst is by migrating the candidate container from the source to a destination server whose current power consumption is below the cap and would not be violated if it accepted the container. A stateful migration method, namely 'CRIU (Check-point/Restore In Userspace)' is employed for migrating containers. e second is by reducing the resources allocated to the candidate container, speci cally the number of CPU cores (reduce one at a time) as CPU usage signi cantly a ects the power consumption of Docker containers [31]. e Docker update command is used to change the number of allo ed CPU cores to the container. e performance of the container will be degraded when using this approach (further considered in Section 6).
Currently, there is support for power capping on the hardware. However, most hardware power capping techniques tweak the voltage and processor frequency, which a ects the potential performance of the entire system and is detrimental to all containers running on the system [22]. However, the proposed so ware power capping technique achieves the power cap without signi cantly a ecting the entire system's performance and only negatively impacts the container that causes the power cap violation.
Violation Detection is to detect when a power cap violation occurs on any server under consideration. e detection algorithm is given in Algorithm 3 and runs on each of the servers. When a violation is detected, Algorithm 4 performs power capping. e model used to compute the power consumption of the container at run-time is considered in Section 4.

Algorithm 3: Power Cap Violation Detection
Input :S, C i j , md is is achieved by collecting the resource usage statistics for each container (line 4) and then passing the data to the power model for predicting power consumed (line 5). e power consumption of all containers running on a server are summed to obtain server power consumption (line 6) a er which power cap violation (if any) is checked for on each individual server (line 9 and 10). If a server crosses the power cap limit then Algorithm 4 is performed with the required inputs. If the power consumed by all containers on the servers is below power cap, then no changes are made (line 13). e last step is Power Capping that uses Algorithm 4 and the input provided by Algorithm 3. A server (line 1) that can accommodate the container without crossing the power cap (line 2) is searched for by checking with a network process that runs on each server. If successful, then the container that violates the power cap on a source server is migrated to the destination server (line 3). If no candidate destination servers are found, the algorithm uses the second option of reducing the allocated CPU cores to a container until the server power consumption falls below the power cap (line 7). For this, the allocated cores are rst reduced by 1 (line 8) and then the resource usage statistics are collected (line 9) and the power consumption is predicted (line 10). Again, if the current power consumption falls below the power cap (line 11) then the algorithm is successful and returns (line 12).
CPU core reduction to achieve power capping will degrade performance of the selected container. is can be compensated for by increasing the CPU cores at a later stage when it may be feasible to do so without exceeding the power cap. is scenario is considered in experimental studies to demonstrate that the impact of CPU core reduction of a container can be compensated when running the parallel component of an application in a cluster.
ere may be a delay in enforcing the power capping limit since the detection algorithm is only executed once every ve minutes (this time is a con gurable parameter of the algorithm to suit any bespoke requirements). Experimentation on the impact of this limit is not presented in this article. It was empirically observed that a er a power cap violation was detected, a few minutes were required for migrating the container to another server. e time taken for migration is due to: (i) Creating checkpoints -a checkpoint freezes a running container and turns its state into a collection of les on disk, (ii) Compressing the checkpoint and transferring it to the selected server, and (iii) Creating a new container and restoring the checkpoint. e majority of the migration time (depends on the size of the container image) is for transferring the checkpoint to the destination server.

PERFORMANCE EVALUATION
e experiments highlight that for WattsApp: (i) the overheads for data collection to estimate power do not signi cantly impact system power consumption, (ii) there is limited error in estimating power using the model, (iii) WattsApp operates across multiple processors, and (iv) the proposed power capping method is more e ective for scheduling than alternate methods, such as Intel's RAPL. e experiments are conducted on two systems with di erent form factors. e rst is workstation Dell Precision 3630 with an Intel Xeon E-2174G processor and 16GB memory. e second is a small form factor Odroid N2 Board with a quad-core ARM Cortex-A73 CPU cluster and a dual core Cortex-A53 cluster and 4GB memory. e system power consumption data is collected using Wa s Up .net hardware power meter. e systems run on Ubuntu 18.04. e containers are created with Ubuntu 18.04 LTS image. Each container is allocated three CPU cores, 4GB memory on workstation and 2 CPU cores and 2GB memory on Odroid. Docker 17.12.0-ce version is used to deploy the containers. Keras 2.2.0 that runs on TensorFlow 1.8.0 is used to build the power model. e hardware power meter Wa s Up .net is used to obtain system power consumption in real-time. Wa s Up .net power is used to collect the power readings using USB from the host OS by a Python script that reads instantaneous power data.  Table 1 are used for evaluating the WattsApp method. ese are scienti c workloads that execute to completion. is paper does not consider alternate workloads, such as Internet-of-ings, stream processing, or sensor-based applications.
e experiments are carried out for single workloads on single containers, multiple workloads on multiple containers, and single workload across multiple containers (a cluster of containers) to thoroughly evaluate the WattsApp method.
Results: e data collection overheads and estimation error in container power prediction is rstly presented. en, the results from scheduling for power capping obtained for single and multiple containers are considered. e average system power consumption overhead when collecting data (CPU, memory, and disk utilization along with power) for a 89 second time period is shown in Figure 4. e blue plot shows the average system power consumption when only power data is collected, and the orange plot shows when both resource utilization and power metrics are collected. On an average, nearly 0.2 Wa s are spent. e graph illustrates that the overhead in terms of system power is negligible (less than 1% of system power consumption).
is is an indicator that Step 1 of the proposed power-aware scheduling method is feasible. Figure 5 shows the data collection overhead on the ARM processor; the overhead is nearly 1.7% of the system power consumption. Figure 6 shows the distribution of error on the Intel Xeon processor in the power values that is estimated for 444 samples using the neural network model. In this experiment, data collected from all workloads (Table 1) is consolidated to build the model that is validated using repeated random sampling by spli ing data into 75% and 25% as training and testing dataset respectively. More than 90% of the samples have an error of less than 10% and nearly 49%  of the samples have less than a 6% error. is highlights that the power model built in Step 2 of the method will have a reasonable accuracy for prediction in Step 3. Figure 7 highlights the Mean Absolute Percentage Error (MAPE) of the model for estimating power of individual workloads executing in a container on the Intel Xeon processor. For this experiment each workload is executed in a single container and the power is estimated for each container. MAPE indicates the average of percentage errors (a lower value indicates that the model estimates the power consumed with a higher accuracy). e average percentage error is between 1% and just over 6%.
Similar experiments are performed on the Odroid board with ARM processor using six workloads from (Table 1) (the four workloads, namely FUZZYKMEANS, KMEANS, KPCA and PCA, are distributed in DCBench with the binaries for the x86 platform). Figure 8 shows the error distribution of estimating power values for 400 samples using the neural network model. In this experiment, data collected from the six workloads is consolidated to build the model that is validated using repeated random sampling by spli ing data into 75% and 25% as training and testing dataset, respectively. More than 80% of the samples have an error of less than 15% and more than 60% of the samples have less than a 10% error. is highlights that the power model from Step 2 will have a reasonable prediction accuracy in Step 3. Figure 9 highlights the Mean Absolute Percentage Error (MAPE) of the model for estimating power of individual workloads executing in a container on the ARM processor. For this experiment each workload is executed in a single container and the power is estimated for each container. MAPE indicates the average of percentage errors (a lower value indicates that the model estimates the power consumed with a higher accuracy). e average percentage error is between 1% and just over 4%. Figure 10 shows the overheads associated with the two techniques adopted in power capping, namely migration and deallocation of resources for di erent sizes of containers. e time taken to migrate using the Checkpoint/Restore in Userspace approach provided by Docker is directly proportional to the size of the container as the container needs to be checkpointed and migrated to an alternate server. However, using the time taken to deallocate resources on a container on the server takes approximately 180 milliseconds. Although migration is a potential option to achieve the power cap, the results show that deallocating resources is a more viable option given the inherent overheads in container migration. In the next set of experiments, power capping results based on only resource deallocation is presented. Migration using containers is a less viable option based on existing technology (if a critical application has to be executed) given large migration overheads although it may be lower than VMs (also not suited for single parallel application executed across a cluster of containers).
In another experiment, a cluster of containers (four on the Intel processor and two on ARM) was created for running the MPI applications from Table 1. Figure 11 shows the results on the Intel processor to demonstrate the feasibility of container power prediction for parallel applications executed across multiple containers. e average MAPE is 3 with error between 1 % and around 5.5%. Figure 12 show the MAPE on ARM processor. e average MAPE 2.6 with error between 1 % and 4 %. e accuracy of WattsApp power estimation is considered for di erent input parameters when the benchmark is executed in a e data for the input parameters were not used during training. e results highlight that the average error percentage is between 0.8% and 4% for both Intel and ARM processors.
Two further experiments were carried out on the Intel processor to identify the potential bene t of the proposed power capping based container scheduling compared against when no power caps and Intel's RAPL-based power cap is employed. e rst experiment is when a single container executes on the server with a given workload. In this experiment, there is only one container running on the server that is likely to violate the power cap. e second experiment is when multiple (three) containers that run the same workload execute on the server. ere are multiple containers running on a given server, and any one of them may violate the power cap. Each container executes the same workload. Figure 15 shows the results for the rst experiment in which a single container executes on the server with a given workload. e graph shows the workload execution time for the proposed power cap method, no power cap, and RAPL's power cap is adopted. In this case, it is noted that the proposed power capping method is more e ective than RAPL's power cap since the total workload execution time is lower in every case. is highlights the bene t of using the proposed power-aware container scheduling method. Figure 16 shows the results for the second experiment in which three containers (C1, C2, C3) with the same workloads execute on the server. ese workloads are more representative of a real world scenario. e graph shows the workload execution time when the proposed power cap method, no power cap, and RAPL's power cap is adopted. It can be clearly seen that when RAPL is employed the workload execution time of all containers increase.
is is because RAPL achieves power capping by reducing the CPU frequency of the server, which in turn a ects the performance of all containers running on the server. On the other hand, it is noted that the proposed power cap technique reduces the allocated number of CPU cores, thus degrading the performance of only one container as opposed to all the running containers. erefore, 'C3' in many cases is noted to take longer than the other containers. In short, only one container out of many (that potentially violates the power cap) is penalized when using the proposed power cap technique. Figure 17 shows the peak power consumption on the Intel Xeon processor of applications (from Table 1) when there is no power (a) For DCBench programs from Table 1 (b) For MPI-C programs from Table 1 Figure 15: Execution time of individual workloads when a single container executes on a server using the power cap technique on the Intel Xeon processor (a) For DCBench programs from Table 1 (b) For MPI-C programs from Table 1  is experiment uses the power cap limit of 55W. e average peak power consumption of the proposed power capping technique is 56.4W which is close to the power cap limit where as the average peak power of RAPL's power cap is 60.2W and signi cantly higher than the power capping limit. e peak power consumption for WattsApp is 60.3W in comparison to the peak power consumption of RAPL's power cap is 65.9W.
Similar experiments are carried out on the Odroid board to demonstrate the e ectiveness of WattsApp power capping on ARM processors. Again only six workloads from Table 1 are used. As RAPL is speci c to Intel, these experiments only compare the WattsApp power cap with no power cap.  Figure 18 shows the results of the experiment in which a single container executes on the ARM processor. e workloads running under WattsApp power cap takes slightly longer time and executes within the power budget de ned by power cap limit of 7W. Figure 19 shows the results when two containers with the same workload executes on the ARM processor. e graph shows the workload execution time under the no power cap and WattsApp power cap. Again workloads running under WattsApp power cap takes longer time and the power budget remains below the power cap limit of 9W. is demonstrate that WattsApp power capping in also e ective on ARM processors. Figure 20 shows the peak power consumption of the six workloads from Table 1 when there is no power capping and under the WattsApp power capping regime. is experiment uses the power cap limit of 9W. e average peak power of the proposed power capping technique is 8.1W (below the power cap limit), but is 9.2W when there is no power cap. Figure 21 and Figure 22 demonstrates another aspect -when a single MPI application is executed across a cluster of containers.
is experiment considers that workloads are running on a cluster of two or more di erent servers. When the CPU cores of the workload need to be reduced on one server, then it is compensated for by increasing the CPU cores allocated on the other server for the workload. e result shows that power capping on one server with compensatory allocation on another server does not signi cantly impact performance. Summary: In short, the experimental results highlight that: (i) e data collection overhead in the proposed power-aware container scheduling method of WattsApp only a ects the system power consumption negligibly. (ii) Nearly 90% of data samples are estimated with less than 10% error. (iii) e MAPE of power estimation using the model that is employed by the proposed power-aware container scheduling method of WattsApp is between 1%-6%. is is relatively low and accurate estimations can be expected from the model. (iv) e power estimation method of WattsApp is also validated on the parallel applications across a cluster of containers. e proposed model estimate the power consumption with MAPE between 1% to 5.5%. e impact of power capping (CPU core reduction) for parallel workloads is minimized during the workload runtime by applying compensation. (v) Deallocation of resources are found to be a more feasible approach than migrating containers for the power capping technique given that the overheads for migration increase with the size of the container. e overheads for deallocating resources is negligible. (vi) When both single and multiple containers are executed on the server, it is noted that the proposed power cap method is more bene cial than when no power cap or RAPL's power cap is employed since the proposed method does not degrade the performance of all running containers to keep the power budget below the cap. WattsApp's power capping is also e ective since the peak power allowed by the WattsApp method is less than that of RAPL's power cap and does not violate the so power cap that may be imposed by administrators.

RELATED WORK
Power modeling of processors and VMs are well explored, but power modeling of containers is still in its early stages. In this section, the impact of virtualization techniques on system power consumption that do not use estimation models is discussed. en container power modeling techniques and power models for processors/servers and VMs are considered. Finally, research on container power capping is discussed.
Impact of virtualization techniques on system power: Power consumed by servers running containers has been experimentally measured without developing power models. e CPU usage of the container has a signi cant contribution to the overall power consumption [31]. Empirical investigation on four virtualization technologies, namely Xen, KVM, Docker and LXC is noted [25]. It was observed that for CPU workloads there is no signi cant di erence in power consumption among the above technologies. However, containers consume less power than other virtualization technologies for network-based workloads.
A comparison of the server power consumption [32] and energy comparison [18] has been presented for virtualization and containerization technologies. e power and energy characteristics of four hypervisors and a container engine including VMware ESXi, Microso Hyper-V, KVM, XenServer and Docker on six different hardware (three mainstream 2U rack servers, one emerging ARM64 server, one desktop server, and one laptop) is considered. It is observed that hypervisors exhibit di erent power and energy consumption pro les when the same workload is executed on the hardware. Although containers are light weight, they are not necessarily more power-e cient than VMs. Similar comparisons of running workloads on containers and bare-metal servers are considered [29]. It is observed that running Docker has an inherent power cost and thus energy consumed is higher than bare-metal.
Container Power Modeling: SmartWa s [13] is a self calibrating so ware power model for containers that relies on hardware performance counters and RAPL's power measurement of CPU and DRAM for estimating power. Using RAPL limits the applicability of SmartWa s to Intel architectures. It also does not capture the impact of disk access and network usage on power that may be the main activity of an I/O or a network based application. e power model of WattsApp on the other hand uses architecture agnostic parameters to model system power and its feasibility on multiple hardware platforms is demonstrated.
Lightweight power models, such as cWa s+ [26] and cWa s++ [27] are developed for containers. cWa s++ is a virtual power model that has two components: a client back-end and a server front-end. e client back-end is installed in the container and accesses the CPU event counters. cWa s++ uses two models, namely an event-based and RAPL-based models.
e event-based model uses CPU performance counters and RAPL-based models uses only RAPL event counters. e evaluation shows that the two power models are useful on workloads obtained from the PARSEC and in-house benchmarks. However, cWa s only uses CPU related metrics to compute container power from server power. CPUs are a major power consuming component of a typical server (nearly one-third [10] and even up to 40% [24] of the total server power), but other components need to be considered. Hence, WattsApp considers memory, IO and the network to account for container power. Moreover, cWa s is intrusive and requires client installations and access to containers (this may not be always feasible depending on access permissions, ownership models, business models etc).
ere is research that accounts for the power consumption of individual threads and application containers [8].
e research relies on power estimation of each CPU core obtained from Intel's RAPL and hardware performance counters (related to CPU events) obtained from the OS. A power-aware consolidation technique of containerized data centers based on a model built using CPU utilization is presented [21]. Both approaches are based on CPU related metrics and do not account for the power consumed by other components [24]). WattsApp on the other hand considers CPU, memory, disk and network related metrics.
Power Models for VM and Processor Power Models: Joulemeter [20] is a so ware power meter for VMs. Joulemeter estimates VM power using linear regression with ordinary least square to correlate VM pro ling data to system power consumption. BITWATTS uses a two-level approach to estimate the process level power estimation based on CPU power consumption is developed [10]. is approach performs pro ling at both system and VM level. e system-level pro ling estimates the system-level power consumption whereas, the VM level pro ling estimates the power consumption of hosted applications. BITWATTS uses a regression technique for learning the process level power models.
Another approach uses a tree regression-based technique for VM power metering [15]. It is suggested that the linear regression methods are not su ciently accurate and therefore an approach that recursively partitions the collected data into easy modeling pieces has been proposed. is method rst, builds the server model using the observation for server and then fairly divides this server power consumption among the virtual machines.
iMeter is a performance counters based VM power model based on polynomial kernel based support vector regression [36]. Principal component analysis is performed to select performance counters that have the most impact on power consumption.
Similarly, there are power models for processors and the components of a server. A two stage cross architecture power model is discussed in [9]. is model takes advantage of both linear regression and support vector machines to provide power estimation with a high accuracy across multiple hardware architectures.
A con gurable learning framework named PowerAPI that automatically builds the power model of a CPU is presented [11]. PowerAPI automatically explores the available hardware performance counters of a CPU and selects the performance counter having the most impact on the power consumption of the server.
ere are a number of other notable processor power consumption models proposed by researchers (for example, [5][6][7]12]), which are not considered within the scope of the discussion of this paper.
However, WattsApp is a so ware power model that uses container resource utilization statistics obtained from the host OS to estimate the container power consumption.
Container Power Capping: ere is limited research in the literature that focuses on power capping for containers. Two power capping techniques are proposed in literature: e rst is a power capping technique (DockerCap) for Docker containers [3]. e system power consumption is obtained from the hardware power meter and RAPL. e CPU quota of all the containers of di erent priority is reduced, thereby a ecting the performance of all the containers. e WattsApp method however uses two techniques, namely container migration, and CPU core reduction to achieve power capping. e merit of the proposed method is that the overall container performance is una ected and only the container that violates the power cap is degraded. e second power capping technique is proposed for Docker containers on the Kubernetes platform [2] by relying on DEEP-mon power monitoring [8]. is technique relies on RAPL and DVFS to manage power cap limits. It is demonstrated in this paper that using RAPL a ects the run-time performance of all containers running on the server. RAPL enforces a power cap on the processor package and DRAM by reducing the CPU frequency and thus degrades the overall system performance. However, WattsApp uses architecture independent metrics to measure resource utilization (CPU, memory, disk and network) making the approach portable. e proposed method is demonstrated to be e ective for both power-aware scheduling and capping.

CONCLUSIONS
is paper proposes WattsApp that is underpinned by a six-step power-aware scheduling method for containers to minimize power cap violations on a server in real-time.
e method relies on a neural network-based power estimation model. e trained model e ectively predicts over 90% of data samples with less than 10% error. By testing on 10 representative benchmark workloads, the approach is able to achieve a MAPE error of less than 6%, and displays minimal overhead during run time scheduling. Unlike hardware-based power capping techniques, such as Intel's RAPL, which are indiscriminate to workloads and degrade the overall performance of all containers running on a server, this so ware-based approach is able to target individual containers running workloads, minimizing overall processing degradation while maintaining a node's power budget. e proposed technique considers multiple scenarios, including (i) single/multiple application, single container and single application, multiple containers. WattsApp has been shown to be feasible and outperforms existing techniques.
Future Work: WattsApp is tested on two popular processor types. However, to deal with heterogeneous platforms the power model will need to be expanded further to accelerator architectures. e current method prioritizes the power budget of an individual server, but not the performance or SLA of the container workloads.
Be er understanding, pro ling, or feedback (for example, as presented in Barrel sh [4]) from an application would enable be er choices within the scheduler and is an avenue for future work.
Finally, this technique will be applied to edge computing, where power is a critical concern. As containers are being increasingly used in this space, this research is directly applicable. Alternate types of workloads, such as stream processing and sensor-based applications will be considered.


Self-labeled Conditional GANs

This paper introduces a novel and fully unsupervised framework for conditional GAN training in which labels are automatically obtained from data. We incorporate a clustering network into the standard conditional GAN framework that plays against the discriminator. With the generator, it aims to find a shared structured mapping for associating pseudo-labels with the real and fake images. Our generator outperforms unconditional GANs in terms of FID with significant margins on large scale datasets like ImageNet and LSUN. It also outperforms class conditional GANs trained on human labels on CIFAR10 and CIFAR100 where fine-grained annotations or a large number of samples per class are not available. Additionally, our clustering network exceeds the state-of-the-art on CIFAR100 clustering.

Introduction
Generative adversarial networks(GANs) [15] are known as one of the most useful subset of generative models. Stan-dard GANs are independent of any kind of human supervision, involving adversarial training of a generator that synthesizes synthetic images given a low dimensional noise vectors, and a discriminator that aims to distinguish between synthetic and real images. They are also called unconditional GANs(uGANs), as there is no common source of information between the discriminator and the generator. Despite the high capability of GANs to generate realistic images, they are notorious for their instability and prone to mode collapse [1].
Conditional GANs(cGANs) [25] have effectively addressed the aforementioned challenges. In the cGAN framework, both the generator and discriminator are fed with an auxiliary input representing some factor that contributes to observations, e.g. object labels or data from different modalities. The discriminator aims to distinguish real image/conditioning term pairs from the fake pairs. Conditional GANs guide the training by considering different modes of the data distributions. When conditioned on labels, cGANs explicitly approximate the joint distribution of observations and labels. For the sake of simplicity, we denote cGANs as those conditioned on class labels throughout 1 the paper.
Although cGANs benefit from a rich prior, their drawback lies in dependency on human labels. To cope with this limitation, we propose Self-labeled Conditional GANs(slcGANs) that learn to assign labels to the images automatically by incorporating an additional clustering network. The benefits are twofold. First, training is independent of human labels. Second, the slcGAN framework provides a rich supervisory signal for our clustering network.
A well known motivation for unsupervised learning from a practical point of view is to alleviate the cost and error of the labeling process. Being independent of human labels not only saves the cost of labeling but also allows us to find a presumably better set of pseudo-labels through which conditional GANs can achieve higher performance on the task of image generation. This is because human labels are based on semantic contents that still include large variety of intra-class variations. This issue is more problematic in datasets like CIFAR10 or LSUN [34], where a fine set of labels is not available, and a large variety of images with different object and background appearances are grouped as a single class. Approximating such complex conditional distributions is still challenging. As it is shown in Figure 1, our clustering network splits the dataset into fine-grained clusters based on abstract factors of variation such as shape, appearance, pose, interaction with other objects and/or background, scale, quantity, etc. Each cluster proposes a simpler subdistribution that includes fewer variations and therefore is easier to approximate for the generator.
Our proposed slcGAN framework includes a clustering network in addition to a standard cGAN. It assigns a Kdimensional probability vector to a real image as input, where K is a given parameter indicating the number of clusters. An image and its corresponding class probability vector is fed to the discriminator as a real pair. The generator takes a randomly selected label, i.e. conditioning term, in addition to a noise vector sampled from a multivariate Gaussian, and generates a fake image. A generated image and its corresponding conditioning term is fed to the discriminator as a fake pair. The clustering network and the generator aim to fool the discriminator such that it fails to distinguish between the real and fake pairs. For this purpose, the clustering network and the generator should follow the same strategy on assigning pseudo-labels to the real images and associating conditioning term to the fake images. The pseudo-label assignment strategy should facilitate the generator task of producing realistic images, as low quality images enable the discriminator to distinguish between the fake and real pairs based on the images alone regardless of the mechanism with which they are tied to the pseudo-labels.
A degenerate solution for the clustering network and the generator of the three player game described above is to hide any dependency between the images and the pseudolabels. To prevent this, we explicitly enforce the generator to establish dependency between the conditioning term and the fake images. Moreover, we provide an auxiliary supervisory signal for the clustering network via multiple view clustering that enforces consistency between the pseudolabels assigned to an image and its augmented versions.
Our experiments show that our proposed slcGAN framework trained with a relatively large number of clusters outperforms unconditional GANs on ImageNet and LSUN datasets with significant margins. Additionally, it outperforms cGANs trained on human labels in terms of FID on CIFAR10 and CIFAR100. Moreover, our clustering network exceeds the state-of-the-art on CIFAR100 in terms of clustering accuracy. Our contributions are: I) We introduce slcGANs, a novel and fully unsupervised framework for joint clustering and conditional GAN training. II) We show that slcGANs outperform uGANs on several large scale datasets, and cGANs conditioned on human labels in terms of FID for image generation on CIFAR10 and CI-FAR100. III) We show that our clustering network exceeds the state-of-the-art for image clustering on the CIFAR100 dataset.

Related work
GANs as a source of supervisory signal. GANs have been exploited before as a source of supervisory signal for learning representations. Radford et al. [29] transfer features of the discriminator of unconditional GANs. Chen et al. [6] investigated the same direction by training the discriminator of unconditional GANs jointly with the auxiliary task of rotation prediction [14]. Adversarially learned inference (ALI) [12] or bidirectional GAN(BiGAN) [2] proposed primarily to augment the standard GAN with an encoder network that maps real data to the latent space. In the case of an optimal discriminator, [2] showed that a deterministic BiGAN acts like an autoencoder, i.e. combination of the encoder and generator, that minimizes 0 reconstruction costs, where the shape of the reconstruction error surface is dictated by a parametric discriminator. Our objective is fundamentally different. We aim to decompose the distribution into smaller subdistributions, each of which is easier for the generator to approximate. The BigBiGAN framework [11] is a recently revised version of BiGAN using the BigGAN [10] generator and a modified discriminator.
Clustering. End-to-end learning for image clustering has been recently explored. A line of methods [33,4,3] follow an iterative approach that alternates between obtaining pseudo-labels via clustering, e.g. k-means, and feature learning by predicting the pseudo-labels obtained from the previous stage. The idea of these methods is tied to the architecture of CNNs as a prior to cluster images. Another line of research [20,19] exploits maximizing the mutual information between an image and its augmentations. More related to our approach, Mukherjee et al. [28] exploit GANs for clustering. They perform clustering in the latent space of unconditional GANs by joint training of an encoder that projects back a generated image to the corresponding latent space. Apart from the complexity of training an optimal encoder in their approach and the suboptimality of unconditional GANs, their encoder never observes real images during the training. Our clustering method obtains the supervisory signal from the generator indirectly to train a separate clustering network. Our clustering network is updated through real images, however, the training signal originates from the generated images via cooperation with the generator.
Conditional GANs without annotations. Lucic et al. [24] exploit the feature space of a pre-trained selfsupervised task along with a fraction of ground truth labels in a semi-supervised setting to alleviate the need for a fully annotated training set. Our approach requires neither a pretrained network nor a partially labeled training set. Very recently and concurrently with us, Liu et al. [23] pursued a similar problem setup as ours. They perform clustering via k-means on the feature space of the discriminator and alternate between clustering and the conditional GAN training. This approach suffers from two main drawbacks. First, their performance relies on the quality of the intermediate representation of the discriminator, which is not necessarily an optimal feature extractor. Second, their method requires an existing clustering algorithm, e.g. k-means. By incorporating a new clustering network to the conditional GAN framework, our method not only is independent of an existing feature extractor but also learns useful features independently of a clustering algorithm. Our experiments show that our method obtains clusters with significantly higher purity, which is essential for the image generation task.

Method
Conditioning on class labels has been effective on alleviating the mode collapse issue of GANs. They enlarge the support of the generator by guiding it to cover all the classes available in the dataset. Our proposed method aims for the same goal in an unsupervised setting in which we obtain the pseudo-labels by incorporating a clustering network. Given a set of unlabeled images, our objective is to train simultaneously a clustering network that assigns a cluster probability vector to each datapoint used as a conditioning term for training conditional GANs. We review the conditional GAN framework in the following and discuss our method in more detail.

Conditional GANs
GANs involve adversarial training of two neural networks, a generator(G) and a discriminator(D). Given a noise vector sampled from a simple low dimensional distribution, the generator aims to approximate the underlying unknown data distribution P r to fool the discriminator, while the discriminator aims to distinguish between real and fake samples.
Conditional GANs extend the GAN framework by exploiting an auxiliary source of information available for G and D as an extra input, where the real joint samples of images and the conditioning terms are empirically available in the training dataset. Class labels are the most widely used conditioning information in Equation 2. We further extend the class conditional GAN framework to not rely on explicit labels in the dataset.

Self-labeled Conditional GANs
Our proposed method involves adversarial training of three networks. In addition to G and D of cGANs, we incorporate a clustering network (C) that assigns pseudo-labels to the images cooperating with G against D. A primary objective function can be obtained simply by substituting labels in the cGAN framework with p(y|x) approximated by C, where K is the number of clusters, 1 K (c) is a Kdimensional one-hot encoding of c sampled from prior distribution, p c , and p(y|x) = softmax(C(x)) is a Kdimensional class probability vector of x determined by C. p z is a multivariate Gaussian and p c is a uniform distribution.
To fool the discriminator, the above objective function requires C and G to match the joint pairs of (x, p(y|x)) sampled from the real distribution to the synthetic pairs of (G(z, c), 1 K (c)). This encourages G to generate realistic images, C to distribute uniformly one-hot encoding probability vectors over the images, and both C and G to follow a shared mapping for associating pseudo-labels with the real and fake images respectively. Moreover, this mapping should assist G with generating more realistic images. The discriminator(D) takes an image and the corresponding pseudo-label as input. A real pair is composed of a real image and the assigned probability vector by C(bottom row). A fake pair is composed of a randomly generated image by the generator(G) and the conditioning term(c) by which it has been created(top row). C and G are trained against D such that it fails to distinguish between real and fake pairs. Mutual information maximization is imposed via a cross entropy loss(top row). C is not updated through this loss, shown in gray. Transformation consistency is imposed by a separate cross entropy loss on real samples(bottom row). Following the BigBiGAN framework, the adversarial loss involves separate terms for image(u) and joint inputs(s).
Preventing random label assignment. Any mismatch on pseudo-label association between C and G to the real and fake images provides D with a signal to distinguish between the fake and real pairs. A degenerate solution for C and G to hide this signal from D is to decouple images from the pseudo-labels. That is, random label assignment to the real images by C and independent fake images of the conditioning term by G.
To prevent this degenerate solution, we explicitly enforce the fake images to depend on the conditioning term. We follow the same approach as the InfoGAN framework [7] by maximizing a variational lower bound of I(c; G(z, c)) parameterized by a neural network, where I denotes mutual information. The InfoGAN framework adapts the discriminator for this purpose by adding a classification head for the fake images in addition to the one for real/fake discrimination.
We however can not follow the same approach as our discriminator is conditional. It takes c as the second input, which makes predicting c a trivial task. We thus use C alternatively by adding the following loss function: where p(y = c|G(z, c)) is estimated by C as it is performed for the real images in Equation 3. Note that C can also be updated by L mi . However, we obtained better performance by training this loss only for G. One reason could be that direct communication between G and C yields degenerate solutions by encoding shortcuts in the fake images by G. These shortcuts allow G and C to minimize L mi without establishing a comprehensive dependency between the fake images and the conditioning term. Furthermore, optimizing L mi only for G provides the generator with another supervisory signal for generating more realistic images consistent with C. Intuitively, it encourages G to generate realistic images such that C behaves on them in a similar manner to real images in terms of pseudo-label prediction. Note that C is only updated via real images throughout the training.
Multiple view clustering. We further provide an auxiliary supervisory signal for C via multiple view clustering. That is, C should assign the same pseudo-label to an image and its augmented versions, Where x t is a randomly augmented version of x via cropping, color jittering, and flipping.
Our approach of multiple view clustering is inline with learning via invariances across augmentations [5], an approach that has shown significant improvements in the field of self-supervised representation learning. These methods mostly rely on contrastive learning [31] to prevent the degenerate solution of constant prediction, which is not required in our method. The supervisory signal originating from the adversarial training enforces C to mimic a similar distribution as p c , which is simply a uniform distribution.

4
Our final objective function is the following.

29: end for
Architecture and adversarial losses. Our generator and discriminator architecture is based on those of Big-GAN [10]. We use a linear transformation to map pseudoclass probability vectors to higher dimensional dense vectors that are later provided to G with class-conditional BatchNorm [13,8], and to D with projection [26]. We use spectral normalization [27] in both C and G and train the adversarial objective functions using the hinge loss [22,32]. Following the BigBiGAN framework, we include a unary term in our adversarial loss that is a function of only images, and split the loss values across two terms. The unary term encourages G towards explicitly generating realistic images, which is in turn implicitly performed via the joint term, i.e. Equation 3. Algorithm 1 demonstrates the training steps of slcGANs in detail.

Experiments
We evaluate our generator in terms of image generation performance and the accuracy of our clustering network. Ground truth labels provide a plausible upper bound for performance in both cases. We conduct experiments on MNIST, CIFAR10, CIFAR100, LSUN, and ImageNet datasets.
Implementation details. We set the channel multiplier to 128 and the number of residual blocks per stage to 1 for both G, D. We use standard ResNet18 [17] as the backbone of our clustering network, Adam [21] optimizer with a batch size of 256 for all networks, a learning rate of 1e − 4, and update the discriminator twice per generator update.

Image synthesis
We evaluate image generation performance quantitatively by reporting the Inception Score(IS) [30] and the Fréchet Inception Distance(FID) [18]. The unconditional GAN, class conditional GAN, and slcGAN are trained with the same architecture. We evaluate our method for small, medium, and large values of K, and perform qualitative evaluations later in section 4.3.
CIFAR. We train our model with (10, 50, 100) on CI-FAR10 and (20, 100, 500) on CIFAR100 as small, medium, and large values of K respectively. As it is shown in Table 1, the slcGAN framework outperforms the unsupervised baselines as well as the cGAN by significant margins. We conjecture that this is due to the number of modes imposed by a large number of clusters, yielding more variant synthetic images, and the flexibility of associating pseudo-labels to the fake images by the generator compared to preassigned labels.

CIFAR10
CIFAR100    LSUN and (100, 1000, 2000) as small, medium, and large values of K respectively. We outperform the unsupervised baselines, closing the gap with the ground truth labels.

Clustering
We evaluate the clustering accuracy when the number of clusters matches the ground truth. For CIFAR100, we consider two cases of 20 superclasses and the original finegrained 100 classes. Although slcGAN does not outperform the baselines on MNIST and CIFAR10, and 20 superclasses on CIFAR100, it outperforms IIC [20] on CIFAR100 when the ground truth includes fine-grained labeling, 15.7% versus 9.5%. 1 We believe this is related to the better generator performance on approximating smaller subdistributions represented by each fine-grained cluster. Our method benefits from a large number of clusters as our generator provides the clustering network with more accurate gradients via the discriminator. The results in Table 3 shows the superiority of our method on fine-grained clustering.
Overclustering. Our generator favors a fine-grained clustering where each cluster represents specific attributes of data. A plausible assessment for this purpose is purity in terms of ground truth labels as images with the same ground truth label are more likely to share the same finegrained attributes. Purity involves assigning each cluster MNIST CIFAR10 CIFAR100 k = 20 k = 100 DCGAN [29] 82.8 31.  Table 3: Clustering evaluations. We find the best permutation that maps our pseudo-labels to the ground truth and measure the accuracy. The numbers of other methods are taken from [20]. We significantly outperform IIC on fine-grained clustering on CIFAR100.
to the class which is most frequent in the cluster and then measuring the accuracy of this assignment by counting the number of correctly assigned datapoints. It is computed as . . , π K } is the set of clusters, C = {c 1 , . . . , c J } is the ground truth classes, and N is the dataset size. Given a dataset with a coarse set of classes, a higher purity indicates dividing each class into fine pseudo-classes.
Purity is affected by the K value as well as cluster distribution prior, i.e. p c in Equation 3, which both are unknowns. To obtain a better insight into how semantics is involved in the clustering, we train a linear classifier on the 512 dimensional features of the layer before the last for image classification using the ground truth labels. The higher performance indicates more semantics involved in the intermediate layers of C for clustering images. As it is shown in Table 4, purity correlates with linear separability, and our method achieves an acceptable performance, Liu et al. [23] achieves a purity of 11.73 on CIFAR10 with K = 100.  Clusters distribution. As it is mentioned in section 3.2, our objective function in Equation 3 encourages C to distribute uniformly one-hot encoding probability vectors over the images. Note that we do not benefit from true ground truth distribution here, since the number of clusters in our training is relatively higher than ground truth. Moreover, it turns out that imposed constraint via adversarial training is not strong. Figure 4 compares our method with k-means in terms of the histogram of the clusters length, the number of samples assigned to the clusters. Our method is trained on LSUN with K = 1000, and k-means is applied on the features of the layer before the last for the same number of clusters. Our method covers a broader range of clusters length than k-means. This observation indicates that our method benefits from the weekly imposed constraint via adversarial training to adjust the distribution of the clusters to facilitate the image generation task of the generator.   Table 4, indicates that the multiple view clustering loss is effective, and more importantly that the main signal originates from the adversarial training.

Qualitative evaluation
We show several synthesised and real clusters of Ima-geNet in Figure 5 and LSUN in Figure 6. Our model is trained on both datasets with 1000 clusters. More qualitative results are available in the supplementary material.
Reconstruction comparison with BigBiGAN. Despite that the BigBiGAN framework can act like an autoencoder, i.e. a combination of the encoder and generator, our method does not reconstruct the copy of a given image. However, our model allows us to reconstruct multiple instances of the same cluster that a given image belongs to. That is, we obtain the pseudo-label of a given image via the clustering network and feed it to the generator with multiple randomly sampled noise vectors. In the case of fine-grained labels, we are able to reconstruct multiple instances with the same style of a given image. Figure 3 shows some examples on the MNIST dataset. Given an image (a), we show BigBi-GAN reconstruction (b), samples generated by our model trained with 10 pseudo-classes (c), and 50 pseudo-classes (d). For each digit we show two samples with different styles. The results show that a fine-grained model recovers the same style of the input, while the coarse-grained model recovers images with the same category. BigBiGAN reconstructs the exact input image.

Conclusion
We have shown that class conditional GANs can be trained independently of the human labels, achieving higher performance than unconditional GANs. Our main idea involves incorporating a clustering network into the class conditional GAN framework that automatically assigns pseudo-7 Generated cluster Real cluster Generated cluster Real cluster Figure 5: Visualization of the generated and real images of several clusters for ImageNet. Each row shows a cluster and the generated images conditioned on the same pseudo-label.
Generated cluster Real cluster Generated cluster Real cluster Figure 6: Visualization of the generated and real images of several clusters for LSUN 20 object categories. Each row shows a cluster and the generated images conditioned on the same pseudo-label.
labels to the images. The benefits of our proposed method are twofold. Our framework not only improves image generation performance on unlabeled datasets but also achieves a useful clustering network that exceeds the state-of-theart for fine-grained clustering. We believe our work develops further the active line of research of generative mod-els towards higher quality and annotation free frameworks. Moreover, we show GANs provide rich supervisory signals for clustering and representation learning.

Generated cluster
Real cluster Figure 7: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. 11 Generated cluster Real cluster Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel.