## LoadCNN: A Low Training Cost Deep Learning Model for Day-Ahead Individual Residential Load Forecasting

Yunyou Huang a,b,c , Nana Wang , Wanling Gao b a,c , Xiaoxu Guo a,c , Cheng Huang a,b,c , Tianshu Hao a,b,c , Jianfeng Zhan a,b,c, ∗

a State Key Laboratory of Computer Architecture, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing 100080, China b University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China c Software Systems Laboratory, ACS, ICT, CAS, Beijing 100080, China

## Abstract

Accurate day-ahead individual residential load forecasting is of great importance to various applications of smart grid on day-ahead market. Deep learning, as a powerful machine learning technology, has shown great advantages and promising application in load forecasting tasks. However, deep learning is a computationallyhungry method, and requires high costs (e.g., time, energy and CO 2 emission) to train a deep learning model, which aggravates the energy crisis and incurs a substantial burden to the environment. As a consequence, the deep learning methods are difficult to be popularized and applied in the real smart grid environment. In this paper, we propose a low training cost model based on convolutional neural network, namely LoadCNN, for next-day load forecasting of individual resident with reduced training cost. The experiments show that the training time of LoadCNN is only approximately 1 / 54 of the one of other state-of-the-art models, and energy consumption and CO 2 emissions are only approximate 1 / 45 of those of other state-of-the-art models based on the same indicators. Meanwhile, the prediction accuracy of our model is equal to that of current state-of-the-art models, making LoadCNN the first load forecasting model simultaneously achieving high prediction accuracy and low training costs. LoadCNN is an efficient green model that is able to be quickly, cost-effectively and environmentally-friendly deployed in a realistic smart grid environment.

Keywords: big data, deep learning, load forecasting, machine learning, neural network, smart meter.

## 1. Introduction

the system security and remove the requirement of expensive energy storage systems [3, 4].

According to the report of National Bureau of Statistics of People's Republic of China, electricity consumption of residents was 907.16 billion kWhs in 2017 [1]. Residents, as important participators of smart grid, are very important to the customeroriented applications, for example demand response (DR), demand side management (DSM), energy storage system(ESS), etc [2]. In these cases, precise day-ahead individual residential load forecasting is significant and essential to balance the generation and consumption, minimize the operating cost and decrease reserved capacity, which helps to maintain

∗ Corresponding author

Email addresses:

huangyunyou@ict.ac.cn

Huang), zhanjianfeng@ict.ac.cn

(Yunyou

(Jianfeng Zhan)

Prediction of individual residential electricity load was firstly reported by Ghofrani et al. in 2011, and it is still a rather new area [2, 5]. Only a few studies on individual residential electric load forecasting have been reported since it is an extremely challenging task. The reason lies in the huge uncertainty and volatility of electricity consumption behaviors of residents, which are difficult to be handled by traditional machine learning methods [6, 7]. Fortunately, the emergence of deep learning provides a solution to this issue, and deep learning models have shown great potential in time series prediction recently [2, 6-11]. Compared with traditional machine learning methods, deep learning shows significant superiority on individual residential electric load forecasting, owing to its power-

Table 1: The estimated costs for training an Encoderdecoder based model LISA [13].

|                             |   Hours | Electricity   | Cloud computing cost   |
|-----------------------------|---------|---------------|------------------------|
| Training without tuning     |     120 | $5            | $52-$175               |
| Training with simple tuning |    2880 | $118          | $1238-$4205            |
| Training with tuning        |  239942 | $9870         | $103k-$350k            |

ful ability to automatically learn complex nonlinear function of relating the input x to the prediction y [17].

However, the deep learning brings an enormous challenge: obtaining a optimal model requires high training costs (e.g., time, energy and CO 2 emission), which aggravates the energy crisis and puts a heavy burden on the environment. This issue has been confirmed by many recent studies, and attracted widespread attention from researchers [1316]. For example, Strubell et al. [13] investigated the training of an encoder-decoder based translation model LISA (the sequence to sequence translation task is very similar to the load forecasting task). As shown in Table 1, the total GPU time for training the model 1 was 239942 hours, and the project spanned a period of 6 month (The model was deployed on 60 GPUs (NVIDIA Titan X (72%) and M40 (28%)).). It is shown in Table 1 that the estimated costs were $9870 for electricity and $103 k -$350 k for Google cloud computing related to researching and developing. In addition, the author pointed out that the CO 2 emission from training the encoder-decoder based model was five times of the CO 2 emission from a car within its whole lifetime [13]. Such a huge consumption of time, energy and the CO 2 emission are serious issues that cannot be ignored.

In fact, there two reasons for the huge training cost of deep learning model. On one hand, the superiority of the deep learning depends on its complex network structure (10 -1000 millions of parameters and several to hundreds of layers) which provides powerful ability to automatically learn complex nonlinear function of relating the input x to the prediction y [17]. However, training such a complex network even for once requires high training costs. On the other hand, it is inevitable that developing

1 The time is required for researching and developing the model

Table 2: The estimated CO 2 emissions [13].

|                                                           |   CO 2 (lbs) |
|-----------------------------------------------------------|--------------|
| Air travel, 1 passenger, NY ↔ SF                          |         1984 |
| Human life, avg, 1 year                                   |        11023 |
| Car, avg incl. fuel, 1 lifetime                           |       126000 |
| Training without tuning for a encoder-decoder based model |          192 |
| Training with tuning for a encoder-decoder based model    |       626155 |

such a complex network requires tens to thousands of experiments to adjust the structure and hyperparameters of the model for following reasons.

- a) Develop a new deep learning model: Plenty of different network structures and hyperparameters must be considered to obtain an optimal model, which requires a number of experiments.
- b) Apply latest deep learning model to individual residential electric load forecasting: Leveraging technology that people already have in their pockets for a specific task is not as simple as it appears [18]. The deep learning model needs a lot of adjustments to suit the individual residential electric load forecasting, which also requires a number of experiments. What's more, due to plenty of training time cost, it is difficult to keep up with the development of deep learning technology . 2
- c) Deploy a deep learning model to a specific real environment: Commonly load forecasting models are only trained on a specific small data set, which makes the model heavily depends on specific local customer behavior and local climate. It needs to adjust and retrain the model on specific data set to deploy the model in other specific environments, which also requires a number of experiments.

However, all of the recent individual residential load forecasting researches have been focusing on promoting the forecasting accuracy and ignoring the huge training costs of deep learning model. Shi et al. [6] firstly developed a deep learning model for individual residential load forecasting in 2017,

2 The update of deep learning technology is very fast as it is the most popular technology in the field of artificial intelligence [19].

which was an encoder-decoder based model with a novel pooling mechanism to overcome over-fitting in deep learning models. Kong et al. [2] proposed a deep learning forecasting framework based on long short-term memory (LSTM), aiming to address the volatility of the electricity consumption behaviors of individual residents. Wang et al [12] developed a gated recurrent unit (GRU) model which was a popular recurrent neural network (RNN) model to forecast the next-day load of individual resident. Wang et al. [8] reported an LSTM model with a new loss function (pinball loss), week information and hour information to forecast the load of the individual resident. Recently, Kong et al. [9] proposed an LSTM-based framework to handle the high volatility and uncertainty of individual residential load. Though RNN-based model is the mainstream model for sequence prediction, and it achieves state-of-the-art performance in individual residential load forecasting tasks [8, 9], the training of RNN-based model is time-consuming and requires a large amount of computing resources since the complex mechanism of RNN. In addition, the RNN-based model is difficult to parallel which requires more training time.

According to the reasons for huge training cost which mention above, two ways are able to reduce the training costs: 1)reduce training cost of each training. 2)reduce the total number of training. In this paper, we propose a convolutional neural network (CNN) based model LoadCNN to reduce the training cost of each training and simultaneously keep an excellent forecasting accuracy. The experiments show that training current state-of-the-art models requires enormous cost, and our model significantly outperforms current stateof-the-art methods. Meanwhile, our model is able to reach the standard of the state-of-the-art performance in forecasting accuracy.

The contributions of this paper are as following four aspects:

- a) New problem: Training costs are firstly considered in load forecasting task, which are important issues that have been ignored in previous researches.
- b) New view: Affected by the life and work habits of residents, many regularities will exist in a day and in the same time periods of different days. In order to make it easier for CNN to obtain the regularity of electricity consumption in the same time periods of different days,
- we convert the electricity load curve into twodimension so that electricity consumption values of the same time period are nearby to each other.
- c) New model: We propose a novel load forecasting model LoadCNN based on CNN. Our model contains two channels to directly extract the consumption regularity among a day and the consumption regularity among a week at the same time period respectively. Besides, LoadCNN only contains 7 layers and all of the convolution layers are one-dimension, which effectively reduces the training costs of our model.
- d) The training time of our model is only approximate 1 / 54 of other state-of-the-art models, the energy consumption and CO 2 emissions are only approximate 1 45 of / other state-of-theart models. Meanwhile, it equals to the current state-of-the-art performance in terms of prediction accuracy.

The rest of this paper is structured as follows. Section 2 introduces our innovative approach. Section 3 describes the experiment deployment. Section 4 presents and discusses the results. Section 5 draws a concluding remark.

## 2. Methodology

In this section, we give a formal definition of day-ahead forecasting and propose a novelty CNNbased model LoadCNN for day-ahead load forecasting.

## 2.1. Day-ahead Individual Residential Load Forecasting

Load curve represents electricity consumption behaviors of individual residents, which is very important to various customer-oriented applications in smart grid. Load curve of an individual resident is denoted as X = { x t | t &gt; 0 and t ∈ R } in this paper. Also, we use the historical load curve of the individual resident X h = { x t | t ≥ T -s , t &lt; T and s &gt; 0 } to predict future load curve of individual resident X f = { x t | t ≥ T , t &lt; T + l and l &gt; 0 } . Here, x is the electricity consumption value in the load curve X s , is the length of the historical load curve X h , l is the length of future load curve X f , and time step T divides load curve into input and output of load forecasting task.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00298v3-figure-5706ba39-4ff6-4f22-9095-b90b676cfa59.jpg)

In this work, we focus on day-ahead load forecasting based on historical load curve of past 7 days, with l = 48 and s = 336 (half an hour interval, 48 data points for a day). Predicted load ˆ X f = { ˆ x t | t ≥ T , t &lt; T + 48 } is defined by Equation 1.

$$\hat { X } _ { f } = f ( X _ { h } ), \quad X _ { h } = \{ x _ { t } \ | \ t \geq T - 3 3 6 \, \ t < T \} \\ \dots \dots \dots \dots \dots \dots \dots$$

- b) The customer ID of individual residential ID is several vectors that are encoded by one hot encoder. Since the number of customer N is generally large ( N = 929), we utilize two vectors to uniquely represent a customer to obtain the vector with a relative smaller size. The size of each vector is 31 = glyph[rightanglenw] 2 √ N glyph[rightanglene] , and the size of ID is 62 = 2 × 31. Similarly, if necessary, we can use k vectors to represent ID .

The object of day-ahead load forecasting task is to minimize prediction loss defined by Equation 2.

$$l o s s ( \hat { X _ { f } }, X _ { f } ) = \sqrt { \frac { 1 } { 4 8 } \sum _ { i = T } ^ { T + 4 8 } ( x _ { i } - \hat { x _ { i } } ) ^ { 2 } } \quad ( 2 )$$

## 2.2. Details of LoadCNN

In this section, we will elaborate our proposed method LoadCNN. In addition, we will also introduce day-ahead individual residential load forecasting algorithm which is based on LoadCNN.

## 2.2.1. Data preparation

Data preprocessing is an essential step of the load forecasting. In our paper, five types of data are fed into LoadCNN, including historical load curve L , individual residential ID , month M , day D , and week W . The details of them are explained as follows:

- a) Historical load curve L is a sequence of energy consumptions of the past 7 days, and the size of L is 336. In order to easily extract the relationship among a day and the relationship between different days by CNN, we reshape L into a matrix with the shape [7,48].
- c) External factors such as lifestyle of customer and weather will have a direct impact on electricity consumption behavior. We consider the month, date, and week information in our model as the following folds: 1) The month index provides some information regarding weather (such as summer is hot). 2) The week index provides some information regarding work (such as weekend is holiday). 3) The date index provides some information regarding special behavior (such as shopping after pay). The month M , day D and week W are encoded by one hot encoder and belong to load curve to predict. The sizes of M D , and W respectively are 12, 31 and 7.

## 2.2.2. Load forecasting model

Compared with RNN, CNN has a simpler neural network structure and achieves state-of-the-art performance in image processing realm [20]. Thus, we seek to develop an energy-saving and efficient green model based on CNN. In this paper, we proposed a CNN-based model named LoadCNN for load forecasting which is shown in Figure 1. The main components of the LoadCNN are horizontal channel and vertical channel.

- a) Horizontal channel: This channel is used to extract the electricity consumption regularity among a day. It contains 6 convolution layers shaped like [1,N] , and 4 maxpooling layers. The convolution layers with 16, 24, 24, 64, 64, and 64 kernels, and all of the convolution layers are activated by the Rectified Linear Unit (ReLU) function.
- b) Vertical channel: This channel is used to extract the electricity consumption regularity between different days. It contains 6 convolution layers shaped like [N,1] , and 4 maxpooling layers. The convolution layers also with 16, 24, 24, 64, 64, and 64 kernels, and all of the convolution layers are activated by the Rectified Linear Unit (ReLU) function.

Besides, the ID, M, D, W and the data from the channels will be contacted and fed into a full connected layer.

Unlike the previous load forecasting works which only simply adjust existing models, we designed LoadCNN according to the electricity consumption habits of residents to extract the electricity consumption regularity and reduce the training costs. The LoadCNN consists of two channels, which respectively extract the consumption regularity among a day and the consumption regularity among a week in the same time period. Thus LoadCNN is able to easily extract the electricity consumption patterns of the consumer. In addition, the one-dimension convolution layers make the structure of LoadCNN more simple and easy to be trained. For example, a two-dimension convolution layers with the shape [4,7] , is able to simultaneously capture the electricity consumption regularity in horizontal and vertical, and to replace the first layer of LoadCNN. However, the twodimension convolutional kernel contains 28 = 4 × 7 elements, much more than the elements of the two one-dimension convolutional kernels in the first layer of LoadCNN, which has 11 = 4 × 1 + 1 × 7 elements. It brings more computation, and raises the training cost of the model.

## 2.2.3. Algorithm

The algorithm designed includes three parts as shown in Algorithm 1: 1) data pre-processing, 2) network training, and 3) evaluation.

Algorithm 1 The algorithm for individual residential electric load forecasting

Input: Load dataset Ψ of residents demand from smart meters.

- Output: The predicted Load of individual residents ˆ X f and the root mean squared error (RMSE), normalised root mean squared error (NRMSE), and mean absolute error(MAE).
- 1: Clean and pre-process the load data and obtain a dataset Ψ 0 = { X ,X h f } . X h is the historical load and X f is the target.
- 2: Divide Ψ 0 into training set Ψ tr , validation set Ψ va , and test set Ψ ts .
- 3: Initialize all learnable parameters θ 0 in LoadCNN.
- 4: The best parameters θ best = θ 0 .
- 5: The best validation loss loss best = 0 55. .
- 6: for Current epoch k th &lt; Max epoch do
- 7: while Any instances in Ψ tr not are selected in this epoch. do

8:

Select a batch of instances Ψ b from Ψ tr .

- 9: Find θ i by minimizing the loss defined by Equation 2 with Ψ . b

10:

if

k

th

% 100 == 0

then

11:

Randomly select a batch of instances Ψ b va from Ψ tr .

- 12: Calculate loss va defined by Equation 2 with Ψ b va .

13:

if loss

best

&lt; loss va

then

14:

15:

16:

17:

loss best

θ

best

=

end if end if

18:

end while

- 19: end for
- 20: Forecast the ˆ X f by LoadCNN with X h ∈ Ψ . ts
- 21: Calculate the RMSE, NRMSE, and MAE with the ˆ X f and X f ∈ Ψ . ts

=

loss va

θ

i

## 3. Experiment Deployment

## 3.1. Data description

To evaluate the performance of LoadCNN, we conduct the experiments on a large-scale smart meter dataset from Smart Metering Electricity Customer Behaviour Trials (CBTs) in Ireland [21]. The data is collected from over 5000 Irish customers for the period of 536 days between 1st July 2009 and 31st December 2010. The smart meter data is halfhourly sampled electricity consumption (kWh) data from each customer.

In CBTs, we selected the customers which meet the condition that residential customers with the controlled stimulus and controlled tariff because of the following two aspects: (1) selected customers were billed on existing flat rate without any DSM stimuli. (2) selected customers are the most representative 3 . Finally, 929 residential customers are selected to verify our method.

To verify our method, we divide the dataset into three sets: training set, validation set, and test set. The test set contains all the data of the last 30 days. The validation set contains data of 60 days which are randomly selected from the 8 th -506 th days. The training set contains all of the rest data.

## 3.2. Experiment Setup

All of models for all customers are built on a server with two Intel Xeon E5-2630 v4 processors, 256 GB of memory and four NVIDIA Titan Xp GPUs. Server system is Linux 3.10.0327.el7.x86 64. In addition, all of the models are implemented by the TensorFlow-gpu 1.10.0v library [22] and Python 3.6.7v.

The parameters for all models are presented as follows: batch size=64, max epoch=65, hidden neuron number of RNN=128, learning rate=0 0015, de-. cay rate=0 96. . All of the RNN-based model are adopted dropout technology, and dropout rate=0 5. . In addition, in order to facilitate the comparison of training time and energy consumption, each model runs on only one GPU.

## 3.3. Metrics

In this work, three widely used metrics are applied to evaluate the accuracy of LoadCNN: root mean squared error (RMSE), normalised root mean

3 The majorities of consumers outside trial are of the type [6]

squared error (NRMSE), and mean absolute error (MAE).

$$R M S E = \sqrt { \frac { \sum _ { i = 1 } ^ { N } ( x _ { i } - \hat { x _ { i } } ) ^ { 2 } } { N } } \quad \quad ( 3 )$$

$$N R M S E = \frac { R M S E } { x _ { m a x } - \hat { x _ { m i n } } } \quad \quad ( 4 )$$

$$M A E = \frac { \sum _ { i = 1 } ^ { N } | ( x _ { i } - \hat { x _ { i } } ) | } { N } \quad \quad ( 5 )$$

Here, ˆ x i is the predicted value, x i is the actual value, x max and x min are the maximum and minimum value of x i respectively. N is the number of point x i in the test set.

Meanwhile, energy efficiency and training efficiency are also needed to measure in our work. Energy consumption ( EC ) is defined in Equation 6 as GPU consumes the most part of energy.

$$E C = \frac { P \times T T \times P U E \times N T } { 1 0 0 0 } \quad \quad ( 6 )$$

Here, P is the power of GPU during training the model. The TT represents the training time of a model for one training. PUE is the power usage effectiveness and accounts for the additional energy that is required to support the compute infrastructure (mainly cooling) [13]. NT is the number of times to train a model. The detailed settings of the parameters are as follows.

- a P : as Figure 2 shows, the differences of the power drawn of a GPU during training a model are negligible. Thus, to simplify the problem and minimize the impact of monitoring procedures on training, we randomly select the average power within 30 minutes during model training as P for model training.
- b PUE : its coefficient is set as 1.58 (global average for data center) according to the study [13].
- c NT : In general, hyperparameter tuning is a big topic and essential to obtain the best forecasting performance [9]. In the recent [23] work, to obtain the best performance of an encoder-decoder model, the author did 4789 trials [13]. The task of the model, which is a task forecasting of sequence to sequence, is similar to the day-ahead individual resident load forecasting. Thus, In our paper, to simplify the problem we assume that NT=1000 trials are required to obtain the best performance of a model.

Figure 2: The power draw of a GPU during training a encoder-decoder model.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00298v3-figure-b4d60d91-c573-44c9-993b-beacf7483249.jpg)

The reasons for assumptions above are as follows: 1) Every model runs on the same sever. 2) Every model runs on a NVIDIA Titan Xp GPU only. 3) Most of the energy consumption of training a model is on GPU.

The CO 2 emissions CO e 2 is estimated by Equation 7 according to U.S. Environmental Protection Agency [13].

$$C O _ { 2 } e = 0. 9 5 4 E C$$

## 3.4. Compared Models Setup

We use 11 models from four types of popular deep learning methods as benchmarks in present work: classic RNN-based model, RNN and CNN-based model, encoder-decoder-based models and CNNbased model.

- · LSTM, a most popular RNN model for time series prediction, is commonly used for load forecasting since 2017 [2]. In our paper, we transform the model into a day-ahead load forecasting model to compare with our model.
- · LSTM-Week is a load forecasting model proposed in 2019. It uses a new loss function and considers the week and hour information [8].
- · LSTM-EID is also a load forecasting model proposed in 2019. It considers the week, record point position in a day and holiday information [9]. In order to compare with our model, we also transform the model into a day-ahead load forecasting model.
- · GRU is another popular RNN model proposed in 2018 for time series prediction and applied

to day-ahead load forecasting [12]. The model considers date, weather and temperature information.

- · Skip-RNN is an RNN model that is able to capture long term dependencies and relieve vanishing gradients when the model is trained on long sequences [24]. Since the length of input in this work is 336 = 7 ∗ 48, the Skip-RNN is considered as a benchmark.
- · LSTM-CNN is a model that mixes typical LSTM and CNN which is similar to a famous model-inception models [25]. The types of LSTM-CNN model have been used to load forecasting on area level and industrial distribution complexes [26, 27]. In order to compare with our model, we transform LSTMCNN model into a day-ahead load forecasting model on individual resident level.
- · seq2seq is a LSTM-based encoder-decoder model which is the most popular model for the forecasting of sequence to sequence.
- · seq2seq-pooling was proposed in 2017 to relieve the overfitting in load forecasting [6]. In order to compare with our model, we transform the model into a day-ahead load forecasting model and use the dropout technology to further relieve the overfitting.
- · seq2seq-attention is an encoder-decoder model that combines the attention mechanism to handle the long sequences [28].
- · Temporal convolutional network (TCN) is recently proposed to handle sequence and achieves the state-of-the-art performance in many sequence modeling tasks [29]. It has been used to load forecasting on individual resident level [30]. In order to compare with our model, we also transform the model into a day-ahead load forecasting model.
- · ResNet, a CNN-based model, is the state-ofthe-art method of image recognition tasks [31].

## 4. Results and discussion

In this section, we present and discuss the results of the experiments in terms of training efficiency, energy consumption, environmental costs and prediction accuracy. In addition, we also investigate

the effect of number of layers in deep learning model since the deeper the model, the more complex the network structure of the model, and the more training time is needed which results in more energy consumed and CO 2 emitted.

## 4.1. Training efficiency, energy consumption and environmental costs

As shown in Table 3, our model not only almost achieves the second highest prediction accuracy, which is only 0.0005 behind the first place, but also obtains superior performance in training time, energy consumption and CO 2 emissions compared with all the other models. Specifically, in training time, LoadCNN takes the shortest time that is only approximate 1 / 54 of other RNN-based models. What's more, LoadCNN is based on CNN and very easy to parallel. Thus, the training time of LoadCNN is able to be further reduced by simply adjusting the code of implementation and adding more GPUs. As for energy consumption and CO 2 emissions, LoadCNN is also only approximate 1 / 45 of other RNN-based models.

The reason for the results is that LoadCNN has a simple network structure which is easy to be trained. However, contrary to our model: 1) The step of RNN-based model is 336 which is a quite large number and makes the structure of the model very complex when training, though the layer of the RNN based model is 3. 2) The other CNN based models are also much more complex than our model because they have more layers.

In addition, compared with the experiment, the training time, energy consumption and CO 2 emissions of the model will be more in reality. In this experiment the training set only contains the data from 929 customers for 439 = 536 -30 -60 -7 days. In the optimal case, the energy consumption and training time of the mainstream models are expected to exceed 16000 kWhs and 150000 = 150 × 1000 h (17 years) respectively (Of course, if there are enough GPUs, we can perform multiple parameter adjustment experiments at the same time to reduce the training time.). However, in real environment the training set should contain hundreds of thousands or even more customers, which will significantly increase the time and energy consumption of the training model. Therefore, training efficient and low-energy models like our model is significant.

## 4.2. The prediction accuracy of different deep learning models.

It is found that the prediction accuracy is hard to improve only by constructing different deep learning models. Specially, as shown in Table 3, the best performance ones of classical RNN-based models, RNN-CNN-based models, and CNN-based models have little difference in the accuracy metrics RMSE NRMSE , , and MAE . Usually, the tiny difference is likely to be eliminated by adjusting hyperparameters. Consistent with Table 3, as shown in Figure 3 except for encoder-decoder based models and TCN model, the prediction performances of other models are not much different. It means that it is difficult to use current deep learning technology to make a major breakthrough in forecasting accuracy of day-ahead individual residential load forecasting.

In order to improve the accuracy of the forecast, we need to pay more attention to obtain the information about personal activities and external information, as the electricity consumption of the household is extremely dependent on the randomness of individual human behaviors and external factors . 4

In addition, it is worth noting that there is a large gap between the performance of the encoderdecoder based models and state-of-the-art models. This can be explained by the mechanism of the decoder. In the day-ahead individual resident load forecasting task, the decoder predicts the value of the current point based on the value of the previous point and the current state of model. Unfortunately, we can not directly obtain the value of the previous point, and the predicted value of the previous point is used to replace the actual value. Therefore, the forecasting errors will be accumulated and amplified.

Finally, it is also worth noting that compared with the previous very short-term (such as 15-minahead) load forecasting work which predicted the electricity consumption more accurately, the dayahead load forecasting tends to predict electricity consumption pattern of the customers. For example, as shown in Figure 3, the early peak and the three later peaks of the actual load curve cannot be accurately predicted.

4 For example, both the business trip of the residents and the change of indoor temperature will cause electricity consumption changes which are difficult to predict only by the historical load curve of the individual resident.

Table 3: Performance comparison.

| Network Architecture   |   TT without tuning (h) | Power (W)   | EC (kWh)   | CO 2 e ( lbs )   | Easy to parallel   |   RMSE (kWh) |   NRMSE (kWh) |   MAE (kWh) | Layers   | Steps   |
|------------------------|-------------------------|-------------|------------|------------------|--------------------|--------------|---------------|-------------|----------|---------|
| LSTM/ 3                |                  164.42 | 66.1656     | 17188.7378 | 16398.0559       | No                 |       0.6192 |        0.0473 |      0.3636 | 3        | 336     |
| LSTM/ 5                |                  239.22 | -           | -          | -                | No                 |       0.6157 |        0.047  |      0.3511 | 5        | 336     |
| LSTM/ 8                |                  365.65 | -           | -          | -                | No                 |       0.7375 |        0.0563 |      0.4085 | 8        | 336     |
| LSTM - Week            |                  164.73 | 68.5650     | 17845.6456 | 17024.7459       | No                 |       0.6246 |        0.0477 |      0.3665 | 3        | 336     |
| LSTM - EID             |                  161.58 | 68.3967     | 17461.4313 | 16658.2055       | No                 |       0.6153 |        0.047  |      0.3639 | 3        | 336     |
| GRU                    |                  170.3  | 64.5683     | 17373.6508 | 16574.4629       | No                 |       0.6156 |        0.047  |      0.3487 | 3        | 336     |
| Skip - RNN             |                  190.33 | 64.1756     | 19298.9762 | 18411.2233       | No                 |       0.6147 |        0.0469 |      0.3477 | 3        | 336     |
| LSTM - CNN             |                  153.2  | 67.3422     | 16300.5835 | 15550.7567       | No                 |       0.6184 |        0.0472 |      0.3583 | 3-8      | 336-1   |
| seq 2 seq/ 3           |                  165.02 | 72.4456     | 18888.8572 | 18019.9698       | No                 |       0.6641 |        0.0507 |      0.4101 | 3-3      | 336-48  |
| seq 2 seq/ 5           |                  274.28 | -           | -          | -                | No                 |       0.6771 |        0.0517 |      0.4806 | 5-5      | 336-48  |
| seq 2 seq/ 8           |                  389.95 | -           | -          | -                | No                 |       0.6881 |        0.0525 |      0.4941 | 8-8      | 336-48  |
| seq 2 seq - pooling/ 3 |                  164.12 | 66.4822     | 17239.4727 | 16446.4570       | No                 |       0.6713 |        0.0513 |      0.3922 | 3-3      | 336-48  |
| seq 2 seq - pooling/ 5 |                  246.22 | -           | -          | -                | No                 |       0.6581 |        0.0503 |      0.4332 | 5-5      | 336-48  |
| seq 2 seq - pooling/ 8 |                  382.95 | -           | -          | -                | No                 |       0.7252 |        0.0554 |      0.5474 | 8-8      | 336-48  |
| seq 2 seq - attention  |                  180.33 | 87.1394     | 24827.8798 | 23685.7973       | No                 |       0.6549 |        0.05   |      0.4005 | 3-3      | 336-48  |
| TCN                    |                   20.55 | 218.3589    | 7089.8951  | 6763.7599        | Yes                |       0.877  |        0.067  |      0.4731 | 8        | 1       |
| ResNet                 |                    7.15 | 187.5428    | 2118.6710  | 2021.2121        | Yes                |       0.6261 |        0.0478 |      0.3673 | 34       | 1       |
| LoadCNN (Our)          |                    2.85 | 80.2228     | 361.2433   | 344.6261         | Yes                |       0.6152 |        0.047  |      0.3469 | 7        | 1       |

## The performance of the models

Figure 3: The forecasting loads by different models and the realistic load.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00298v3-figure-7b3854dc-3b94-4568-829a-10c68a81e4f2.jpg)

## 4.3. Effect by the number of layer in model

The recent revival of neural networks has benefited from the development of computer hardware that has made neural networks deeper and deeper which is the main cause of the high complexity of deep learning model.

In general, the deeper the neural network is the more precise the prediction is. However, as shown in the Table 3, deeper models of LSTM, seq2seq, seq2seq-pooling, CNN do not perform better than shallow models. What's more, 8-layer LSTM, seq2seq and seq2seq-pooling models have terrible performance. It means that on the one hand, the most powerful means of deep learning - increasing depth can no longer help improve the accuracy of the model. On the other hand we need to develop new technologies to solve the over-fitting problem.

As a conclusion, it is unnecessary to make the model deeper, which leads to a more complex network structure that will translate into more training time, more energy consumption and more CO 2 emission.

## 5. Conclusions

Day-ahead individual residential load forecasting is very important to real applications (such as demand response) of smart grid. Deep learning models have became commonly used methods in load forecasting. However, the deep learning models are computationally-hungry, which requires plenty of training costs. All of the previous load forecasting works have been only focusing on improving prediction accuracy and ignore training costs.

To save resources and promote the application of deep learning models, we propose and develop an efficient green CNN-based model LoadCNN. It not only equal to the current state-of-the-art performance but also has huge advantages in training efficiency, energy consumption and environmental

costs. The experimental results on public and largescale dataset show that the training time of our model is only approximate 1 / 54 of other state-ofthe-art models, the energy consumption and CO 2 emissions are only approximate 1 / 45 of other stateof-the-art models. In addition, it is found that it is difficult to improve the accuracy by simply adjusting the hyperparameters or structure of deep learning models. In the future, to improve the accuracy, we must obtain more relevant information (such as human activities).

## Acknowledgements

We are very grateful to CER Smart Metering Project -Electricity Customer Behaviour Trial, 2009-2010 and ISSDA. This work is supported by the Major Program of National Natural Science Foundation of China (Grant No. 61432006).

## Competing interests

The authors declare no competing interests.

## References

- [1] N. B. of Statistics of China, Annual data, http://data. stats.gov.cn/easyquery.htm?cn=C01 , accessed June 12, 2019.
- [2] W. Kong, Z. Y. Dong, D. J. Hill, F. Luo, Y. Xu, Shortterm residential load forecasting based on resident behaviour learning, IEEE Transactions on Power Systems 33 (1) (2017) 1087-1088.
- [3] A. Tascikaraoglu, B. M. Sanandaji, Short-term residential electric load forecasting: A compressive spatiotemporal approach, Energy and Buildings 111 (2016) 380-392.
- [4] K. Chen, K. Chen, Q. Wang, Z. He, J. Hu, J. He, Short-term load forecasting with deep residual networks, IEEE Transactions on Smart Grid.
- [5] M. Ghofrani, M. Hassanzadeh, M. Etezadi-Amoli, M. S. Fadali, Smart meter based short-term load forecasting for residential customers, in: 2011 North American Power Symposium, IEEE, 2011, pp. 1-5.
- [6] H. Shi, M. Xu, R. Li, Deep learning for household load forecastinga novel pooling deep rnn, IEEE Transactions on Smart Grid 9 (5) (2017) 5271-5280.
- [7] M. H. Alobaidi, F. Chebana, M. A. Meguid, Robust ensemble learning framework for day-ahead forecasting of household based energy consumption, Applied energy 212 (2018) 997-1012.
- [8] Y. Wang, D. Gan, M. Sun, N. Zhang, Z. Lu, C. Kang, Probabilistic individual load forecasting using pinball loss guided lstm, Applied Energy 235 (2019) 10-20.
- [9] W. Kong, Z. Y. Dong, Y. Jia, D. J. Hill, Y. Xu, Y. Zhang, Short-term residential load forecasting based on lstm recurrent neural network, IEEE Transactions on Smart Grid 10 (1) (2019) 841-851.
- [10] Y. Peng, Y. Wang, X. Lu, H. Li, D. Shi, Z. Wang, J. Li, Short-term load forecasting at different aggregation levels with predictability analysis, arXiv preprint arXiv:1903.10679.
- [11] M. Cai, M. Pipattanasomporn, S. Rahman, Day-ahead building-level load forecasts using deep learning vs. traditional time-series techniques, Applied Energy 236 (2019) 1078-1088.
- [12] Y. Wang, M. Liu, Z. Bao, S. Zhang, Short-term load forecasting with multi-source data using gated recurrent unit neural networks, Energies 11 (5) (2018) 1138.
- [13] E. Strubell, A. Ganesh, A. McCallum, Energy and policy considerations for deep learning in nlp, in: 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy, 2019.
- [14] R. Schwartz, J. Dodge, N. A. Smith, O. Etzioni, Green ai, arXiv preprint arXiv:1907.10597.
- [15] A. Dalgren, Y. Lundeg˚rd, Greenml: a A methodology for fair evaluation of machine learning algorithms with respect to resource consumption (2019).
- [16] J. Dodge, S. Gururangan, D. Card, R. Schwartz, N. A. Smith, Show your work: Improved reporting of experimental results, in: 9th International Joint Conference on Natural Language Processing (IJCNLP), Hong Kong, China, 2019.
- [17] X.-X. Zhou, W.-F. Zeng, H. Chi, C. Luo, C. Liu, J. Zhan, S.-M. He, Z. Zhang, pdeep: Predicting ms/ms spectra of peptides with deep learning, Analytical chemistry 89 (23) (2017) 12690-12697.
- [18] Y. Chen, C. Hu, B. Hu, L. Hu, H. Yu, C. Miao, Inferring cognitive wellness from motor patterns, IEEE Transactions on Knowledge and Data Engineering 30 (12) (2018) 2340-2353.
- [19] H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, P.A. Muller, Deep learning for time series classification: a review, Data Mining and Knowledge Discovery (2019) 1-47.
- [20] A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis, Deep learning for computer vision: A brief review, Computational intelligence and neuroscience 2018.
- [21] C. E. Smart, Metering customer behaviour trials (cbt) findings report.
- [22] Google, Tensorflow, https://tensorflow.google.cn/ versions/r1.10/api\_docs/python/tf , accessed June 21, 2019.
- [23] E. Strubell, P. Verga, D. Andor, D. Weiss, A. McCallum, Linguistically-informed self-attention for semantic role labeling, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 5027-5038.
- [24] V. Campos Camunez, B. Jou, X. Gir´ o Nieto, J. Torres Vi˜als, n S.-F. Chang, Skip rnn: learning to skip state updates in recurrent neural networks, in: Sixth International Conference on Learning Representations: Monday April 30-Thursday May 03, 2018, Vancouver Convention Center, Vancouver:[proceedings], 2018, pp. 1-17.
- [25] C. Szegedy, S. Ioffe, V. Vanhoucke, A. A. Alemi, Inception-v4, inception-resnet and the impact of residual connections on learning, in: Thirty-First AAAI Conference on Artificial Intelligence, 2017.
- [26] C. Tian, J. Ma, C. Zhang, P. Zhan, A deep neural network model for short-term load forecast based on long short-term memory network and convolutional neural

network, Energies 11 (12) (2018) 3493.

[27] J. Kim, J. Moon, E. Hwang, P. Kang, Recurrent inception convolution neural network for multi short-term load forecasting, Energy and Buildings 194 (2019) 328341.

[28] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473.

[29] S. Bai, J. Z. Kolter, V. Koltun, An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv preprint arXiv:1803.01271.

[30] M. Voß, C. Bender-Saebelkampf, S. Albayrak, Residential short-term load forecasting using convolutional neural networks, in: 2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), IEEE, 2018, pp. 1-6.

[31] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
Yunyou Huang Nana Wang Wanling Gao Xiaoxu Guo Cheng Huang Tianshu Hao Jianfeng Zhan
LoadCNN: A Low Training Cost Deep Learning Model for Day-Ahead Individual Residential Load Forecasting
Accurate day-ahead individual residential load forecasting is of great
importance to various applications of smart grid on day-ahead market. Deep
learning, as a powerful machine learning technology, has shown great advantages
and promising application in load forecasting tasks. However, deep learning is
a computationally-hungry method, and requires high costs (e.g., time, energy
and CO2 emission) to train a deep learning model, which aggravates the energy
crisis and incurs a substantial burden to the environment. As a consequence,
the deep learning methods are difficult to be popularized and applied in the
real smart grid environment. In this paper, we propose a low training cost
model based on convolutional neural network, namely LoadCNN, for next-day load
forecasting of individual resident with reduced training cost. The experiments
show that the training time of LoadCNN is only approximately 1/54 of the one of
other state-of-the-art models, and energy consumption and CO2 emissions are
only approximate 1/45 of those of other state-of-the-art models based on the
same indicators. Meanwhile, the prediction accuracy of our model is equal to
that of current state-of-the-art models, making LoadCNN the first load
forecasting model simultaneously achieving high prediction accuracy and low
training costs. LoadCNN is an efficient green model that is able to be quickly,
cost-effectively and environmentally-friendly deployed in a realistic smart
grid environment.