text:
\section{Introduction} \label{sec_introduction}
 Mobile  edge computing (MEC) provides a promising solution to enabling cloud computing services in privacy-persevering mobile edge computing-based Internet of Thing (EdgeIoT) \cite{mao2017survey, sun2016edgeiot, Deng9001216}. The IoT devices can offload their local computation-intensive tasks to computationally powerful edge servers \cite{mach2017mobile, 9352033}. In EdgeIoT, offloading the source data of the IoT devices to the edge server is vulnerable to eavesdropping attacks \cite{Liangxiao,9183799}. To prevent private data leakage of the IoT devices, federated learning (FL) \cite{mcmahan2017communication} is used to train a global data learning model at the edge server by aggregating the data structure parameters of the IoT devices, while the source data remains at the IoT devices.

Fig. \ref{fig:1-1} depicts an FL-enabled EdgeIoT, where the IoT devices are deployed to sense and process private information,  e.g., health reports of patients \cite{xiao2018iot}.  Specifically, mobile phones use image classification models to classify pictures or images. FL can help multiple mobile phones to cooperatively train an effective global image classification model, without the need of sharing the images and pictures used for the training \cite{Ji2019}. Likewise, vehicles can train an accurate autonomous driving model by  aggregating the local models trained separately by multiple vehicles  \cite{LiYijing2021}.

 A local  model (e.g., the weight vector \cite{McMahanMRA16, kang2019, Lim2021}  or gradient \cite{ji2019learning}) is trained on the sensing  data of an IoT device to sense and process private information, such as private health reports of patients. Next, the edge server aggregates the local models of all IoT devices to create a comprehensive and effective global model without collecting the private data of the IoT devices.  A global model is obtained at the edge server, e.g., by applying FedAvg \cite{McMahanMRA16} to  the local models. The edge server broadcasts the  global model back to all the IoT devices. According to the global model, each of the IoT devices renews the training of its local model. By iteratively training the local model at the IoT device and updating the global model at the edge server, the learning accuracy of FL on the data classification and event prediction can be progressively improved \cite{9220780}.

 While selecting the IoT devices with large training datasets can improve the learning accuracy of FL \cite{Dinesh}, it can often result in fast depletion of the batteries at the devices. On the other hand, selecting the IoT devices with small data for training the local  models  can save the battery energy of the IoT devices, but likely leads to a low accuracy of the global model. In this sense,  balancing the learning accuracy of FL and energy consumption of the IoT devices is crucial to EdgeIoT.

\begin{figure}[htbp!]
\centerline{\includegraphics[width = 0.5\textwidth]{Fl_framework.png}}
 \caption{FL-enabled EdgeIoT. A local model is trained with the dataset at the IoT device.  The local model is aggregated by an edge server, where the global model is trained and returned to the IoT devices.}
 \label{fig:1-1}
 \end{figure}

 Moreover, the FL process is time-slotted by design and  sequential decision making is required. Hence, in this paper, we propose an online resource allocation optimization to balance the learning accuracy of FL and energy consumption of the IoT devices. In practice, the instantaneous information of data size, transmit power, and link qualities between the edge server and the IoT devices is unlikely to be known.  The optimization is formulated as a partially observable Markov decision process (POMDP),  where the network state consists of the source data size of the IoT devices, channel conditions between the edge server and the IoT devices, bandwidth and remaining energy of the selected IoT devices. All the entries of a network state are continuous. The action space includes the discrete selection of IoT devices and the continuous transmit power allocation. Given the continuous network states and actions, a large number of IoT nodes lead to a large state and action space, and complex network state transitions. This results in difficulty in the joint optimization of IoT device selection and transmit  power allocation. Due to a large and continuous state and action space in the formulated POMDP, a new  deep reinforcement learning-based device selection and transmit power allocation algorithm is proposed to maximize the ratio of the learning accuracy of FL and  the energy consumption of the IoT devices. The major contributions of this article are summarized as follows:
\begin{itemize}
\item We propose to jointly optimize the selection of IoT devices and their transmit powers in an FL-empowered edge IoT system, thereby balancing  the learning accuracy of FL and energy consumption of the IoT devices. The optimization is online, adapting to the time-varying arrivals of training data and the energy budget of the IoT devices, and the changing channel conditions between the IoT devices and the base station (i.e., the model aggregator).

\item FL-DLT3 is proposed to learn the network state dynamics while maximizing the ratio of the learning accuracy of FL to the energy consumption of the IoT devices. Considering the continuous transmit powers of the IoT devices, FL-DLT3 optimizes the edge server's selection of IoT devices for each round of FL and the transmit powers of the IoT devices,  based on Twin Delayed Deep Deterministic Policy Gradient (TD3).

\item A new long short term memory (LSTM) layer is designed in coupling with the proposed FL-DLT3 to predict the time-varying network states, e.g., data size, bandwidth, channel gain, and remaining energy of the IoT devices. The LSTM layer estimates the unobserved states at every training iteration of the TD3. To the best of our knowledge, this is the first time that LSTM is employed in coupling  with TD3 for the resource allocation of FL-enabled EdgeIoT.

\item FL-DLT3 is implemented in PyTorch. The effectiveness of FL-DLT3 is validated with the experimental data. Numerical results show that FL-DLT3 achieves fast convergence (less than 100 iterations) while the FL-accuracy-to-energy-consumption ratio is improved by 51.8\%, as compared to the state of the art.
\end{itemize}

The rest of this paper is structured as follows. The literature on FL-based resource allocation in MEC is reviewed in Section~\ref{sec_related-work}. Section~\ref{sec_FL-based}  presents the FL protocol and system models. The resource allocation optimization for EdgeIoT is formulated in Section~\ref{sec_problem_formu}. Section~\ref{sec_drl-based} proposes the FL-DLT3 framework which conducts deep reinforcement learning (DRL) based EdgeIoT devices selection and resource allocation. Section~\ref{section-numerical} evaluates the proposed FL-DLT3 framework. Finally, Section~\ref{sec_conclusion} concludes this paper.

\section{Related Work} \label{sec_related-work}
This section presents the literature on resource allocation with FL in MEC.

Assuming that the IoT devices have the same computational resources and wireless channel conditions, Federated averaging (FedAvg) \cite{McMahanMRA16} randomly selects IoT devices to participate FL training and synchronously aggregates local models. This process is repeated until a desirable training accuracy is achieved. In \cite{Nishio2019},  different IoT devices owns different computing capabilities and wireless channel conditions, resulting in different local model training time and upload time. The authors develop a FL protocol called FedCS, which allows the server to aggregate as many local model updates as possible to improve image classification accuracy.

To improve the training accuracy of FL systems in the
context of wireless channels and energy arrivals of
mobile devices, the authors \cite{Shunfeng2021}  model the transmission power allocation and mobile device selection of the FL training as a constrained Markov decision process. Due to a high complexity, stochastic learning methods and Lagrange multipliers are used to simplify the model and  to obtain an efficient policy for all mobile device. In \cite{AnhLNKW19}, the devices with limited battery energy, CPU computations, and bandwidths, are considered in a mobile crowd network. A deep Q-learning-based resource allocation for data, energy, and CPU cycles is developed to reduce the energy consumption of FL-based mobile devices and training time of the FL.  Given limited computation and communication resources at the devices, \cite{wang2019adaptive} analyzes the convergence bound of distributed gradient descent. A control algorithm is developed to determine the tradeoff between local update and global parameter aggregation. Some IoT devices have limited communication and computing resources and fail to complete training tasks, which leads to many discarded learning rounds affecting the model accuracy \cite{Wahab2021}. The authors study multicriteria-based approach for IoT device selection in FL that reducing the number of communication rounds to reach the intended accuracy and increasing the number of selected  IoT devices in each round.

The authors of \cite{yoshida2020mab} study a trial-and-error based IoT device selection  based on  multi-armed bandit (MAB), where computation tasks, traffic, and link qualities are unknown. The MAB-based IoT device selection balances the IoT device selection according to the selection frequency and IoT devices' resources.  In \cite{xu2021online}, the IoT device scheduling in MEC is formulated without the IoT devices' channel and computing information. To reduce the training latency, the IoT device scheduling is formulated as a MAB problem, where $\epsilon$-greedy is used to reduce the learning accuracy. In \cite{xia2020}, the authors develop a MAB-based IoT device scheduling framework to reduce the training latency of FL. Given the known independent and identically distributed (i.i.d.) local data at the IoT devices, an FL-based IoT device scheduling algorithm is designed.

 The authors of \cite{zhu2021federated} aim to reduce the
average age of data sources by controlling the IoT devices,
scheduling the data and allocating the bandwidth. An actor-critic learning framework is developed, where the IoT devices learn the scheduling strategies based on their local observations. To reduce the  training time and energy consumption  of IoT devices, the authors of \cite{zhan9139873} design an experience-driven algorithm based on proximal policy optimization and produce sub-optimal results.  To improve the network throughput,  the authors of \cite{kwon9067847} present  a  cell association and base station allocation method. Multi-agent deep deterministic policy gradient  is used to handle unexpected events and unreliable channels in underwater wireless networks.  In \cite{MeixiaTao}, the IoT devices are selected to improve the FL accuracy and reduce the training time and energy consumption. To balance the training accuracy and delay  of FL and  energy consumption, Twin Delayed Deep Deterministic policy gradient algorithm (TD3) is employed to capture the interplay between function approximation error in both policy and value updates, and produce the  IoT devices scheduling policy, the CPU frequency allocated for training, and the transmit power allocation.

An FL-based device selection optimization is developed in our preliminary work \cite{zheng2021federated} to balance the energy consumption of the IoT devices and the
learning accuracy of FL. The optimization model takes
advantage of the a-priori knowledge of the network state
information, e.g., data size, bandwidth, and channel gain. Due to NP-hardness of the optimization, an energy efficiency-FL accuracy balancing heuristic algorithm (FedAECS) was presented in \cite{zheng2021federated} to approximate
the optimal IoT device selection policy offline. In contrast, this paper considers a practical scenario without the prior  information about data size, bandwidth, channel gain, and remaining energy of the IoT devices. Due to a large state and action space, we propose a new FL-DLT3 to balance FL accuracy and energy consumption of the IoT devices online, where LSTM is leveraged to predict the hidden state of the IoT devices as input state of TD3. In addition,  we also compare the performance of the proposed FL-DLT3 with the FedAECS in \cite{zheng2021federated}.

\section{System Model}  \label{sec_FL-based}
In this section, we study the training protocol and energy model of FL. The notations used in the paper are summarized in Table \ref{tab:table3-1}.
\begin{table}[h!]
\small
\begin{center}
    \caption{The list of key variables defined in system model}
    \label{tab:table3-1}

    \begin{tabular}{|c|c|}
     \hline
     \textbf{Notation} & \textbf{Definition} \\

      \hline
      $K$  & The total number of IoT devices \\
        \hline
        $k$ & Index of IoT device \\
            \hline
       $T$ &  The total number of the rounds\\
       \hline
        $t$ & Index of the rounds \\
       \hline
         $D_{t,k}$ &  IoT device $k$ owns data size \\
           \hline
       $\mathbf{x}_{ki}$ & Input of the FL model \\
         \hline
       $y_{ki}$ & Output of the FL model \\
         \hline
      $\mathbf{w}$ & Weight parameter of FL training\\
       \hline
    $ \beta_{t,k}$ &  Whether IoT device $k$ is selected\\
      \hline
      $f_k$ &   CPU frequency of IoT device $k$\\
       \hline
      $\zeta_k$  & Effective capacitance coefficient\\
     \hline
     $L$ & Number of local iterations of FL training  \\
       \hline
      $E_{t,k1}^{cmp}$ & \makecell[c]{ Energy consumption for computing \\ $c_{k}D_{t,k}$ CPU cycles} \\
      \hline
      $  E_{t,k}^{cmp} $ & \makecell[c]{Total computation energy at \\ IoT device $k$ }  \\
      \hline
       $r_{t,k}^{up}$ &  Achievable uplink transmit rate \\
       \hline
      $b_{t.k}$ & \makecell[c]{Bandwidth allocated to IoT \\ device $k$ by the server }\\
      \hline
         $P_{t,k}$ & IoT device $k$ transmit power \\
       \hline
     $G_{t,k}$ & Uplink channel gain \\
      \hline
    $r_{t,k}^{down}$ & \makecell[c]{Achievable downlink transmit rate \\ of IoT device $k$  }\\
    \hline
    $H_{t,k}$ &  Downlink channel gain \\
     \hline
       $P_t^s$ &    Transmit power of the edge server \\
       \hline
    $\tau_{t,k}^{down}$ & \makecell[c]{Downloading time of the global model \\ at IoT device $k$ } \\
     \hline
       $\mathfrak{S}_{d}$ &   Size of the global model \\
        \hline
    $\tau_{t,k}^{up}$ & Transmission time of the local model at $t$ \\
    \hline
       $\mathfrak{S}_{u}$ &   Size of the local model \\
     \hline
     $E_{t,k}^{up}$ & \makecell[c]{ Energy consumption of device $k$ \\ on the local model transmission }\\
     \hline
   $E_{t,k}^{c}$ &  \makecell[c]{Total energy consumption  of \\ the selected  device $k$ } \\
     \hline
   $E_{t,k}$ &  Remaining battery energy of device $k$ \\
     \hline
    $\Delta E_{t,k}$ &  Amount of harvested energy \\
    \hline
    $\Delta_t$ &  Data evenness of selected devices \\
    \hline
       $\tau_{t,k} $ & \makecell[c]{Completion time of IoT \\ device $k$ in $t$-th round}\\
       \hline

      $\mu_{k}$  & System parameter \\
      \hline
       $\nu$ & \makecell[c]{Constant value}\\

       \hline
    \end{tabular}
 \end{center}
\end{table}

\subsection{FL Protocol with MEC}
Fig.~\ref{fig:3-1} depicts the FL protocol, where the global model at the edge server and the local models at the IoT devices are trained in $T$ rounds. Each FL round is composed of resource request, IoT device selection, global model aggregation, global model download, local model update and upload. The edge server  initializes the hyperparameters of the global model, e.g., learning rate, batch size, and the weights of the global model.

\begin{figure*}[htbp!]
\centerline{\includegraphics[width = 0.8\textwidth]{time-cost.png}}
 \caption{The whole training process of FL, in each round, the time consumption of IoT device $k$ includes global model download, local model update and upload.}
 \label{fig:3-1}
 \end{figure*}

\begin{itemize}

 \item \texttt{Resource Request}: The IoT devices send to the edge server the information needed, i.e., data size, for the device selection.

  \item \texttt{IoT Device Selection}: The edge server selects the IoT devices to upload the local model for the training of the global model, where the details will be presented in the next section.

 \item  \texttt{Global Model Aggregation}:  The edge server aggregates the local models of the selected IoT devices to produce  the global model.

\item \texttt{Global Model Download}: All the IoT devices download the the global model from the edge server.

\item  \texttt{Local Model Update and Upload}:
The selected IoT devices individually train their local models according to  the FL parameters of the global model.
\end{itemize}

We consider $K$ number of IoT devices, where $k \in [1, K]$. Let   $x_{ki}$ and $y_{ki}$ denote the input (e.g., pixels of an image) and the output (e.g., labels of the image) of the FL model \cite{Adnan-2101-07511}, respectively. The dataset of device $k$ is denoted as  $\mathcal{D}_{t,k} = \{ \mathbf{x}_{ki}, y_{ki} \}_{i=1}^{D_{t,k}}$, where $D_{t,k}$ is the size of the dataset of device $k$ in the $t$-th round and data sample $i$ in device $k$. Let $f(\mathbf{w}, \mathbf{x}_{ki}, y_{ki})$ denote the loss function of FL, which captures  approximation errors over the input $\mathbf{x}_{ki}$ and the output $y_{ki}$. $\mathbf{w}$ is the weight parameter of the loss function of the neural network being trained according to the FL procedure. Given $D_{t,k}$, the loss function at IoT device $k$ can be specified as,

\begin{equation}
   F_{t,k}(\mathbf{w}) = \frac{1}{D_{t,k}} \sum_{i = 1}^{D_{t,k}} f(\mathbf{w}, \mathbf{x}_{ki}, y_{ki}),
    \label{each-device-in-epoch-loss}
\end{equation}

where $ f(\mathbf{w}, \mathbf{x}_{ki}, y_{ki}) $ can be specified according to the FL structure. For example, $f(\mathbf{w}, \mathbf{x}_{ki}, y_{ki}) =  \frac{1}{2} (\mathbf{x}_{ki}^T \mathbf{w}- y_{ki})^2$  is used to model linear regression, or $f(\mathbf{w}, \mathbf{x}_{ki}, y_{ki}) = -\log(1 + \exp{(y_{ki} \mathbf{x}_{ki}^T \mathbf{w})})$  is for the model of  logistic regression \cite{shalev2014understanding}. Since the training of FL aims to minimize the weighted global loss function, we have

\begin{equation}
\begin{split}
   \mathop{\min}\limits_{\mathbf{w}}  F_t(\mathbf{w}) & =  \sum_{k=1}^{K} \frac{  D_{t,k}}{D_{t}} F_{t,k}(\mathbf{w})  \\ & = \frac{1}{D_t} \sum_{k=1}^{K} \sum_{i = 1}^{D_{t,k}} f(\mathbf{w}, \mathbf{x}_{ki}, y_{ki}),

 \end{split}
\end{equation}
where  $D_{t} =  \sum_{k=1}^{K}  D_{t,k}$, it represents the total amount of data in the $t$-th round.

\subsection{Energy Model}
The energy consumption of the IoT devices accounts for the local training of the dataset and the transmissions of the local model. We assume that training a data sample at the IoT device  requires $c_k$ CPU cycles per bit. Given the data size of $D_{t,k}$, the number of CPU cycles for the local model training is  $c_{k}D_{t,k}$. We denote $f_k$ as the computation capacity of IoT device $k$, which is measured in CPU cycles per second. According to \cite{Dinh2021}, the computation time of training the local model at device $k$ in the each $t$-th round \cite{Dinh2021}, we have

\begin{equation}
\tau_{t,k}^{train} = \frac{ c_{k} D_{t,k} L}{f_k}, \label{time-localtran}
\end{equation}
where $L$ is the number of local iterations of FL training.

According to \cite{yang2020energy} and \cite{mao2016dynamic}, the energy consumption on $c_{k}D_{t,k}$ CPU cycles at IoT device $k$ is
\begin{equation}
    E_{t,k1}^{cmp} = \zeta_k c_kD_{t,k} f_{k} ^2,
\end{equation}
where $\zeta_k $ is the effective capacitance coefficient of computing chipset for device $k$. To compute the local model, IoT device $k$ needs to compute $c_{k}D_{t,k}$ CPU cycles. Thus, the total computation energy at IoT device $k$ in the $t$-th round can be given as

\begin{equation}
    E_{t,k}^{cmp} =  L  E_{t,k1}^{cmp} = L \zeta_k c_kD_{t,k} f_{k}^2.
    \label{computation-energy}
\end{equation}

Furthermore, the achievable uplink transmit rate of IoT device $k$ is given by
\begin{equation}
r_{t,k}^{up} = b_{t,k} \log_{2} (1 + \frac{P_{t,k} G_{t,k}}{N_0 b_{t,k}}),
\label{transmitrate-up}
\end{equation}
where $b_{t,k}$ is the bandwidth allocated to IoT device $k$ by the edge server in the $t$-th round, $P_{t,k}$ is the power consumption of data transmit for IoT device $k$,  $G_{t,k}$ is the uplink channel gain between device $k$ and the edge server, and $N_0$ is the power spectral density of the Gaussian noise.

The achievable downlink transmit rate of IoT device $k$ is
\begin{equation}
r_{t,k}^{down} = b_{t,k} \log_{2} (1 + \frac{P_{t}^{s} H_{t,k}}{N_0 b_{t,k}}),
\label{transmitrate-down}
\end{equation}
where $P_{t}^{s}$ denotes the transmit power of the edge server, and $H_{t,k}$ is the downlink wireless channel gain from the edge server to device $k$.

The downloading time of the global model at device $k$ in the $t$-th round is
\begin{equation}
  \tau_{t,k}^{down} = \frac{\mathfrak{S}_{d}}{ r_{t,k}^{down}},
 \label{time-download}
\end{equation}
where $ \mathfrak{S}_{d}$ denotes the  size of the global model. The transmission time of the local model at the $t$-th round can be given by

\begin{equation}
  \tau_{t,k}^{up} = \frac{\mathfrak{S}_{u}}{ r_{t,k}^{up}}, \label{time-upload}
\end{equation}
where $\mathfrak{S}_{u}$ is the size of the local model. By substituting \eqref{transmitrate-up} to \eqref{time-upload}, the energy consumption of an IoT device on the local model transmission is

\begin{equation}
E_{t,k}^{up} = P_{t,k} \tau_{t,k}^{up}  = \frac{ P_{t,k} \mathfrak{S}_{u} }{b_{t,k} \log_{2} (1 + \frac{P_{t,k} G_{t,k}}{N_0b_{t,k}})}.
\label{energy-transmit}
\end{equation}

The total energy consumption $E_{t,k}^{c}$ of the selected IoT device $k$ in the $t$-th round is

\begin{equation}
E_{t,k}^{c} =  E_{t,k}^{cmp} + E_{t,k}^{up}
\label{energy-consumption-of-device}
\end{equation}

The remaining battery energy of the selected IoT device $k$ in the $t$-th round is
\begin{equation}
E_{t,k} = E_{t-1,k}- E_{t-1,k}^{c} + \Delta E_{t,k},
\end{equation}
where $\Delta E_{t,k}$ is the amount of harvested energy.

 \section{Problem Formulation}  \label{sec_problem_formu}
 In this section, we study the IoT device selection and transmit power allocation to maximize  the ratio of the learning accuracy of FL to the energy consumption of IoT devices.

 Let $\beta_{t,k}$ be a binary indicator. If IoT device $k$ is selected by the edge server in the $t$-th round, $\beta_{t,k} = 1$; otherwise, $\beta_{t,k} = 0$.  We define the accuracy of FL as the
the fraction of predictions FL model got right. According to \cite{zhan2020learning, lim2020towards, khan2020federated}, the accuracy of FL, denoted by $\Gamma(\beta_{t,k})$,  can be simplified as,
\begin{equation}
    \Gamma(\beta_{t,k} ) =   \log (1 +  \sum_{k=1}^{K} \mu_{k}\beta_{t,k} D_{t,k}) \quad \forall t \in \mathcal{T},
    \label{accuracy}
\end{equation}
where $\mu_{k} > 0$ is a system parameter ~\cite{lim2020towards}.
To improve the evenness of data size of the selected IoT devices, similar to \cite{Lyu2019}, we define the expectation of the difference between the total amount of data for all devices and the amount of data for the selected devices at the $t$-th training round, and normalize the expectation, we have,
\begin{equation}
   \Delta_t = \frac{\mathbb{E} \Big[\nu \sum_{k=1}^K D_{t,k}  - \beta_{t,k} D_{t,k}  \Big]}{\sum_{k=1}^K D_{t,k} } \quad \forall t \in \mathcal{T},
\end{equation}
where $ \nu  \in (0, 1]$ is a constant value, the arrival data $D_{t, k}$ of device $k$ follows uniform distribution or normal distribution. The objective function can be determined as  $ \frac{\Gamma(\beta_{t,k})}{ \sum_{k=1}^{K} \beta_{t,k} E_{t,k}^c} - \Delta_t$, which aims to balance the learning accuracy of FL and energy consumption of the
IoT device and improve the evenness of data size of the selected IoT devices. We formulate the optimization  as \textbf{P1}.
\begin{equation*}
\textbf{P1:} \quad \mathop{\max} \limits_{\beta_{t,k}, P_{t,k}} \   \sum_{t=1}^{T}  \bigg[     \frac{\Gamma(\beta_{t,k})}{ \sum_{k=1}^{K} \beta_{t,k} E_{t,k}^c} - \Delta_t \bigg]
 \end{equation*}
\begin{equation}
  \mathbf{s.t.}:    \beta_{t,k}  E_{t,k}^c \leq  E_{t,k}, \ (t \in[1, T], k \in [1, K]). \label{energy-constr}
\end{equation}

\begin{equation}
    \Gamma(\beta_{t,k}) \geq \epsilon_0, \ ( \epsilon_0  \in (0, 1]).
     \label{con-accuracy}
\end{equation}

\begin{equation}
    \sum_{k=1}^{K} \beta_{t,k}  b_{t,k} \leq  B, \ (t \in[1, T], k \in [1, K]).
    \label{con-bandwidth}
\end{equation}

\begin{equation}
    1 \leq \sum_{k=1}^{K} \beta_{t,k} \leq K, \ (t \in[1, T], k \in [1, K]).
  \label{con-selection}
\end{equation}

\begin{equation}
\beta_{t,k} \tau_{t,k} \leq t_{d}, \ (t \in[1, T], k \in [1, K]).
 \label{con-time}
\end{equation}

\begin{equation}
P_{k}^{min} \leq P_{t,k} \leq P_{k}^{max}, \ (t \in[1, T], k \in [1, K]).
\end{equation}

\begin{equation}
\beta_{t,k} \in \{0, 1\}, \ (t \in [1, T],  k \in [1, K]). \label{variable-constr}
\end{equation}

Specifically,
\begin{itemize}
     \item Constraint ($\beta_{t,k}  E_{t,k}^c \leq  E_{t,k}$) guarantees that the selected IoT device has sufficient energy to complete the local model training in $t$.
     \item  Constraint ($\Gamma(\beta_{t,k}) \geq \epsilon_0 $) specifies the minimum requirement of the FL accuracy in $t$, where $\epsilon_0 \in (0, 1]$ defines the lower bound threshold.
     \item  Constraint ($\sum_{k=1}^{K} \beta_{t,k}  b_{t,k} \leq  B$)
     guarantees that the total bandwidth of the selected IoT devices is smaller than the bandwidth capacity $B$.
     \item  Constraint ($1 \leq \sum_{k=1}^{K} \beta_{t,k} \leq K $) describes that at least one IoT device is selected for FL.
    \item Constraint ($ \beta_{t,k} \tau_{t,k} \leq t_{d} $) ensures that the download, computation and transmit delay of the selected device has to be less than the duration of the round $t_{d}$. As shown in Fig.~\ref{fig:3-1}, the each $t$-th round contains the downloading time of the global model, and local computation and transmission time of the local model.  According to \eqref{time-localtran}, \eqref{time-download}, and \eqref{time-upload}, we can obtain the time delay $\tau_{t, k}$, i.e., $\tau_{t, k} = \tau_{t,k}^{train} + \tau_{t,k}^{up} + \tau_{t,k}^{down}$.  Note that the IoT device selection time and training time of the global model at the edge server can be neglected since the edge server supports more powerful CPUs than the IoT device.

     \item  Constraint ($P_{k}^{min} \leq P_{t,k} \leq P_{k}^{max}$)  indicates the upper and lower bounds of IoT devices' transmit power.
\end{itemize}

Problem \textbf{P1} involves nonlinear problems with continuous and integer variables \cite{sahinidis2019mixed}. The typical 0-1 Multidimensional Knapsack Problem (MKP)  \cite{kellerer2004multidimensional} is a special case of problem \textbf{P1}. Assume that transmit power is a constant. In this case, the only variable is the selected IoT device. The items to be put in the knapsack are the IoT devices with energy consumption $E_{t,k}$, data size $D_{t,k}$, and bandwidth $b_{t,k}$. The capacity of the knapsack is equal to the total bandwidth, the optimization variable $\beta_{t,k}$ is a binary indicator of item (IoT device) $k$ selection. $\beta_{t,k}$ is set to 1 to indicate that item $k$ is selected. Otherwise, $\beta_{t,k}$ is set to 0. The total reliability  of the knapsack has lower bounds which are equal to the minimum requirement of accuracy constraint \eqref{con-accuracy}. Note that every item to be put into the knapsack must obey the time constraint \eqref{con-time}. Coupled with the energy consumption of IoT devices on the transmission is a non-linear function of the transmit power, which has a continuous variable. Therefore, the proposed optimization is NP-hard. It is also mentioning that  problem \textbf{P1} involves  a long time horizon with  random and unpredictable data and energy arrivals. This leads to an intractable large state space of the problem require online optimization of selection and allocation decisions.

\section{DRL-based IoT Devices Selection and Resource Allocation} \label{sec_drl-based}
\subsection{POMDP Formulation for FL-based IoT Device Selection and Resource Allocation} \label{sec_pomdp}
 The considered resource allocation can be formulated as a POMDP that is a generalization of a Markov decision process (MDP) with only partially observable states \cite{kaelbling1998planning}. A POMDP can be represented by a 6-tuple $(S, A, P, R, O, \Omega)$, where $S$ is the state space, $A$ is the action space, $P$ is the transition probability, $R$ is the reward function, $O$ is the observation space and $\Omega$ is the observation model. The edge server cannot observe the underlying state. Instead, an observation $S_{\alpha^{'}}^{o} \in O$ is received after a state transition to the next state $S_{\alpha^{'}}$ with the probability $\Omega (S_{\alpha^{'}}^{o} | S_{\alpha^{'}})$.

 \textit{State and Action Space}: According to problem \textbf{P1}, the state space of the POMDP consists of  data size, uplink channel gains, downlink channel gains, bandwidth and remaining energy of the IoT devices. The network state $S_{\alpha}$ is defined as
\begin{equation}
  S_{\alpha} = \{(D_{\alpha, k}, G_{\alpha, k}, H_{\alpha, k}, b_{\alpha, k}, E_{\alpha, k}), k = 1, \cdots, K \}.
\end{equation}

The action of the POMDP is the selection of the IoT devices for FL, denoted by  $\beta_{\alpha,k}$,  and the transmit powers of the selected IoT devices,  $P_{\alpha,k}$, as given by
\begin{equation}
  \mathcal{A} \in \{ (\beta_{\alpha,k}, P_{\alpha,k}), k = 1, \cdots, K \},
\end{equation}
where $\beta_{\alpha, k} \in \{0, 1 \}$ and $P_{\alpha,k} \in [P_k^{min}, P_k^{max}]$.

\textit{Observation Space}: At each state  $S_{\alpha}$, the edge server can observe partially the network state from the selected IoT devices, where the state observation $S^o_{\alpha}$ can be packed in its uploaded local model. Particularly, the state observation $S_\alpha^o  \in S_{\alpha}$ is given by
\begin{equation}
  S_\alpha^o = \{(D_{\alpha, k}, G_{\alpha, k}, H_{\alpha, k}, b_{\alpha, k}, E_{\alpha, k})_{o}, k = 1, \cdots, K \}.
\end{equation}
Note that the state of unselected IoT devices cannot be observed by the edge server.

\textit{Reward}: Let $ R(S_{\alpha^{'}}^o | S_\alpha^o, A_{\alpha} )$ denote the  immediate reward received when the action $A_{\alpha}  \in  \mathcal{A} $ is taken at state  $S_{\alpha}$.  The reward is defined to consist of the  AE gain that is the ratio of the FL accuracy to the energy consumption of the selected IoT devices, and the penalty resulting from the data unevenness among the selected IoT devices, i.e.,
  \begin{equation}
    R(S_{\alpha^{'}}^o | S_\alpha^o, A_{\alpha} ) =  \frac{\Gamma( \beta_{\alpha,k})}{ \sum_{k=1}^{K} \beta_{\alpha,k} E_{\alpha,k}^c}  - \Delta_{\alpha},
  \end{equation}
where $   R(S_{\alpha^{'}}^o | S_\alpha^o, A_{\alpha} ) $ indicates that the state observation transits to subsequent $S_{\alpha^{'}}^o$  from the current state observation $S_\alpha^o$. For illustration convenience,  we use $R_{\alpha}$ to denote the reward in the following sections.

To evaluate the action selected by a policy $\pi_{\theta}$ with  parameters $\theta$, where $\pi_{\theta}$ is a mapping from state observations to actions, and the set of all policies is defined as $\Pi$. We aim to maximize the expected total reward denoted as action-value function $Q_{\pi_{\theta}}(S_\alpha^o, A_{\alpha})$,
\begin{equation}
    Q_{\pi_{\theta}}(S_\alpha^o, A_{\alpha}) = \mathop{\max}\limits_{\pi \in \Pi}  \mathbb{E}_{S_\alpha^o}^{\pi_{\theta}} \bigg\{  \sum_{n = 0}^{\infty}  \gamma^{n}  R_{\alpha} \bigg\},
    \label{action-value function}
\end{equation}
where $\gamma \in [0, 1]$ is a  discount factor for future state observations. $\mathbb{E}_{S_\alpha^o}^{\pi_{\theta}} \{\cdot\}$ takes the expectation with respect to policy $\pi_{\theta}$ and state observation $S_\alpha^o$.  According to the Bellman equation \cite{bellman1966dynamic}, the optimal action-value function \eqref{action-value function} of  a state-action pair $(S_\alpha^o, A_{\alpha})$ and the value of the subsequent state-action pair $( S_{\alpha^{'}}^o, A_{\alpha^{'}})$ can be further rewritten as
\begin{equation}
\begin{split}
    Q_{\pi_{\theta}}(S_\alpha^o, A_{\alpha}) =  \mathop{\max}\limits_{\pi_{\theta} \in \Pi}  \mathbb{E}_{
    S_\alpha^o}^{\pi_{\theta}} \bigg\{ R_{\alpha} +  \gamma Q_{\pi_{\theta}}( S_{\alpha^{'}}^o, A_{\alpha^{'}})  \bigg\}.
\end{split}
\label{bellman-equation}
\end{equation}

The optimal action, namely, $A_{\alpha}^{*}$,  which satisfies \eqref{bellman-equation}, can be given by
\begin{equation}
\begin{split}
    A_{\alpha}^{*} =  \arg \mathop{\max}\limits_{\pi_{\theta} \in \Pi}  \mathbb{E}_{S_\alpha^o}^{\pi_{\theta}} \bigg\{ R_{\alpha} + \gamma Q_{\pi_{\theta}}( S_{\alpha^{'}}^o, A_{\alpha^{'}})  \bigg\},
\end{split}
\label{optimal-action}
\end{equation}
where $A_{\alpha}^{*}$ provides the maximized AE gain.

Given a practical scenario where the edge server has no prior
knowledge on the transition probabilities, we propose a FL-DLT3 framework that joint IoT device selection and transmit power allocation algorithm that utilizes TD3, one of the deep reinforcement learning techniques, to maximize the AE gain.

The state and action space increases dramatically with the number of devices, and the action space has both continuous and discrete actions. Moreover, the network state, in practice, is not always observable at the edge server because the data size and remaining energy information of IoT devices are always kept locally before the server takes the devices selection. In view of these challenges, it is difficult to find the exact solution of \textbf{P1}. To this end, we develop FL-DLT3 framework to find the near optimal solution of problem \textbf{P1}.

In general, FL-DLT3 consists of the TD3-based  deep reinforcement learning and the LSTM-based state characterization layer, as depicted in Fig.~\ref{FL-DLT3}. FL-DLT3 leverages the actor-critic neural network structure to develop the TD3-based  deep reinforcement learning \cite{fujimoto2018addressing}. The TD3 at the edge server is trained to optimize IoT device selection and transmit power allocation in a continuous action space, where the edge server has no priori information on the state transition probabilities. The LSTM is used to predict the hidden state of the IoT devices as input state of TD3. The AE gain is maximized over the large continuous state and action spaces. Specially, the global model is trained iteratively to improve the learning accuracy of the local model. With the growth of FL iterations,  FL-DLT3 optimally selects the IoT devices to maximize the accuracy of FL while ensuring the energy consumption requirement.

\begin{figure*}[htbp]
\centering
\includegraphics[scale=0.5]{FL-DLT3.png}
 \caption{ An illustration of the proposed FL-DLT3 framework, where a policy network, a policy target network, two critic networks and two critic target networks are trained. Each of the networks consists of a feedforward branch and recurrent branch. A training round of FL is performed by the selected IoT devices with the allocated transmit power. Then, the edge server gets the reward $R_{\alpha}$ and the state update $S_{\alpha^{'}}^{o}$ of the IoT devices. The transition $\{(A_{\alpha^{-}},  S_\alpha^o, A_{\alpha},  R_\alpha, S_{\alpha^{'}}^o)\}$  is stored into the replay buffer $\mathcal{B}$, and a mini-batch of transitions is randomly sampled from $\mathcal{B}$ to train the policy and critic networks on the edge server.}
\label{FL-DLT3}
\end{figure*}

\subsection{TD3 on the Edge Server}
TD3 utilizes the deterministic policy gradient algorithm (DPG) \cite{silver2014deterministic} to optimally update the current policy by deterministically mapping network states to a specific action of the edge server. The critic is used to approximate the action-value function, which is related to the update of the actor, also known as policy. Moreover, the edge server stores the transition  about  previous actions of the edge server,
current observation, actions of the edge server, AE gain, next observations, i.e., $(A_{\alpha^{-}},  S_\alpha^o, A_{\alpha},  R_\alpha, S_{\alpha^{'}}^o)$,  into the replay buffer $\mathcal{B}$ at each training step. A mini-batch of $N$ transitions are randomly sampled from $\mathcal{B}$ to train the actor-critic networks. The action will be selected according to the policy network with exploration noise after the first $M$ time steps. Since the IoT device selection action output from the policy network may not  be binary, we classify them as binary by setting thresholds. Based on the TD3 framework, the policy and critic networks can be trained to approximate the optimal policy of the formulated POMDP problem.

For the continuous IoT device selection and transmit power allocation, the objective is to find the optimal policy $\pi_{\theta}$, which maximizes the expected AE gain $J(\theta) = \mathbb{E}_{S_\alpha^o}^{\pi_{\theta}}  \big[ R(S_0^o, A_0) \big] $, the parameterized policies $\pi_{\theta}$ can be updated by taking the gradient of the expected return with respect to $\theta$, i.e.,
\begin{equation}
    \nabla_{\theta} J(\theta) =  \mathbb{E}_{\pi_{\theta}}  \big[ \nabla_{A_{\alpha}} Q_{\pi_{\theta}}(S_\alpha^o, A_{\alpha})|_{A_{\alpha} = \pi(S_\alpha^o)} \nabla_{\theta} \pi_{\theta}(S_\alpha^o) \big],
    \label{update-theta}
\end{equation}
where the optimal action-value function can be approximated by the critic neural network \cite{mnih2015human} $Q_{\omega}(S_\alpha^o, A_{\alpha})$ with parameters $\omega$, which obtains the AE gain. To update $Q_{\omega}(S_\alpha^o, A_{\alpha})$, the critic neural
network minimizes the approximation loss between the
current target value and $Q_{\omega}(S_\alpha^o, A_{\alpha})$ by adjusting the parameter $\omega$,
\begin{equation}
\begin{split}
\mathop{\min}\limits_{\omega}  \mathbb{E} \Big[  \Big( R_{\alpha}  +   \gamma Q_{\omega^{'}} \big(S_{\alpha^{'}}^o, \pi_{\theta}(S_{\alpha^{'}}^o) \big) - Q_{\omega}(S_\alpha^o, A_{\alpha}) \Big)^2 \Big],
\end{split}
\end{equation}
where $\omega^{'}$ stands for the periodically updated parameters of the critic target network, $\pi_{\theta}(S_{\alpha^{'}}^o)$ is the policy target network takes action in the next state observation  $S_{\alpha^{'}}^o$.

However, the update of the critic neural network with the above value function may result in overestimation, which can produce sub-optimal policies of the policy network. To deal with the overestimation bias problem, TD3 uses two approximately independent critic networks $\{ Q_{\omega_1}, Q_{\omega_2} \}$ to estimate the value function, i.e.,
\begin{equation}
\begin{split}
   & y_{\alpha, 1} = R_{\alpha}  +  \gamma Q_{\omega_1^{'}}(S_{\alpha^{'}}^o, \pi_{\theta}(S_{\alpha^{'}}^o)),  \\
   & y_{\alpha, 2} = R_{\alpha}  +  \gamma Q_{\omega_2^{'}}(S_{\alpha^{'}}^o, \pi_{\theta}(S_{\alpha^{'}}^o)),
\end{split}
\end{equation}
where the minimum of the two estimators is utilized in the update of the value function, i.e., $y_{\alpha} = \min \{ y_{\alpha, 1}, y_{\alpha, 2} \}$.

Moreover, FL-DLT3 follows TD3 to update the policy and targets less frequently than the Q functions. Delayed policy updates consists of only updating the policy and  critic target network every $d$ intervals. And we also add noise to the target action, which makes the policy network less likely to exploit actions with high Q-value estimates. The proposed FL-DLT3 IoT device selection and transmit power allocation policy is shown in Algorithm \ref{FL-DLT3-algorithm}.   Since the network system has $K$ IoT devices, each state of the IoT devices is made up of the source of data size, uplink channel gain, downlink channel gain, bandwidth, and the remaining energy of the IoT device. It takes $T$ rounds of iterations to before the algorithm terminates.  Therefore, the complexity of FL-DLT3 is $\mathcal{O}\big( T[ 7K(n_{pa}+n_{pc1} + n_{pc2}) + (l_{pa}-1)n_{pa}^{2}  +  (l_{pc1}-1)n_{pc1}^{2} + (l_{pc2}-1)n_{pc2}^{2}  + 12 \times (7K\times N_{lstm} + (7K)^2 + 7K)] \big)$, where $n_{pa},n_{pc1}$ and $n_{pc2}$ are the number of neurons in the hidden layer of policy network, critic network 1 and critic network 2, respectively; $l_{pa},l_{pc1}$ and $l_{pc2}$ are the number of the hidden layers of policy network, critic network 1 and critic network 2, respectively; $N_{lstm}$ is the hidden size of LSTM.

\subsection{Network Architecture}
 Each network follows two-branch structure as in \cite{peng2018sim}, which consists of a feedforward branch and recurrent branch (as shown in  Fig.~\ref{FL-DLT3}). The feedforward branch and recurrent branch both  use five fully-connected layers neural network of 512 hidden nodes. The recurrent branch consists of an embedding layer of 512 fully connected units followed by 512 LSTM units. For each hidden layer (apart from the LSTM),  ReLU activations are used between the two hidden layers for both the policy and critic. The final tanh unit and linear unit following the output of the policy and critic, respectively.

\subsection{LSTM Layer}
The unknown network observation transitions resulting from time-varying data size, channel gains, bandwidth, and energy harvesting, which increase learning uncertainties and reduce learning accuracy. The edge server running the FL-DLT3 cannot observe the complete states of all the IoT devices.  It can only make the observation of a IoT device when the device is selected and uploads its state information to the edge server. The learning efficiency and accuracy of the TD3-based FL-DLT3 can be compromised by the imperfect knowledge of the states of the IoT devices. Motivated by this fact, we develop a state characterization layer to predict the states of the IoT devices which are not observable, and feed the predicted states into the every neural network of FL-DLT3.

For every policy (target) network, two critic (target) networks and memory replay, we use LSTM to predict their respective hidden states. The LSTM-based neural network is used to find out the hidden state in the environment, e.g.,  the state of a device that has not been selected previously. Moreover, the LSTM based state characterization layer helps to  accelerate the convergence of the FL. At time state $i$, the hidden states $h_{i}^{hid}$ is calculated by the following composite function
\begin{equation}
h_{i}^{hid} = o_i  \tanh(C_i),
\end{equation}
\begin{equation}
o_{i} = \sigma( W_0 \cdot [C_i, h_{i-1}^{hid},  A_i ] + e_0),
\end{equation}
\begin{equation}
C_{i} = F_i C_{i-1} + p_i \tanh ( W_c \cdot [h_{i-1}^{hid},  A_i ] + e_c),
\end{equation}
\begin{equation}
F_{i} = \sigma ( W_f \cdot [h_{i-1}^{hid}, C_{i-1},  A_i ] + e_f),
\end{equation}
\begin{equation}
p_i = \sigma ( W_p \cdot [h_{i-1}^{hid}, C_{i-1},  A_i ] + e_p),
\end{equation}
where $o_{i}, C_{i}, F_{i}$, and $p_i$ denote the output gate, cell activation vectors, forget gate, and input gate of the LSTM layer, respectively.  $\sigma$, and $ \tanh$ refer to logistic sigmoid function and the hyperbolic tangent function, respectively. $\{ W_0, W_c,W_f,W_p \}$ are the weight matrix, and $\{ e_0, e_c,e_f,e_p \}$ are the bias matrix.

\begin{algorithm}
\SetAlgoLined
	\caption{The developed FL-DLT3 IoT device selection and resource allocation policy}
  Initialize critic networks $\{ Q_{\omega_1}(S_\alpha^o, A_{\alpha}, A_{\alpha^{-}}),  Q_{\omega_2}(S_\alpha^o, A_{\alpha}, A_{\alpha^{-}}) \}$ and policy network $\pi_{\theta}(S_\alpha^o, A_{\alpha^{-}})$ with random parameters $\omega_1, \omega_2, \theta$. \\
  Initialize target networks $\omega_1 ^ {'} \leftarrow \omega_1, \omega_2 ^ {'} \leftarrow \omega_2, \theta^{'} \leftarrow \theta$. \\
  Initialize  environment $S_{\alpha}^o$, previous actions $A_{\alpha^{-}} \leftarrow 0 $, replay buffer $\mathcal{B}$.\\
  \For{$\alpha = 1,  \cdots, T$}
	{
	\eIf{$\alpha \leq N$}{
	Explore $N$ steps with random policy, obtain $R_{\alpha}$ and $S_{\alpha^{'}}^o$, and storage the transition $(A_{\alpha^{-}}, S_\alpha^o, A_{\alpha}, R_{\alpha}, S_{\alpha^{'}}^o)$ of the $N$ steps into $\mathcal{B}$.
	}{
	Select action with exploration noise $A_{\alpha} \sim \pi_{\theta}(S_\alpha^o, A_{\alpha^{-}}) + \rho, \rho \sim \mathcal{N}(0, \sigma)$. \\
    Server allocates the selected IoT devices $P_{\alpha,k}$. \\
    Server performs model aggregation to obtain updated global model and distributes it to all selected devices in the next round. \\
    Server observes new observation $S_{\alpha^{'}}^o$, calculates $R_{\alpha}$, and stores the experience $(A_{\alpha^{-}}, S_\alpha^o, A_{\alpha}, R_{\alpha}, S_{\alpha^{'}}^o)$ into $\mathcal{B}$. \\
     Sample mini-batch of $N$ experiences $\{(A_{\alpha^{-}}, S_\alpha^o, A_{\alpha}, R_{\alpha}, S_{\alpha^{'}}^o)_i \}_{i=1}^N$ from $\mathcal{B}$.\\
    Obtain target action $\widehat{A}_{\alpha^{'}} \leftarrow \pi_{\theta^{'}}(S_{\alpha^{'}}^o, A_{\alpha} ) +  \widehat{\rho}, \widehat{\rho} \sim \rm{Clip}(\mathcal{N}(0, \widehat{\sigma}), -c, c )$. \\
$\widehat{y} \leftarrow R_{\alpha} + \gamma \mathop{\min} \limits_{m=1,2} Q_{\omega_m}(S_{\alpha^{'}}^o,\widehat{A}_{\alpha^{'}}, A_{\alpha})$. \\
Update critics $\omega_m \leftarrow \arg \mathop{\min} \limits_{\omega_m} \mathbb{E}(\widehat{y}  - Q_{\omega_m}(S_\alpha^o, A_{\alpha^{-}} ))^2$. \\
  \If{$\alpha$ mod $d$ = 0}
  {
  Update $\theta$ by the deterministic policy gradient using equation \eqref{update-theta}. \\
Update target networks: \\
$ \omega_{m^{'}} \leftarrow  \varphi \omega_m + (1 - \varphi)\omega_{m^{'}}$. \\
 $ \theta^{'} \leftarrow \varphi \theta + (1 - \varphi) \theta^{'}$. \\
  }
	}
   }
	\label{FL-DLT3-algorithm}
\end{algorithm}

\section{Numerical Experiments} \label{section-numerical}
In this section, we present the implementation of the proposed
FL-DLT3 on PyTorch, which is an open source machine learning library based on the Torch library. We compare FL-DLT3 with benchmarks in terms of network size, data size, and communication bandwidth. Table \ref{tab:experiment} specifies the configuration of simulation parameters.
\begin{table}[h!]
\small
\begin{center}
    \caption{Simulation parameters}
    \label{tab:experiment}

    \begin{tabular}{l|l}
     \hline
     \textbf{Parameters} & \textbf{Values} \\

      \hline

     Number of rounds ($T$) & 1000 \\
     Number of local iterations ($L$) & 4   \\
    \makecell[l]{For training one data sample \\ CPU  cycles per bit ($c_k$) } & 20 cycles / bit \\
    \makecell[l]{Computation capacity of IoT \\ device $k$ ($f_k$)}  & [2, 4] GHZ\\
     Transmit power of IoT device $k$ ($ P_{t,k}) $ &  [0.1, 60] W\\
     Transmit power of the edge server ($ P_{t}^{s}$) &  [100,1000] W\\
     Uplink channel gain  ($G_{t,k}$ ) & $[10^{-3}, 10^{-1}]$ dB\\
     Downlink channel gain ($H_{t,k}$ ) & $[10^{-1}, 10]$ dB\\
    \makecell[l]{Power spectral density of \\ the Gaussian noise ($ N_0$)}  & $1.0 \times 10^{-8}$ \\
     Parameter size of the global model ($S_d$) &  $ 1 \times 10^{4} $ bit \\
     Upload data size of IoT device  ($\mathfrak{S}_{u}$)  & $ 5 \times 10^{4} $ bit \\
     \makecell[l]{Binary indicator of device \\ selection ($\beta_{t,k}$ )} &  \{0, 1\} \\
     System parameter  ($\mu_{k}$) &  $4.2 \times 10^{-9}$\\
     Effective capacitance coefficient  ($\epsilon_k$) & $ 1.2 \times 10^{-28}$\\
     The amount of harvested energy ($\Delta E_{t,k}$) & [50, 200] J\\
     The constant value ($\nu$) & 1.0 \\

     Critic network learning rate  & $3 \times 10^{-4}$\\
     Policy network learning rate & $3 \times 10^{-4}$\\
     Neural network weight coefficient ($\phi$) & $5 \times 10^{-3}$ \\

    \makecell[l]{Interval of policy target \\ network update ($d$)}  & 10 \\
     Discount factor ($\gamma$) & 0.99 \\
     Batch size ($N$) & 45 \\
     Replay buffer size ($|\mathcal{B}|$) & $5 \times 10^{5}$ \\
     Exploration noise ($\sigma$) & $0.5$ \\
    Clipped normal noise ($c$) & $0.5$ \\
    \hline
    \end{tabular}
 \end{center}
\end{table}

\subsection{Implementation and Training of FL-DLT3}
We implement the proposed FL-DLT3 with Python 3.9. PyTorch is set up on a Linux workstation with 64-bit Ubuntu 18.04. FL-DLT3 trains the resource allocation with FL on 2 Nvidia's GPUs, one is GeForce GTX 1060 with 3 GB memory, the other is GeForce RTX 2060 with 6 GB memory.

The experience replay memory can store  $5 \times 10^{5}$ training samples in terms of \textit{previous actions of the edge server, current observation, actions of the edge server, AE gain, next observations}. The previous actions and current observation are combined to feed into policy network and critic networks to infer the hidden state. Both state space and action space are updated by using Adam optimizer, where the learning rate is $3 \times 10^{-4}$. At one training step, the policy and critic networks are trained with a mini-batch of 45 transitions, sampled from the experience replay memory.

The policy target network is implemented by adding $\rho \sim \mathcal{N}(0, 0.5) $ to the actions chosen by the policy target network, clipped to (0, 60). The delayed policy updates the policy and  critic target networks every $d$ intervals, where $d = 10$.

\subsection{AE Gain Performance}
For performance validation, we compare FL-DLT3 with existing state-of-the-art FL-based device scheduling approaches,  i.e., FedAECS \cite{zheng2021federated}, FedCS \cite{Nishio2019} and FedAvg \cite{McMahanMRA16}.
\begin{itemize}
    \item \textbf{FedAECS:} The edge server selects the IoT devices to fulfill a predetermined ratio of FL accuracy to energy consumption, while meeting the requirement of accuracy and bandwidth.
    \item \textbf{FedCS:}  Given the  bandwidth limit, the edge server selects  the maximum number of IoT devices for FL.
    \item  \textbf{FedAvg:}  Given the limit of the  bandwidth, the edge server determines the number of IoT devices for FL training, and randomly selects the IoT devices.
\end{itemize}

Fig. \ref{fig:experment-1} shows the AE gains, where the $t$-th round is from 1 to 1000 and $K$ = 40. The data size $D_{t,k}$ and bandwidth $b_{t,k}$ of the $K$ devices vary in [2, 10] MB and [10, 50] KHz, respectively. In general, the proposed FL-DLT3 achieves the highest AE gain, as compared to the existing FedAECS, FedCS, and FedAvg, improved by 51.8\%, 82.4\% and 85.0\% respectively.  The reason is that FL-DLT3 leverages experience replay and predict the states of the IoT devices which are not observable, however, FedAECS, FedCS, and
FedAvg are unable to predict the states of the IoT devices.
\begin{figure}[htbp]
\centering
\includegraphics[scale=0.5]{Compare_with_benchmarks.png}
\caption{Comparison of the AE gains, where $T = 1000$ and $K = 40$.}
\label{fig:experment-1}
\end{figure}

FL-DLT3 outperforms FL-DLT3 without the LSTM layer with a gain of 39.8\%. This is because the LSTM layer efficiently predicts the unknown network observation transitions, which enriches the training environment for FL and TD3.

Fig.~\ref{fig:experment-2} studies the AE gain of the proposed FL-DLT3, where $K$ increases from 10 to 80. In general, the proposed FL-DLT3 achieves the highest AE gain   11.4557, as compared to the existing FedAECS (2.7212), FedCS (1.3952), and FedAvg (1.3498) given $K = 10$.
\begin{figure}[htbp]
\centering
\includegraphics[scale=0.55]{AE_gain_with_number_of_devices.png}
\caption{Comparison of the AE gain obtained by using FL-DLT3, FL-DLT3 without LSTM, FedAECS, FedCs and FedAvg with different number of IoT devices}
\label{fig:experment-2}
\end{figure}

In Fig.~\ref{fig:experment-2sub}, it can be observed that the energy consumption of FL-DLT3 dominates the performance when $K$ increase from 10 to 50. When $K >$   60, the FL accuracy dominates the performance. This confirms that the AE gain's fluctuation in Fig.~\ref{fig:experment-2} when K increase from 10 to 50.
The reasons is that  FL-DLT3 can select more IoT devices, hence the FL accuracy and energy consumption increase monotonically with the number of devices.

\begin{figure}[htbp]
\centering
\includegraphics[scale=0.55]{FL_DLT3_accuracy_energy_vs_td3_no_lstm.png}
\caption{Compare the FL accuracy and energy consumption of FL-DLT3 and FL-DLT3 without LSTM as the number of devices changes}
\label{fig:experment-2sub}
\end{figure}

Fig.~\ref{fig:experment-3} shows the AE gain  given 1000  rounds. With an increase of $K$, the AE gain achieved by FL-DLT3 decreases from 5.3411 to 3.7066. This also validates the performance in Fig. ~\ref{fig:experment-2}.
\begin{figure}[htbp]
\centering
\includegraphics[scale=0.5]{AE_gain_with_different_number_of_devices.png}
\caption{The AE gain achieved by the proposed FL-DLT3 given $K$ =  20, 40, or 80.}
\label{fig:experment-3}
\end{figure}

 Fig.~\ref{fig:experment-4} depicts the AE gain of FL-DLT3 with regards to the average data size. In general, the AE gain increases with the growth of the data size. This is because the FL accuracy rises in a high rate while the energy consumption of the IoT devices on the training of FL-DLT3 slightly increases. Moreover, the AE gain obtained by FL-DLT3 is about twice that obtained by FedAECS, thanks to the transmit power allocation in FL-DLT3.
 \begin{figure}[htbp]
\centering
\includegraphics[scale=0.55]{average_datasize_varying_with_AE_gain.png}
\caption{AE gain varying with data size following normal distribution while keeping the same variance (0.2 MB) }
\label{fig:experment-4}
\end{figure}

 Fig.~\ref{fig:experment-5} describes the AE gain of FL-DLT3 with regards to the average bandwidth. Overall, the AE gain raises with an increase of the average bandwidth since the energy consumption of the IoT devices on the local models transmission is reduced,  in other words, a better channel quality leads to smaller packet retransmission and energy consumption.  In addition,  FL-DLT3 achieves the highest AE gain given different bandwidths. This is achieved by the LSTM layer is integrated into the FL-DLT3 to predict the time-varying bandwidth.

\begin{figure}[htbp]
\centering
\includegraphics[scale=0.55]{AE_gain_varying_with_channel.png}
\caption{AE gain varying with bandwidth following normal distribution while keeping the same variance (4 KHz) }
\label{fig:experment-5}
\end{figure}

Fig.~\ref{fig:experment-5sub} shows  the FL accuracy and energy consumption of FL-DLT3 and FL-DLT3 without LSTM in regard to the average bandwidth size of IoT devices changes. Generally, the energy consumption of FL-DLT3 decreases with an increase of the  bandwidth since the retransmission of the local models is reduced.

\begin{figure}[htbp]
\centering
\includegraphics[scale=0.55]{FL_DLT3_accuracy_energy_channel_varying.png}
\caption{Compare the FL accuracy and energy consumption of FL-DLT3 and FL-DLT3 without LSTM as the average channel size of IoT devices changes}
\label{fig:experment-5sub}
\end{figure}

 Fig.~\ref{fig:experment-6} shows the runtime of FL-DLT3, where $K$ is set to 10, 40, 60, or 80. In the first 45 rounds, the runtime of FL-DLT3 is about 0.018 s. This is because the experience replay buffer is initialized in which the training of FL-DLT3 is not conducted yet.  Once the learning experience is sufficient and FL-DLT3 carries out the training, the runtime raises to 0.74 s given $K$ = 10. When K increases to 80, the training of FL-DLT3 takes 1.3286 seconds. The reason is that an increase of the IoT devices results in a large state and action space, hence the learning time of FL-DLT3 increases.  In addition, the runtime of FL-DLT3 randomly fluctuates. This is due to random interruptions from other program executions that are concurrently operated on the server.

\begin{figure}[htbp]
\centering
\includegraphics[scale=0.55]{runtime_varying_with_t_round.png}
\caption{Runtime with $t$-th round}
\label{fig:experment-6}
\end{figure}

\section{Conclusion} \label{sec_conclusion}
In this paper, we proposed FL-DLT3, which is a new deep reinforcement learning based resource allocation with FL for EdgeIoT. Given the large state and action space, FL-DLT3 learns the network state dynamics while maximizing the ratio of the learning accuracy of FL to the energy consumption of the IoT devices in a continuous domain. To  improve  the  learning  accuracy,  a  new  state  characterization  layer  based  on  LSTM  is  developed  in  FL-DLT3  to  predict  the time-varying  data  size,  bandwidth,  channel  gain,  and remaining energy of the IoT devices. FL-DLT3 is implemented in PyTorch. The effectiveness of FL-DLT3 is validated with the experimental data.

\section*{Acknowledgements}
This work was supported in part by the National Funds through FCT/MCTES (Portuguese Foundation for Science and Technology), within the CISTER Research Unit under Grant UIDP/UIDB/04234/2020, and in part by national funds through FCT, within project PTDC/EEI-COM/3362/2021 (ADANET).

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}


text:
\section{Introduction}

The \textit{Kepler} mission was launched on March 6, 2009. It was designed explicitly to be able to detect earth-size planets, including those in the habitable zone. The detection method for \textit{Kepler} to achieve this goal is measuring planetary transits to a high level of photometric precision. To improve the chances of detection, a large number of stars have to be monitored for a long period of time, preferably continuously. It was quickly realised that \textit{Kepler} would also provide extremely valuable data to study stellar pulsations. Data for this purpose are obtained from a subset of the about 500 stars that are sampled every 58.8 seconds (short cadence), out of a much larger set of about 150 000 stars observed at a rate of 29.4 minutes \citep{borucki2008}.

{
The satellite orbits in an earth-trailing heliocentric orbit (ETHO), with a 55$^\circ$ Sun avoidance angle, and continuously observes a patch of the sky centred on the Cygnus-Lyra region. The data are organised into quarters; after each quarter the photometer is rolled 90$^\circ$ to keep the solar arrays pointed at the Sun. This causes a 42 hour data gap \citep{haas2010}. This work uses Q0-Q11 data of HAT-P-7, which has been continuously observed in \textit{Kepler}'s short cadence mode.
}

HAT-P-7 (KIC 10666592, visual magnitude 10.46, spectral type F8) has been identified as a planet host by ground-based observations prior to the start of the \textit{Kepler} mission \citep{pal2008}. It is one of the most extensively studied star-planet systems known to date. A study of the stellar pulsations has been carried out by \cite{hatp7joergen}, based on early \textit{Kepler} data (Q0-Q1). After the initial discovery of its exoplanet HAT-P-7b \citep{pal2008}, phase variations and planetary occultations were discovered in \textit{Kepler}'s Q0 data \citep{hatp7science}, and consequently the planetary temperature and atmosphere were studied \citep{hatp7hotdayside,hatp7atmosphere}. Additional ellipsoidal variations were later discovered in the light curve \citep{hatp7ellipsoidalvariations} and the system is currently the only planetary case where this effect has been found. HAT-P-7 has a surprising inclination angle: a study of the Rossiter-McLaughlin effect in radial velocity data has found it to be retrograde or polar \citep{hatp7thirdbody}. These authors also suggest the presence of a third body. A study of transit timings based on EPOXI data found no evidence of an additional planet \citep{hatp7additionalplanets}.

Here, we make a preliminary anaysis of the system based on almost three years of \textit{Kepler} data. We derive stellar pulsation frequencies and perform an asteroseismic modelling to derive the properties of the host star. The planetary system is then studied making use of these newly determined stellar properties.

\section{Stellar properties}

Stellar pulsations cause temperature variations, which lead to flux variations. The frequencies are observed from the power spectrum. Different models are then compared with the observed frequencies and the best fit leads to the stellar properties. The procedure followed here resembles the asteroseismic analysis of HAT-P-7 based on only the first month of \textit{Kepler} data \citep{hatp7joergen}.

\subsection{Observations and data analysis}

The raw \textit{Kepler} data are detrended and normalised, and transit-like features are removed using a specifically designed median filter. The filtered photometry is then used to calculate a power spectrum. A spectrum with a sampling rate of 0.005 $\mu$Hz is used, which means an oversampling of about three times. A median filter with a period of 1 $\mu$Hz is used to smooth the power spectrum, to remove the fine structure of the spectrum, caused by the finite lifetime of the modes.

The location of individual frequencies is governed by the asymptotic equation \citep{tassoul1980}:

\begin{equation}\label{eq:asymptotic}
 \nu_{n,l} \approx \Delta \nu \left( n + \frac{1}{2}l + \epsilon \right) - \delta \nu_{0l}.
\end{equation}
In this equation, $\nu_{n,l}$ is the frequency with radial order $n$ and angular degree $l$ (neglecting rotational splitting so there is no dependence on $m$), and $\Delta \nu$ and $\delta \nu_{0l}$ the large and small separations while $\epsilon$ is an offset coming from second order and near-surface effects. This theoretical relation is used to identify the individual modes, from the smoothed power spectrum. The entire procedure is illustrated in Figure \ref{frequency_observations}.

\begin{figure}
\includegraphics[width=1\columnwidth]{hatp7_powerspectrum.eps}
\includegraphics[width=1\columnwidth]{hatp7_powerspectrum_zoom.eps}
\caption{Top figure: oversampled (red) and smoothed power spectrum (black), showing regular spacings governed by the asymptotic equation (eq. \ref{eq:asymptotic}). Bottom figure (zoom): mode identification for three observed frequencies. An algorithm determines the maxima in the smoothed spectrum, the relevant maxima (vertical lines) are selected manually making use of the asymptotic equation. Sometimes, an average of two maxima is used (e.g. $l = 2$ mode in this figure).}
\label{frequency_observations}
\end{figure}

We have determined 14 ($l = 0$), 13 ($l = 1$) and 16 ($l = 2$) frequencies, which constitute a total of ten more frequencies than the initial asteroseismic analysis \citep{hatp7joergen}. The frequencies were determined with an average standard error estimated to be $\sigma_\nu = $ 0.75 $\mu$Hz (compared to a previous accuracy of 1.4 $\mu$Hz). They are illustrated in Figure~\ref{fig:echelle}.

\subsection{Modelling}

To calculate evolution models and adiabatic oscillation frequencies, we use the Aarhus codes \citep{ASTEC,ADIPLS}. Evolution tracks were calculated for a grid of models with mass range 1.33-1.70 M$_\odot$ and metallicity 0.015-0.035. The convective core was modelled with overshoot in units of the pressure scale height of $\alpha_{\textrm{ov}} = 0$, 0.1 and 0.2. Based on the observed large separation an estimate of the density is obtained \citep{temperaturescalingrelation}, which is used to select relevant time steps in the stellar evolution tracks, for which frequencies are then calculated.

The calculated frequencies are compared with the observations, after correcting for near-surface effects that are not modelled properly. The empirical correction law is given by \citep{surfacecorrection}:

\begin{equation}
 \nu_{\rm{obs}}(n) - \nu_{\rm{best}}(n) = a \left[\frac{\nu_{\rm{obs}}(n)}{\nu_0}\right]^b,
\end{equation}
where $\nu_0$ is a reference frequency and $a$ and $b$ are parameters to be determined. In addition, we set $\nu_{\rm{best}}(n) = r\nu_{\rm{ref}}(n)$, where $r$ should be close to one as the reference model should be close to the initial best model. In practice, we set $b = 4.90$ (the solar value) and we find the best fit for $r$ and $a$ (for each model individually). We can then calculate the reference model, for comparison with the observed frequencies. The best model is then picked by comparing the observed frequencies with the corrected model frequencies. We use a $\chi^2$ minimisation to determine which model fits best:

\begin{equation}
 \chi^2 = \frac{1}{(N-1)\sigma_\nu^2} \sum_{nl} \left( \nu_{\rm{nl}}^{\rm{(obs)}} - \nu_{\rm{nl}}^{\rm{(mod)}} \right)^2.
\end{equation}

\begin{figure}
\includegraphics[width=1\columnwidth]{hatp7_echelle_model.eps}
\caption{An \'echelle diagram showing the observed frequencies (red) and the best model (blue).}
\label{fig:echelle}
\end{figure}

The frequencies of the best model are shown together with the observational frequencies in the \'echelle diagram in Figure~\ref{fig:echelle}. With the exception of a number of frequencies (between 1200 and 1400 $\mu$Hz) for $l = 1$, the frequencies are modelled very well, resulting in $\chi^2 = 3.65$. Gaussian noise was added to the observational frequencies to create 1000 different frequency sets. For each, the best model is calculated and the stellar parameters are calculated. The average and standard deviation of these parameters are presented in Table~\ref{tab:hatp7_parameterlist}.

\begin{table}
 \caption{Parameters obtained for star (HAT-P-7) and its planet (HAT-P-7b).
\label{tab:hatp7_parameterlist}}
\begin{center}
\begin{tabular}[width=1\columnwidth]{ccc}
\hline
  \multicolumn{2}{c}{HAT-P-7}  \\
\hline
  Mass (M$_\odot$) 		& 1.361 $\pm$ 0.021 		\\
  Z$_0$				& 0.01904 $\pm$ 0.0015 		\\
  $[\rm{Fe}/\rm{H}]$		& 0.13				\\
  Age (Gyr) 			& 2.19 $\pm$ 0.12 		\\
  Radius (R$_\odot$) 		& 1.904 $\pm$ 0.010 		\\
  Temperature (K) 		& 6259 $\pm$ 32 		\\
  Density (g/cm$^{-3}$) 	& 0.2781 $\pm$ 0.0017		\\
  Luminosity (L$_\odot$) 	& 4.996 $\pm$ 0.098		\\
  $\log g$ (cgs) 		& 4.98 $\pm$ 0.13		\\
  Ellips. var. $d$ (ppm)	& 59 $\pm$ 1			\\
\hline
\hline
  \multicolumn{2}{c}{HAT-P-7b}  \\
\hline
Orbital period (d) 			& 2.20473506	(11)	\\
Orbital inclination i ($^\circ$) 	& 86.68 $\pm$ 0.14	\\
$M_{\rm p}$ (M$_J$) 			& 1.741 $\pm$ 0.028	\\

$R_{\rm p}$ (R$_J$)			& 1.431 $\pm$ 0.011	\\
$T_{\rm{day}}$	(K)			& 2470 $\pm$ 10 	\\
$T_{\rm{night}}$ (K)			& 1772 $\pm$ 10		\\
$A_{\rm{g}}$ 				& 0.193 $\pm$ 0.002	\\
\hline
\end{tabular}
\end{center}
\end{table}

\section{Planetary properties}

\subsection{Period}

The time of each individual transit is measured and used to determine the planetary period. Assuming a perfectly Keplerian orbit, we fit a straight line through all transit times (taking gaps into account; whenever a transit is measured incompletely it is ignored). Due to the large amount of measured transits (286), the period is determined very accurately. An O-C plot has been calculated, which compares the observed and the calculated transit times to look for a gravitational influence from additional bodies in the system. No sinuisoidal pattern was detected, and variations are at the level of the observational error level for individual transit times ($\approx 0.00015$ days). After the period has been calculated, the time series is folded (calculated modulo the period).

\subsection{Transit shape}

From the folded time series, the transit is modelled. The transit duration determines the impact parameter, which is directly related to the orbital inclination. The transit depth determines the relative planetary radius, which can be combined with the stellar radius (derived in the previous section), to obtain the absolute planetary radius. The shape of the bottom of the transit depends on the limb darkening, modelled by a  quadratic intensity profile \citep{exoplanetsbook_chapterwinn}:

\begin{equation}
I(X,Y) \propto 1 - u_1 (1-\mu) - u_2 (1-\mu)^2,
\end{equation}
with $\mu = \sqrt{1 - X^2 - Y^2}$ and $(X,Y)$ the coordinates on a unit circle. The coefficients $u_k$ are constants defining the precise shape of the limb-darkening law. We have simply used them as fitting parameters. The modelled transit is shown in Figure~\ref{fig:folded_flux}. We have also made a new estimate of the planetary mass, using the radial velocity values derived by \cite{pal2008}.

\subsection{Phase variations}

The planetary system is tidally locked, which causes the star to always `see' the same side of the planet, while a distant observer sees the planetary dayside and nightside at different times. However, for HAT-P-7 the situation is slightly complicated. The close orbit of the planet has induced a tidal distortion on the star: rather than being oblate due to rotation, it has its longest axis towards the planet and the shortest axis perpendicular to the orbital plane. This causes ellipsoidal variations (which peak at phases near 0.25 and 0.75), which should be added to the phase variations \citep{pfahl2008,hatp7ellipsoidalvariations}. HAT-P-7 is the only currently known system with ellipsoidal variations caused by a planet. We model the flux throughout one orbital phase as

\begin{eqnarray}
f(\Phi, i) &=& A_{\rm g} \left(\frac{R_{\rm p}}{a}\right)^2 \frac{\sin(\alpha) + (\pi-\alpha)\cos(\alpha)}{\pi}\\
 & & - d \frac{\cos(2\alpha)}{\pi} \nonumber
\end{eqnarray}
with the orbital phase $\Phi \in$ [0,1] taken as 0 at maximum radial velocity of the star, so that the phase angle $\alpha$ is defined as $\cos \alpha = - \sin i \sin 2\pi \Phi.$ The first part of $f(\Phi, i)$ is the Lambert law for a sphere \citep{planetaryreflection}, where $A_{\rm g}$ is the planetary albedo and $a$ is the semi-major axis. A non-zero value for $d$ corresponds to the ellipsoidal variations. We refer to Table~\ref{tab:hatp7_parameterlist} for values and to Figure~\ref{fig:folded_flux} for the best fit.

\subsection{Occultation}

When the planet moves behind the star, an occultation occurs. The depth $\delta_{\rm{flux}}(\lambda)$ gives information about the planetary dayside temperature. By combining the information with the phase variations, we can also obtain the nightside temperature. We approximate both star and planet as a blackbody and use

\begin{equation}
 \delta_{\rm{flux}}(\lambda) = \left(\frac{R_{\rm p}}{R_*}\right)^2 \frac{B_\lambda (T_{\rm p})}{B_\lambda (T_\star)},
\end{equation}
where $B_\lambda (T)$ is the Planck function \citep{exoplanetsbook_chapterwinn}, which we integrate over the \textit{Kepler} bandpass. The exact occultation depth is calculated by simply comparing the averages of data points inside and just outside the occultation, as shown in Figure~\ref{fig:folded_flux}. We find an occultation depth of 71.85 $\pm$ 0.23 parts per million. We assume that all planetary flux comes from radiation (albedo zero), so the quoted temperature value is an upper limit. The indicated accuracy of the quoted temperatures should be seen as the internal error of using a black-body spectrum to describe the spectrum of the exoplanet. Since the black-body flux in the \textit{Kepler} bandpass for exoplanet temperatures will change a lot even for a small change in temperature one can estimate the black-body temperature precisely. We find the occultation to be 4.90 $\pm$ 0.25 ppm deeper than the value just outside transit.

\begin{figure}
\includegraphics[width=1\columnwidth]{hatp7_transit_fit_report.eps}
\includegraphics[width=1\columnwidth]{hatp7_phasevariation_fit.eps}
\includegraphics[width=1\columnwidth]{hatp7_occultation.eps}
\caption{After the planetary period is determined, the observations (blue) are folded. This allows for a detailed study of the planetary transit (top figure), phase variations (caused by planetary day-nightside and ellipsoidal variations, middle figure), and planetary occultation (bottom). The black line shows a data fit, the colored data points are used in calculating the occultation depth.
}
\label{fig:folded_flux}
\end{figure}

\section{Discussion}

The asteroseismic analysis of the host star has followed largely the same methods as applied by \cite{hatp7joergen} for \textit{Kepler}'s Q0 and Q1 data. With eleven quarters now available, all parameters are determined to higher accuracy. The agreement is fairly good, but we have derived a slightly lower mass, radius and metallicity. We have not been able to model a few frequencies properly (see Figure~\ref{fig:echelle}), a problem that did not present itself in the earlier analysis by \cite{hatp7joergen} because of the fewer frequencies observed, and a lower observational accuracy. Ellipsoidal variations were found by \cite{hatp7ellipsoidalvariations}, and we confirm that the phase curve cannot be fit without including this effect. \cite{hatp7ellipsoidalvariations} find a variation of 37 ppm (no errors quoted, based on \textit{Kepler} Q0-Q1 data), which is considerably lower than our value of $59 \pm 1 {\rm \, ppm}$. All planetary values have improved accuracy. Most are in general agreement with earlier work. Our planetary temperatures are significantly lower than all earlier temperature determinations \citep{pal2008,hatp7science,hatp7ellipsoidalvariations,hatp7atmosphere}. They all used an occultation depth of $130 \pm 11 {\rm \, ppm}$ \citep{hatp7science}, which is almost twice as high as the value derived in this work; the analysis was done based only on \textit{Kepler} Q0 data, and the modelling did not include ellipsoidal variations as they were not yet discovered in HAT-P-7 at the time. On the other hand, the temperature in this work assumes a blackbody, which is an oversimplification and also causes the temperature error to be underestimated. \cite{hatp7ellipsoidalvariations} and \cite{hatp7atmosphere} used more realistic atmosphere models, and it should be interesting to see how they are affected by the revised occultation depth.

This work shows the excellent quality \textit{Kepler} can now deliver, both for studying stellar pulsations and planetary transits. Asteroseismology is likely to take up an increasingly important role in studying host stars of exoplanets.

\acknowledgements
The research leading to these results has received funding from the European
Research Council under the European Community's Seventh Framework Programme
(FP7/2007--2013)/ERC grant agreements n$^\circ$227224 (PROSPERITY) and n$^\circ$267864 (ASTERISK). Funding for the Stellar Astrophysics Centre is provided by The Danish National Research Foundation. We are grateful to the entire \textit{Kepler} team for the efforts leading to this highly successful mission.

redpajama_set_name: RedPajamaArXiv