Telecommunication Systems
April 2018 , Volume 67, Issue 4, pp 763–780 | Cite as
Optimal energy-delay tradeoff for opportunistic spectrum access in cognitive radio networks
Oussama Habachi
Yezekael Hayel
Rachid El-Azouzi
First Online: 29 August 2017
Cognitive radio (CR) has been considered as a promising technology to enhance spectrum efficiency via opportunistic transmission at link level. Basic CR features allow secondary users (SUs) to transmit only when the licensed channel is not occupied by primary users (PUs). However, waiting for an idle time slot may lead to large packet delays and high energy consumption. We further consider that the SU may decide, at any moment, to use another dedicated way of communication (4G) in order to transmit his packets. Thus, we consider an Opportunistic Spectrum Access (OSA) mechanism that takes into account packet delay and energy consumption. We formulate the OSA problem as a Partially Observable Markov Decision Process (POMDP) by explicitly considering the energy consumption as well as packets' delay, which are often ignored in existing OSA solutions. Specifically, we consider a POMDP with an average reward criterion. We derive structural properties of the value function and we show the existence of optimal strategies in the class of the threshold strategies. For implementation purposes, we propose online learning mechanisms that estimate the PU activity and determine the appropriate threshold strategy on the fly. In particular, numerical illustrations validate our theoretical findings.
Cognitive radio POMDP OSA Online learning
Appendix 1: Proof of Lemma 1
In this lema, we prove that the belief update function \(\varOmega ^{ns}\) is increasing with the belief. First, \(\varOmega ^{ns}\) is linear with the belief since \(\varOmega ^{ns}(\lambda )=\beta +(\alpha -\beta )\lambda \). Second, let us prove that \(\varOmega ^{ns}(\lambda )\ge \lambda \) if \(\lambda \le \pi (0)\) by induction on the belief.
We have the initial condition: \(\beta \le \pi (0)=\frac{\beta }{1-\alpha +\beta }\) and \(\varOmega ^{ns}(\beta )=\beta +(\alpha -\beta )\beta \ge \beta \).
We assume that \(\varOmega ^{ns}(\lambda )\ge \lambda \) for a given \(\lambda \le \pi (0)\).
The induction operator gives:
$$\begin{aligned} \varOmega ^{ns}(\varOmega ^{ns}(\lambda ))= & {} \beta +(\alpha -\beta )\varOmega ^{ns} (\lambda )\\\ge & {} \beta +(\alpha -\beta )\lambda =\varOmega ^{ns}(\lambda ) \end{aligned}$$
Thus, \(\varOmega ^{ns}(\lambda )\ge \lambda \) for all \(\lambda \le \pi (0)\). The analysis for \(\lambda \ge \pi (0)\) is similar. Note that we have considered the case where \(\alpha \ge \beta \), then the update function is increasing with the belief. \(\square \)
Appendix 2: Proof of Proposition 1
The proof of the proposition 1 is similar to [22] where the authors consider the finite time horizon problem. Hence, we briefly describe the procedure for this proof. Considering the maximum packet delay \(l^*\) and for all belief vector \(\lambda \), the value function \(V(\varvec{\lambda },l^*)\) is linear with the belief because
$$\begin{aligned} V(\varvec{\lambda },l^*)= & {} Q_2(\varvec{\lambda },l^*)-g_u,\\= & {} -g_u+\varPhi -c_s-P_{4G}+V(\varOmega ^{s}(\varvec{\lambda }|\theta =1),1)\\&+\lambda _{n^*}(P_{4G}-P_p+V(\varOmega ^{s}(\varvec{\lambda }|\theta =0),1)\\&-V(\varOmega ^{s}(\varvec{\lambda }|\theta =1),1)). \end{aligned}$$
Then the value function \(V(\varvec{\lambda },l^*)\) can be rewritten as an inner product of the belief vector and a \(\varUpsilon \)-vector. As \(Q_2(\varvec{\lambda },l)=Q_2(\varvec{\lambda },l^*)\), for all l, the action-value function \(Q_2(\varvec{\lambda },l)\) can be also rewritten as an inner product of the belief vector and a \(\varUpsilon \)-vector. We suppose that Proposition 1 holds for all packet delays higher than \(l+1\) and we prove that the proposition is true for packet delay l. After some algebra, we can rewrite the action-value functions given in (6) and (8) in terms of \(\varUpsilon \)-vector:
$$\begin{aligned} Q_0(\varvec{\lambda },l)= & {} -f(l)+\max \limits _{\varUpsilon \in \varGamma _{l+1}}<\varOmega ^{ns}(\varvec{\lambda }|\theta ),\varUpsilon >\nonumber \\= & {} -f(l)+\sum \limits _{s\in {\mathcal {S}}}\omega _{s}\left[ \sum \limits _{s^{\prime }\in {\mathcal {S}}}P(s^{\prime }|s)\varUpsilon _{l+1}^{\varOmega ^{ns} (\varvec{\lambda }|\theta )}\right] , \end{aligned}$$
$$\begin{aligned} Q_1(\varvec{\lambda },l)= & {} -c_s+\lambda _{n^*}(\phi -P_p+V({\varOmega ^{s} (\varvec{\lambda }|\theta =0)},1))\nonumber \\&+(1-\lambda _{n^*})\nonumber \\&\left( -f(l)+\max \limits _{\varUpsilon \in \varGamma _{l+1}}<{\varOmega ^{s} (\varvec{\lambda }|\theta =1)},\varUpsilon >\right) \nonumber \\= & {} -c_s+\lambda _{n^*}(\phi -P_p+V(\varOmega ^{s}(\varvec{\lambda }| \theta =0),1))\nonumber \\&+(1-\lambda _{n^*})\nonumber \\&\left( -f(l)+\sum \limits _{s\in {\mathcal {S}}}\omega _{s} \left[ \sum \limits _{s^{\prime }\in {\mathcal {S}}}P(s^{\prime }|s) \varUpsilon _{l+1}^{\varOmega ^{s}(\varvec{\lambda }|\theta =1)}\right] \right) ,\nonumber \\ \end{aligned}$$
where \(\varUpsilon _{l+1}^{\varOmega ^{ns}(\varvec{\lambda }|\theta )}\) and \(\varUpsilon _{l+1}^{\varOmega ^{s}(\varvec{\lambda }|\theta =1)}\) are, respectively, the \(\varUpsilon \)-vectors for the regions containing belief vectors \(\varOmega ^{ns}(\varvec{\lambda }|\theta )\) and \(\varOmega ^{s}(\varvec{\lambda }|\theta =1)\), respectively. Each term in the square brackets of (9) and (10) are elements \(\varUpsilon _{\lambda ,l}\) of a \(\varUpsilon \)-vector \(\varUpsilon _{l}\). Then the action-value functions can be rewritten as an inner product of the belief vector and a \(\varUpsilon \)-vector \(\varUpsilon _{l}\). Moreover, there are only a finite number of such \(\varUpsilon \)-vector \(\varUpsilon _{l}\) since we have a finite set of belief for all l. As the maximum of a finite set of piecewise linear and convex functions is also piecewise linear and convex, the Proposition 1 holds. \(\square \)
Let us prove first that the value function \(V(\varvec{\lambda },l)\) is monotonically decreasing with the packet delay l for all belief vector \(\varvec{\lambda }\). The SU takes the action 2 for all \(\varvec{\lambda }\) when the packet delay is \(l^*\), thus we have:
$$\begin{aligned} V(\varvec{\lambda },l^*)= & {} \varPhi -c_s+\lambda _{n^*}(-P_p+V(\varOmega ^{s} (\varvec{\lambda }|\theta =0),1))\\&+(1-\lambda _{n^*})(-P_{4G}+V(\varOmega ^{s}(\varvec{\lambda }|\theta =1),1)). \end{aligned}$$
The SU chooses the action that maximizes its average utility and thus:
$$\begin{aligned}&V(\varvec{\lambda },l^*-1)=\\&\max _{a}Q_a(\varvec{\lambda },l^*-1)-g_u\\&\quad \ge Q_2(\varvec{\lambda },l^*-1)-g_u,\\&\quad =\varPhi -c_s+\lambda _{n^*}(-P_p+V(\varOmega ^{s}(\varvec{\lambda }|\theta =0),1))\\&\qquad +(1-\lambda _{n^*})(-P_{4G}+V(\varOmega ^{s}(\varvec{\lambda }|\theta =1),1))\\&\qquad -g_u,\\&\quad =V(\varvec{\lambda },l^*). \end{aligned}$$
Let us prove that this propriety holds for all packet delays using a backward induction on l:
Initial condition: For all belief vector \(\lambda \), \(V(\varvec{\lambda },l^*)\le V(\varvec{\lambda },l^*-1)\),
Suppose that \(V(\varvec{\lambda },l+2)\le V(\varvec{\lambda },l+1)\), \(\forall \varvec{\lambda }\).
$$\begin{aligned} Q_0(\varvec{\lambda },l)= & {} -f(l)+V(\varOmega ^{ns}(\varvec{\lambda }|\theta ),l+1),\\ \nonumber\ge & {} -f(l+1)+V(\varOmega ^{ns}(\varvec{\lambda }|\theta ),l+2),\\= & {} Q_0(\varvec{\lambda },l+1).\\ Q_1(\varvec{\lambda },l)= & {} -c_s+\lambda _{n^*}(\varPhi -P_p+V(\varOmega ^{s} (\varvec{\lambda }|\theta =0),1))\\&+(1-\lambda _{n^*})(-f(l) +V(\varOmega ^{s}(\varvec{\lambda }|\theta =1),\\&l+1)),\\\ge & {} -c_s+\lambda _{n^*}(\varPhi -P_p+V(\varOmega ^{s} (\varvec{\lambda }|\theta =0),1))\\\nonumber&+(1-\lambda _{n^*}) (-f(l+1)+V(\varOmega ^{s}(\varvec{\lambda }|\theta =1),\\&l+2)),\\= & {} Q_1(\varvec{\lambda },l+1).\\ Q_2(\varvec{\lambda },l)= & {} -c_s+\varPhi -P_{4G}+V(\varOmega ^{s} (\varvec{\lambda }|\theta =1),1)\\&+\lambda _{n^*}(P_{4G} -P_p+V(\varOmega ^{s}(\varvec{\lambda }|\theta =0),1)\\&-V(\varOmega ^{s} (\varvec{\lambda }|\theta =1),1)),\\\ge & {} Q_2(\varvec{\lambda },l+1). \end{aligned}$$
The inequalities come from the induction assumption and the monotonicity of the penalty function f(l). Thus, we have: \(\forall \lambda , \quad V(\lambda ,l)\ge V(\lambda ,l+1)\).
The value function is therefore decreasing with the packet delay. \(\square \)
We have the following inequality:
$$\begin{aligned} -P_p+V(\alpha ,1)\ge -P_{4G}+V(\beta ,1). \end{aligned}$$
Proof of Lemma 2
We prove this lemma by contradiction, so we suppose that \(-P_p+V(\alpha ,1)<-P_{4G}+V(\beta ,1)\). We first prove that:
$$\begin{aligned} g_u+V(\alpha ,1)\ge & {} Q_2(\alpha ,1),\\ g_u+V(\alpha ,1)\ge & {} -c_s+\alpha (\phi -P_p+V(\alpha ,1))\\&+(1-\alpha )(\phi -P_{4G}+V(\beta ,1)),\\ g_u+V(\alpha ,1)\ge & {} -c_s+\phi -P_p+V(\alpha ,1),\\ g_u> & {} \varPhi -c_s-P_p. \end{aligned}$$
and we make the assumption that the immediate reward when the channel is idle is positive, i.e. \(\varPhi -c_s-P_p\ge 0\). We know that the SU takes the action 2 in the state \((\lambda ,l^*)\) for all belief vector \(\lambda \), i.e \(a^*(\lambda ,l^*)=2, \forall \lambda \). We have:
$$\begin{aligned} g_u+V(\lambda ,l^*)= & {} -c_s+\lambda (\phi -P_p+V(\alpha ,1))\\&+(1-\lambda )(\phi -P_{4G}+V(\beta ,1)). \end{aligned}$$
Let us focus on the packet delay \(l^*-1\). If \(\lambda \le \pi (0)\), we have:
$$\begin{aligned} Q_0(\lambda ,l^*-1)= & {} -f(l^*-1)+V(\varOmega ^{ns}(\lambda ),l^*),\\= & {} -g_u-f(l^*-1)-c_s+\varOmega ^{ns}(\lambda )\\&\times (\phi -P_p+V(\alpha ,1))\\&+(1-\varOmega ^{ns}(\lambda ))(\phi -P_{4G}+V(\beta ,1)),\\= & {} V(\lambda ,l^*)-f(l^*-1)+(\varOmega ^{ns}(\lambda )-\lambda )\\&(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)),\\< & {} V(\lambda ,l^*). \end{aligned}$$
The inequality is due to the assumption that \(-P_p+V(\alpha ,1)<-P_{4G}+V(\beta ,1)\), \(\varOmega ^{ns}(\lambda )\ge \lambda \) and \(f(l^*-1)\) is positive. As the value function \(V(\lambda ,l)\) is decreasing with the packet delay l (see Proposition 2), then \(Q_0(\lambda ,l^*-1)<V(\lambda ,l^*)<V(\lambda ,l^*-1)\). As we proved that \(g_u\ge 0\), the SU does not take the action 0 when the packet delay is \(l^*-1\). For the action 1, we have:
$$\begin{aligned} Q_1(\lambda ,l^*-1)= & {} -c_s+\lambda (\phi -P_p+V(\alpha ,1))\\&+(1-\lambda )(-f(l^*-1)+V(\beta ,l^*)),\\= & {} -c_s+\lambda (\phi -P_p+V(\alpha ,1))+(1-\lambda )\\&\left( \right. \phi -g_u-f(l^*-1)-c_s \\&\times +\beta (-P_p+V(\alpha ,1))\\&\left. +(1-\beta )(-P_{4G}+V(\beta ,1))\right) ,\\< & {} -c_s+\lambda (\phi -P_p+V(\alpha ,1))+(1-\lambda )\\&(\phi -g_u-f(l^*-1)-c_s-P_{4G}\\&+V(\beta ,1)),\\< & {} -c_s+\lambda (\phi -P_p+V(\alpha ,1))\\&+(1-\lambda )(\phi -P_{4G}+V(\beta ,1)),\\= & {} Q_2(\lambda ,l^*-1). \end{aligned}$$
The first inequality is due to the assumption that \(-P_p+V(\alpha ,1)<-P_{4G}+V(\beta ,1)\) and the second one is because \(g_u\), \(f(l^*-1)\) and \(c_s\) are positive. Thus, the optimal strategy is to take the action 2 when the packet delay is \(l^*-1\).
Let us prove now by backward induction on l that the optimal action is the action 2 for all belief vector \(\lambda \le \pi (0)\).
If the SU takes the action 2 when the packet delay is \(l^*\), then it takes also the action 2 when the packet delay is \(l^*-1\).
We suppose that SU takes the action 2 when the packet delay is \(l<l^*-1\).
We have the following inequalities:
$$\begin{aligned} Q_0(\lambda ,l-1)= & {} -f(l-1)+V(\varOmega ^{ns}(\lambda ),l),\\= & {} -g_u-f(l-1)-c_s+\varOmega ^{ns}(\lambda )(\phi -P_p\\&+V(\alpha ,1))+(1-\varOmega ^{ns}(\lambda ))\\&(\phi -P_{4G}+V(\beta ,1)),\\= & {} V(\lambda ,l)-f(l-1)+(\varOmega ^{ns}(\lambda )\\&-\lambda )(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)),\\< & {} V(\lambda ,l). \end{aligned}$$
The inequality is due to the assumption that \(-P_p+V(\alpha ,1)<-P_{4G}+V(\beta ,1)\) and \(\varOmega ^{ns}(\lambda )\ge \lambda \), and \(f(l-1)\) is positive. As the value function is decreasing with the packet delay (see Proposition 2), then \(Q_0(\lambda ,l-1)<V(\lambda ,l-1)+g_u\), i.e. the SU does not take the action 0 with the packet delay \(l-1\).
$$\begin{aligned} Q_1(\lambda ,l-1)= & {} -c_s+\lambda (\phi -P_p+V(\alpha ,1))\\&+(1-\lambda )(-f(l-1)+V(\beta ,l)),\\= & {} -c_s+\lambda (\phi -P_p+V(\alpha ,1))\\&+(1-\lambda )\left( \right. \phi -g_u-f(l-1)-c_s \\&+\beta (-P_p+V(\alpha ,1))+(1-\beta )\\&\left. (-P_{4G}+V(\beta ,1))\right) ,\\< & {} -c_s+\lambda (\phi -P_p+V(\alpha ,1))+(1-\lambda )\\&(\phi -g_u-f(l-1)-c_s-P_{4G}\\&+V(\beta ,1)),\\< & {} -c_s+\lambda (\phi -P_p+V(\alpha ,1))\\&+(1-\lambda )(\phi -P_{4G}+V(\beta ,1)),\\= & {} Q_2(\lambda ,l-1). \end{aligned}$$
The first inequality is due to the assumption that \(-P_p+V(\alpha ,1)<-P_{4G}+V(\beta ,1)\) and the second one is because \(g_u\), \(f(l-1)\) and \(c_s\) are positive. Thus, The optimal strategy is to take action 2 when the packet delay is \(l-1\). Thus, the SU does not take the action 1 with the packet delay \(l-1\). Finally, the SU takes action 2 for all packet delays and beliefs lower than \(\pi (0)\).
We now look at the action-value function \(Q_2(\alpha ,1)\) when the packet delay is \(l=1\).
$$\begin{aligned} Q_2(\alpha ,1)= & {} -c_s+\alpha (\phi -P_p+V(\alpha ,1))\\&+(1-\alpha )(\phi -P_{4G}+V(\beta ,1)),\\ Q_2(\alpha ,1)= & {} \phi -c_s-P_{4G}+V(\beta ,1)\\&+\alpha (P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)),\\ -g_u+Q_2(\alpha ,1)= & {} -g_u+V(\alpha ,1)-P_p+\phi -c_s\\&+(\alpha -1)(P_{4G}-P_p+V(\alpha ,1)\\&-V(\beta ,1)). \end{aligned}$$
As the SU takes the action 2 also for the state \((\beta ,1)\), we have:
$$\begin{aligned} g_u+V(\beta ,1)= & {} -c_s+\beta (\phi -P_p+V(\alpha ,1))\\&+(1-\beta )(\phi -P_{4G}+V(\beta ,1)),\\ g_u+V(\beta ,1)= & {} \phi -c_s-P_{4G}+V(\beta ,1)\\&+\beta (P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)),\\ g_u= & {} \phi -c_s-P_{4G}+\beta (P_{4G}\\&-P_p+V(\alpha ,1)-V(\beta ,1)). \end{aligned}$$
Thus, we obtain:
$$\begin{aligned} -g_u+Q_2(\alpha ,1)= & {} V(\alpha ,1)+P_{4G}-P_p+(\alpha -\beta -1)\\&(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)). \end{aligned}$$
As we assumed that \(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)<0\), and \(P_{4G}>P_p\), then we obtain \(V(\alpha ,1)+g_u\le Q_2(\alpha ,1)\) and therefore the SU takes also the action 2 in the state \((\alpha ,1)\). Then we get:
$$\begin{aligned} g_u+V(\alpha ,1)= & {} Q_2(\alpha ,1)=-c_s+\alpha (\phi -P_p+V(\alpha ,1))\\&+(1-\alpha )(\phi -P_{4G}+V(\beta ,1)). \end{aligned}$$
Let us evaluate finally the difference \(V(\alpha ,1)-V(\beta ,1)\):
$$\begin{aligned} V(\alpha ,1)-V(\beta ,1)= & {} (\alpha -\beta )(P_{4G}-P_p+V(\alpha ,1)\\&-V(\beta ,1)),\\ V(\alpha ,1)-V(\beta ,1)< & {} 0. \end{aligned}$$
$$\begin{aligned}&V(\alpha ,1)-V(\beta ,1)=(\alpha -\beta )(P_{4G}-P_p+V(\alpha ,1)\\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-V(\beta ,1)),\\&(V(\alpha ,1)-V(\beta ,1))(1-\alpha +\beta )=(\alpha -\beta )(P_{4G}-P_p),\\&V(\alpha ,1)-V(\beta ,1)=\frac{(\alpha -\beta )(P_{4G}-P_p)}{1-\alpha +\beta },\\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>0. \end{aligned}$$
which leads to a contradiction, and therefore, \(-P_p+V(\alpha ,1)\ge -P_{4G}+V(\beta ,1)\). The analysis is similar when \(\lambda > \pi (0)\). \(\square \)
Let us prove that the value function \(V(\lambda ,l)\) is increasing with the belief vector \(\lambda \) for any packet delay l. For all \(\lambda _1\le \lambda _2\), we have that:
$$\begin{aligned} V(\lambda _1,l^*)= & {} -g_u-c_s+\varPhi -P_{4G}+V(\beta ,1)\\&+\lambda _1(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)),\\\le & {} -g_u-c_s+\varPhi -P_{4G}+V(\beta ,1)\\&+\lambda _2(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)),\\= & {} V(\lambda _2,l^*). \end{aligned}$$
This inequality result from the Lemma 2. Let us prove that this propriety holds for all packet delays l using backward induction:
Initial condition: There exists a packet delay \(l^*\) such that \(V(\lambda _1,l^*)\le V(\lambda _2,l^*),\) \(\forall \lambda _1\le \lambda _2\),
We suppose that \(V(\lambda _1,l+1)\le V(\lambda _2,l+1),\) \(\forall \lambda _1\le \lambda _2\),
First case We assume that \(\varPhi +f(l)-P_p+V(\alpha ,1)-V(\beta ,l+1)\ge 0\), then:
$$\begin{aligned} Q_0(\lambda _1,l)= & {} -f(l)+V(\varOmega ^{ns}(\lambda _1|\theta ),l+1),\\\le & {} -f(l)+V(\varOmega ^{ns}(\lambda _2|\theta ),l+1),\\= & {} Q_0(\lambda _2,l). \end{aligned}$$
The inequality is a direct result from the induction assumption and the Lemma 1. We have also:
$$\begin{aligned} Q_1(\lambda _1,l)= & {} -c_s-f(l)+V(\beta ,l+1)\\&+\lambda _1(\varPhi +f(l)-P_p+V(\alpha ,1)\\&-V(\beta ,l+1)),\\\le & {} -c_s-f(l)+V(\beta ,l+1)\\&+\lambda _2(\varPhi +f(l)-P_p+V(\alpha ,1)\\&-V(\beta ,l+1)),\\= & {} Q_1(\lambda _2,l).\\ Q_2(\lambda _1,l)= & {} -c_s+\varPhi -P_{4G}+V(\beta ,1)\\&+\lambda _1(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)),\\\le & {} -c_s+\varPhi -P_{4G}+V(\beta ,1)\\&+\lambda _2(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)),\\= & {} Q_2(\lambda _2,l). \end{aligned}$$
The inequalities comes from the Lemma 2. Thus, we have proved that \(V(\lambda _1,l)\le V(\lambda _2,l)\).
Second case We suppose that \(\varPhi +f(l)-P_p+V(\alpha ,1)-V(\beta ,l+1)<0\), then for all \(\lambda \) we have:
$$\begin{aligned} Q_1(\lambda ,l)= & {} -c_s+\lambda (\phi -P_p+V(\alpha ,1))\\&+(1-\lambda )(-f(l)+V(\beta ,l+1)),\\\le & {} -c_s-f(l)+V(\beta ,l+1),\\\le & {} -f(l)+V(\beta ,l+1),\\\le & {} -c_s-f(l)+V(\varOmega ^{ns}(\lambda |\theta ),l+1),\\\le & {} Q_0(\lambda ,l). \end{aligned}$$
In fact, we have that \(\beta \le \varOmega ^{ns}(\lambda |\theta )\) for all belief vector \(\lambda \) and the value function \(V(\lambda ,l)\) is increasing with the belief for the packet delay \(l+1\) (induction assumption). Thus, \(g_u+V(\lambda ,l)=\max {\{Q_0(\lambda ,l),Q_2(\lambda ,l)\}}\). Moreover, we have:
The inequality is a direct result from the induction assumption. Finally, we have that:
$$\begin{aligned} Q_2(\lambda _1,l)= & {} -c_s+\varPhi -P_{4G}+V(\beta ,1)\\&+\lambda _1(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)),\\\le & {} -c_s+\varPhi -P_{4G}+V(\beta ,1)\\&+\lambda _2(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)),\\= & {} Q_2(\lambda _2,l). \end{aligned}$$
The inequality comes from the Lemma 2.
Thus, \(V(\lambda _1,l)\le V(\lambda _2,l)\) for belief vectors \(\lambda _1 \le \lambda _2\) and for all packet delay l. \(\square \)
In this proposition, we determine explicitly the best action \(a^*(\lambda ,l)\) for the SU depending on the belief \(\lambda \) and the packet delay l. At each time slot and for a given information state \((\lambda ,l)\), the secondary use will decide to take the action 0 if \(Q_0(\lambda ,l)\ge \max {\{Q_{1}(\lambda ,l),Q_{2}(\lambda ,l)\}}\).
First we assume that \(Q_1(\lambda ,l)>Q_2(\lambda ,l)\), then, let us compare \(Q_0(\lambda ,l)\) and \(Q_{1}(\lambda ,l)\). The inequality \(Q_0(\lambda ,l)\ge Q_{1}(\lambda ,l)\) is equivalent to:
$$\begin{aligned}&-f(l)+V(\varOmega ^{ns}(\lambda |\theta ),l+1)\\&\quad \ge -c_s+\lambda (\varPhi -P_p +V(\alpha ,1))\\&\qquad +(1-\lambda )(-f(l)+V(\beta ,l+1)), \\&V(\varOmega ^{ns}(\lambda |\theta ),l+1)\\&\quad \ge V(\beta ,l+1) -c_s+\lambda (f(l)\\&\qquad +\,\varPhi -P_p+V(\alpha ,1)-V(\beta ,l+1)). \end{aligned}$$
As the value function \(V(\lambda ,l) \)is decreasing with the packet delay l and increasing with the belief \(\lambda \), we have \(V(\alpha ,1)\ge V(\beta ,l+1)\). As we assumed that the immediate reward \(\phi \) is higher than the cost \(P_p\), we obtain that \(f(l)+\varPhi -P_p+V(\alpha ,1)-V(\beta ,l+1)\) is positive. Then, we have the following equivalence:
$$\begin{aligned}&Q_0(\lambda ,l)\ge Q_{1}(\lambda ,l) \Leftrightarrow \\&V(\varOmega ^{ns}(\lambda |\theta ),l+1) \ge V(\beta ,l+1)-c_s+\lambda (f(l)+\varPhi \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-P_p+V(\alpha ,1)-V(\beta ,l+1)). \end{aligned}$$
Define the functions F and G as follow:
$$\begin{aligned} F(\lambda ,l)= & {} {V(\varOmega ^{ns}(\lambda |\theta ),l+1)},\\ G(\lambda ,l)= & {} V(\beta ,l+1)-c_s+\lambda (f(l)+\varPhi \\&-P_p+V(\alpha ,1)-V(\beta ,l+1)). \end{aligned}$$
We proved in Proposition 1 that the value function is Piecewise linear and convex. Therefore, for all packet delays, the function \(F(\lambda ,l)\) is PWLC and increasing with \(\lambda \), and the function \(G(\lambda ,l)\) is linear and increasing with \(\lambda \). Note that
If \(F(\lambda ,l)\ge G(\lambda ,l)\), then \(Q_0(\lambda ,l)\ge Q_1(\lambda ,l)\) and therefore the best action is 0.
If \(F(\lambda ,l)< G(\lambda ,l)\), then \(Q_0(\lambda ,l)< Q_1(\lambda ,l)\) and therefore the best action is 1.
Let us focus on \(F(\pi (0),l)\) and \(G(\pi (0),l)\).
Let us prove that \(g_u>-f(l)\). We have:
$$\begin{aligned}&g_u+V(\alpha ,1)\ge Q_0(\alpha ,1),\\&g_u+V(\alpha ,1)\ge -f(l)+V(\varOmega ^{ns}(\alpha ),l+1),\\&g_u+V(\alpha ,1)-V(\varOmega ^{ns}(\alpha ),l+1)\ge -f(l),\\&g_u>-f(l). \end{aligned}$$
The inequality is because of the monotonicity of the value function and \(\varOmega ^{ns}(\alpha )<\alpha \). Suppose that the SU chooses the action 0 for the state \((\pi (0),l)\). We have:
$$\begin{aligned}&g_u+V(\pi (0),l)=-f(l)+V(\varOmega ^{ns}(\pi (0)),l+1),\\&g_u+V(\pi (0),l)\le -f(l)+V(\varOmega ^{ns}(\pi (0)),l),\\&g_u+V(\pi (0),l)\le -f(l)+V(\pi (0),l),\\&g_u\le -f(l). \end{aligned}$$
This leads to a contradiction as \(g_u>-f(l)\). Thus, \(Q_0(\lambda ,l)< Q_1(\lambda ,l)\) and therefore, \(F(\pi (0),l)<G(\pi (0),l)\). Therefore, the cases 1, 3, 5 and 6 are eliminated. Finally, the optimal policy is a kind of threshold and is depicted in the following:
The SU takes the action 0 for all beliefs lower than the following threshold
$$\begin{aligned} Th1(\lambda ,l)=\frac{V(\varOmega ^{ns}(\lambda |\theta ),l+1) -V(\beta ,l+1)+c_s}{f(l)+\varPhi -P_p+V(\alpha ,1)-V(\beta ,l+1)}, \end{aligned}$$
and take the action 1 otherwise.
Second, we assume that \(Q_2(\lambda ,l)>Q_1(\lambda ,l)\) and then, we have to compare the action 0 and 2, which is equivalent to compare the action-value functions \(Q_0(\lambda ,l)\) and \(Q_{2}(\lambda ,l)\). The SU takes the action 0 instead of the action 2 if \(Q_0(\lambda ,l)\ge Q_{2}(\lambda ,l)\), which is equivalent to:
$$\begin{aligned}&-f(l)+V(\varOmega ^{ns}(\lambda |\theta ),l+1)\\&\quad \ge -c_s +\lambda (\varPhi -P_p+V(\alpha ,1))\\&\qquad +(1-\lambda )(\phi -P_{4G}+V(\beta ,1)), \\&V(\varOmega ^{ns}(\lambda |\theta ),l+1)\\&\quad \ge V(\beta ,1)+\varPhi +f(l)-c_s-P_{4G}\\&\quad \quad +\lambda (P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)). \end{aligned}$$
We have from the Lemma 2, that \(P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)\ge 0\). Then, we can provide the same analysis presented in the previous case with the function \(F(\lambda ,l)={V(\varOmega ^{ns}(\lambda |\theta ),l+1)}\) and the function \(G(\lambda ,l)=V(\beta ,1)+\varPhi +f(l)-c_s-P_{4G}+\lambda (P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1))\). The latter is linear increasing in \(\lambda \). We obtain the following threshold policy:
The SU takes the action 0 for all beliefs lower than the following threshold:
$$\begin{aligned} Th2(\lambda ,l)=\frac{V(\varOmega ^{ns}(\lambda |\theta ),l+1) -V(\beta ,1)-\varPhi -f(l)+c_s+P_{4G}}{P_{4G}-P_p+V(\alpha ,1)-V(\beta ,1)}, \end{aligned}$$
\(\square \)
We have from the Lemma 1 that if \(\lambda >\pi (0)\) then \(\varOmega ^{ns}(\lambda )\le \lambda \). Suppose that the SU takes the action 0 for a belief \(\lambda \) and packet delay l. Thus we have
$$\begin{aligned}&g_u+V(\lambda ,l)=-f(l)+V(\varOmega ^{ns}(\lambda ),l+1),\\&g_u+V(\lambda ,l)\le -f(l)+V(\varOmega ^{ns}(\lambda ),l),\\&g_u+V(\lambda ,l)\le -f(l)+V(\lambda ,l),\\&g_u\le -f(l). \end{aligned}$$
This leads to a contradiction as \(g_u>-f(l)\). The first inequality is because the value function is decreasing with the packet delay and the second one is because that the value function is increasing with the belief and \(\varOmega ^{ns}(\lambda )\le \lambda \). Thus, if \(\lambda >\pi (0)\), then the SU never takes the action 0 and then \(Q_0(\lambda ,l)< \max {\{Q_{1}(\lambda ,l),Q_{2}(\lambda ,l)\}}\).
Let us compare the value-action functions \(Q_1(\lambda ,l)\) and \(Q_2(\lambda ,l)\) for all belief vector \(\lambda \) and packet delay l. The SU waits for next time slot after sensing if \(Q_1(\lambda ,l)\ge Q_{2}(\lambda ,l)\), which is equivalent to:
$$\begin{aligned}&-c_s+\lambda (\varPhi -P_p+V(\alpha ,1))\\&\quad +(1-\lambda )(-f(l)+V(\beta ,l+1))\\&\quad \ge -c_s+\lambda (\varPhi -P_p+V(\alpha ,1))\\&\qquad +(1-\lambda )(\phi -P_{4G}+V(\beta ,1)),\\&\qquad -f(l)+V(\beta ,l+1)\phi -P_{4G}+V(\beta ,1)\ge 0. \end{aligned}$$
Remark that this condition depends only on the packet delay l and not on the belief vector \(\lambda \). \(\square \)
Appendix 8: Proof of Corollary 1
If \(-f(l)\) is lower than \(\varPhi -P_{4G}\), then \(-f(l)-\varPhi +P_{4G}+V(\beta ,l+1)-V(\beta ,1)\) is always negative. In fact, \(V(\beta ,2)-V(\beta ,1)\) is negative and \(-f(l)-\varPhi +P_p+V(\beta ,l+1)-V(\beta ,1)\) is decreasing with l. Therefore, the previous expression is negative for all \(l\ge 1\).
Hossain, E., Niyato, D., & Han, Z. (2009). Dynamic spectrum access and management in cognitive radio networks. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Mitola, J. (2000). Cognitive radio: An integrated agent architecture for software defined radio. Ph.D. Dissertation, Royal Institute of Technology (KTH), Stockholm.Google Scholar
Akyildiz, F., Lee, W.-Y., et al. (2006). NeXt generation dynamic spectrum access cognitive radio wireless networks: A survey. Computer Networks, 50(13), 2127–2159.CrossRefGoogle Scholar
Jaganathan, K., Menache, I., Modiano, E., & Zussman, G. (2011). Non-cooperative spectrum access—The dedicated vs. free spectrum choice. In Proceedings of ACM MOBIHOC'11.Google Scholar
Zhao, Q., Tong, L., Swami, A., & Chen, Y. (2007). Decentralized cognitive MAC for opportunistic spectrum access in ad hoc networks: A POMDP framework. IEEE Journal on Selected Areas in Communications, 25(3), 589–600.CrossRefGoogle Scholar
Zheng, H., & Peng, C. (2005). Collaboration and fairness in opportunistic spectrum access. In Proceedings of IEEE international conference on communication (ICC).Google Scholar
Shi, Y., Hou, Y., Zhou, H., & Midkiff, S. (2012). Distributed cross-layer optimization for cognitive radio networks. IEEE Transactions on Vehicular Technology, 59(8), 4058–4069.CrossRefGoogle Scholar
Min, A., Kim, K., Singh, J., & Shin, K. (2011). Opportunistic spectrum access for mobile cognitive radios. In Proceedings of IEEE Infocom.Google Scholar
Hoang, A. T., Liang, Y. C., Wong, D. T. C., Zeng, Y., & Zhang, R. (2008). Opportunistic spectrum access for energy-constrained cognitive radios. IEEE Transactions on Wireless Communications, 8(3), 1206–1211.CrossRefGoogle Scholar
Chen, Y., Zhao, Q., & Swami, A. (2009). Distributed spectrum sensing and access in cognitive radio networks with energy constraint. IEEE Transactions on Signal Processing, 57(2), 783–797.CrossRefGoogle Scholar
Sultan, A. (2012). Sensing and transmit energy optimization for an energy harvesting cognitive radio. IEEE Wireless Communication Letters, 1, 500–503.CrossRefGoogle Scholar
Garcia-Saaverdra, A., Serrano, P., & Banchs, A. (2014). Energy-efficient optimization for distributed opportunistic scheduling. IEEE Communications Letters, 18(6), 1083–1086.CrossRefGoogle Scholar
Xiong, C., Lu, L., & Li, G. (2014). Energy-efficient spectrum access in cognitive radios. IEEE Journal on Selected Areas in Communications, 32(3), 550–562.CrossRefGoogle Scholar
Tsiropoulos, G. I., Dobre, O. A., Ahmed, M. H., & Baddour, K. E. (2016). Radio resource allocation techniques for efficient spectrum access in cognitive radio networks. IEEE Communications Surveys & Tutorials, 18(1), 824–847.CrossRefGoogle Scholar
Althunibat, S., Di Renzo, M., & Granelli, F. (2015). Towards energy-efficient cooperative spectrum sensing for cognitive radio networks: An overview. Telecommunication Systems, 59, 77–91.CrossRefGoogle Scholar
Wu, Y., Tsang, D., & Qian, L. (2012). Energy-efficient delay-constrained transmission and sensing for cognitive radio systems. IEEE Transactions on Vehicular Technology, 61(7), 3100–3113.CrossRefGoogle Scholar
Pei, Y., Liang, Y., Teh, K., & Li, K. (2011). Energy-efficient design of sequential channel sensing in cognitive radio networks: Optimal sensing strategy, power allocation, and sensing order. IEEE Journal on Selected Areas in Communication, 29(8), 1648–1659.CrossRefGoogle Scholar
Mahapatra, R., Nijsure, Y., Kaddoum, G., Hassan, N. U., & Yuen, C. (2016). Energy efficiency tradeoff mechanism towards wireless green communication: A survey. IEEE Communications Surveys & Tutorials, 18(1), 686–705.CrossRefGoogle Scholar
Habachi, O., El Azouzi, R., & Hayel, Y. (2013). A Stackelberg model for opportunistic sensing in cognitive radio networks. Transactions on Wireless Communications, 12(5), 2148–2159.CrossRefGoogle Scholar
Kaelbling, L., Littman, M., & Cassandra, A. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence Journal, 101, 99–134.CrossRefGoogle Scholar
Putterman, M. L. (2005). Markov decision process discrete stochastic dynamic programming. London: Wiley.Google Scholar
Smallwood, R., & Sondik, E. (1973). The optimal control of partially observable Markov decision processes over a finite horizon. Operations Research, 21, 1071–1088.CrossRefGoogle Scholar
Lovejoy, W. S. (1987). Some monotonicity results for partially observed Markov decision processes. Operations Research, 35(5), 736–743.CrossRefGoogle Scholar
Shellhammer, S. J., Sadek, A. K., & Zhang, W. (2009). Technical challenges for cognitive radio in the TV white space spectrum. In 2009 Information Theory and Applications Workshop (pp. 323–333). San Diego, CA.Google Scholar
© Springer Science+Business Media, LLC 2017
1.XLIMUniversity of LimogesLimogesFrance
Habachi, O., Hayel, Y. & El-Azouzi, R. Telecommun Syst (2018) 67: 763. https://doi.org/10.1007/s11235-017-0370-8
First Online 29 August 2017