text:
Brendan Rodgers Admits He Would Love to Have Monaco Loanee Youri Tielemans at Leicester Next Year
Leicester City manager Brendan Rodgers has confessed his desire to bring on-loan midfielder Youri Tielemans back to the club next season.
The 21-year-old joined from Ligue 1 giants Monaco in January until the end of the current campaign, with the Foxes deciding not to include an option to buy in the deal. He had underwhelmed at Stade Louis II after joining from Belgian outfit Anderlecht for €25m two years ago. However, Tielemans has been reborn since moving to Leicester, showcasing an exquisite skill-set and impressing onlookers.
The playmaker has made a remarkable start to life in the Premier League despite question marks over his form for the French club earlier in the year. In just seven appearances for Leicester he has already racked up four assists and hit his first goal for the team against Fulham at the start of March.
He has received countless plaudits for his superb displays, with Rodgers amongst those left mesmerised by his performances. When asked by the Daily Mail about the possibility of Tielemans' move being made permanent, the Foxes boss said: "I can't worry about that. He's a top-class young talent. But I always have to respect that he's a Monaco player.
"I'm sure there will be discussions between the two of us. He's certainly a player who I would love to continue working with. He's only going to get better."
Youri Tielemans' last  matches:
5 points - Brighton (H)
5 points - Watford (A)
9 points - Fulham (H)
5 points - Burnley (A)
7 points - AFC Bournemouth (H)#FPL pic.twitter.com/f2CHj8hYsI
— Fantasy Premier League (@OfficialFPL) April 1, 2019
The Northern Irishman was evidently keen to have the Belgium international as part of his squad for the upcoming season after his excellent contributions in recent months.
However, the midlands side could be priced out of a permanent deal for Tielemans. During the winter transfer window Monaco told Leicester it would take a £40m sum to acquire Tielemans, the latter instead deciding to take at look at him over the course of a loan spell.


text:
Home > Research > Migrancy Research Group > Publications & Outputs
Migrancy Research Group
Sort by: Author last name
Firth, G. E., 1/07/2016, Textbook on immigration and asylum law. Clayton, G. (ed.). 7th ed. Oxford: Oxford University Press
Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSN › Chapter
Personhood, best interests and the asylum seeking child
Firth, G. E. & Mauthe, B., 2016, (Unpublished).
Research output: Contribution to conference - Without ISBN/ISSN › Conference paper
Building fortress Europe: the Polish-Ukrainian frontier
Follis, K., 2012, Philadelphia: University of Pennsylvania Press. 284 p. (Democracy, Citizenship and Constitutionalism )
Research output: Book/Report/Proceedings › Book
Repossession: Notes on Restoration and Redemption in Ukrainian-Polish Borderlands
Follis, K., 12/05/2008, In: Cultural Anthropology. 23, 2, p. 329-360 32 p.
Are the European Union's new borders like the Iron Curtain? 1989, borders and freedom of movement in Poland and Ukraine
Follis, K., 09/2009, In: International Journal of Politics, Culture, and Society. 22, 3, p. 385-400 16 p.
The awkward divide: paradoxes on transnationality on the Polish-Ukrainian border
Follis, K., 2011, Transnational Europe: problems, paradox, limits. DeBardeleben, J. & Hurrelmann, A. (eds.). Basingstoke: Palgrave, p. 239-256 18 p. (Palgrave studies in European Union politics).
What is an economic migrant? Europe's new borders and the politics of classification
Follis, K., 2011, Citizenship, borders and human needs. Smith, R. M. (ed.). Philadelphia: University of Pennsylvania Press, p. 115-131 17 p. (Democracy, citizenship, and constitutionalism).
Socio-Legal Integration: Polish Post-2004 EU Enlargement Migrants in the United Kingdom by Agnieszka Kubal
Follis, K., 2013, In: Slavic Review. 72, 3, p. 625-626 2 p.
Post-territory and surveillance: new technologies of vision in European migration management
Follis, K., 2013, (Unpublished).
Human rights as pedagogy and indignation: accounting for migrants' deaths at sea in the Council of Europe
Previous 12 3 4 5 6 7 8 9 ...24 Next

redpajama_set_name: RedPajamaCommonCrawl


text:
Tom Bateman Witnessed A Murder On The Orient Express
Photographed by Danielle Kosann
It's not every day we find ourselves sitting down to chat with one of Hollywood's most talked about new actors. Tom Bateman, breakout star of Murder on the Orient Express, is a tried and true Potatohead. Bateman is guilty of the same so-called Golden-Aged Thinking we suffer from: He longs for a different, more romantic time and urges people to put down their phones on dates. We chatted with Bateman about everything from his favorite time period (Elizabethan England) to what he's giving as gifts this holiday season. Oh, and did we mention he's British? Swoon…
What's your ideal food day?
An ideal food day would be boiled eggs and toast for breakfast, my mum and dad's roast dinner for lunch and anything as long as it's with good company in the evening. It would be Italian, preferably. I love their sense of family and food. And wine. Lots of wine.
What was it like working with the cast on Murder on the Orient Express? Can you tell us something funny about a few of your cast members we don't know already?
Filming Murder on the Orient Express was probably the best job I've ever had. Ken put together a hugely talented team – one that, for a young actor like myself, was a dream to watch and learn from – but also the kindest. I don't think I've ever laughed or loved so much. Something you may not have known is that Marwan Kenzari does not appreciate me super gluing his breakfast bowl shut and his water bottles to the countertop…I found that out the hard way.
Murder on the Orient Express takes place in the 1930s, what elements in the film of your character (wardrobe, way of speaking, movement etc) are very 30s?
The whole world lends itself to the time period. It was such a joy to film because the attention to detail was so precise. We were completely immersed in the world of the Orient Express and that influenced how we moved, spoke and held ourselves. Once the set, costumes and props were done, the work was almost done for us.
Did you always know you wanted to be an actor? Did you ever want to be anything else?
I only realized at about age fifteen that I wanted to be an actor. Before then, I suppose I wanted to be one but I just hadn't realized it yet, in that I wanted to be a policeman, a lawyer, a vet, a skydiving instructor, a scientist – you name it, I wanted to be it. I suppose the attraction was that different lives and worlds appealed to me. Acting satisfies that itch.
Do you still want to be anything else?
I don't think I could do anything else now. People who do what they love for a living are very lucky and blessed in that our jobs are also our hobbies. I'd do this for free – don't tell any of the producers that!
Do you personally like mysteries? Why or why not?
I love mysteries. I think the appeal of the genre is that audiences can become actively involved and try to crack the case instead of being a more passive viewer. That's what I love about them anyway!
Is there another time period and place you've always loved? What is it and why?
I love Elizabethan London – the whole grit and immediacy of it. There was a time when you couldn't build higher than St. Paul's Cathedral, and I think London must have looked so beautiful then. I suppose it's easy to look back and admire a time in history – we only learn about the beautiful bits. I'm quite happy where and when I am now, but what makes me sad is our lack of connection to what is around us now. People see the world through their phones, now, not their eyes. I miss a time without those.
Would you consider yourself a romantic? Why or why not?
I'm not sure a person can call themselves romantic. It's a trait others have to see, I think. I love people, in all walks of life. Sitting opposite a person, talking and listening to them could be seen as romantic – or just how we should all be living.
What's your idea of the perfect first date?
For the perfect first date, see above. I think there's something wonderful about walking, too. You get to see the world around you and be a part of that with someone.
With the holidays coming up, have any amazing gift ideas you can give to people shopping for boyfriends, husbands, partners, brothers, etc.?
I actually think that people have too much stuff. I try, the best I can, to give people experiences. I bought my brother a plane flying experience for his birthday. That's something that will last forever and not simply add to consumerism.
What was the funniest thing Amy Schumer did on the set of Snatched?
Amy is actually a lot more serious and concentrated on the work than you might think! That said, she constantly took the piss and made me laugh.
What's the best advice you have ever received? The worst?
The best advice I have ever received is that you have to be true to yourself. That was given to me regarding acting, but I think can be taken larger than that even – to our whole lives.
What would your last meal be and who would it be with?
My last meal would be with all the people I love in the world around a huge table in the countryside eating vast amounts of pie and mashed potatoes with wine.
If you could give us a quote of the day to sum up your mood and mode, what would it be?
Do something – see what happens. I just went for an evening walk around New York in the rain. Just to do something and see what might happen.
6 thoughts on "Tom Bateman Witnessed A Murder On The Orient Express"
Archie Cogollos says:
Interesting!!!!!…I agree with the romantic part….when you're in love you can talk to each other for hours and never even notice the passage of time…I know I have…
Patricia Caruana says:
Excellance!!!!!!!!!!!!!!!!!!!! He had all the right answers.
Tom sounds like my kind of Man. Except for the fact I am 70 years old and married. 🙂
He sounds wonderful. a great guy; thanks for the interview.
I like his philosophy
Nan Joyzee says:
enjoyed him immensely on the limited series "TThe Tunnel"
Outlander's Caitriona Balfe Is A Cynic, Like Me
Moulin Rouge's Karen Olivo Is The Queen of Manifesting
Oakes Fegley Is Hooked On Greek Mythology


Best HGH Clinic in Joliet.
At Optimal Health Medical in Joliet, our hormone replacement physicians use only natural or bio-identical hormones in your HGH Therapy program. Natural hormones have proven to be a safer and more effective hormone treatment than animal derived hormone shots. If you are a man or women suffering from a hormonal imbalance, menopause or andropause - contact us to speak with a hormone specialist.
If you live in the Joliet County area, you can get an HGH prescription legally and safely at one of our Local HGH Clinics in or near Joliet. You can purchase real Human Growth Hormone in Joliet which includes your somatropin HGH shots, injection syringe kit, or HGH pen device with somatropin HGH cartridges for self-administration of your medications.
For more Joliet HGH treatment info, to find out the cost of HGH Therapy or to buy HGH Injections in Joliet, fill out the Hormone Therapy Info Form.
The best Human Growth Hormone Doctors & HGH Treatment Clinics serving HGH Joliet, HGH Rockdale, HGH Fairmont IL, HGH Ingalls Park, HGH Preston Heights, HGH Crest Hill, HGH Lockport, HGH Shorewood, HGH New Lenox, HGH Romeoville, HGH Plainfield, HGH Plattville, HGH , HGH Elwood, HGH Manhattan, HGH Oswego, HGH Frankfort, HGH Mokena, HGH Tinley Park, HGH Wilton Center, HGH Orland Park, HGH Naperville, HGH Aurora, HGH Downers Grove, HGH Garden Prairie Illinois, Joliet area.
For more HGH treatment info, to find out the cost of HGH Therapy or to buy HGH Injections in Joliet at the best price, fill out the Quick Info Form.
HGH Clinics in Joliet provide HGH and Testosterone Therapy Services. Call for the cost of a somatropin prescription and best GH injection prices.
Test for Low Human Growth Hormone and get Doctor prescribed HGH Injections in Joliet.


text:
Gov't to Dispatch Humanitarian Assistance from Communities to Pakistan Early October
By Office of Assistant to Deputy Cabinet Secretary for State Documents & Translation Date 26 September 2022
Read: 142 Views
Humanitarian assistance for Pakistan (Photo by: PR/Rahmat)
The Indonesian Government will provide planes to dispatch humanitarian assistance from all walks of life in the country for victims of floods and landslides in Pakistan.
"The Government will facilitate to dispatch [the assistance] with a third plane," Head of the National Agency for Disaster Management (BNPB) Suharyanto said after the dispatch of humanitarian assistance for Pakistan at Halim Perdanakusuma Air Force Base, Monday (09/26).
The Head of BNPB added that the next plane will depart in early October. "We've received assistance from private organizations, communities, NGOs, and so on. The deadline is October 3," he said.
Furthermore, Suharyanto also stated that the Government has dispatched two planes to Pakistan transporting government assistance including tents, adult clothes, children's clothes, blankets, plastic mats, sleeping bag, mosquito nets, antigen test kits, cleaning equipment, medicines, media materials, and generators.
In addition, the Government will also send out medical team to Pakistan, he said. "The medical team to be dispatched will consist of doctors and nurses as well as personnel from Ministry of Health, the Indonesian National Defense Forces, the National Police. The team will be adjusted according to the needs of the Pakistani Government," he said.
On that occasion, Deputy Ambassador of Pakistan Muhammad Faisal Fayyaz expressed his gratitude to the Indonesian Government for providing humanitarian assistance.
"It's an honour to be here and I would like to express gratitude on behalf of the government and the people of Pakistan," he said, adding that the two countries have always stood side by side when facing difficult times. Indonesia has also assisted Pakistan after the 2005 earthquake and the 2010 floods.
"This shows a deep and promising bilateral relationship in the future," he stated. (MAY/RF/UN) (FI/LW)


A podcast series exploring the hope that is Jesus Christ, helping all people understand some of the basic facts of Christianity. Exploring Hope features expert scholars in their fields of study or work talking on the subjects they know best. Southeastern is excited to present Exploring Hope to you as a learning resource.


text:
\section{Introduction}

In this work we are interested in solving the strongly convex, composite, unconstrained optimization problem,
\begin{equation}\label{eq:Problem}
	\min_{x\in \R^n} \{F(x) \eqdef f(x) + h(x)\}.
\end{equation}
We use $x^*$ to denote the optimal solution of \eqref{eq:Problem}, and $F^*:= F(x^*)$ to denote the associated optimal function value. It is assumed that $h(x)$ is a convex and possibly nonsmooth function. Furthermore, throughout the paper we make the following assumption regarding the function $f$.
\begin{assumption}\label{A_SCL}
The function $f(\cdot)$ is $\mu$-strongly convex and $L$-smooth, i.e., for all $x,y\in\R^n$, it holds that
\begin{align}
	f(x) \geq f(y) + \ve{\nabla f(y)}{x-y} + \tfrac{\mu}{2} \tnorm{x-y}, \label{eq:ass1} \\
	f(x) \leq f(y) + \ve{\nabla f(y)}{x-y} + \tfrac{L}{2} \tnorm{x-y}. \label{eq:ass2}
\end{align}
\end{assumption}
It is straightforward to show that strong convexity of $f(x)$ implies strong convexity of $F(x)$.

For problems of the form \eqref{eq:Problem}, which satisfy Assumption~\ref{A_SCL}, it is well known that Nesterov's methods \cite{Nesterov04,Nesterov07,Nesterov13} converge linearly, with the accelerated variants converging at the optimal rate of $(1-\sqrt{\mu/ L})$.

Nesterov's acceleration approach, and the idea of adding momentum, has led to the extensive analysis of accelerated first order methods in a variety of settings. This includes a recent surge of interest in investigating stochastic gradient methods \cite{robbins1951stochastic,schmidt2017minimizing,johnson2013accelerating} and their accelerated variants \cite{cotter2011better,shalev2013accelerated,kingma2014adam,nitanda2014stochastic}. Coordinate descent methods \cite{nesterov2012efficiency,richtarik2014iteration} are another class of algorithms that have proved extremely popular, largely because they can take advantage of modern parallel computing architecture \cite{jaggi2014communication,ma2015adding}, and this has also inspired much research into studying their accelerated versions \cite{fercoq2015accelerated,allen2016even,ACOCOA}.
However, while the theoretical and practical performance of Nesterov's methods is well established, a satisfactory geometric interpretation of these approaches has been elusive.

Recently the authors of \cite{Bubeck15,Drusvyatskiy16} proposed algorithms for smooth functions (i.e., $h(x) = 0$ in \eqref{eq:Problem}) that enjoy the same optimal rate of convergence as Nesterov's accelerated method, but also have a novel geometric intuition. Specifically, the geometric descent algorithm \cite{Bubeck15} achieves the optimal linear convergence rate, and shares a geometric intuition similar to that of ellipsoidal methods. The authors illustrate that the optimal rate is achieved by appropriately shrinking two balls that contain $x^*$ (the minimizer of $f(x)$) at each iteration.

Motivated by \cite{Bubeck15}, the paper \cite{Drusvyatskiy16} proposed the Optimal Quadratic Averaging (OQA) algorithm. This algorithm maintains a sequence of quadratic lower bounds o the objective function, and at each iteration the new quadratic lower bound is formed as the optimal average of the current lower bound and the lower bound from the previous iteration. The gap between the function value $f(x_k)$ and the minimum value of lower bound, $\phi_k^*$ say, converges to zero at the optimal rate. Importantly, the lower bound also acts a natural stopping criterion for the algorithm, and when $f(x_k) - \phi_k^* \leq \epsilon$, where $\epsilon >0$ is some stopping tolerance, then the user has a certificate of $\epsilon$-optimality, i.e., it is guaranteed that $f(x_k) - f^* \leq \epsilon$. In practice, the OQA algorithm can be equipped with historical information to achieve further speed up. However, the OQA algorithm and its history based variant need at least two calls of a line search process at every iteration, which can pose a heavy computational burden in terms of function evaluations. The authors in \cite{Drusvyatskiy16} also briefly describe how their \emph{unaccelerated OQA algorithm} can be extended to composite functions, and left as an open problem the possibility of deriving \emph{accelerated} proximal variants.

Very recently, the authors of \cite{Chen16} successfully addressed the open problem in \cite{Drusvyatskiy16} and presented an accelerated algorithm for composite problems of the form \eqref{eq:Problem}, that achieves the optimal linear rate of convergence. Their algorithm, called the geometric proximal gradient (GeoPG) method also has a satisfying geometrical interpretation similar to that in \cite{Bubeck15}. Unfortunately, a major drawback of GeoPG in \cite{Chen16} is that the algorithm is rather complicated, and requires a couple of inner loops to determine necessary algorithm parameters. For example, for GeoPG one must find the root of a specific function and one is also required to compute a minimum enclosing ball via some iterative process; both of these steps must be carried out at every iteration, which is expensive.

In this paper we propose several new algorithms to solve problem \eqref{eq:Problem} that are motivated by, and extend, the previously mentioned works. In particular, we present four algorithms: a Gradient Descent (GD) type algorithm for smooth problems, an accelerated GD type algorithm for smooth problems, a proximal GD type algorithm for composite problems, and an accelerated proximal GD type algorithm for composite problems. Our algorithms all converge linearly, and the accelerated variants converge at the optimal linear rate. These algorithms blend the positive features of Nesterov's methods \cite{Nesterov04,Nesterov07,Nesterov13} and the OQA algorithm \cite{Drusvyatskiy16}, and thus enjoy the advantages of both approaches. First, similarly to Nesterov's methods, no line search is needed by any of our algorithms as long as we make the standard assumption that the Lipschitz constant $L$ is known or is easily computable. Hence, there are no `inner-loops' in any of our algorithm variants, which ensures that the computational cost is low and is fixed at every iteration. Secondly, our algorithms incorporate quadratic lower bounds so they have natural stopping conditions; a feature that is similar to OQA. However, our algorithms update the quadratic lower bound at each iteration by taking a convex combination of the previous two lower bounds, which is different from OQA.

Another contribution of this work is that we also propose the concept of an UnderEstimate Sequence (UES), which is a natural extension of Nesterov's Estimate Sequence \cite{Nesterov83}. Perhaps surprisingly, estimate sequences initially appeared to be largely overlooked, but since Nesterov's work on smoothing techniques in the early 2000s \cite{Nesterov05}, they have seen a significant revival in popularity. For example, the work of Baes in \cite{Baes09}, the development of a randomized estimate sequence in \cite{Lu15} and an approximate estimate sequence in \cite{Lin15}. To the best of our knowledge, this is the first work which proposes estimate sequences that form \emph{lower bounds} on the objective function. The UES framework is the powerhouse of our convergence analysis; we prove that each of our proposed algorithms generates a UES, and subsequently the algorithms converge (linearly) to the optimal solution of problem \eqref{eq:Problem}. While we describe 4 new algorithms in this work, we stress that the UES framework is general, and it allows a plethora of algorithms to be developed. Moreover, any developed algorithm whose iterates generate a UES is guaranteed to converge linearly to the optimal solution $F^*$.

\subsection{Contributions}
In this section we state the main contributions of this paper (listed in no particular order).
\begin{itemize}
\item \textbf{Underestimate Sequence.} We introduce the concept of an UnderEstimate Sequence (UES), which extends Nesterov's work on estimate sequences in \cite{Nesterov83}. The UES consists of three sequences $\{x_k\}_{k=0}^\infty$, $\{\phi_k(x)\}_{k=0}^\infty$ and $\{\alpha_k\}_{k=0}^\infty$, where for all $k$, $\phi_k(x)$ is a \emph{global lower bound} on the objective function $F(x)$. While there have been several extensions and variants of Nesterov's work \cite{Nesterov83}, to the best of our knowledge this is the first time that the estimate sequence framework has been adapted to act as a \emph{lower bound} or \emph{under}estimate of $F(x)$. The UES framework is general, conceptually simple, and it allows the construction of a wide variety of algorithms to solve \eqref{eq:Problem}.
\item \textbf{New algorithms.} We present 4 new algorithms that are computationally efficient and adhere to the UES framework. Crucially, two of our algorithms solve the \emph{composite} problem \eqref{eq:Problem}. The algorithms are: (i) SUESA, a GD type algorithm for smooth problems; (ii) ASUESA, an accelerated GD type algorithm for smooth problems; (iii) CUESA, a proximal GD type algorithm for composite problems, and (iv) ACUESA, an accelerated proximal GD type algorithm for composite problems.
\item \textbf{Algorithms with optimal convergence rate.} Each of the algorithms generate iterates that form a UES, so all four algorithms are guaranteed to converge linearly to the optimal solution of \eqref{eq:Problem}. Moreover, the accelerated algorithms (ASUESA and ACUESA) are guaranteed to converge linearly \emph{at the optimal rate.}
\item \textbf{Algorithms with convergence certificates.}  The underestimate sequence builds a global lower bound of $F(x)$ at each iteration, and the gap between the (minimum of the) lower bound and $F(x_k)$ tends to zero. Thus, this difference acts as a kind of surrogate ``duality gap'', and once this gap falls below some (user defined) stopping tolerance $\epsilon$, it is guaranteed that the point returned by the algorithm is $\epsilon$-optimal.
\item \textbf{No line search.} The algorithms developed in this work are computationally efficient and do not involve any `inner loops'. In contrast, the methods in \cite{Bubeck15,Drusvyatskiy16,Chen16} all involve an exact linesearch or a root finding process to determine necessary algorithmic parameters, which comes with an additional computational cost.
\end{itemize}

\subsection{Paper Outline}
The paper is organized as follows. In the next section we introduce the concept of an Underestimate Sequence (UES), and present a proposition which shows that if one has a UES, then it is guaranteed that $F(x_k) - F^* \to 0$ at a linear rate. Section~\ref{sec:lb} is dedicated to the discussion of lower bounds for the function $F(x)$ (in both the smooth and composite cases), and these lower bounds are a critical part of the underestimate sequences framework. In Section~\ref{sec:smooth} we propose two algorithms for solving \eqref{eq:Problem} in the smooth case ($h=0$) and in Section~\ref{sec:composite} we present two algorithms for solving composite problems of the form \eqref{eq:Problem} ($h\neq 0$). All algorithms in Sections~\ref{sec:smooth} and \ref{sec:composite} are supported by convergence theory, which shows that they are guaranteed to converge to the optimal solution of \eqref{eq:Problem} at a linear rate. In Section~\ref{sec:adaptiveL} we present another algorithm which uses an adaptive Lipschitz constant, rather than the true Lipschitz constant. Section~\ref{sec:numericalexperiments} presents numerical experiments to demonstrate the practical advantages of our proposed algorithms, and we give concluding remarks in Section~\ref{sec:conclusion}.

\section{Underestimate Sequence}\label{sec:UES}

In this section, we present the definition of an Underestimate Sequence (UES) and a proposition showing that if one has a UES then $F(x_k) - F^* \to 0$.

\begin{definition}\label{def:UES}
	A series of sequences $\{x_k\}_{k=0}^\infty$, $\{\phi_k(x)\}_{k=0}^\infty$ and $\{\alpha_k\}_{k=0}^\infty$, where $\alpha_k\in (0,1)$ for all $k\geq0$, is called an Underestimate Sequence (UES) of the function $F(x)$ if, for all $x\in\R^n$ and for all $k\geq 0$ we have,
	\begin{align}
		\phi_k(x) &\leq F(x),  \label{eq:def1}\\
		F(x_{k+1}) -\phi_{k+1}^* &\leq (1-\alpha_k) (F(x_{k}) - \phi_{k}^*), \label{eq:def2}
	\end{align}
	where $\phi_k^* := \min_x \phi_k(x)$.
\end{definition}

\begin{proposition}\label{prop:UES}
	If $\{x_k\}_{k=0}^\infty$, $\{\phi_k(x)\}_{k=0}^\infty$ and $\{\alpha_k\}_{k=0}^\infty$ is an UES of $F(x)$, then
	\begin{equation}\label{eq:linconverge}
		F(x_k) -\phi_k^* \leq \lambda_k(F(x_0) - \phi_0^*),
	\end{equation}
	where $\lambda_k =  \prod_{i=0}^k  (1-\alpha_k) .$ Furthermore, since both $\phi_k^*\leq \phi_k(x^*) \leq F^*$ and $\lambda_k\in(0,1)$ hold for all $k\geq0$, the above inequality implies that $\{F(x_k) - F^*\}_{k=0}^\infty$ converges to $0$.
\end{proposition}

Definition~\ref{def:UES} is different from Nesterov's Estimate Sequence (ES) in two ways. Firstly, both our UES and Nesterov's ES contain a sequence of estimators $\{\phi_k(x)\}_{k=0}^\infty$ for $F(x)$, and $\phi_k^*$ converges to $F^*$ as $k$ increases. However, in Definition~\ref{def:UES} $\phi_k(x)$ must be a \emph{lower/under estimator} of $F(x)$ for all $k\geq0$, while this does not necessarily hold for an ES. Nesterov's proof is based on the fact that $F(x_{k}) \leq \phi^*_{k}$, but this \emph{does not hold} in our case. Secondly, the definition of an ES only contains two sequences, while the UES has an extra sequence of points $\{x_k\}_{k=0}^\infty$. This enables us to show that the gap between the function value at $x_k$ and $\phi_k^*$ decreases in the $k$th iteration.

Proposition~\ref{prop:UES} shows that any sequences that form a UES (i.e., any sequences that satisfy Definition~\ref{def:UES}) are guaranteed to converge to the optimal solution of problem \eqref{eq:Problem} \emph{and} the estimate of the duality gap $F(x_k) -\phi_k^*$ is also guaranteed to converge at a linear rate. Thus, the UES construction provides a general framework for determining whether an optimization algorithm for problem \eqref{eq:Problem} will converge (linearly). In particular, if the iterates generated by an optimization algorithm satisfy Definition~\ref{def:UES}, then that algorithm is not only convergent, but also achieves a linear rate of convergence.

The UES framework is not only interesting from a theoretical perspective, but it also provides a major practical advantage. In particular, $F(x_k) -\phi_k^*$ provides a natural stopping criterion when designing algorithms, due to the fact that $F(x_k)$ and $\phi_k^*$ are upper and lower bounds for $F^*$, respectively. This difference is a kind of surrogate for the duality gap, and subsequently, algorithms that adhere to the UES framework are provided with a certificate of optimality, which is a highly desirable attribute.

\section{Lower Bounds via Quadratic Averaging}\label{sec:lb}

The purpose of this section is to introduce (global) lower bounds for the function $F(x)$ defined in \eqref{eq:Problem}, in both the smooth ($h(x)=0$) and nonsmooth cases. Lower bounds are the cornerstone of the UES set up, as seen in \eqref{eq:def1} in Definition~\ref{def:UES}. Being able to efficiently construct global lower bounds for $F(x)$ will allow the development of practical algorithms whose convergence is guaranteed via the UES framework.

Before stating the lower bounds, several technical results are presented that will be used throughout this paper.

\subsection{Preliminary Technical Results}

The proximal map is defined as
\begin{equation}\label{eq:proxmap}
\prox{x}{\gamma} \eqdef \arg\min_{u} \{ h(u) + \tfrac \gamma 2 \tnorm{x-u} \},
\end{equation}
and the proximal gradient is
\begin{equation}\label{eq:proxgrad}
	G_\gamma(x) \eqdef \gamma\left(x - \prox{x- \tfrac{1}{\gamma} \nabla f(x) }{\gamma}\right).
\end{equation}
Definitions \eqref{eq:proxmap} and \eqref{eq:proxgrad} will be used with $\gamma \equiv L$. Given some point $x\in \R^n$, a short step and a long step are denoted by
\begin{eqnarray}
    x^{+}  &\eqdef& x - \tfrac{1}{L} G_L(x),\label{eq:shortproxstep}\\
	x^{++} &\eqdef& x - \tfrac{1}{\mu} G_L(x). \label{eq:longproxstep}
\end{eqnarray}
In the smooth case ($h\equiv0$), the proximal gradient is simply the gradient $\nabla f(\cdot)$, so the short and long steps (\eqref{eq:shortproxstep} and \eqref{eq:longproxstep}) simplify as
\begin{eqnarray}
  x^+ &=& x - \tfrac1 L \nabla f(x) \label{eq:shortstep}\\
  x^{++} &=& x - \tfrac{1}{\mu} \nabla f(x).\label{eq:longstep}
\end{eqnarray}

The following Lemma characterizes elements of the subdifferential of $h(x^+)$.
\begin{lemma}\label{smalllemma}
Let $G_L(x)$ and $x^+$ be defined in \eqref{eq:proxgrad} and \eqref{eq:shortproxstep}, respectively. Then, for all $x\in\R^n$, $G_L(x) - \nabla f(x)\in \partial h(x^+)$.
\end{lemma}

\begin{proof} For a given point $x\in\R^n$,
	\begin{eqnarray*}
		x^+ &\overset{\eqref{eq:shortproxstep}}{=}&  x - \tfrac{1}{L} G_L(x)\\
		    &\overset{\eqref{eq:proxgrad}}{=}& x - \tfrac{1}{L}L (x - \prox{x - \tfrac{1}{L} \nabla f(x)}{L} ) \\
		    &=& \prox{x - \tfrac{1}{L} \nabla f(x)}{L} \\
		    &\overset{\eqref{eq:proxmap}}{=}& \arg\min_u \left\{h(u) + \tfrac L2 \tnorm{u- (x - \tfrac{1}{L} \nabla f(x))}  \right\}.
	\end{eqnarray*}
	This gives
	$$ 0\in  \tfrac 1L \partial h(x^+) + x^+ - (x - \tfrac{1}{L} \nabla f(x) ) \overset{\eqref{eq:shortproxstep}}{=} \tfrac 1L \partial h(x^+) - \tfrac 1L (G_L(x) -\nabla f(x)).$$
Multiplying through by $L$, and rearranging, gives the result.\qed
\end{proof}

\subsection{A Lower Bound for Smooth Functions}\label{s:lbsmooth}

For any point $y\in \R^n$, one can define a lower bound
\begin{equation}\label{eq:lowerboundDF}
	\phi(x;y) := f(y) - \tfrac{1}{2\mu}\tnorm{\nabla f(y)} + \tfrac{\mu}{2} \tnorm{x- y^{++} } \leq f(x),
\end{equation}
which holds with equality $\phi(x;y) = f(x)$ if and only if $x = y$. The lower bound in \eqref{eq:lowerboundDF} is a consequence of the assumption in \eqref{eq:ass1} and the equivalence
\begin{eqnarray}\label{eq:normequiv}
  \tfrac{\mu}2\tnorm{x- y^{++}} \overset{\eqref{eq:longstep}}{=} \tfrac{\mu}2\tnorm{x-y} + \langle x-y,\nabla f(y)\rangle + \tfrac1{2\mu}\tnorm{\nabla f(y)}.
\end{eqnarray}

Now, a sequence of lower bounds $\{\phi_k(x)\}_{k=0}^\infty$ can be defined in the following way. Using \eqref{eq:lowerboundDF} and a given initial point $x_0$, define the function
\begin{equation}\label{eq:phi0}
  \phi_0(x) \eqdef \phi(x; x_0) = \phi_0^* + \tfrac{\mu}{2}\tnorm{x-v_0},
\end{equation}
where
\begin{eqnarray}\label{eq:c0v0}
  \phi_0^* = f(x_0) - \tfrac{1}{2\mu}\tnorm{\nabla f(x_0)} \quad \text{and}\quad  v_0 = x_0^{++}.
\end{eqnarray}
Differentiating the expression in \eqref{eq:phi0} w.r.t. $x$ shows that $\phi_0^*$ and $v_0$ in \eqref{eq:c0v0} are the minimum value and minimizer of $\phi_0(x)$, respectively. This motivates the following construction:
\begin{enumerate}
  \item $\phi_0(x) \eqdef \phi(x; x_0) = \phi_0^* + \tfrac{\mu}{2}\tnorm{x-v_0}$
  \item For $k\geq 0$, $\alpha_k \in (0,1)$, and some point $y_k$ between $x_k$ and $v_k$, recursively define
  \begin{equation}\label{eq:zzzzz4}
	\phi_{k+1}(x) := (1- \alpha_k) \phi_k(x) + \alpha_k \phi(x; y_k).
\end{equation}
\end{enumerate}

\begin{lemma}
  For all $k\geq0$, $\phi\kp$ can be written in the canonical form
  \begin{equation}\label{eq:phicanonical}
    \phi\kp(x) = \phi\kp^* + \tfrac{\mu}2 \|x - v\kp\|^2,
  \end{equation}
  where $\alpha_k \in (0,1)$ and
  \begin{eqnarray}
    v_{k+1} &\eqdef& (1- \alpha_k) v_k + \alpha_k y_k^{++} \label{eq:zzzzzz0}\\
    \phi_{k+1}^* &\eqdef& (1- \alpha_k)(\phi_k^* +\tfrac{\mu}{2} \tnorm{v_{k+1}-v_k } ) \notag\\
	    	 &&+\alpha_k\left( f(y_k) - \tfrac{1}{2\mu}\tnorm{\nabla f(y_k)} + \tfrac{\mu}{2} \tnorm{v_{k+1} - y_k^{++}}\right).\label{eq:zzz1}
  \end{eqnarray}
\end{lemma}
\begin{proof}
  Using the definitions \eqref{eq:zzzzzz0} and \eqref{eq:zzz1}, for all $k\geq 0$ and all $x\in\R^n$, $\phi\kp(x)$ can be expressed in the form
  \begin{eqnarray*}
    \phi\kp(x) &=& (1-\alpha_k)\left(\phi_k^* + \tfrac{\mu}2 \|x - v_k\|^2 \right)\\
    &&+ \alpha_k\left(f(y_k) - \tfrac{1}{2\mu}\tnorm{\nabla f(y_k)} + \tfrac{\mu}{2} \tnorm{x - y_k^{++}}\right)\\
    &=& \phi\kp^* + \tfrac{\mu}2 \|x - v\kp\|^2.
  \end{eqnarray*}
  By taking the derivative of \eqref{eq:phicanonical} w.r.t. $x$, we see that the minimizer of $\phi_k(x)$ is $x = v\kp$, where $v\kp$ is defined in \eqref{eq:zzzzzz0}. Substituting this minimizer into \eqref{eq:phicanonical} gives the minimum value $\phi_{k+1}^*$ as in \eqref{eq:zzz1}.\qed
\end{proof}
\begin{lemma}\label{l:equivphikstar}
  An equivalent expression for $\phi_{k+1}^*$ in \eqref{eq:zzz1} is
\begin{align}\label{eq:phistarequiv}
    \phi_{k+1}^* &= (1-\alpha_k)\left(\phi_k^* + \alpha_k\tfrac{\mu}2\|v_{k}-y_k^{++}\|_2^2\right) +\alpha_k( f(y_k) - \tfrac{1}{2\mu}\tnorm{\nabla f(y_k)}).
\end{align}
\end{lemma}
\begin{proof}
  Using \eqref{eq:zzzzzz0} gives the equivalences
\begin{eqnarray}\label{eq:vkp1vk}
  \|v_{k+1}-v_{k}\|^2 &=& \|(1-\alpha_k)v_k + \alpha_k y_k^{++}-v_{k}\|^2 = \alpha_k^2\|v_{k}-y_k^{++}\|^2
\end{eqnarray}
and
\begin{eqnarray}
\notag
  \|v_{k+1} - y_k^{++}\|^2 &=& \|(1-\alpha_k)v_k + \alpha_k y_k^{++} - y_k^{++}\|^2\\
  &=& (1-\alpha_k)^2\|v_k - y_k^{++}\|^2.\label{eq:vkp1xkpp}
\end{eqnarray}
Combining \eqref{eq:vkp1vk} and \eqref{eq:vkp1xkpp} gives
\begin{eqnarray}
\notag
  &&\hspace{-5mm}(1-\alpha_k)\|v_{k+1}-v_{k}\|_2^2 + \alpha_k\|v_{k+1} - y_k^{++}\|^2\\
  \notag
  &=&\alpha_k^2(1-\alpha_k)\|v_{k}-y_k^{++}\|_2^2 + \alpha_k(1-\alpha_k)^2\|v_k - y_k^{++}\|^2\\
  \notag
  &=&\alpha_k(1-\alpha_k)(\alpha_k + (1-\alpha_k)) \|v_{k}-y_k^{++}\|_2^2 \\
  &=&\alpha_k(1-\alpha_k)\|v_{k}-y_k^{++}\|_2^2.\label{eq:vkequiv}
\end{eqnarray}
Substituting \eqref{eq:vkequiv} into \eqref{eq:zzz1} gives the result.\qed
\end{proof}

The following Lemma shows that $\phi_k(x)$ is a (global) lower bound for $f(x)$.
\begin{lemma}\label{lem:lb}
	For all $k\geq0$, let $\alpha_k \in (0,1)$. Then, for all $x\in\R^n$, $\phi_k(x)\leq f(x)$.
\end{lemma}
\begin{proof}
We proceed by induction. When $k=0$, the result holds trivially. Now, assume that $\phi_k(x) \leq f(x)$. Then
\begin{eqnarray*}
\phi_{k+1}(x) &\overset{\eqref{eq:zzzzz4}}{=}& (1- \alpha_k) \phi_k(x) + \alpha_k \phi(x; y_k),\\
	&\overset{\eqref{eq:lowerboundDF}}{\leq}& (1- \alpha_k) f(x) + \alpha_k f(x) = f(x).
\end{eqnarray*}
\qed
\end{proof}

\subsection{A Lower Bound for Composite Functions}\label{s:lbcomposite}

Here, the previous results are extended from the smooth to the composite setting, so it is assumed that $h(x)$ is not equivalent to the zero function.

The following Lemma defines a lower bound for $F(x)$ in \eqref{eq:Problem}. The lower bound is the same as that presented in \cite{Drusvyatskiy16} and \cite{Chen16}, with the roles of $x$ and $y$ reversed here; the proof is included for completeness.
\begin{lemma}[Lemma~6.1 in \cite{Drusvyatskiy16}; Lemma~3.1 in \cite{Chen16}]
  Given a point $y \in \R^n$, let $G_L(y)$ and $y^+$ be defined in \eqref{eq:proxgrad} and \eqref{eq:shortproxstep}, respectively. Then for all $x\in \R^n$
  \begin{eqnarray}\label{eq:lbcomposite}
    \varphi(x;y) \eqdef F(y^+) + \langle G_L(y),x-y\rangle + \tfrac{\mu}2\|x-y\|^2 + \tfrac1{2L}\|G_L(y)\|^2 \leq F(x).
  \end{eqnarray}
\end{lemma}
\begin{proof}
   By Assumption~\ref{A_SCL} ($\mu$-strongly convex)
\begin{eqnarray}\label{eq:mustrong}
 f(y) + \ve{\nabla f(y) }{x-y} + \tfrac \mu 2 \tnorm{x-y} \leq f(x), \qquad \forall x,y\in\R^n,
\end{eqnarray}
and ($L$-smooth)
\begin{eqnarray}\label{eq:Lsmooth}
\notag
  f(y^+) &\leq& f(y) + \langle \nabla f(y), y^+ - y \rangle + \tfrac L{2 }\| y^+-y  \|_2^2\\
  &\overset{\eqref{eq:shortproxstep}}{=}& f(y) -\tfrac1L \langle \nabla f(y), G_L(y)\rangle + \tfrac1{2L}\|G_L(y)\|_2^2.
\end{eqnarray}
Combining \eqref{eq:mustrong} and \eqref{eq:Lsmooth} gives
\begin{eqnarray*}
\notag
  F(y^+) &\leq&  F(x) - \ve{\nabla f(y) }{x-y} - \tfrac \mu 2 \tnorm{x-y} -\tfrac1L \langle \nabla f(y), G_L(y)\rangle\\
  && + \tfrac1{2L}\|G_L(y)\|_2^2 + (h(y^+) - h(x))\\
  &=&  F(x) - \ve{\nabla f(y) }{x-y^+} - \tfrac \mu 2 \tnorm{x-y} \\
  && + \tfrac1{2L}\|G_L(y)\|_2^2 + (h(y^+) - h(x))\\
  &=&  F(x) - \ve{\nabla f(y) - G_L(y)}{x-y^+} - \tfrac \mu 2 \tnorm{x-y}\\
  && + \tfrac1{2L}\|G_L(y)\|_2^2 + (h(y^+) - h(x)) - \ve{G_L(y)}{x-y^+}\\
  &\leq&  F(x)- \tfrac \mu 2 \tnorm{x-y} + \tfrac1{2L}\|G_L(y)\|_2^2 - \ve{G_L(y)}{x-y^+}\\
  &=&  F(x)- \tfrac \mu 2 \tnorm{x-y} - \tfrac1{2L}\|G_L(y)\|_2^2 - \ve{G_L(y)}{x-y}.
\end{eqnarray*}
Rearranging gives the result.\qed
\end{proof}

Before stating the next result, which shows that $\varphi(x;y)$ is a quadratic lower bound, we give the following equivalence, which is the composite version of \eqref{eq:normequiv},
\begin{eqnarray}\label{eq:normequivcomposite}
  \tfrac{\mu}2\tnorm{x- y^{++}} \overset{\eqref{eq:longproxstep}}{=} \tfrac{\mu}2\tnorm{x-y} + \langle x-y,G_L(y)\rangle + \tfrac1{2\mu}\tnorm{\nabla G_L(y)}.
\end{eqnarray}

\begin{lemma}\label{lem:varphixycanon}
  For all $x,y\in\R^n$, the lower bound \eqref{eq:lbcomposite} has the canonical form
  \begin{equation}\label{eq:varphixycanon}
  \varphi(x;y) = \varphi^* + \tfrac{\mu}{2} \tnorm{x-y^{++}},
\end{equation}
where
 \begin{equation}\label{eq:varphiminval}
  \varphi^* = F(y^+) + \left(\tfrac1{2L} - \tfrac1{2\mu} \right)\|G_L(y)\|^2.
\end{equation}
\end{lemma}
\begin{proof}
Minimizing $\varphi(x;y)$ in \eqref{eq:lbcomposite} w.r.t. $x$, and using the definition in \eqref{eq:proxmap}, yields the minimizer
\begin{equation}\label{eq:varphiminimizer}
  y^{++} = \arg\min_x \varphi(x;y).
\end{equation}
The corresponding  minimal value is
\begin{align*}
 \varphi^* &\eqdef \min_x\varphi(x;y) = \varphi(y^{++};y)\\
 &\overset{\eqref{eq:lbcomposite}}{=} F(y^+) + \langle G_L(y),y^{++}-y\rangle + \tfrac{\mu}2\|y^{++}-y\|^2 + \tfrac1{2L}\|G_L(y)\|^2\\
 &\overset{\eqref{eq:longproxstep}}{=} F(y^+) - \tfrac1{\mu}\langle G_L(y),G_L(y) + \tfrac{\mu}2\|\tfrac1{\mu}G_L(y)\|^2 + \tfrac1{2L}\|G_L(y)\|^2\\
 &= F(y^+) + \left(\tfrac1{2L} - \tfrac1{2\mu} \right)\|G_L(y)\|^2,
\end{align*}
which is equivalent to \eqref{eq:varphiminimizer}. (Note also that \eqref{eq:varphiminimizer} and \eqref{eq:varphiminval} are the minimizer and minimum value of \eqref{eq:varphixycanon}, respectively.) Furthermore,
\begin{eqnarray*}
  \varphi(x;y) &\overset{\eqref{eq:varphixycanon}}{=}& \varphi^* + \tfrac{\mu}{2} \tnorm{x-y^{++}}\\
  &=& F(y^+) + \left(\tfrac1{2L} - \tfrac1{2\mu} \right)\|G_L(y)\|^2 + \tfrac{\mu}{2} \tnorm{x-y^{++}}\\
  &\overset{\eqref{eq:normequivcomposite}}{=}& F(y^+) + \left(\tfrac1{2L} - \tfrac1{2\mu} \right)\|G_L(y)\|^2\\
  && + \tfrac{\mu}{2} \tnorm{x-y}+ \tfrac1{2\mu} \tnorm{G_L(y)} + \langle G_L(y), x-y\rangle\\
  &=&F(y^+) + \tfrac1{2L}\|G_L(y)\|^2 + \tfrac{\mu}{2} \tnorm{x-y} +\langle G_L(y), x-y\rangle,
\end{eqnarray*}
which confirms that \eqref{eq:varphixycanon} is equivalent to \eqref{eq:lbcomposite}.\qed
\end{proof}
\begin{remark}
  Lemma~\ref{lem:varphixycanon} shows that the lower bound \eqref{eq:lbcomposite} (equivalently \eqref{eq:varphixycanon}) is a quadratic lower bound for $F(x)$.
\end{remark}
Now, a sequence of lower bounds $\{\varphi_k(x)\}_{k=0}^\infty$ can be defined in the following way. Using \eqref{eq:lbcomposite} and a given initial point $x_0$, define the function
\begin{equation}\label{eq:compositecanonical}
  \varphi_0(x) \eqdef \varphi(x;x_0) = \varphi_0^* + \tfrac{\mu}2\|x-v_0\|_2^2,
\end{equation}
where
\begin{eqnarray}\label{eq:minvalminmize}
  \varphi_0^* \eqdef F(x_0^+) + \left(\tfrac1{2L} - \tfrac1{2\mu} \right)\|G_L(x_0)\|^2,\quad v_0 = x_0^{++}.
\end{eqnarray}
Differentiating \eqref{eq:compositecanonical} w.r.t. $x$ shows that the minimum value and minimizer of $\varphi_0(x)$ are given by \eqref{eq:minvalminmize}. This motivates the following construction
\begin{enumerate}
  \item $\varphi_0(x) \eqdef \varphi(x;x_0) = \varphi_0^* + \tfrac{\mu}2\|x-v_0\|_2^2,$
  \item For $k\geq 0$ and some point $y_k$ between $x_k$ and $v_k$ we recursively define
  \begin{equation}\label{eq:varphiconvexcomb}
    \varphi_{k+1}(x) \eqdef (1-\alpha_k) \varphi_k(x) + \alpha_k \varphi(x;y_k).
  \end{equation}
\end{enumerate}

\begin{lemma}
  For all $k \geq 0$, $\varphi_{k+1}$ can be written in the canonical form
  \begin{eqnarray}\label{eq:canon}
    \varphi_{k+1}(x) \eqdef \varphi_{k+1}^* + \tfrac{\mu}2\|x-v_{k+1}\|_2^2,
  \end{eqnarray}
  where
  \begin{eqnarray}
    v_{k+1} &\eqdef&(1-\alpha_k)v_k + \alpha_ky_k^{++}\label{eq:vkcomp}\\
    \varphi_{k+1}^* &\eqdef& (1-\alpha_k)(\varphi_k^* + \tfrac{\mu}2\|v_{k+1}-v_{k}\|_2^2)\label{eq:phikcomp}\\
    \notag
     &&+\; \alpha_k\left(F(y_k^+) + \left(\tfrac1{2L} - \tfrac1{2\mu}\right)\|G_L(y_k)\|^2 + \tfrac{\mu}2\|v_{k+1} - y_k^{++}\|^2\right).
  \end{eqnarray}
\end{lemma}
\begin{proof}
Using \eqref{eq:varphiconvexcomb} and \eqref{eq:canon}, for all $k\geq 0$ and all $x\in \R^n$, $\varphi_{k+1}(x)$ can be expressed in the form
  \begin{eqnarray*}
    \varphi_{k+1}(x) &=& (1-\alpha_k)(\varphi_k^* + \tfrac{\mu}2\|x-v_{k}\|_2^2)\\
     &&+ \; \alpha_k\left(F(y_k^+)+ \left(\tfrac1{2L} - \tfrac1{2\mu}\right)\|G_L(y_k)\|^2 + \tfrac{\mu}2\|x - y_k^{++}\|^2\right)\\
     &=& \varphi_{k+1}^* + \tfrac{\mu}2\|x-v_{k+1}\|_2^2.
  \end{eqnarray*}
  Taking the derivative of \eqref{eq:canon} w.r.t. $x$ shows that the minimizer of $\varphi_k(x)$ is $x = v_{k+1}$. Substituting $x = v_{k+1}$ into the above gives \eqref{eq:phikcomp}.\qed
\end{proof}

\begin{lemma}
  An equivalent expression for $\varphi_{k+1}^*$ in \eqref{eq:phikcomp} is
\begin{eqnarray}\label{eq:varphistarequiv}
    \varphi_{k+1}^* &=& (1-\alpha_k)\left(\varphi_k^* + \alpha_k\tfrac{\mu}2\|v_{k}-y_k^{++}\|_2^2\right)\\
    \notag
    &&+ \alpha_k\left(F(y_k^+) +  \left(\tfrac1{2L} -\tfrac1{2\mu} \right)\|G_L(y_k)\|^2\right).
\end{eqnarray}
\end{lemma}
\begin{proof}
The proof follows the same arguments as for Lemma~\ref{l:equivphikstar}; noting that \eqref{eq:zzzzzz0} and \eqref{eq:vkcomp} are equivalent, and then combining \eqref{eq:vkequiv} and \eqref{eq:phikcomp} gives the result. \qed
\end{proof}

\begin{lemma}
  For all $k\geq 0 $ and $\forall x\in\R^n$, $\varphi_k(x) \leq F(x)$.
\end{lemma}
\begin{proof}
  When $k=0$, the result holds trivially. Now assume that $\varphi_k(x) \leq F(x)$. Then
  \begin{eqnarray*}
    \varphi_{k+1}(x) &=&(1-\alpha_k)\varphi_k(x) + \alpha_k\varphi(x;y_k)\\
    &\leq& (1-\alpha_k)F(x) + \alpha_kF(x) = F(x).
  \end{eqnarray*}
  This completes the proof.\qed
\end{proof}

\section{Algorithms and Convergence Guarantees for Smooth Functions}\label{sec:smooth}

The purpose of this section is to demonstrate that the UES framework, and the previously presented lower bounds, are \emph{useable} definitions that give rise to \emph{efficient implementable algorithms}. Throughout this section we consider smooth optimization problems (problems of the form \eqref{eq:Problem} with $h\equiv 0$) and, as for all results in this work, we suppose that Assumption~\ref{A_SCL} holds.

We present two algorithms whose iterates fit the Underestimate Sequence framework described in Section~\ref{sec:UES}, and use the lower bounds developed in Section~\ref{s:lbsmooth}. The first algorithm is a gradient descent type method, while the second algorithm is a gradient descent type method that incorporates an acceleration strategy. As will be shown, both algorithms are supported by convergence guarantees, which are established via the UES framework.

\subsection{An Underestimate Sequence Algorithm for Smooth Functions}

We are now ready to present an algorithm that fits our UES framework; a brief description follows.

\begin{algorithm}[h!]
	\caption{Smooth Underestimate Sequence Algorithm (SUESA)}
	\label{alg:SUESA}
	\begin{algorithmic}[1]
        \STATE Initialization: Set $k=0$, $\epsilon >0$, initial point $x_0\in\R^n$ and compute $\mu$, $L$.
		\STATE Set $\phi_0(x)$ as in \eqref{eq:phi0}, with $v_0$ and $\phi_0^*$ as in \eqref{eq:c0v0}, and let $\alpha_k= \frac{\mu}{L}$.
		\WHILE {$f(x_k) - \phi_k^* > \epsilon$}
        \STATE Set $y_k = x_k$ and $y_k^{++} = x_k^{++}$.
		\STATE Set $x_{k+1} = x_k - \frac{1}{L} \nabla f(x_k)$.
        \STATE Update $v_{k+1}$ and $\phi_{k+1}^*$ as in \eqref{eq:zzzzzz0} and \eqref{eq:zzz1}, respectively.
		\STATE $k=k+1$.
		\ENDWHILE
	\end{algorithmic}
\end{algorithm}

The Smooth (functions) UnderEstimate Sequence Algorithm (SUESA) presented in Algorithm~\ref{alg:SUESA} solves the problem \eqref{eq:Problem} in the smooth case, i.e., when $h = 0$. The algorithm proceeds as follows. First, an initial point $x_0\in \R^n$ is chosen, as well as some stopping tolerance $\epsilon>0$. Secondly, the point $v_0 = x_0^{++}$ (i.e., $v_0$ is the long step from $x_0$) is constructed, as well as the lower bound $\phi_0(x)$ with minimum value $\phi_0^*$. The algorithm uses a fixed step size of $\alpha = \mu/L$ at every iteration. Next, the main loop commences and an iteration proceeds as follows. One sets $y_k = x_k$ (i.e., $y_k$ is not explicitly used in SUESA); $x_k$ is updated by taking a gradient descent step with the step size $\tfrac{1}{L}$, resulting in the new point $x_{k+1}$; the point $v_{k+1} = x_{k+1}^{++}$ is constructed and the lower bound $\phi_{k+1}(x)$ is updated.

The algorithm constructs two points at every iteration, namely $x_k$ and $v_k$, and the values $\phi_{k}(x)$ and $\phi_{k}^*$. The point $v_k$ and the value $\phi_{k}^*$ are used for the lower bound, which is essential for the stopping criterion. The stopping condition $f(x_k) - \phi_k^* \leq \epsilon$ provides a certificate of optimality; once the stopping condition is satisfied, it is guaranteed that $x_k$ gives a function value $f(x_k)$ that is at most $\epsilon$ from the true solution $f^*$.

If Step~5 is considered in isolation, then one sees that at every iteration of SUESA, the point $x_k$ is updated via a standard gradient descent step. That is, a step of size $1/L$ in the direction of the negative gradient is taken from the current point $x_k$, resulting in the new point $x_{k+1}$. However, SUESA is different from the standard gradient descent method, because SUESA also involves several other ingredients, including the points $v_k$ and lower bound values $\phi_k^*$.

The following result provides a convergence guarantee for SUESA. In particular, Theorem~\ref{thm1} shows that the iterates generated by Algorithm~\ref{alg:SUESA} form an underestimate sequence (i.e., they satisfy Definition~\ref{def:UES}) and therefore, Algorithm~\ref{alg:SUESA} is guaranteed to converge (linearly) to the solution of problem \eqref{eq:Problem}.

\begin{theorem}\label{thm1}
	Let Assumption~\ref{A_SCL} hold. The sequences $\{x_k\}_{k=0}^\infty$, $\{\phi_k(x)\}_{k=0}^\infty$ and $\{\alpha_k\}_{k=0}^\infty$ generated by SUESA (Algorithm~\ref{alg:SUESA}) form a UES.
\end{theorem}
\begin{proof}
	We must show that the iterates generated by Algorithm~\ref{alg:SUESA} satisfy the conditions of Definition~\ref{def:UES}. Note that, $\alpha_k = \mu/L \in (0,1)$ for all $k \geq 0$, so by Lemma~\ref{lem:lb}, \eqref{eq:def1} holds. Thus, it remains to prove \eqref{eq:def2}. Combining the definition of $x_{k+1}$ (Step~5 in Algorithm~\ref{alg:SUESA}) with \eqref{eq:ass2}, gives
	\begin{eqnarray}\label{eq:zzzz3}
		f(x_{k+1}) &\leq& f(x_k) + \ve{\nabla f(x_k)}{-\tfrac{1}{L} \nabla f(x_k)}+\tfrac{L}{2} \tnorm {\tfrac{1}{L} \nabla f(x_k)} \notag\\
		&=& f(x_k)  - \tfrac{1}{2L} \tnorm{\nabla f(x_k)}.
	\end{eqnarray}
	Subtracting $\phi_{k+1}^*$ in \eqref{eq:phistarequiv} from both sides of the above gives,
	\begin{eqnarray*}
		f(x_{k+1}) - \phi_{k+1}^*
&\leq& f(x_k) - \tfrac{1}{2L} \tnorm{\nabla f(x_k)}
			 - (1-\alpha_k)\left(\phi_k^* + \alpha_k\tfrac{\mu}2\|v_{k}-y_k^{++}\|_2^2\right)\\
&& -\alpha_k( f(y_k) - \tfrac{1}{2\mu}\tnorm{\nabla f(y_k)})\\
		&\leq & (1- \alpha_k) (f(x_k) -\phi_k^*)  - (\tfrac{\alpha_k}{2\mu} - \tfrac{1}{2L} )\tnorm{\nabla f(x_k)} \\
		    &&- (1-\alpha_k)\alpha_k\tfrac{\mu}2\|v_{k}-y_k^{++}\|_2^2 \\
		&\leq & (1- \alpha_k) (f(x_k) -\phi_k^*),
	\end{eqnarray*}
where the last step follows because $\alpha_k = \tfrac{\mu}{L}$, so $\tfrac{\alpha_k}{2\mu} - \tfrac{1}{2L}=\tfrac{\mu}{2\mu L} - \tfrac{1}{2L}=0$, and $(1-\alpha_k)\alpha_k\tfrac{\mu}2\|v_{k}-y_k^{++}\|_2^2\geq 0$.
    Therefore, the sequences $\{x_k\}_{k=0}^\infty$, $\{\phi_k(x)\}_{k=0}^\infty$ and $\{\alpha_k\}_{k=0}^\infty$ generated by Algorithm~\ref{alg:SUESA}
	form a UES. \qed
\end{proof}

\begin{corollary}\label{coro1}
Let Assumption~\ref{A_SCL} hold.  Then the sequences $\{x_k\}_{k=0}^\infty$, $\{\phi_k(x)\}_{k=0}^\infty$ and $\{\alpha_k\}_{k=0}^\infty$ generated by SUESA (Algorithm~\ref{alg:SUESA}) form a UES, so SUESA converges at a linear rate
	\begin{align}
		 f(x_k) -\phi_k^* \leq (1- \tfrac{\mu}{L})^k (f(x_0) -\phi_0^*).
	\end{align}
\end{corollary}

Corollary~\ref{coro1} is simply a consequence Proposition~\ref{prop:UES}, which states that if $\{x_k\}_{k=0}^\infty$, $\{\phi_k(x)\}_{k=0}^\infty$ and $\{\alpha_k\}_{k=0}^\infty$ form an underestimate sequence, then \eqref{eq:linconverge} holds (i.e., linear convergence). Theorem~\ref{thm1} shows that SUESA (Algorithm~\ref{alg:SUESA}) generates iterates forming a UES, implying that SUESA converges linearly to the optimal solution. Moreover, $\alpha_k = \tfrac{\mu}L$ for all $k$ in SUESA, so recalling the definition of $\lambda_k$ in Proposition~\ref{prop:UES}, confirms the rate $(1- \tfrac{\mu}{L})$ in Corollary~\ref{coro1}.

We remark that there are other ways to prove convergence of Algorithm~\ref{alg:SUESA}. For example, one can proceed by proving that the distance between $x_k$ and the minimizer of the lower bound in $k$th iteration shrinks at a fixed rate. That is, since $\alpha_k = \tfrac{\mu}{L}$, we have the following equality,
\begin{align}\label{eq:xkvk}
	x_{k+1} - v_{k+1} &= \left(x_k - \tfrac{1}{L} \nabla f(x_k)\right) - \left((1- \alpha_k)v_k - \alpha_k (x_k - \tfrac{1}{\mu} \nabla f(x_k) )\right) \notag\\
	& = (1- \tfrac{\mu}{L}) (x_k - v_k).
\end{align}
Equation \eqref{eq:xkvk} illustrates that, after each iteration of Algorithm~\ref{alg:SUESA}, the line joining $x_{k+1}$ and $v_{k+1}$ is parallel to the line joining $x_k$ and $v_k$ from the previous iteration (see the blue lines in Figure~\ref{fig:a1o1}). Moreover, the distance between the two points is reduced by precisely $(1- \tfrac{\mu}{L})$ at every iteration. Intuitively, the solution $x_k$ and the minimizer $v_k$ are becoming ever closer, and eventually they both converge to the optimal solution $x^*.$

One can visualize the fact above using the following toy example. Consider the (smooth) regularized logistic regression problem, i.e., problem \eqref{eq:Problem} with $h=0$ and
\begin{align*}
	f(x) = \sum_{i=1}^m \log(1+\exp(-y_i \ve{x}{a_i})) + \tfrac \lambda 2 \|x\|^2,
\end{align*}
where $a_i \in \R^n$ is the $i$th feature vector with corresponding (binary) label $y_i\in \R$.
For this example we randomly generate 100 two dimensional data points with binary labels $\{a_i, y_i\}$ (so $m=100$ and $n=2$) as shown in the left hand plot in Figure~\ref{fig:a1o1}. (Each point $a_i$ is plotted on a 2D grid, and the point is colored green or red to highlight its label $y_i$). Parameter $\lambda= 0.01$, so the strong convexity constant is $\mu=0.01$. Algorithm~\ref{alg:SUESA} is used to solve this problem, starting from the point $x_0 = (-20,10)^T$, and the iterates are shown in the right hand plot in Figure~\ref{fig:a1o1}.
\begin{figure}[H]
\centering
\includegraphics[scale=.35]{fig1.eps}
\includegraphics[scale=.3]{a1.eps}
\caption{Left: Randomly generated two classes of 2D data. Right: A 2D illustration for Option 1. The blue, red and green points represent $\{x_k\}, \{v_k\}, \{x_k^{++}\}$ respectively.}
\label{fig:a1o1}
\end{figure}

\subsection{An Accelerated Underestimate Sequence Algorithm for Smooth Functions}

We now present an accelerated first order algorithm for solving problems of the form \eqref{eq:Problem} when $h=0$; a description will follow.

\begin{algorithm}
	\caption{Accelerated Smooth Underestimate Sequence Algorithm (ASUESA)}
	\label{alg:ASUESA}
	\begin{algorithmic}[1]
		\STATE Initialization: Set $k=0$, $\epsilon >0$, initial point $x_0\in\R^n$ and compute $\mu$, $L$.
\STATE Set $\phi_0(x)$ as in \eqref{eq:phi0}, with $v_0$ and $\phi_0^*$ as in \eqref{eq:c0v0}.	 Let $\alpha_k= \sqrt{\frac{\mu}{L}}$, $\beta_k = \frac{1}{1+\alpha_k}$.
		\WHILE {$f(x_k) - \phi_k^* > \epsilon$}
		\STATE Set $y_k = \beta_k x_k + (1- \beta_k) v_k$.
		\STATE Set $x_{k+1} = y_k - \frac{1}{L} \nabla f(y_k).$
		\STATE Update $v_{k+1}$ and $\phi_{k+1}^*$ as in \eqref{eq:zzzzzz0} and \eqref{eq:zzz1}, respectively.
		\STATE $k=k+1$.
		\ENDWHILE
	\end{algorithmic}
\end{algorithm}

The Acclerated Smooth UnderEstimate Sequence Algorithm (ASUESA) presented in Algorithm~\ref{alg:ASUESA} solves \eqref{eq:Problem} in the smooth case, i.e., when $h = 0$, and can be described as follows. Algorithm initialization is similar to that of SUESA (Algorithm~\ref{alg:SUESA}), where an initial point $x_0\in \R^n$ and some stopping tolerance $\epsilon>0$ are chosen, the point $v_0 = x_0^{++}$ is constructed and the lower bound $\phi_0(x)$ and minimum value $\phi_0^*$ are evaluated. For ASUESA one sets $\alpha_k = \sqrt{\mu/L}$ and the parameter $\beta_k = \frac{1}{1+\alpha_k}$ is also used. Parameter $\alpha_k$ is fixed for all iterations, and subsequently so too is $\beta_k$. The main loop proceeds as follows. At every iteration one sets $y_k$ to be a convex combination of the points $x_k$ and $v_k$; a gradient descent step is taken \emph{from} $y_k$, resulting in the new point $x_{k+1}$; the point $v_{k+1}$ is constructed using \eqref{eq:zzzzzz0} and the lower bound $\phi_{k+1}(x)$ is updated via \eqref{eq:zzz1}.

Notice that Algorithm~\ref{alg:ASUESA} can be viewed as an accelerated version of Algorithm~\ref{alg:SUESA}. In contrast to Algorithm~\ref{alg:SUESA}, ASUESA constructs \emph{three} points at every iteration, namely $x_k$, $v_k$ and $y_k$, where the intermediate vector $y_k$ is a convex combination of the points $x_k$ and $v_k$ (i.e., for ASUESA $x_k \neq y_k$.) Notice also that $x_{k+1}$ is the result of a gradient descent step taken from the point $y_k$.  The variable $\phi_{k}^*$ is also maintained and is used in the stopping condition.

The following result provides a convergence guarantee for ASUESA. Theorem~\ref{thm2} shows that the iterates generated by Algorithm~\ref{alg:ASUESA} fit the UES framework (i.e., they satisfy Definition~\ref{def:UES}) and therefore, Algorithm~\ref{alg:ASUESA} is guaranteed to converge (linearly at the optimal rate) to the solution of problem \eqref{eq:Problem} (see Corollary~\ref{coro2}).

\begin{theorem}\label{thm2}
Let Assumption~\ref{A_SCL} hold. The series of sequences $\{x_k\}_{k=0}^\infty$, $\{\phi_k(x)\}_{k=0}^\infty$ and $\{\alpha_k\}_{k=0}^\infty$ generated by ASUESA in Algorithm~\ref{alg:ASUESA} form an UES.
\end{theorem}
\begin{proof}
At every iteration of ASUESA the function value is reduced as follows
\begin{align}
\notag
f(y_k^+) = f(y_k - \tfrac{1}{L}\nabla f (y_k))&\leq f(y_k) + \langle\nabla f(y_k),\tfrac{-\nabla f(y_k)}{L}\rangle + \tfrac{L}{2}\tnorm{ \tfrac{\nabla f(y_k)}{L}}\\
& = f(y_k) -\tfrac{1}{2L}\tnorm{\nabla f(y_k) }. \label{eq:ibuij}
\end{align}

Moreover,
\begin{eqnarray*}
  f(x_{k+1})-\phi_{k+1}^* &\overset{\eqref{eq:phistarequiv}}{=} &f(y_k^+) -(1-\alpha_k) \phi_k^*\\
		 &&- \alpha_k\left(f(y_k) - \tfrac{\tnorm{\nabla f(y_k) }}{2\mu} + \tfrac{\mu}{2}(1-\alpha_k )\tnorm{v_k -y_k^{++}}\right)  \\
		&\overset{\eqref{eq:ibuij}}{\leq} & f(y_k) - \tfrac{1}{2L}\tnorm{\nabla f(y_k) }  - (1-\alpha_k)\phi_k^* \\
		&&- \alpha_k\left(f(y_k) - \tfrac{\tnorm{\nabla f(y_k) }}{2\mu} + \tfrac{\mu}{2}(1-\alpha_k )\tnorm{v_k -y_k^{++}}\right)\\
        &=& (1-\alpha_k)(f(y_k) - \phi_k^*) - \left(\tfrac{1}{2L} - \tfrac{\alpha_k}{2\mu}\right)\tnorm{\nabla f(y_k) }\\
        && -  \alpha_k\tfrac{\mu}{2}(1-\alpha_k )\tnorm{v_k -y_k^{++}}.
\end{eqnarray*}
	By completing the square term one obtains
\begin{eqnarray}
&&\hspace{-4mm}f(x_{k+1})-\phi_{k+1}^* \notag\\
		&= & (1-\alpha_k)\left(f(y_k)-\phi_k^* \right)- \left(\tfrac{1}{2L} - \tfrac{\alpha_k}{2\mu}\right)\tnorm{\nabla f(y_k) } \notag \\
		&&-\tfrac{\mu}{2}\alpha_k(1-\alpha_k) \Big(\tnorm{v_k - y_k}+\tnorm{\tfrac{1}{\mu}\nabla f(y_k) }
		 + \tfrac{2}{\mu} \langle\nabla f(y_k),v_k -y_k\rangle\Big) \notag\\
    &= & (1-\alpha_k)\left(f(y_k)-\phi_k^* \right)- \left(\tfrac{1}{2L} - \tfrac{\alpha_k}{2\mu}+ \tfrac{\alpha_k(1-\alpha_k)}{2\mu}\right)\tnorm{\nabla f(y_k) } \notag \\
		&&-\alpha_k(1-\alpha_k) \langle\nabla f(y_k),v_k -y_k\rangle \notag\\
		&\leq & (1-\alpha_k)\left(f(y_k)-\phi_k^* \right) - \left(\tfrac{1}{2L} - \tfrac{\alpha_k^2}{2\mu}\right)\tnorm{\nabla f(y_k) } \notag\\
		&& - \alpha_k(1-\alpha_k)\langle\nabla f(y_k),v_k -y_k\rangle\notag\\
        &=& (1-\alpha_k)\left(f(y_k)-\phi_k^* \right)- \alpha_k(1-\alpha_k)\langle\nabla f(y_k),v_k -y_k\rangle,\label{eq:fphi}
\end{eqnarray}
where the last step follows because $\alpha_k = \sqrt{\frac{\mu}{L}}$ in ASUESA, so $\tfrac{1}{2L} - \tfrac{\alpha_k^2}{2\mu} = 0$.

Now, rearranging the expression for $y_k$ in Step~6 in Algorithm~\ref{alg:ASUESA} gives
  \begin{equation}\label{eq:vk}
    v_k = \tfrac1{1-\beta_k}(y_k - \beta_k x_k) = y_k + \tfrac{\beta_k}{1-\beta_k}(y_k - x_k),
  \end{equation}
and notice also that $\beta_k = \tfrac{1}{1+\alpha_k}$ for all $k$, so
  \begin{eqnarray}\label{eq:alphabeta1}
    1-\beta_k =  1-\tfrac{1}{1+\alpha_k} = \tfrac{\alpha_k}{1+\alpha_k} =\alpha_k\beta_k \quad \Rightarrow \quad \tfrac{\alpha_k\beta_k}{1-\beta_k}= 1.
  \end{eqnarray}
Thus, by the convexity of $f$ we have
	\begin{eqnarray}\label{eq:pqr}
		-\langle\nabla f(y_k),v_k -y_k\rangle \notag
		&=& -\langle\nabla f(y_k),y_k + \tfrac{\beta_k}{1- \beta_k} (y_k-x_k) -y_k\rangle \notag\\
		&\leq& \tfrac{\beta_k}{1- \beta_k} \big(f(x_k) - f(y_k)  \big).
	\end{eqnarray}
	Using \eqref{eq:pqr} in \eqref{eq:fphi} gives
	\begin{align*}
		f(x_{k+1})-\phi_{k+1}^*
		\leq &(1-\alpha_k)\left(f(y_k)-\phi_k^* \right)+\alpha_k(1-\alpha_k)\tfrac{\beta_k}{1-\beta_k}(f(x_k)-f(y_k))\\
        \overset{\eqref{eq:alphabeta1}}{=} &(1-\alpha_k)\left(f(y_k)-\phi_k^* \right)+(1-\alpha_k)(f(x_k)-f(y_k))\\
		= & (1-\alpha_k) (f(x_k)-\phi_k^*).
	\end{align*}
\qed
\end{proof}

\begin{corollary}\label{coro2}
Let Assumption~\ref{A_SCL} hold. Then, the sequence of iterates $\{x_k\}_{k\geq0}$ generated by Algorithm \ref{alg:ASUESA} exhibits the optimal linear rate of convergence
	\begin{align*}
		 f(x_k) -\phi_k^* \leq \left(1- \sqrt{\tfrac{\mu}{L}}\right)^k (f(x_0) -\phi_0^*).
	\end{align*}
\end{corollary}

Corollary~\ref{coro2} shows that ASUESA converges linearly at the optimal rate. The difference in convergence rates between Algorithms~\ref{alg:SUESA}~and~\ref{alg:ASUESA} is essentially explained by the quadratic term $\tnorm{v_k -y_k^{++}}$, which is entirely ignored in the proof of Theorem \ref{thm1}. Thus, in the proof of Theorem~\ref{thm2}, one is able to incorporate another term containing $\tnorm{\nabla f(y_k)}$, which leads to a larger allowable value of $\alpha_k$, and ultimately, a tighter bound for Algorithm~\ref{alg:ASUESA}.

\section{Algorithms and Convergence Guarantees for Composite Functions}\label{sec:composite}

The purpose of this section is to extend the results presented in Section~\ref{sec:smooth} from the smooth to the composite setting, i.e., here we suppose that $h(x) \neq 0$. In particular, we present two algorithms whose iterates fit the Underestimate Sequence framework described in Section~\ref{sec:UES}, and use the lower bounds developed in Section~\ref{s:lbcomposite}. Both algorithms appear to fit the composite setting very naturally; the first algorithm is a proximal gradient descent type method, while the second algorithm is an accelerated proximal gradient variant. The algorithms also incorporate stopping conditions that provide a certificate of optimality. We establish convergence guarantees for both algorithms via the UES framework and for all results we suppose that Assumption~\ref{A_SCL} holds.

\subsection{A Composite Underestimate Sequence Algorithm}

We now present an algorithm to solve \ref{eq:Problem}, which is based on the UES framework. A brief description will follow.

\begin{algorithm}
	\caption{Composite UES Algorithm (CUESA)}
	\label{alg:CUESA}
	\begin{algorithmic}[1]
		\STATE Initialization: Set $k=0$, $\epsilon >0$, initial point $x_0\in\R^n$ and compute $\mu$, $L$.
		\STATE Set $\varphi_0(x)$ as in \eqref{eq:compositecanonical}, with $v_0$ and $\varphi_0^*$ as in \eqref{eq:minvalminmize}. Let $\alpha_k= \frac{\mu}{L}$.
		\WHILE {$F(x_k) - \varphi_k^* > \epsilon$}
		\STATE Set $x_{k+1} = x_k - \tfrac{1}{L} G_L (x_k),$
        \STATE Set $y_k = x_k$, $y_k^+ = x_k^+$, and $y_k^{++} = x_k^{++}$
		\STATE Update $v_{k+1}$ and $\varphi_{k+1}^*$ as in \eqref{eq:vkcomp} and \eqref{eq:phikcomp} respectively.
		\STATE $k=k+1$.
		\ENDWHILE
	\end{algorithmic}
\end{algorithm}

The Composite (functions) UnderEstimate Sequence Algorithm (CUESA) presented in Algorithm~\ref{alg:CUESA} solves problem~\eqref{eq:Problem} when $h \neq 0$. The algorithm is described now. First, an initial point $x_0\in \R^n$ is chosen, as well as some stopping tolerance $\epsilon>0$. Secondly, the point $v_0 = x_0^{++}$ is constructed, as well as the lower bound $\varphi_0(x)$ with minimum value $\varphi_0^*$. The algorithm uses a fixed step size of $\alpha_k = \tfrac{\mu}{L}$ at every iteration. Next, the main loop commences and an iteration proceeds as follows. One sets $y_k = x_k$ (so $y_k$ is not explicitly used in CUESA); $x_k$ is updated by taking a \emph{proximal} gradient descent step with the step size $\tfrac{1}{L}$, resulting in the new point $x_{k+1}$; the point $v_{k+1} = x_{k+1}^{++}$ is constructed and the lower bound $\varphi_{k+1}(x)$ is updated.

The algorithm utilizes two points at every iteration, namely $x_k$ and $v_k$, as well as the values $\varphi_{k}(x)$ and $\varphi_{k}^*$. The point $v_k$ and the value $\varphi_{k}^*$ are used for the lower bound, which is essential for the stopping criterion.

Considering only Step~5, one sees that at every iteration of CUESA the point $x_k$ is updated via a proximal gradient descent step. That is, a step of size $1/L$ in the direction of the negative proximal gradient is taken from the current point $x_k$, resulting in the new point $x_{k+1}$. What makes CUESA distinct from a standard proximal gradient method is the inclusion of several other ingredients related to the lower bound $\varphi_k(x)$, which guarantee an $\epsilon$-optimal solution.

Now we present a convergence guarantee for CUESA. Theorem~\ref{thm3} shows that the iterates generated by Algorithm~\ref{alg:SUESA} form an underestimate sequence (i.e., they satisfy Definition~\ref{def:UES}) and therefore, Algorithm~\ref{alg:SUESA} is guaranteed to converge (linearly) to the solution of problem \eqref{eq:Problem}.

\begin{theorem}\label{thm3}
  Let Assumption~\ref{A_SCL} hold. The sequences $\{x_k\}_{k=0}^\infty$, $\{\varphi_k(x)\}_{k=0}^\infty$ and $\{\alpha_k\}_{k=0}^\infty$ generated by CUESA (Algorithm~\ref{alg:CUESA}) form a UES.
\end{theorem}
\begin{proof}
From Step~5 in CUESA, one sees that $y_k = x_k$ for all $k$, so it also follows that $y_k^+ = x_{k+1}$ for all $k$. Now, using $y = x= x_k$ in the lower bound \eqref{eq:lbcomposite} gives
\begin{eqnarray}\label{eq:compFreduce}
  F(x_{k+1}) \leq F(x_k) - \tfrac1{2L} \|G_L(x_k)\|_2^2.
\end{eqnarray}
Thus,
\begin{align*}
&F(x_{k+1})-\varphi_{k+1}^*\\
&=(1-\alpha) F(x_{k+1}) +\alpha F(x_{k+1}) -\varphi_{k+1}^*\\
&\overset{\eqref{eq:phistarequiv}}{=} (1-\alpha) F(x_{k+1}) +\alpha F(x_{k+1}) -
    (1-\alpha_k)\left(\varphi_k^* + \alpha_k\tfrac{\mu}2\|v_{k}-y_k^{++}\|_2^2\right)\\
    &\qquad-\alpha_k\left(F(y_k^+) +  \left(\tfrac1{2L} -\tfrac1{2\mu} \right)\|G_L(y_k)\|^2\right)\\
&= (1-\alpha) F(x_{k+1})  -
    (1-\alpha_k)\left(\varphi_k^* + \alpha_k\tfrac{\mu}2\|v_{k}-y_k^{++}\|_2^2\right)\\
    &\qquad-\alpha_k\left(\tfrac1{2L} -\tfrac1{2\mu} \right)\|G_L(y_k)\|^2\\
    &\leq (1-\alpha) \left(F(x_{k+1}) - \varphi_k^*\right)-\alpha_k\left(\tfrac1{2L} -\tfrac1{2\mu} \right)\|G_L(y_k)\|^2
\end{align*}
\begin{align*}
&\overset{\eqref{eq:compFreduce}}{\leq}
(1-\alpha)\left( F(x_{k}) - \tfrac1{2L} \|G_L(x_k)\|^2 - \phi_k^*\right)- \alpha\left(\tfrac1{2L}- \tfrac{1}{2\mu}\right)\|G_L(x_k)\|^2\\
&\leq
(1-\alpha)
\left( F(x_{k})- \phi_k^*\right)+ \left(
  \tfrac{\alpha}{2\mu} -\tfrac{\alpha}{2L}
 -(1-\alpha) \tfrac1{2L}
  \right) \|  G_L(x_k)\|^2\\
&\leq (1-\alpha_k)(F(x_k)-\varphi_k^*),
\end{align*}
where the last step follows because $\alpha_k = \tfrac{\mu}{L}$ so
$\tfrac{\alpha_k}{2\mu} -\tfrac1{2L}= \tfrac{1}{2L}-\tfrac1{2L} = 0.$\qed
\end{proof}
\begin{corollary}\label{coro3}
Let Assumption~\ref{A_SCL} hold. Then, the sequence of iterates $\{x_k\}_{k\geq0}$ generated by Algorithm \ref{alg:CUESA} exhibits a linear rate of convergence
	\begin{align*}
		 F(x_k) -\varphi_k^* \leq \left(1- \tfrac{\mu}{L}\right)^k (F(x_0) -\varphi_0^*).
	\end{align*}
\end{corollary}

\subsection{An Accelerated Composite UES Algorithm}

An accelerated algorithm for convex composite problems is now presented.
\begin{algorithm}
	\caption{Accelerated Composite UES Algorithm (ACUESA)}
	\label{alg:ACUESA}
	\begin{algorithmic}[1]
		\STATE Initialization: Set $k=0$, $\epsilon >0$, initial point $x_0\in\R^n$ and compute $\mu$, $L$.
		\STATE Set $\varphi_0(x)$ as in \eqref{eq:compositecanonical}, with $v_0$ and $\varphi_0^*$ as in \eqref{eq:minvalminmize}.
                Let $\alpha_k= \sqrt{\frac{\mu}{L}}$, $\beta_k = \frac{1}{1+\alpha_k}$.
		\WHILE {$F(x_k) - \varphi_k^* > \epsilon$}
		\STATE Set $y_k = \beta_k x_k + (1- \beta_k) v_k$.
		\STATE Set $x_{k+1} = y_k - \frac{1}{L}  G_L(y_k).$
		\STATE Update $v_{k+1}$ and $\varphi_{k+1}^*$ as in \eqref{eq:vkcomp} and \eqref{eq:phikcomp} respectively.
		\STATE $k=k+1$.
		\ENDWHILE
	\end{algorithmic}
\end{algorithm}

The Acclerated Composite UnderEstimate Sequence Algorithm (ACUESA) presented in Algorithm~\ref{alg:ACUESA} solves \eqref{eq:Problem} when $h \neq 0$. The algorithm proceeds as follows. ACUESA is initialized with a starting point $x_0\in \R^n$, a stopping tolerance $\epsilon>0$, the point $v_0 = x_0^{++}$ as well as the construction of the lower bound $\varphi_0(x)$ and minimum value $\varphi_0^*$. For ACUESA one sets $\alpha_k = \sqrt{\mu/L}$ and the parameter $\beta_k = \frac{1}{1+\alpha_k}$ is also used. Notice that parameters $\alpha_k$ and $\beta_k$ are fixed for all iteration. The main loop proceeds as follows. At every iteration one sets $y_k$ to be a convex combination of the points $x_k$ and $v_k$; a gradient descent step is taken \emph{from} $y_k$, resulting in the new point $x_{k+1}$; the point $v_{k+1}$ is constructed and the lower bound $\varphi_{k+1}(x)$ is updated.

Algorithm~\ref{alg:ACUESA} can be viewed as the accelerated version of Algorithm~\ref{alg:CUESA}. In contrast to Algorithm~\ref{alg:CUESA}, ACUESA constructs \emph{three} points at every iteration, namely $x_k$, $v_k$ and $y_k$, where the intermediate vector $y_k$ is a convex combination of the points $x_k$ and $v_k$ (i.e., for ACUESA $x_k \neq y_k$.) Notice also that $x_{k+1}$ is the result of a gradient descent step taken from the point $y_k$.  The variable $\varphi_{k}^*$ are also maintained and is used in the stopping condition.

The following result provides a convergence guarantee for ACUESA. Theorem~\ref{thm4} shows that the iterates generated by Algorithm~\ref{alg:ACUESA} fit the UES framework (i.e., they satisfy Definition~\ref{def:UES}), so Algorithm~\ref{alg:ACUESA} is guaranteed to converge (linearly at the optimal rate) to the solution of problem \eqref{eq:Problem} (see Corollary~\ref{coro4}).

\begin{theorem}\label{thm4}
	Let Assumption~\ref{A_SCL} hold. The sequences $\{x_k\}_{k=0}^\infty$, $\{\varphi_k(x)\}_{k=0}^\infty$ and $\{\alpha_k\}_{k=0}^\infty$ generated by ACUESA (Algorithm~\ref{alg:ACUESA}) form a UES.
\end{theorem}
\begin{proof}
From Step~7 in ACUESA,
\begin{equation}\label{eq:xkp1yp}
  x_{k+1} = y_k - \tfrac1LG_L(y_k) \overset{\eqref{eq:shortproxstep}}{\equiv} y_k^+.
\end{equation}
Hence,
  \begin{eqnarray}
    F(x_{k+1}) - \varphi_{k+1}^* &=& (1-\alpha_k)F(x_{k+1}) + \alpha F(x_{k+1})  - \varphi_{k+1}^*\notag\\
    &\overset{\eqref{eq:phistarequiv}}{=}&(1-\alpha_k)F(x_{k+1}) + \alpha F(x_{k+1}) - (1-\alpha_k)\varphi_k^* -  \alpha_kF(y_k^+)\notag\\
    && -\alpha(1-\alpha)\tfrac{\mu}2\|v_{k} - y_k^{++}\|_2^2  -  \alpha_k\left(\tfrac1{2L} - \tfrac1{2\mu} \right)\|G_L(y_k)\|^2)\notag\\
     &\overset{\eqref{eq:xkp1yp}}{=}&(1-\alpha_k)F(x_{k+1})- (1-\alpha_k)\varphi_k^*\notag\\
     && -\alpha_k(1-\alpha_k)\tfrac{\mu}2\|v_{k} - y_k^{++}\|_2^2+\left(\tfrac{\alpha_k}{2\mu} -\tfrac{\alpha_k}{2L}\right)\|G_L(y_k)\|^2\notag\\
     &=&(1-\alpha_k)\left(F(x_{k+1})-\varphi_k^* -\alpha_k\tfrac{\mu}2\|v_{k} - y_k^{++}\|_2^2\right)\notag\\
     &&+\left(\tfrac{\alpha_k}{2\mu} -\tfrac{\alpha_k}{2L}\right)\|G_L(y_k)\|^2 \label{eq:Fkp11}
  \end{eqnarray}
  By considering the expression for $y_k$ in Step~6 in Algorithm~\ref{alg:ACUESA} and noticing that $\beta_k = \tfrac{1}{1+\alpha_k}$ for all $k$, \eqref{eq:vk} and \eqref{eq:alphabeta1} hold. Thus, combining \eqref{eq:vk} and \eqref{eq:longproxstep} gives
  \begin{align}
    &\tfrac{\alpha_k\mu}2\|v_k - y_k^{++}\|^2\notag \\
    &= \tfrac{\alpha_k\mu}2\|\tfrac{\beta_k}{1-\beta_k}(y_k - x_k) + \tfrac1{\mu}G_L(y_k)\|^2\notag\\
    &= \tfrac{\alpha_k\mu}2\tfrac{\beta_k^2}{(1-\beta_k)^2}\|x_k-y_k\|^2 + \tfrac{\alpha_k}{2\mu}\|G_L(y_k)\|^2
    - \tfrac{\alpha_k\beta_k}{1-\beta_k}\langle G_L(y_k),x_k-y_k\rangle\notag\\
    &\overset{\eqref{eq:alphabeta1}}{=} \tfrac{\mu}2\tfrac{\beta_k}{1-\beta_k}\|x_k-y_k\|^2 + \tfrac{\alpha_k}{2\mu}\|G_L(y_k)\|^2 - \langle G_L(y_k),x_k-y_k\rangle.\label{eq:vkyknorm}
  \end{align}
  Substituting \eqref{eq:vkyknorm} into \eqref{eq:Fkp11} results in
  \begin{eqnarray*}
    && \hspace{-5mm} F(x_{k+1}) - \varphi_{k+1}^*\\
    &=& (1-\alpha_k)\left(F(x_{k+1})-\varphi_k^* -\tfrac{\mu}2\tfrac{\beta_k}{1-\beta_k}\|x_k-y_k\|^2  + \langle G_L(y_k),x_k-y_k\rangle \right)\\
     && -(1-\alpha_k)\tfrac{\alpha_k}{2\mu}\|G_L(y_k)\|^2+\left(\tfrac{\alpha_k}{2\mu} -\tfrac{\alpha_k}{2L}\right)\|G_L(y_k)\|^2\\
     &=&(1-\alpha_k)\left(F(x_{k+1})-\varphi_k^* -\tfrac{\mu}2\tfrac{\beta_k}{1-\beta_k}\|x_k-y_k\|^2  + \langle G_L(y_k),x_k-y_k\rangle \right)\\
     && + (1-\alpha_k)\tfrac{1}{2L}\|G_L(y_k)\|^2.
  \end{eqnarray*}
 Using a rearrangement of the lower bound \eqref{eq:lbcomposite}, and \eqref{eq:xkp1yp}, gives
  \begin{eqnarray*}
    && \hspace{-5mm}F(x_{k+1}) - \varphi_{k+1}^*\\
     &\leq&(1-\alpha_k)\left(F(x_k)- \langle G_L(y_k),x_k-y_k\rangle - \tfrac{\mu}2\|x_k-y_k\|^2 - \tfrac1{2L}\|G_L(y_k)\|^2 -\varphi_k^*\right) \\
     && +(1-\alpha_k)\left(-\tfrac{\mu}2\tfrac{\beta_k}{1-\beta_k}\|x_k-y_k\|^2  + \langle G_L(y_k),x_k-y_k\rangle  + \tfrac{1}{2L}\|G_L(y_k)\|^2\right)\\
     &=&(1-\alpha_k)(F(x_k) -\varphi_k^*) +(1-\alpha_k)\left( - \tfrac{\mu}2\|x_k-y_k\|^2 -\tfrac{\mu}2\tfrac{\beta_k}{1-\beta_k}\|x_k-y_k\|^2\right)\\
     &\leq&(1-\alpha_k)(F(x_k) -\varphi_k^*)  - \tfrac{\mu}2(1-\alpha_k)\tfrac{1}{1-\beta_k}\|x_k-y_k\|^2\\
     &\leq&(1-\alpha_k)(F(x_k) -\varphi_k^*).
  \end{eqnarray*}
  Thus, the iterates generated by ACUESA form a UES.\qed
\end{proof}

\begin{corollary}\label{coro4}
Let Assumption~\ref{A_SCL} hold. Then, the sequence of iterates $\{x_k\}_{k\geq0}$ generated by Algorithm \ref{alg:ACUESA} exhibits the optimal linear rate of convergence
	\begin{align*}
		 F(x_k) -\varphi_k^* \leq \left(1- \sqrt{\tfrac{\mu}{L}}\right)^k (F(x_0) -\varphi_0^*).
	\end{align*}
\end{corollary}

\section{An algorithm with adaptive $L$}\label{sec:adaptiveL}

In the algorithms presented so far, the Lipschitz constant $L$ is explicitly used in each algorithm. However, by studying the convergence proofs for Algorithms~\ref{alg:SUESA}--\ref{alg:ACUESA} one notices that the role of the Lipschitz constant $L$ is to enforce a reduction in the function value from one iteration to the next (see the first step in the proofs of Theorems~\ref{thm1}--\ref{thm4}). Thus, it is natural to ask the question, `Can an \emph{adaptive} Lipschitz constant, $L_k$ say, be used in place of the true Lipschitz constant $L$?'. In this section we show that, using a strategy similar to that proposed by Nesterov in \cite{Nesterov07,Nesterov13}, it is possible to employ an adaptive Lipschitz constant while preserving convergence guarantees.

\subsection{The Inequality}

When the Lipschitz constant $L$ is unknown, or is expensive to compute, it may be preferable to employ an `adaptive' Lipschitz constant, $L_k$ say, i.e., determine a value $L_k$ that approximates $L$ locally. This approach has been previously studied by Nesterov in \cite{Nesterov07,Nesterov13}, and it has the added advantage that $L_k$ may be smaller than the true Lipschitz constant $L$, which can lead to large step sizes. Throughout the algorithm certain inequalities must hold to ensure that convergence guarantees are maintained. The relevant inequalities are as follows.

\paragraph{Smooth case.} For smooth functions, \eqref{eq:zzzz3} and \eqref{eq:ibuij} must hold for SUESA and ASUESA, respectively. This means that at every iteration, if $L_k$ satisfies
\begin{equation}\label{eq:smoAdapL}
f\left(y_k - \tfrac1{L_k} \nabla f(y_k) \right)
\leq f(y_k ) - \tfrac1{2 L_k} \|\nabla f(y_k)\|^2,
\end{equation}
then convergence guarantees for SUESA and ASUESA are maintained.
If $L_k$ is chosen to satisfy \eqref{eq:smoAdapL}
then we will show that we have the improvement $\alpha_k = \mu/L_k$ (or $\alpha_k = \sqrt{\mu/L_k}$ for the accelerated case) at every iteration.

\paragraph{Non-smooth case.}
For composite functions, \eqref{eq:compFreduce} and \eqref{eq:Fkp11} must hold for CUESA and ACUESA, respectively. So, if $L_k$ satisfies
\begin{eqnarray}
\label{eq:comAdapL}
  F\left(y_k - \tfrac1{L_k} G_{L_k}(y_k)\right) \leq F(y_k) - \tfrac1{2L_k} \|G_{L_k}(y_k)\|_2^2,
\end{eqnarray}
then the algorithms are still guaranteed to converge. This also implies the improvement $\alpha_k = \mu/L_k$ (or $\alpha_k = \sqrt{\mu/L_k}$ for the accelerated case) at every iteration.

With these two inequalities in mind, the adaptive Lipschitz process can be described briefly as follows. When initializing Algorithms~\ref{alg:SUESA}--\ref{alg:ACUESA}, choose an initial estimate $L_0>0$, and increase and decrease factors $u>1$ and $d>1$ respectively. To find the appropriate $L_k$, at iteration $k$, one starts with the value $L_{k-1}$ (i.e., the adaptive Lipschitz constant from the previous iteration) and increases it via multiplication with $u$, or decreases it via division by $d$, until \eqref{eq:smoAdapL} (or \eqref{eq:comAdapL}) is satisfied. At iteration $k$, once an $L_k$ is found such that \eqref{eq:smoAdapL} (or \eqref{eq:comAdapL} in the composite case) holds, then the iteration proceeds with $L_k$ used in place of $L$.

Note that, using this process, it is possible that at some iteration, $L_k < L$, i.e., $L_k$ may be smaller than $L$. In this case, the stepsize $1/L_k$ is used, which is larger than $1/L$.

The psuedocode is presented in Algorithm~\ref{alg:adapL}. Note that determining the adaptive Lipschitz constant occurs as an inner loop within one of Algorithms~\ref{alg:SUESA}--\ref{alg:ACUESA}. Thus, we use the iteration counter $s$ in Algorithm~\ref{alg:adapL} to distinguish it from the outer loop counter $k$.
\begin{algorithm}
	\caption{Finding $L_k$ in iteration $k$ of Algorithms~\ref{alg:ASUESA} and \ref{alg:ACUESA}.}
	\label{alg:adapL}
	\begin{algorithmic}[1]
		\STATE Input: $x_k, v_k$ $u>1, d>1$ and $L_{k-1}$.
		\STATE Initialize: If $k=0$ let $L_{s} = L_0$, or if $k\geq1$ then $L_{s} = L_{k-1}/d$.
		\FOR{$s = 0,1,2,\dots$}
		\STATE  $\alpha_{s} = \sqrt{\tfrac{\mu}{L_{s}}}$, $\beta_{s} = \tfrac{1}{1+\alpha_{s}}$.\label{ref:mmmmm0}
		\STATE Set $y_s = \beta_{s} x_k + (1- \beta_{s}) v_k$.
		\STATE Set $x_{s} = y_s - \frac{1}{L_s} G(y_s).$
\IF{\eqref{eq:smoAdapL} or \eqref{eq:comAdapL} holds}
    \STATE Break.
  \ELSE
    \STATE $L_{s+1}$ = $u \cdot L_s$.\label{ref:mmmmm1}
  \ENDIF
\ENDFOR
\STATE Output: $L_k = L_s$, $\alpha_k = \alpha_{s}$, $\beta_k = \beta_{s}$, $y_k = y_s$ and $x_{k+1} = x_s$.
	\end{algorithmic}
\end{algorithm}
Note that the strategy above holds for  Algorithms~\ref{alg:ASUESA} and \ref{alg:ACUESA}, but it is straightforward to adapt it to Algorithms~\ref{alg:SUESA} and \ref{alg:CUESA}  by modifying the variables $\alpha_s$ and $\beta_s$.

We now present several theoretical result related to this setup.
\begin{lemma}
Let $u,d >1$. If $L_0 \leq L$ then for all $k$, $L_k \leq L \cdot u$. If $L_0 \geq L$ then for the first $k=\frac{\log \frac{L}{L_0}}{\log d}$ iterations,
$L_k = L_0 \cdot d^k$.
\end{lemma}
\begin{proof}
Note that, if $L_k \geq L$ then $L_k$ is guaranteed to satisfy \eqref{eq:smoAdapL} or \eqref{eq:comAdapL}.

Now suppose that at the start of iteration $k$, we have a trial value $L_s < L$. If $L_s$ satisfies either \eqref{eq:smoAdapL} or \eqref{eq:comAdapL} then one sets $L_k=L_s$ and terminates the inner loop, noting that $L_k < L$. If $L_s$ does not satisfy \eqref{eq:smoAdapL} or \eqref{eq:comAdapL} then it is increase by multiplication with $u$. Suppose that $L_s$ is the largest possible trial value satisfying $L_s < L$ and $L_s\cdot u \geq L$, but with \eqref{eq:smoAdapL} or \eqref{eq:comAdapL} not holding. Then multiplying $L_s$ by $u$ once results in $L_s\cdot u \geq L$ and $L_s\cdot u \leq L\cdot u$, and this value is guaranteed to satisfy \eqref{eq:smoAdapL} or \eqref{eq:comAdapL} so we set $L_k = L_s$.

On the other hand, suppose that at iteration $k=0$, the initial value happens to satisfy $L_0 > L$. Then this $L_0$ will satisfy \eqref{eq:smoAdapL} or \eqref{eq:comAdapL} so it will be accepted with $L_k = L_0$ at iteration $k=0$. At iteration $k=1$ the first trial value is set to $L_s = L_{k-1}/d = L_0/d$. If $L_s \geq L$, then one accepts $L_k = L_1 = L_s$ and the iteration proceeds. At the start of any iteration, if the previous value $L_{k-1}\geq L$, then the first trial value is divided by $d>1$. Thus, it is guaranteed that $\frac{L_0}{d^K} \geq L, $
where
$$  K  = \left \lfloor{\frac{\log \frac{L_0}{L}}{\log d}}\right \rfloor.$$
In other words, $L_k\geq L$ must hold for $k=\{1,2,...,K\}$.

\end{proof}
\begin{proposition}
The maximum number of times that Lines~\ref{ref:mmmmm0}~to~\ref{ref:mmmmm1} are executed during the first $K$ iterations is bounded by
\begin{equation}\label{eq:case1adapL}
\left \lceil{\frac{\max\{\log (Ld/L_0),0\}}{\log u}}\right \rceil + (K-1)\left \lceil{\frac{\log d}{\log u}+1}\right \rceil.
\end{equation}
\end{proposition}

\begin{proof}
	In the first iteration, at most we need to execute the procedure from line \ref{ref:mmmmm0} to line \ref{ref:mmmmm1} of Algorithm \ref{alg:adapL} $\left \lceil{\frac{\log (Ld/L_0)}{\log u}}\right \rceil $ times, assuming $L_0/d <L$. In the case of $L_0/d\geq L$, this procedure is carried out once. In the $k$th iteration when $k\geq 2$, since we know that $L_{k-1}\geq L$, the above procedure should be run $\left \lceil{\frac{\log d}{\log u}+1}\right \rceil $ times if $L_{k-1}/d <L$ occurs. Thus, we obtain \eqref{eq:case1adapL}.
\end{proof}

\section{Numerical Experiments}\label{sec:numericalexperiments}

In this section, we present numerical results to compare our proposed algorithms with several other methods that have an optimal convergence rate. The algorithms are as follows, and are summarized in Table~\ref{table-DiscAlgs}.

\emph{OQA.} The Optimal Quadratic Averaging algorithm (OQA) \cite{Drusvyatskiy16}, which builds upon the work in \cite{Bubeck15}, maintains a quadratic lower bound on the objective function value at every iteration. The quadratic lower bound is called `optimal' because it is the `best' lower bound that can be obtained as a convex combination of the previous 2 quadratic lower bounds. In OQA, $x_{k+1}$ is set to be the minimizer of $f(x)$ on the line joining the points $x_k^+$ and the minimizer of the current quadratic lower bound. In \cite{Drusvyatskiy16} the author suggest a variant of OQA, which we call OQA+ here, that computes $x_{k}^+$ via a line search that does not use the true Lipschitz constant $L$. We compare both OQA and OQA+ in our experiments.

\emph{NEST.} We use NEST to denote the algorithm described in Chapter 2 of \cite{Nesterov04}. Further, NEST+ is a variant of NEST in which the Lipschitz constant $L$ is adaptively update via the strategy in \cite{Nesterov07,Nesterov13}.

\emph{GD.} We also implement a Gradient Descent (GD) method which uses a fixed stepsize of $\tfrac{1}{L}$. Note that this is similar to Algorithm~\ref{alg:SUESA}, although GD does not maintain any kind of lower bound. As the only non-optimal algorithm, Gradient Descent provides a benchmark that will enable us to observe any performance advantages of the optimal methods.

\begin{table}[H]\tiny
	\centering
	\begin{tabular}{l|l} \toprule
		{Algorithm} & {Description}  \\ \midrule
		{OQA} &  {Optimal Quadratic Averaging Algorithm} \\
		{OQA+}  & {Optimal Quadratic Averaging Algorithm with $x_k^+= \text{line-search} (x_{k},x_k-\nabla f(x_k))$}   \\
		{ASUSEA}  & {Accelerated Smooth Underestimate Sequence Algorithm}\\
		{ASUSEA+}  & {Accelerated Smooth Underestimate Sequence Algorithm with adaptive Lipschitz constant} \\
		{SUESA}  & {Smooth Underestimate Sequence Algorithm}   \\
		{NEST}   & {Algorithm described in Chapter 2 of \cite{Nesterov04}} \\
		{NEST+}  & {Algorithm described in Chapter 2 of \cite{Nesterov04} with adaptive Lipschitz constant}  \\\midrule
		{CUSEA}  &  {Composite Underestimate Sequence Algorithm}     \\
		{CUESA+} &  {Composite Underestimate Sequence Algorithm with adaptive Lipschitz constant} \\
		{ACUESA} &  {Accelerated Composite Underestimate Sequence Algorithm}  \\
		{ACUESA+} &  {Accelerated Composite Underestimate Sequence Algorithm with adaptive Lipschitz constant} \\

		{CNEST}  &  {Algorithm (4.9) described in \cite{Nesterov07} with fixed Lipschitz constant}     \\
		{CNEST+}   & {Algorithm (4.9) described in \cite{Nesterov07} with adaptive Lipschitz constant} \\\bottomrule
	\end{tabular}
	\normalsize
	\caption{Description of implemented algorithms}
\label{table-DiscAlgs}
\end{table}

\subsection{Empirical Risk Minimization}

We consider two Empirical Risk Minimization (ERM) problems, which are popular in the machine learning literature. In particular, we study ERM with a squared hinge loss
\begin{equation}\label{eq:squaHingeLoss}
f(x) =  \frac{1}{m} \sum_{i=1}^{m} \big(\max \{0,y_i - a_i^Tx\}\big)^2 +  \tfrac{\lambda}{2}\tnorm{x},
\end{equation}
and ERM with a logistic loss (also called logistic regression)
\begin{equation}\label{eq:logisticRe}
f(x) = \frac{1}{m} \sum_{i=1}^{m} \log\big( 1+ e^{-y_ia_i^Tx}\big)^2 +  \tfrac{\lambda}{2}\tnorm{x}.
\end{equation}
In each case $y_i \in\{-1,+1\}$ is the label and $a_i \in \R^n$ represents the training data for $i = \{1,2,...,m\}$. All the datasets in our experiments come from LIBSVM database \cite{Chang11}. Also note that for all experiments we have $\mu = \lambda$.

\subsubsection*{Comparison on Decreasing Objective Values}

In the first experiment we compare the OQA, ASUESA and NEST algorithms (both the standard and adaptive Lipschitz variants) and investigate how the objective function values behave on several test problems. The test problems considered in this experiment are the \texttt{ala} dataset with a squared hinge loss and a value $\lambda = 10^{-4}$, the \texttt{rcv1} dataset with a logistic loss and a value $\lambda = 10^{-4}$, and the \texttt{covtype} dataset with a squared hinge loss and a value $\lambda = 10^{-5}$.

\begin{figure*}[h!]
	\centering
	\includegraphics[scale=.15]{1a1a_2_1e-4_2.eps}
    \includegraphics[scale=.15]{1rcv1_1_1e-4_2.eps}
    \includegraphics[scale=.15]{1covtype_2_1e-5_2.eps}
	\includegraphics[scale=.15]{1a1a_2_1e-4_3.eps}
	\includegraphics[scale=.15]{1rcv1_1_1e-4_3.eps}
	\includegraphics[scale=.15]{1covtype_2_1e-5_3.eps}
	\caption{Evolution of the gap $f(x_k)-\phi_k^*$ for each algorithm compared with the number of function evaluations and cputime.}
	\label{fig:exp11}
\end{figure*}

In Figure~\ref{fig:exp11} we plot the gap $f(x_k)-\phi_k^*$ vs the number of function evaluations and the gap $f(x_k)-\phi_k^*$ vs the cpu time. The figure shows the advantages of using an adaptive Lipschitz constant with the adaptive methods performing better than their original versions in most cases. Figure~\ref{fig:exp11} also shows that ASUESA+ performs very well, being the best algorithm on the first dataset, and the second best algorithm on the other two datasets.

\subsubsection*{Theory and Practice for OQA and ASUESA}
In this numerical experiment we study ASUESA and OQA and investigate how their practical performance compares with that predicted by theory. For the OQA algorithm a line search is needed to determine a necessary algorithmic variable, and to ensure that theory for OQA holds, the line search should be exact. In this experiment we will use bisection to compute this variable, but we will restrict the number of bisection steps allowed to $b=2,5,20$. Figure~\ref{fig:bisection} plots the ratio $(f(x_k) - \phi^*_{k})/(f(x_{k-1}) - \phi^*_{k-1})$ for ASUESA and for three instances of OQA, where each instance uses a different number of bisection steps $b=2,5,20$. We also plot $1-\sqrt{\tfrac{\mu}{L}}$ (black dots), which is the amount of decrease in the gap $f(x_k)-\phi_k^*$ at each iteration predicted by the theory. (In theory, we should have $(f(x_k) - \phi^*_{k})/(f(x_{k-1}) - \phi^*_{k-1}) = 1-\sqrt{\tfrac{\mu}{L}}$ for all $k\geq 0$.)

\begin{figure*}[h!]
	\centering
	\includegraphics[scale=.15]{11gap_a1a_1_1e-4_1.eps}

	\includegraphics[scale=.15]{11gap_covtype_2_1e-5_1.eps}
	\caption{Comparison of  $\tfrac{f(x_k) - \phi^*_{k}}{f(x_{k-1}) - \phi^*_{k-1}}$ for ASUESA and for OQA with different numbers of bisection steps ($b=2,5,20$). The black dots are $1-\sqrt{\tfrac{\mu}{L}}$.}
	\label{fig:bisection}
\end{figure*}

From the plots in Figure~\ref{fig:bisection} we see that ASUESA performs very well, and as predicted by the theory, with the ratio $(f(x_k) - \phi^*_{k})/(f(x_{k-1}) - \phi^*_{k-1})$ always strictly below the theoretical bound. On the other hand, the quality of line search affects OQA significantly. The fewer the number of line search (bisection) iterations, the more likely it is for OQA to violate the theoretical results. Note that this is not necessarily surprising because the theory for OQA requires the exact minimizer along a line segment to be found, so 2 or 5 iterations of bisection may be simply too few to find it. Notice that when $b=2$, the green line shows that OQA behaves erratically, with the ratio $(f(x_k) - \phi^*_{k})/(f(x_{k-1}) - \phi^*_{k-1})$ being greater than 1 on many iterations, indicating that the gap is growing on those iterations. When we use OQA with $b=5$ steps of bisection at each iteration (light blue line), the algorithm performs better, and often, but not always, the ratio is less than 1. Finally, the dark blue line shows the behaviour of OQA when $b=20$ steps of bisection at each iteration. The dark blue line is always below the theoretical bound of $1-\sqrt{\tfrac{\mu}{L}}$, indicating good algorithmic performance (often better than predicted by theory). However, the line search needed by OQA comes at an additional computational cost, which can still mean that the overall runtime is longer for OQA than for ASUESA, as we now show.

Here a similar experiment is performed to compare the theoretical and practical performance of SUESA and ASUESA. We have already seen that the theoretical results for ASUESA give a proportional reduction of $1-\sqrt{\tfrac{\mu}{L}}$ in the gap at every iteration. However, for SUESA, the proportional reduction in the gap is $1-\tfrac{\mu}{L}$. We investigate how these theoretical bounds compare with the practical performance of each of these algorithms. We use the \texttt{ala}, \texttt{rcv1} and \texttt{covtype} datasets for this experiment, and for each of the three datasets we form both a logistic loss, and a squared hinge loss to create 6 problem instances. The results are shown in Figure~\ref{fig:thvsprac}.
\begin{figure*}[h!]
	\centering
	\includegraphics[scale=.15]{1gap_a1a_1_1e-4.eps}
	\includegraphics[scale=.15]{1gap_rcv1_1_1e-4.eps}
    \includegraphics[scale=.15]{1gap_covtype_1_1e-5.eps}
    \includegraphics[scale=.15]{1gap_a1a_2_1e-4.eps}
	\includegraphics[scale=.15]{1gap_rcv1_2_1e-4.eps}
	\includegraphics[scale=.15]{1gap_covtype_2_1e-5.eps}
	\caption{Comparison of  $\tfrac{f(x_k) - \phi^*_{k}}{f(x_{k-1}) - \phi^*_{k-1}}$ for SUESA and ASUESA and $1 - \tfrac{\mu}{L}$(green line) and $1-\sqrt{\tfrac{\mu}{L}}$ (black line). }
	\label{fig:thvsprac}
\end{figure*}

Figure~\ref{fig:thvsprac} presents the ratio $\tfrac{f(x_k) - \phi^*_{k}}{f(x_{k-1}) - \phi^*_{k-1}}$ for SUESA and ASUESA. Also displayed is the theoretical (unaccelerated) rate $1 - \tfrac{\mu}{L}$ (the green line) and the theoretical (accelerated) rate $1-\sqrt{\tfrac{\mu}{L}}$ (the black line). One sees that the practical performance of SUESA is very similar to that predicted by the theory because the blue line matches the green line closely. Another observation is that for the accelerated algorithm (ASUESA), in practice, the reduction in the gap $f(x_k) - \phi^*_{k}$ is often more optimistic than the theoretical rate.

\subsection{Experiments on composite functions}

In this section we perform several numerical experiments on problems with a composite objective. Specifically, we consider the elastic net problem, which is problem \eqref{eq:Problem} with
\begin{equation}\label{eq:squaHingeLoss}
F(x) =  \frac{1}{n} \sum_{i=1}^{n} (a_i^Tx -y_i)^2 +  \tfrac{\lambda_1}{2}\|x\|_2^2 + \tfrac{\lambda_2}{2}\|x\|_1.
\end{equation}
Notice that the first two terms in \eqref{eq:squaHingeLoss} are smooth, while the $\ell_1$-norm term makes \eqref{eq:squaHingeLoss} nonsmooth overall.
We compare our Algorithm \ref{alg:CUESA} and \ref{alg:ACUESA} (CUESA and ACUESA) with the one proposed in \cite{Nesterov07} (NEST). As stated previously, each of these algorithms can be implemented with either a fixed $L$ or an adaptive $L$, and we will compare each algorithm under both of these two options.

For these experiments we again use the 3 datasets \texttt{ala}, \texttt{rcv1} and \texttt{covtype}. For the \texttt{ala} data the regularization parameters were set to $\lambda_1= \lambda_2 =  10^{-4}$, for the \texttt{rcv1} data the regularization parameters were set to $\lambda_1 = 10^{-4}$ and $ \lambda_2 = 10^{-5}$, and for the \texttt{covtype} data the regularization parameters were set to $\lambda_1 = 10^{-4}$ and $ \lambda_2 = 10^{-6}$.

\begin{figure}[htbp]
	\centering
	\includegraphics[scale=.15]{a1a_3_1e-4_1e-4_2.eps}
    \includegraphics[scale=.15]{rcv1_3_1e-4_1e-5_2.eps}
    \includegraphics[scale=.15]{covtype_3_1e-4_1e-6_2.eps}
	\includegraphics[scale=.15]{a1a_3_1e-4_1e-4_3.eps}
	\includegraphics[scale=.15]{rcv1_3_1e-4_1e-5_3.eps}
	\includegraphics[scale=.15]{covtype_3_1e-4_1e-6_3.eps}
 	\caption{Comparison of how gaps between the objective values and the minimum amount of the lower bounds decreases for different algorithms. We observe the advantage of ACUESA+ in both number of function evaluations and running time.}
	\label{fig:comp}
\end{figure}

The results of this experiment are presented in Figure~\ref{fig:comp}, and they show the clear practical advantage of the ACUESA algorithm. The ACUESA algorithm outperforms the CNEST algorithm in all problem instances. Interestingly, on the \texttt{rcv1} dataset, the CUESA+ algorithm (CUESA with an adaptive Lipschitz constant) performs better than the accelerated ACUESA algorithm, although the ACUESA+ (accelerated plus adaptive Lipschitz constant) algorithm is still the best overall.

In the final numerical experiment presented here, we investigate the theoretical vs practical performance of CUESA and ACUESA. We set up three problems using each of the 3 datasets already described, and the results are presented in Figure~\ref{fig:comptheoryvsprac}.
\begin{figure}[H]
	\centering
	\includegraphics[scale=.15]{Non_gap_a1a_3_1e-4_1e-5.eps}
	\includegraphics[scale=.15]{Non_gap_covtype_3_1e-5_1e-6.eps}
	\includegraphics[scale=.15]{Non_gap_rcv1_3_1e-4_1e-6.eps}
	\caption{Comparison of  $\tfrac{f(x_k) - \phi^*_{k}}{f(x_{k-1}) - \phi^*_{k-1}}$ for CUESA and ACUESA and $1 - \tfrac{\mu}{L}$(green line) and $1-\sqrt{\tfrac{\mu}{L}}$ (black line). Here we have a similar observation as in Figure \ref{fig:comp}.}
	\label{fig:comptheoryvsprac}
\end{figure}
As before, the green line represents the theoretical (unaccelerated) rate $1 - \tfrac{\mu}{L}$ and the black line represents the theoretical (accelerated) rate $1 - \sqrt{\tfrac{\mu}{L}}$. Note that the practical performance of CUESA closely matches the theoretical rate. We also observe that the practical performance of ACUESA is always at least as good as the theoretical rate, and can often get better decrease in the gap per iteration than $1 - \sqrt{\tfrac{\mu}{L}}$.

All the numerical results presented in this section strongly support the practical success of the SUESA, ASUESA, CUESA and ACUESA algorithms.

\section{Conclusion}\label{sec:conclusion}

In this paper we studied efficient algorithms for solving the strongly convex composite problem \eqref{eq:Problem}. We proposed four new algorithms --- SUESA, ASUESA, CUESA and ACUESA --- to solve \eqref{eq:Problem} in both the smooth and composite cases. All of these algorithms maintain a global lower bound on the objective function value, which can be used as an algorithm stopping condition to provide a certificate of convergence. Moreover, we proposed a new underestimate sequence framework that incorporates three sequences, one of which is a global lower bound on the objective function, and this framework was used to establish convergence guarantees for the algorithms proposed here. Our algorithms have a linear rate of convergence, and the two accelerated variants (ASUESA and ACUESA) converge at the optimal linear rate. We also presented a strategy to adaptively select a local Lipschitz constant for the situation when one does not wish to, or cannot, compute the true Lipschitz constant. Numerical experiments show that our algorithms are computationally competitive when compared with other state-of-the-art methods including Nesterov's accelerated gradient methods and optimal quadratic averaging methods.

\bibliographystyle{plain}

redpajama_set_name: RedPajamaArXiv


text:
Q: Django custom decorator not wrapping I'm trying to create a custom decorator in Django and apply it to a class based view in the URLs file.  This is the decorator in decorators.py.  It uses functools.  I'm not able to get the wrapper function to execute the second log command.
def user_isnt_blocked(func):
    logger.info("user_not_blocked call " + func.__name__)
    @wraps(func)
    def wrapper(self, request, *args, **kwargs):
        logger.info("user_not_blocked wrapper")
        return func(self, request, *args, **kwargs)
    return wrapper

Right now, the decorator does nothing as I'm first trying to figure out how to get it to do the wrapping part - i.e. it is not logging the "user_not_blocked wrapper".  The initial call is logged - it does print out "user_not_blocked call".  The pertinent url pattern for the class-based view is coded as follows:
urlpatterns = [
    url(r'update-profile/$', user_isnt_blocked(views.UpdateProfile.as_view()), name="update-profile"),]

The class-based view is as follows:
class UpdateProfile(View):

    @method_decorator(csrf_exempt)
    def dispatch(self, request, *args, **kwargs):
        return super(UpdateProfile, self).dispatch(request, *args, **kwargs)

    def get(self, request):
        return super(UpdateProfile, self).dispatch(request, *args, **kwargs)

    def post(self, request):
        ...

Any help with what I'm doing wrong would be appreciated.

redpajama_set_name: RedPajamaStackExchange