The luxury industry is inherently international and brands need to be able to adapt to new markets while maintaining the prestige of your image.
Successfully conveying the brand voice, while localizing the content to local audiences requires the expertise of a specialist.
TextMaster has an in-depth experience working with luxury brands to create engaging and persuasive content for new markets.
We go beyond the services of a traditional translation agency by incorporating advanced technology in our translation offering. Coupled with the expertise of translators in different luxury sectors such as jewelry, leather goods, fine watchmaking, and "haute couture", this combination significantly streamlines the translation process.
TextMaster provides high quality translation by offering both industry-leading translation technology as well as a network of experienced translators.
Our network of experienced translators By combining their extensive experience with a thorough knowledge of your sector, our translators are able to deliver work that meets the highest standards.
TextMaster assembles a team of qualified translators to work on your projects in the long term. This allows them to gain a thorough understanding of your brand and your requirements, thereby ensuring that their translations match your needs perfectly.
TextMaster's project managers can set up your translation projects and can also take charge of them from start to finish, from uploading the content, assigning the translators, creating a detailed briefing, coordinating the project, etc. With this turnkey solution, you have a dedicated contact person that you can turn to throughout your project.
TextMaster ensure the quality of the final translations through a systematic review process. Translators undergo a rigorous recruitment process in order to join our network of professionals. Their work is continuously reviewed to make sure that it follows the project guidelines. It is then proofread and evaluated in order to make sure that it meets our quality standards.
Our translation platform makes it easy to order multilingual content online in more than 50 languages 24/7. You can choose from a variety of custom options to suit your project requirements.
With TextMaster, you can create your own glossaries that will automatically integrate your preferred vocabulary in the translations. By maintaining a coherent writing style, you can ensure that you have a consistent brand voice in your translations.
Memento, TextMaster's real-time translation memory, suggests previously-translated text so that your translators don't needlessly translate repetitions of content that has already been approved by you. Making small updates to your content is also easier because Memento is able to identify the changes that need to be translated.
Your team can work together on the TextMaster platform with our multi-account feature. You can manage billing, monitor project statuses and follow up on approvals through these different accounts.


text:
THE WRIGHT STUFF FESTIVAL - PERMANENCE
NAFA STUDIO THEATRE
151 Bencoolen St, Singapore
The Wright Stuff is a playwright incubation programme by Toy Factory, where new playwrights and their first plays are chosen from an open call to be developed under the tutelage of Toy's Artistic Directors. It is an invaluable platform for these new writers' plays to be mentored and eventually staged - with professional directors helming them.
Luggage in tow and drink in hand, Blanche is back home. Tired of running and longing for more, she steps back into her old life, colliding with everything she once left behind. Permanence weaves in and out Blanche's life at different points in time, following her frantic journey as she struggles to survive, fighting the dread of domesticity, the fraughtness of relationships, and the toxicity of a past that she can't seem to shake off. Through it all, she comes face to face with herself from the past, the present and the future, with a front-row seat in seeing how a life slowly breaks across time. (This Production contains mature content)

redpajama_set_name: RedPajamaCommonCrawl


Australia-China Young Scientists Exchange Programme (YSEP) is jointly funded by Australian and Chinese Governments for early to mid-career researchers to have a chance to visit each other every year under the Australia-China Science and Research Fund (ACSRF).
YSEP provides an opportunity for young Australian and Chinese researchers to visit China and Australia respectively on a two week exchange where they could develop their expertise in international collaboration by increasing their understanding of the cultures of the two countries, the science and research practices and systems of the two countries and by providing a catalyst for future collaboration. YSEP aims to support future long term science, technology and innovation collaboration between Australia and China, by bringing together future science leaders from both countries to foster long term relationships.
For the 2017 round, 15 Chinese researchers visited Australian institutions during July and later 14 Australian researchers visited Chinese institutions successfully.
Any scientists interested in this initiative may contact Industry.Beijing@dfat.gov.au.


MARGATE CITY - The governing bodies of Longport, Margate, Ventnor and Atlantic City have all joined Brigantine in passing resolutions that discourage residents from using single-use bags, specifically plastic bags, and encouraging the use of reusable bags.
Brigantine was the first municipality in New Jersey to pass the resolution back in 2011. Egg Harbor City also adopted the same resolution earlier this month.
"As coastal communities, it really is our responsibility to set the bar high and reduce, if not eliminate, plastic bags," said Atlantic City Council President Frank Gilliam. "We now know that plastic is the most common type of marine litter worldwide, and that plastics comprise up to 90 percent of floating marine debris. These facts cannot be ignored," said Gilliam.


The power of Metaphysics explored!
It's always great to get together with friends, but when you are with a group of like minded people miracles happen quickly. Our Metaphysics workshops are the perfect catalyst for sharing your passion and manifesting your dream career, ideal relationships and financial abundance. Once you join us you won't want to leave.
Traditionally we start off with a suggested topic and let the meeting be whatever it will be. You will help others and be helped at the same time. Let the Universe be your teacher through open reciprocal communication at this life altering workshop.


text:
package org.elasticsearch.snapshots;

import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.unit.ByteSizeUnit;
import org.elasticsearch.test.ESBlobStoreRepositoryIntegTestCase;

import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;

public class FsBlobStoreRepositoryIT extends ESBlobStoreRepositoryIntegTestCase {
    @Override
    protected void createTestRepository(String name) {
        assertAcked(client().admin().cluster().preparePutRepository(name)
            .setType("fs").setSettings(Settings.settingsBuilder()
                .put("location", randomRepoPath())
                .put("compress", randomBoolean())
                .put("chunk_size", randomIntBetween(100, 1000), ByteSizeUnit.BYTES)));

    }
}

redpajama_set_name: RedPajamaGithub


text:
Kerch Bay (, , ) is a bay of the Sea of Azov and Strait of Kerch in the eastern Crimea region.

It is located at the eastern Kerch Peninsula, near the municipality of Kerch.

Archaeology

In 2017, a head of an ancient Greek God was discovered during underwater construction operations at the Kerch Bay. It was crafted sometime between the 5th and 3rd centuries BC.

References

Bays of Crimea
Kerch Peninsula
Bays of the Sea of Azov
Kerch Strait

redpajama_set_name: RedPajamaWikipedia


text:
Liverpool set their sights on Benfica jewel Fernandez - for €120m?
The first season in Europe has started very well for Enzo Fernández.
29 September 2022 Thursday 03:58
The first season in Europe has started very well for Enzo Fernández. He has already become a regular at Benfica, so he was also allowed to make his debut for Argentina. That's already getting Liverpool FC on the scene.
It was only this summer that Benfica secured the services of Enzo Fernández. He came to Portugal from the Argentinian club River Plate for around twelve million euros.
The change has already paid off for the 21-year-old. At the club from the Portuguese capital, he is already part of the established permanent staff. He has played all previous games from the starting XI and is only rarely substituted in games. In addition, he has been able to prove his goal threat several times, although he plays in central midfield.
The fact that he was also able to celebrate his national team debut for Argentina rounds off what has been such a successful start to the season for him personally.
But it is precisely because he shows early on that he can keep up in top European football and even go ahead that other clubs are paying attention to him - even though he only joined Benfica in the summer.
Above all, Liverpool are said to have a growing interest in Fernández. This is reported by the Portuguese sports newspaper O Jogo. Accordingly, the England club is considering making an offer for the youngster in the future.
Irrespective of this, Fernández and Benfica should already have clarified that he will definitely end the current season in Lisbon. A premature winter change is thus excluded. A farewell could then become an issue in the summer at the earliest.
This point in time, after just one year, would also come very early. But there is a release clause in the contract for such matters: Liverpool would have to pay a whopping 120 million euros to sign the Benfica jewel. A scenario that seems very unlikely. Especially since the Reds are already considering other, costly transfers. Like Jude Bellingham's.
No matter how much interest there may be from Liverpool or other clubs, Benfica are in a very good position. Either there will be a transfer and with it a big profit of more than 100 million euros, or we can continue to count on Fernández's skills. The Reds will certainly give priority to Bellingham. If you get nothing from the young BVB star, your gaze could quickly turn to Enzo Fernández.
This article was originally posted on 90min.com as Liverpool set their sights on Benfica jewel Fernandez - for €120m? released.
1 Ratings: 5.66 million viewers watch the ARD crime... 2 "The TV total Wok World Cup" on ProSieben:... 3 Gas consumption: Saving gas: Federal Network Agency... 4 German unity: salaries in eastern Germany still lower... 5 Farewell: "Gangsta's Paradise" rapper... 6 MacKenzie Scott: Jeff Bezos' ex-wife is getting... 7 Oktoberfest: History of the Dirndl: Two people from... 8 218th day of war: Moscow wants to annex four Ukrainian... 9 Nord Stream Pipelines: Lithuania's Foreign Minister:... 10 Action by the outfitter: "color of mourning":...
Handball: before the end of the World Cup: national coach Gislason praises goalkeeper Wolff
Tennis: Australian Open: A dream final for the history books
Bundesliga: RB art shooter Szoboszlai: "Should continue to annoy me"
NHL: Two assists in Ottawa's win over Toronto
FC Bayern Munich: Matthäus protects Gnabry after his fashion trip
Ratings: 5.66 million viewers watch the ARD crime...


This wonderfully illustrated set of Alphabet flashcards have been designed for beginner readers as an aid to learn the letters of the alphabet. The illustrations and sounds provide a fun and interactive style of learning!
Use one card at a time, or create a beautiful wall mount for revision!
Includes set of 26 durable, plastic coated cards.


text:
Voices of the Sea
A couple's love for each other and their island home is put to the test in this unforgettable and intimately observed documentary about ordinary Cuban lives. Mariela's first husband drowned trying to flee Cuba on a raft. Her second husband is the wise and witty Orlando, a fisherman. Together with their four children they subsist, often lacking the basics, caught in a Cold War time warp of hardship. While Mariela weighs leaving the family and risking her life escaping, the aging Orlando devotes himself to fishing on the open sea, and passing on his craft to their teenage son—a craft integral to his own identity as a proud Cuban, his code of behavior as a man, and nature's order.
Preceded by the short film Ángela.
Director: Kim Hopkins
Country: United Kingdom, Cuba
Type: Feature Films
Born to an American father and a British mother, Kim Hopkins currently resides in York, England. Kim helped form the Documentary Department at EICTV, Cuba. Kim is a graduate of the National Film and Television School, UK. Kim spent a decade making television before co-founding Labor of Love Films Ltd., a UK based independent production company specializing in feature documentaries. Kim's broadcast credits include: BBC1,2,4, World, C4, A+E, WDR, Discovery Channel, National Geographic, NRK1.


text:
Last Day of Classes
Session II and 8-Week Classes
Final Examinations
Summer Final Grades Due
"A Little Bit of Everything"
Art Opening Reception - Corey Fuller
First Faculty Meeting
New Student Orientation/Welcome Week
Enrollment Now Open for Fall Music Lessons and Ensembles at OBU
OBU Hosts Annual Awards Assembly May 8, Recognizes Outstanding Students
OBU Broadcast Journalism Students Compete in OBEA Annual Media Competition
OBU Bison Debate Finishes Third in Nation, Takes Home Individual National Honors
Bison Bicycle Classic Slated for Sept. 26
Oklahoma Baptist University's Division of Music, in conjunction with the Pedalers Racing Team and the Greater Shawnee Area Convention and Visitors Bureau, announces the Bison Bicycle Classic to be held in Shawnee on Saturday, Sept. 26, 2009. The event will include 10-, 25- and 50-mile bicycle tours, as well as a 50-mile non-sanctioned road race.
All proceeds for the weekend will go to Habitat for Humanity of Shawnee Inc. to build the Division of Music HFH House. This project's goal is to raise funds to build a complete home for the local HFH affiliate. In addition to the Bison Bicycle Classic, the Division of Music also holds an annual concert fundraiser for Habitat for Humanity.
St. Gregory's University will also sponsor a 5K run/race to offer an event for those who would rather run than ride a bike. Last year's combined events brought in nearly $3,000 for Habitat for Humanity.
"This is the ninth year for the Bison Bicycle Classic, and we are always amazed at the number in the community that come out to volunteer and participate," said James Vernon, event coordinator and OBU associate professor of music. "We are pleased to be able to participate with our students in serving this worthy cause."
The Bicycle Classic will include 10-, 30- and 62-mile fully-supported bike tours for cyclists who wish not to race. In addition to the sporting events, Downtown Shawnee will host a "Back on Bell Street Jazz Festival" for local and out-of-town participants to enjoy at the conclusion of the sporting events.
For more information regarding volunteering or participating in the Bison Bicycle Classic, contact Jim Vernon or Thresa Swadley at (405) 878-2305 or e-mail music@okbu.edu.


We have seen many hacks and scams in the our previous posts which helped you to be aware of it and be protected from these kind of frauds. Today we are going to talk about the recent scams and hacks which were used by the fraudsters.
If your bank calls or texts you, don't trust it and definitely don't give out any of your details. If in doubt, contact your bank yourself so that you can be sure you are actually communicating with it.
The Fake Microsoft Warning browser hijack is an internet virus that can be hard to spot. The first symptoms may be an unexpected change in your default browser or homepage search engine, but you'll know you are infected when pop-up banners bombard your screen, becoming almost impossible not to click. What's the motivation for these hackers? Well, the more ads you click on the more money they make.
Which celebrity you look alike? Which Harry Potter house would you be in? Social media quizzes are undeniably fun. But sometimes things that seem very innocent aren't. Some quizzes are designed to steal your data in an outright scam by asking permissions. By hiding embedded links in the quizzes hackers can steal information from your social media account, and in some instances that of your 'friends'.
In May 2018 the authorities warned about a Russian malware called VPNFilter. The hacking campaign affected over half a million routers in 54 countries worldwide, creating a massive botnet and directly spying on and manipulating web activity.
In June 2018 the email addresses of an undisclosed number of Reddit users were accessed by hackers. Reddit reported the issue to law enforcement and is cooperating with the investigation. Reddit also messaged user accounts if there was a chance the credentials taken reflected the account's current password.
In December 2018, question-and-answer website Quora announced that its data records had been hacked. The names, email addresses and encrypted passwords of over 100 million users – about one third of its active monthly user base – are at risk. Quora has emailed the users who have been affected.
While it took place a few years before, one of the most controversial data breach reveals of 2018 was that data analytics firm Cambridge Analytica had harvested 50 million Facebook profiles, collecting personal information and data on individual's engagement via an app, in order to predict and influence the choices of voters in political elections.


I think the most common times people get injured is when they start a new activity, or when they make a big change to the amount of exercise they are doing, even if it is something they have done before. To give you an example, if you've never included running in your fitness program but start doing so, that's a time when your body is adjusting to the new stimulus and as such you're prone to injury. Or if you usually run 5km three times a week but change to 10km five times a week, that's also a time when you're prone to injury.
It's a fine line; you want to push yourself to get fitter, but if you push too hard you can hurt yourself. When you're exercising, there inevitably comes a time when you will feel a bit of discomfort. When you reach that point you might wonder whether you can keep going and push through it, or if doing so could cause an injury. There is such a thing as 'good pain', which is the pain you feel during exercise but in the absence of an injury. Then there's 'bad pain', which is the pain you feel when you have suffered an injury.
What does good pain feel like? Think about when you climb a really tall set of stairs or a steep hill. As you start to breathe harder, you get less oxygen to your muscles. When that happens you start to develop lactic acid. At the beginning you feel almost nothing, then it becomes slightly uncomfortable, then if you're able to keep going, it becomes really uncomfortable. So we would say that one point of differentiation from bad pain is that good pain has more of a gradual build-up.
The location of the pain is also a clue. Good pain will be generalised to a region, whereas bad pain is quite specific. For example, if you're 'feeling the burn' in your quads from climbing stairs (good pain), then most likely the entire front of your leg will feel it, all the way from your knees to your hips. On the other hand, if you tear or strain your quadriceps (bad pain), the location of the pain will be a lot more specific – you'll feel sharp pain right at the point of the tear and in a small area surrounding it.
I'm sure you've all watched something like the Olympic Games, where you can see prime examples of the two types of pain. For instance, in the 100m when someone tears their hamstring, they grab their leg as if they've been shot. They feel sharp, shooting pain that has a sudden onset, the leg gives way underneath them and they're unable to continue.
Then you see someone in a longer-distance event with a grimace on their face as they push on and on. They're definitely in pain, but the pain has built up gradually and doesn't affect their ability to keep running (well, for a while at least). While this is not a nice feeling, this is the 'good pain' we're talking about.
There's a fine line between pushing yourself and hurting yourself. It just comes down to using a bit of common sense. Build up your training gradually. Warm up a little before going too hard. If something doesn't feel right, you're better off missing part of a session and being okay for your next session than trying to get a little bit more done and having to miss the rest of the week.


text:
\section{Technical proofs}
\begin{lemma}\label{lem:smoothbnd}
For any $H$-smooth non-negative function $f : \ensuremath{\mathbb{R}} \mapsto \ensuremath{\mathbb{R}}$ and any $t, r \in \ensuremath{\mathbb{R}}$ we have that
$$
\left(f(t) - f(r)\right)^2 \le 6 H ( f(t) + f(r) )  (t - r)^2
$$
\end{lemma}
\begin{proof}
We start by noting that by the mean value theorem for any $t, r \in \ensuremath{\mathbb{R}}$ there exists $s$ between $t$ and $r$  such that
\begin{equation}\label{eq:meanval}
f(t) - f(r) = f'(s) (t - r)
\end{equation}
By smoothness we have that
$$
\abs{f'(s) - f'(t)} \le H\abs{t-s} \leq H \abs{t - r}.
$$
Hence we see that
\begin{equation}\label{eq:lipder}
\abs{f'(s)} \le \abs{f'(t)} + H \abs{t - r}
\end{equation}
We now consider two cases: \\
\noindent{\bf Case I:} If $\abs{t - r} \le \frac{\abs{f'(t)}}{5 H}$ then
by Equation \eqref{eq:lipder}, $\abs{f'(s)} \le 6/5 \abs{f'(t)}$,
and combining this with Equation \eqref{eq:meanval} we have:
\begin{align}\label{eq:int1}
(f(t) - f(r))^2 & \le f'(s)^2 (t-r)^2 \leq \frac{36}{25} f'(t)^2 (t -
r)^2 \notag
\intertext{But Lemma \ref{lemma:smooth} ensures $f'(t)^2 \leq 4 H
  f(t)$ yielding:}
& \le \frac{144}{25} H f(t) (t - r)^2 < 6 H f(t) (t-r)^2
\end{align}
\noindent{\bf Case II:} On the other hand, when $\abs{t - r} >
\frac{\abs{f'(t)}}{5 H}$, we have from Equation \eqref{eq:lipder} that
$\abs{f'(s)} \le 6 H \abs{t-r}$.  Plugging this into Equation
\eqref{eq:meanval} yields:
\begin{align}
(f(t) - f(r))^2 & = \abs{f(t)-f(t)} \cdot \abs{f(t)-f(r)} \le \abs{f(t) - f(r)} (\abs{f'(s)} \abs{t-r}) \notag \\
& \le \abs{f(t) - f(r)} (6 H \abs{t-r} \cdot \abs {t - r}) = 6
H \abs{f(t)-f(r)} (t-r)^2 \notag \\
& \le 6 H \max\{f(t), f(r)\} (t - r)^2 \label{eq:int2}
\end{align}
Combining the two cases, we have from Equations \eqref{eq:int1} and
\eqref{eq:int2}) and the non-negativity of $f(\cdot)$, that in either
case:
\begin{equation*}
(f(t) - f(r))^2 \le 6 H \left(f(t) + f(r) \right) (t - r)^2 \qedhere
\end{equation*}
\end{proof}
\iffalse
Using a more cumbersome proof, breaking up the function to
quasi-convex pieces, the constant $6$ in the above lemma can be
reduced to $4$.
\fi

\textbf{Relating Fat-shattering Dimension and Rademacher complexity : }

The following lemma upper bounds the fat-shattering dimension at scale
$\epsilon \ge \mc{R}_n(\H)$ in terms of the Rademacher Complexity of
the function class. The proof closely follows the arguments of
Mendelson \cite[discussion after Definition 4.2]{Mendelson02}.

\begin{lemma}\label{lem:fatrad}
For any hypothesis class $\H$, any sample size $n$ and any $\epsilon > \mc{R}_n(\H)$ we have that
$$
\mathrm{fat}_\epsilon(\H) \le \frac{4\ n\ \mc{R}_n(\H)^2}{\epsilon^2}
$$
\end{lemma}
In particular, if $\mc{R}_n(\H) = \sqrt{R/n}$ (the typical case), then
$\mathrm{fat}_\epsilon(\H) \leq R/\epsilon^2$.
\begin{proof}
  Consider any $\epsilon \ge \mc{R}_n(\H)$. Let
  $x^*_1,\ldots,x^*_{\mathrm{fat}_\epsilon}$ be the set of $\mathrm{fat}_\epsilon$
  shattered points. This means that there exists
  $s_1,\ldots,s_{\mathrm{fat}_\epsilon}$ such that for any $J \subset
  [\mathrm{fat}_\epsilon]$ there exists $h_J \in \H$ such that $\forall i \in
  J, h_J(x_i) \ge s_i + \epsilon$ and $\forall i \not\in J, h_J(x_i)
  \le s_i - \epsilon$.  Now consider a sample $x_1,\ldots,x_{n'}$ of
  size $n' = \lceil \frac{n}{\mathrm{fat}_\epsilon}\rceil \mathrm{fat}_\epsilon$,
  obtained by taking each $x^*_i$ and repeating it $\lceil
  \frac{n}{\mathrm{fat}_\epsilon}\rceil$ times, i.e.~$x_i =
  x^*_{\lfloor \frac{i}{\mathrm{fat}_\epsilon} \rfloor}$. Now, following
  Mendelson's arguments:
\begin{align*}
\mc{R}_{n'}(\H) & \ge \Es{\sigma \sim \mathrm{Unif}\{\pm 1\}^{n'}}{\frac{1}{n'} \sup_{h \in \H}\left| \sum_{i=1}^{n'} \sigma_i h(x_i)\right|} \\
& \ge \frac{1}{2} \Es{\sigma \sim \mathrm{Unif}\{\pm 1\}^{n'}}{\frac{1}{n'} \sup_{h, h' \in \H}\left| \sum_{i=1}^{n'} \sigma_i (h(x_i) - h'(x_i))\right|} ~~~~~~~~~~~ \textrm{(triangle inequality)}\\
& = \frac{1}{2} \Es{\sigma \sim \mathrm{Unif}\{\pm 1\}^{n'}}{\frac{1}{n'} \sup_{h, h' \in \H}\left| \sum_{i=1}^{\mathrm{fat}_\epsilon} \left(\sum_{j=1}^{\lceil n/\mathrm{fat}_\epsilon \rceil} \sigma_{(i-1) \mathrm{fat}_\epsilon + j}\right) \left(h(x^*_i) - h'(x^*_i)\right)\right|}\\
& \ge \frac{1}{2} \Es{\sigma \sim \mathrm{Unif}\{\pm 1\}^{n'}}{\frac{1}{n'}\left| \sum_{i=1}^{\mathrm{fat}_\epsilon} \left(\sum_{j=1}^{\lceil n/\mathrm{fat}_\epsilon \rceil} \sigma_{(i-1) \mathrm{fat}_\epsilon + j}\right) \left(h_{R}(x^*_i) - h_{\overline{R}}(x^*_i)\right)\right|}
\intertext{
where for each $\sigma_1,\ldots,\sigma_{n'}$, $R \subseteq
[\mathrm{fat}_\epsilon]$ is given by $R = \left\{i \in [\mathrm{fat}_\epsilon]
  \middle|  \mathrm{sign}\left(\sum_{j=1}^{\lceil n/\mathrm{fat}_\epsilon
      \rceil} \sigma_{(i - 1) \lceil n/\mathrm{fat}_\epsilon\rceil + j}\right)
  \ge 0 \right\}$, $h_R$ is the function in $\H$ that
$\epsilon$-shatters the set $R$ and $h_{\overline{R}}$ be the function
that shatters the complement of set $R$.}
& \ge  \frac{1}{2} \Es{\sigma \sim \mathrm{Unif}\{\pm 1\}^{n'}}{\frac{1}{n'} \sum_{i=1}^{\mathrm{fat}_\epsilon} \left|\sum_{j=1}^{\lceil n/\mathrm{fat}_\epsilon \rceil} \sigma_{(i-1) \mathrm{fat}_\epsilon + j}\right| 2 \epsilon }\\
& \ge  \frac{\epsilon}{n'} \sum_{i=1}^{\mathrm{fat}_\epsilon} \Es{\sigma \sim \mathrm{Unif}\{\pm 1\}^{n'}}{ \left|\sum_{j=1}^{\lceil n/\mathrm{fat}_\epsilon \rceil} \sigma_{(i-1) \mathrm{fat}_\epsilon + j}\right|}\\
& \ge  \frac{\epsilon\ \mathrm{fat}_\epsilon}{n'}  \sqrt{\frac{\lceil n/\mathrm{fat}_\epsilon \rceil}{2}} \hspace{2.5in} \textrm{(Kinchine's inequality)}\\
& =  \sqrt{\frac{\epsilon^2\ \mathrm{fat}_\epsilon}{2\ n'}}.
\end{align*}
We can now conclude that:
\begin{align*}
\mathrm{fat}_\epsilon \le \frac{2 n' \mc{R}^2_{n'}(\H)}{\epsilon^2} \le \frac{4 n \mc{R}^2_{n}(\H)}{\epsilon^2}
\end{align*}
where last inequality is because Rademacher complexity decreases with
increase in number of samples and $n \leq n' \le 2n$ (because $\epsilon \ge
\mc{R}_n(\H)$ which implies that $\mathrm{fat}_\epsilon < n$). \qedhere

\end{proof}

\textbf{Dudley Upper Bound : }

In order to state and prove the following Lemma, we shall find it
simpler to use the empirical Rademacher complexity for a given sample
$x_1,\ldots,x_n$ \cite{BartlettMe02}:
\begin{equation}
  \label{eq:empraddef}
\hat{R}_n(\H) =  \Es{\sigma \sim \text{Unif}(\{\pm 1\}^n)}{\sup_{h \in \H} \frac{1}{n} \left|\sum_{i=1}^n  h(x_i) \sigma_i \right|}.
\end{equation}
and the $L_2$ covering number at scale $\epsilon > 0$ specific to a sample $x_1,\ldots,x_n$, denoted by $N_2\left(\epsilon,\mathcal{F},(x_1,\ldots,x_n)\right)$ as the size of a minimal cover $\mathcal{C}_\epsilon$ such that
$$\forall f \in \mathcal{F}, \exists f_\epsilon \in
\mathcal{C}_\epsilon\ \textrm{s.t.}~\sqrt{\frac{1}{n}
  \sum_{i=1}^n (f(z_i) - f_\epsilon(z_i))^2} \le
\epsilon~.$$ We will also denote $\hE{f^2} = \frac{1}{n} \sum_{i=1}^n f^2(x_i)$.

The following Lemma is stated in terms of the empirical Rademacher
complexity and covering numbers.  Taking a supremum over samples of
size $n$, we get the same relationship between the worst-case
Rademacher complexity and covering numbers, as is used in Section \ref{sec:rad}.

\begin{lemma}[\cite{SrebroSr10} following \cite{Mendelson02}]\label{lem:cover}
For any function class $\mathcal{F}$ containing functions $f : \mathcal{X} \mapsto \mathbb{R}$, we have that
$$
\hat{R}_n(\mathcal{F}) \le \inf_{\alpha \ge 0}\left\{ 4 \alpha + 10 \int_{\alpha}^{\sup_{f \in {F}} \sqrt{\hE{f^2}}} \sqrt{\frac{\log \mathcal{N}\left(\epsilon,\mathcal{F},(x_1,\ldots,x_n)\right)}{n}} d \epsilon \right\}
$$
\end{lemma}
\begin{proof}
Let $\beta_0 = \sup_{f \in {F}}\sqrt{ \hE{f^2}} $ and for any $j \in \mathbb{Z}_+$ let $\beta_j = 2^{-j} \sup_{f \in {F}} \sqrt{\hE{f^2}}$. The basic trick here is the idea of chaining. For each $j$ let $T_i$ be a (proper) $L_2$-cover  at scale $\beta_j$ of $\mathcal{F}$ for the given sample. For each $f \in \mathcal{F}$ and $j$, pick an $\hat{f}_i \in T_i$ such that $\hat{f}_i$ is an $\beta_i$ approximation of $f$. Now for any $N$, we express $f$ by chaining as
$$
f = f - \hat{f}_N + \sum_{i=1}^{N}\left(\hat{f}_{i} - \hat{f}_{i-1}\right)
$$
where $\hat{f}_0 = 0$. Hence for any $N$ we have that
\begin{align}
\hat{R}_n(\mathcal{F}) &= \frac{1}{n}\Ep{\sigma}{\sup_{f \in \mathcal{F}} \sum_{i=1}^n  \sigma_i \left(f(\ensuremath{\mathbf{x}}_i) - \hat{f}_N(\ensuremath{\mathbf{x}}_i) + \sum_{j=1}^N \left( \hat{f}_j(\ensuremath{\mathbf{x}}_i) - \hat{f}_{j-1}(\ensuremath{\mathbf{x}}_i)\right) \right)} \notag \\
& \le \frac{1}{n}\Ep{\sigma}{\sup_{f \in \mathcal{F}} \sum_{i=1}^n  \sigma_i \left(f(\ensuremath{\mathbf{x}}_i) - \hat{f}_N(\ensuremath{\mathbf{x}}_i) \right)} + \sum_{j=1}^N \frac{1}{n} \Ep{\sigma}{\sup_{f \in \mathcal{F}} \sum_{i=1}^n \sigma_i \left( \hat{f}_j(\ensuremath{\mathbf{x}}_i) - \hat{f}_{j-1}(\ensuremath{\mathbf{x}}_i)\right)} \notag \\
& \le \frac{1}{n} \sqrt{\sum_{i=1}^n \sigma_i^2}\ \sup_{f \in \mathcal{F}} \sqrt{ \sum_{i=1}^n (f(x_i) - \hat{f}_N(x_i)^2} + \sum_{j=1}^N \frac{1}{n} \Ep{\sigma}{\sup_{f \in \mathcal{F}} \sum_{i=1}^n \sigma_i \left( \hat{f}_j(\ensuremath{\mathbf{x}}_i) - \hat{f}_{j-1}(\ensuremath{\mathbf{x}}_i)\right)} \notag\\
& \le \beta_N +  \sum_{j=1}^N \frac{1}{n} \Ep{\sigma}{\sup_{f \in \mathcal{F}} \sum_{i=1}^n \sigma_i \left( \hat{f}_j(\ensuremath{\mathbf{x}}_i) - \hat{f}_{j-1}(\ensuremath{\mathbf{x}}_i)\right)} \label{eq:chain}
\end{align}
where the step before last is due to Cauchy-Shwarz inequality and $\mathbf{\sigma} = \left[ \sigma_1, ...,\sigma_n \right]^\top$. Now note that
\begin{align*}
\frac{1}{n} \sum_{i=1}^n (\hat{f}_j(x_i) - \hat{f}_{j-1}(x_i))^2 & = \frac{1}{n} \sum_{i=1}^n \left( (\hat{f}_j(x_i)) - f(x_i)) + (f(x_i) - \hat{f}_{j-1}(x_i)) \right)^2 \\
& \le \frac{2}{n} \sum_{i=1}^n \left( \hat{f}_j(x_i)) - f(x_i)\right)^2 +  \frac{2}{n} \sum_{i=1}^n \left(f(x_i) - \hat{f}_{j-1}(x_i) \right)^2 \\
& \le 2 \beta_j^2 + 2 \beta_{j-1}^2 = 6 \beta_j^2
\end{align*}
Now Massart's finite class lemma \cite{Massart00} states that if for any function class $\mathcal{G}$, $\sup_{g \in \mathcal{G}} \sqrt{\frac{1}{n} \sum_{i=1}^n g(x_i)^2 } \le R$, then
$\hat{R}_{n}(\mathcal{G}) \le \sqrt{\frac{2 R^2 \log(|\mathcal{G}|)}{n}}$. Applying this to function classes $\{f - f' : f \in T_j ,\ f' \in T_{j-1}\}$ (for each $j$) we get from Equation \eqref{eq:chain} that for any $N$,
\begin{align*}
\hat{R}_n(\mathcal{F}) & \le \beta_N +  \sum_{j=1}^N  \beta_j \sqrt{\frac{12 \log(|T_j|\ |T_{j-1}|)}{n}} \\
& \le \beta_N +   \sum_{j=1}^N  \beta_j \sqrt{\frac{24 \ \log\ |T_j|}{n}} \\
& \le \beta_N +  10 \sum_{j=1}^N  (\beta_j - \beta_{j+1}) \sqrt{\frac{\log\ |T_j|}{n}} \\
& \le \beta_N +  10 \sum_{j=1}^N  (\beta_j - \beta_{j+1}) \sqrt{\frac{\log\ \mathcal{N}\left(\beta_j,\mathcal{F},(x_1,\ldots,x_n)\right)}{n}} \\
& \le \beta_N +  10 \int_{\beta_{N+1}}^{\beta_{0}} \sqrt{\frac{\log\ \mathcal{N}\left(\epsilon,\mathcal{F},(x_1,\ldots,x_n)\right)}{n}} d \epsilon
\end{align*}
where the third step is because $2 (\beta_j - \beta_{j+1}) = \beta_j$ and we bounded $\sqrt{24}$ by $5$. Now for any $\alpha > 0$, pick $N = \sup\{j : \beta_j > 2 \alpha\}$. In this case we see that by our choice of $N$, $\beta_{N+1} \le 2 \alpha$ and so $\beta_N = 2 \beta_{N+1} \le 4 \epsilon$. Also note that since $\beta_{N} > 2 \alpha$, $\beta_{N+1} = \frac{\beta_N}{2}> \alpha$. Hence we conclude that
 \begin{align*}
\hat{R}_n(\mathcal{F}) & \le 4 \alpha +  10 \int_{\alpha}^{\sup_{f \in {F}} \sqrt{\hE{f^2}}} \sqrt{\frac{\log\ \mathcal{N}\left(\epsilon,\mathcal{F},(x_1,\ldots,x_n)\right)}{n}} d \epsilon
\end{align*}
Since the choice of $\alpha$ was arbitrary we take an infimum over $\alpha$.
\end{proof}

\textbf{Detailed Proof of Main Result : }

\begin{proof}[\bf Detailed proof of Theorem \ref{thm:mainRad}]
By Theorem 6.1 of \cite{Bousquet02} (specifically the displayed equation prior to the last one in the proof of the theorem) we have that if $\psi_n$ is any sub-root function that satisfies for all $r > 0$,  $\mc{R}_n(\mc{L}_\phi(r)) \le \psi_n(r)$ then,
\begin{equation} \label{eq:olivier}
\L{h}  \le \hat{L}(h) + 45 r^*_n + \sqrt{\L{h}} \left( \sqrt{8 r^*_n} + \sqrt{\frac{4 b(\log(1/\epsilon) + 6 \log \log n )}{n}} \right) + \frac{20 b(\log(1/\epsilon) + 6 \log \log n )}{n}
\end{equation}
where $r^*_n$ is the largest solution to equation $\psi_n(r) = r$.
Now by Lemma \ref{lem:key} we have that $\psi_n(r)=56 \sqrt{H r} \log^{1.5} n \hat{\mathcal{R}}_n(\H)$ satisfies the property that for all $r > 0$,  $\mc{R}_n(\mc{L}_\phi(r)) \le \psi_n(r)$ and so using this we see that
$$
r^*_n = 56^2 H \log^3\!n \mc{R}_n(\H)
$$
and for this $r^*_n$ Equation \eqref{eq:olivier} holds. Now using the simple fact that for any non-negative $A,B,C$,
$$
A \le B + C\sqrt{A} \Rightarrow A \le B + C^2 + \sqrt{B}C
$$
we conclude,
\begin{equation*}
\L{h} \le \hat{L}(h) + 106\ r^*_n  + \frac{48 b}{n}  \left(\log\tfrac{1}{\epsilon} + \log \log\ n \right) + \sqrt{\hat{L}(h) \left(8 r^{*}_n + \frac{4 b}{n}  \left(\log\tfrac{1}{\epsilon} + \log \log\ n \right) \right)}
\end{equation*}
plugging in $r^*_n$ we get the required statement.
\end{proof}

\section{Introduction}

Consider empirical risk minimization for a hypothesis class $\H = \{ h:\mathcal{X}
\rightarrow \ensuremath{\mathbb{R}}\}$ w.r.t.~some non-negative loss function $\phi(t,y)$.
That is, we would like to learn a predictor $h$ with small risk
$$\L{h}=\E{\phi(h(X),Y)}$$ by minimizing the empirical risk
$$\hat{L}(h)=\frac{1}{n}\sum_{i=1}^n\phi(h(x_i),y_i)$$
of an i.i.d.~sample $(x_1,y_1),\ldots,(x_n,y_n)$.

Statistical guarantees on the excess risk are well understood for {\em
  parametric} (i.e.~finite dimensional) hypothesis classes.  More
formally, these are hypothesis classes with finite VC-subgraph
dimension \cite{Pollard84} (aka pseudo-dimension).  For such classes learning
guarantees can be obtained for any bounded loss function
(i.e.~s.t.~$\abs{\phi}\leq b < \infty$) and the relevant measure of
complexity is the VC-subgraph dimension.

Alternatively, even for some non-parametric hypothesis classes
(i.e.~those with infinite VC-subgraph dimension), e.g.~the class of
low-norm linear predictors
$$\H_B =\left\{ h_{w}:\ensuremath{\mathbf{x}}\mapsto\ip{\ensuremath{\mathbf{w}}}{\ensuremath{\mathbf{x}}} \middle| \norm{\ensuremath{\mathbf{w}}}\leq B
\right\},$$ guarantees can be obtained in terms of {\em
  scale-sensitive} measures of complexity such as fat-shattering
dimensions \cite{AlonBeCeHa93}, covering numbers \cite{Pollard84} or Rademacher complexity
\cite{BartlettMe02}.  The classical statistical learning theory approach for
obtaining learning guarantees for such scale-sensitive classes is to
rely on the Lipschitz constant $D$ of $\phi(t,y)$ w.r.t.~$t$
(i.e.~bound on its derivative w.r.t.~$t$).  The excess risk can then
be bounded as (in expectation over the sample):
\begin{equation}
  \label{eq:lip}
  \L{\hat{h}} \leq L^* + 2 D R_n(\H) = L^* + 2 \sqrt{D^2 \frac{R}{n}}
\end{equation}
where $\hat{h}=\arg\min \hat{L}(h)$ is the empirical risk minimizer (ERM),
$L^*=\inf_h \L{h}$ is the approximation error, and $R_n(\H)$ is the
Rademacher complexity of the class, which typically scales as
$R_n(\H)=\sqrt{R/n}$.  E.g.~for $\ell_2$-bounded linear predictors,
$R = B^2 \sup \norm{X}_2^2$.  The Rademacher complexity can be bounded
by other scale-sensitive complexity measures, such as the
fat-shattering dimensions and covering numbers, yielding similar
guarantees in terms of these measures.

In this paper we address two deficiencies of the guarantee
\eqref{eq:lip}.  First, the bound applies only to loss functions with
bounded derivative, like the hinge loss and logistic loss popular for
classification, or the absolute-value ($\ell_1$) loss for regression.
It is not directly applicable to the squared loss
$\phi(t,y)={\scriptstyle\tfrac{1}{2}}(t-y)^2$, for which the second
derivative is bounded, but not the first.  We could try to simply
bound the derivative of the squared loss in terms of a bound on the
magnitude of $h(x)$, but e.g.~for norm-bounded linear predictors
$\H_B$ this results in a very disappointing excess risk bound of the
form $O(\sqrt{ B^4 (\max \norm{X})^4 / n })$.  One aim of this paper
is to provide clean bounds on the excess risk for smooth loss
functions such as the squared loss with a bounded second, rather then
first, derivative.

The second deficiency of \eqref{eq:lip} is the dependence on
$1/\sqrt{n}$.  The dependence on $1/\sqrt{n}$ might be unavoidable in
general.  But at least for finite dimensional (parametric) classes, we
know it can be improved to a $1/n$ rate when the distribution is
separable, i.e.~when there exists $h \in \H$ with $\L{h}=0$ and so
$L^*=0$.  In particular, if $\H$ is a class of bounded functions with
VC-subgraph-dimension \iffalse\footnote{A generalization of the
  VC-dimension for real valued functions, also known as the
  pseudo-dimension, and that captures the dimensionality (number of
  free parameters) in a parametric class}\fi $d$
(e.g.~$d$-dimensional linear predictors), then (in expectation over
the sample) \cite{Panchenko02}:
\begin{equation}
  \label{eq:panchenko}
  \L{\hat{h}} \leq L^* + O\left( \frac{ d D \log n}{n} + \sqrt{ \frac{d D L^* \log n}{n}} \right)
\end{equation}
The $1/\sqrt{n}$ term disappears in the separable case, and we get a
graceful degredation between the $1/\sqrt{n}$ non-separable rate and
the $1/n$ separable rate.  Could we get a $1/n$ separable rate, and
such a graceful degradation, also in the non-parametric case?

As we will show, the two deficiencies are actually related.  For
non-parametric classes, and non-smooth Lipschitz loss, such as the
hinge-loss, the excess risk might scale as $1/\sqrt{n}$ and not $1/n$,
{\em even in the separable case}.  However, for $H$-smooth non-negative loss
functions, where the second derivative of $\phi(t,y)$ w.r.t.~$t$ is
bounded by $H$, a $1/n$ separable rate {\em is} possible.  In Section
\ref{sec:rad} we obtain the following bound on the excess risk (up to
logarithmic factors):
\begin{align}
  \label{eq:intromain}
  \L{\hat{h}} &\leq L^* + \tilde{O}\left( HR^2_n(\H) + \sqrt{H L^*}
    R_n(\H) \right) \notag \\
  &= L^* + \tilde{O}\left( \frac{H R}{n} +
    \sqrt{\frac{H R L^*}{n}} \right) \;\;\leq\;\; 2L^* +\tilde{O}\left(\frac{H R}{n}\right).
\end{align}
In particular, for $\ell_2$-norm-bounded linear predictors $\H_B$ with $\sup
\norm{X}_2^2 \leq 1$, the excess risk is bounded by $\tilde{O}(HB^2/n +
\sqrt{HB^2L^*/n})$.  Another interesting distinction between
parametric and non-parametric classes, is that even for the
squared-loss, the bound \eqref{eq:intromain} is tight and the
non-separable rate of $1/\sqrt{n}$ is unavoidable.  This is in
contrast to the parametric (fine dimensional) case, where a rate of
$1/n$ is always possible for the squared loss, regardless of the
approximation error $L^*$ \cite{LeeBaWi98}.  The differences between
parametric and scale-sensitive classes, and between non-smooth, smooth
and strongly convex (e.g.~squared) loss functions are discussed in
Section \ref{sec:tight} and summarized in Table \ref{tab:tight}.

The guarantees discussed thus far are general learning guarantees for
the stochastic setting that rely only on the Rademacher complexity of
the hypothesis class, and are phrased in terms of minimizing some
scalar loss function.  In Section \ref{sec:convex} we consider also
the online setting, in addition to the stochastic setting, and present
similar guarantees for online and stochastic convex optimization
\cite{Zinkevich03,ShalevShSrSr09}.  The guarantees of Section
\ref{sec:convex} match equation \eqref{eq:intromain} for the special
case of a convex loss function and norm-bounded linear predictors, but
Section \ref{sec:convex} capture a more general setting of optimizing
an arbitrary non-negative convex objective, which we require to be
smooth (there is no separate discussion of a ``predictor'' and a
scalar loss function in Section \ref{sec:convex}).  Results in Section
\ref{sec:convex} are expressed in terms of properties of the norm,
rather then a measure of concentration like the Radamacher complexity
as in \eqref{eq:intromain} and Section \ref{sec:rad}.  However, the
online and stochastic convex optimization setting of Section
\ref{sec:convex} is also more restrictive, as we require the objective
be convex (in Section \ref{sec:rad} and for the bound
\eqref{eq:intromain} we make no assumption about the convexity of the
hypothesis class $\H$ nor the loss function $\phi$).

Specifically, for a non-negative $H$-smooth {\em convex} objective
(see exact definition in Section \ref{sec:convex}), over a domain
bounded by $B$, we prove that the average online regret (and so also
the excess risk of stochastic optimization) is bounded by $O(HB^2/n +
\sqrt{HB^2L^*/n})$.  Comparing with the bound of $O(\sqrt{D^2 B^2/n})$
when the loss is $D$-Lipschitz rather then $H$-smooth
\cite{Zinkevich03,NemirovskiYu78}, we see the same relationship
discussed above for ERM.  Unlike the bound \eqref{eq:intromain} for
the ERM, the convex optimization bound avoids polylogarithmic factors.
The results in Section \ref{sec:convex} also generalize to smoothness
and boundedness with respect to non-Euclidean norms.

Studying the online and stochastic convex optimization setting
(Section \ref{sec:convex}), in addition to ERM (Section
\ref{sec:rad}), has several advantages.  First, it allows us to obtain
a learning guarantee for an efficient single-pass learning methods,
namely stochastic gradient descent (or mirror descent), as well as for
the non-stochastic regret.  Second, the bound we obtain in the convex
optimization setting (Section \ref{sec:convex}) is actually better
then the bound for the ERM (Section \ref{sec:rad}) as it avoids all
polylogarithmic and large constant factors.  Third, the bound is
applicable to other non-negative online or stochastic optimization
problems beyond classification, including problems for which ERM is
not applicable (see, e.g.,~\cite{ShalevShSrSr09}).

\section{Empirical Risk Minimization with Smooth Loss}\label{sec:rad}

Recall that the Rademacher complexity of $\H$ for any $n \in \mathbb{N}$ given by \cite{BartlettMe02}:
\begin{equation}
  \label{eq:raddef}
\mc{R}_n(\H) = \sup_{x_1,\ldots , x_n \in \mathcal{X}} \Es{\sigma \sim \text{Unif}(\{\pm 1\}^n)}{\sup_{h \in \H} \frac{1}{n} \left|\sum_{i=1}^n  h(x_i) \sigma_i \right|}.
\end{equation}
Throughout we shall consider the ``worst case" Rademacher complexity.

Our starting point is the learning bound \eqref{eq:lip} that applies
to $D$-Lipschitz loss functions, i.e. such that $\abs{\phi'(t,y)}\leq D$ (we
always take derivatives w.r.t.~the first argument).  What type of
bound can we obtain if we instead bound the second derivative
$\phi''(t,y)$?  We will actually avoid talking about the second
derivative explicitly, and instead say that a function is $H$-smooth
iff its derivative is $H$-Lipschitz.  For twice differentiable $\phi$,
this just means that $\abs{\phi''}\leq H$.  The central observation, which
allows us to obtain  guarantees for smooth loss functions, is that for
a smooth loss, the derivative can be bounded in terms of the function value:
\begin{lemma}\label{lemma:smooth}
For an $H$-smooth non-negative function $f : \mathbb{R} \mapsto \mathbb{R}$, we have:
$\abs{f'(t)}  \le \sqrt{4 H f(t)}$
\end{lemma}
\begin{proof}
For any $t, r$, we have $t<s<r$ for
which $f(r) = f(t) + f'(s)(r- t)$.  Now:
\begin{align*}
0 & \le  f(r) = f(t) + f'(t)(r - t) + (f'(s) - f'(t))(r- t) \\
& \le f(t) + f'(t)(r - t) + H \abs{s-t}\abs{r - t} \le f(t) + f'(t)(r - t) + H (r - t)^2
\end{align*}
Setting $r = t - \frac{f'(t)}{2 H}$ yields the desired bounds.
\end{proof}

This Lemma allows us to argue that close to the optimum value, where
the {\em value} of the loss is small, then so is its derivative.
Looking at the dependence of \eqref{eq:lip} on the derivative bound
$D$, we are guided by the following heuristic intuition: Since we
should be concerned only with the behavior around the ERM, perhaps it
is enough to bound $\phi'(\hat{\ensuremath{\mathbf{w}}},x)$ at the ERM $\hat{\ensuremath{\mathbf{w}}}$.  Applying
Lemma \ref{lemma:smooth} to $L(\hat{h})$, we can bound
$\abs{\E{\phi'(\hat{\ensuremath{\mathbf{w}}},X)}} \leq \sqrt{4 H L(\hat{h})}$.  What we
would actually want is to bound each $\abs{\phi'(\hat{\ensuremath{\mathbf{w}}},x)}$
separately, or at least have the absolute value {\em inside} the
expectation---this is where the non-negativity of the loss plays an
important role.  Ignoring this important issue for the moment and
plugging this instead of $D$ into \eqref{eq:lip} yields $L(\hat{h})
\leq L^* + 4\sqrt{H L(\hat{h})} R_n(\H)$.  Solving for $L(\hat{h})$
yields the desired bound \eqref{eq:intromain}.

This rough intuition is captured by the following Theorem:

\begin{theorem}\label{thm:mainRad}
  For an $H$-smooth non-negative loss $\phi$ s.t.$\forall_{x,y,h} \abs{\phi(h(x),y)}\leq
  b$, for any $\delta > 0$ we have that with probability at least $1 -
  \delta$ over a random sample of size $n$, for any $h \in \H$,
\begin{align*}
\L{h} \le \hat{L}(h) +K\left( \sqrt{\hat{L}(h)} \left(\sqrt{H} \log^{1.5}\! n\  \mc{R}_n(\H)  + \sqrt{\frac{b \log(1/\delta)}{n}}\right)+ H \log^{3}\!n\  \mc{R}_n^2(\H)  + \frac{b \log(1/\delta)}{n}\right)
\end{align*}
and so:
\vspace{-0.1in}
\begin{align*}
\L{\hat{h}} \le L^* + K \left( \sqrt{L^*} \left(\sqrt{H} \log^{1.5}\!n\ \mc{R}_n(\H)  + \sqrt{\frac{b \log(1/\delta)}{n}}\right)+  H \log^{3}\!n\  \mc{R}_n^2(\H)  + \frac{b \log(1/\delta)}{n}\right)
\end{align*}
where $K < 10^5$ is a numeric constant derived from \cite{Mendelson02} and \cite{Bousquet02}.
\end{theorem}
Note that only the ``confidence'' terms depended on $b=\sup
\abs{\phi}$, and this is typically not the dominant term---we
believe it is possible to also obtain a bound that holds in
expectation over the sample (rather than with high probability) and
that avoids a direct dependence on $\sup\abs{\phi}$.

To prove Theorem \ref{thm:mainRad} we use the notion of Local
Rademacher Complexity \cite{BartlettBoMe05}, which allows us to focus
on the behavior close to the ERM.  To this end, consider the following
empirically restricted loss class
$$
\mc{L}_\phi(r) := \left\{(x,y) \mapsto \phi(h(x),y) : h \in \H , \hat{L}(h) \le r\right\}
$$
Lemma \ref{lem:key}, presented below, solidifies the heuristic
intuition discussed above, by showing that the Rademacher complexity
of $\mc{L}_\phi(r)$ scales with $\sqrt{Hr}$.  The Lemma can be seen as a
higher-order version of the Lipschitz Composition Lemma
\cite{BartlettMe02}, which states that the Rademacher complexity of
the {\em unrestricted} loss class is bounded by $D R_n(\H)$.  Here,
we use the second, rather then first, derivative, and obtain a bound
that depends on the empirical restriction:

\begin{lemma}\label{lem:key}
For a non-negative $H$-smooth loss $\phi$ bounded by $b$ and any function class $\H$ bounded by $B$:
$$
\mc{R}_n(\mc{L}_\phi(r)) \le \sqrt{12 H r}\ \mc{R}_n(\H) \left( 16 \log^{3/2}\left(\frac{n B}{\mc{R}_n(\H)}\right) - 14 \log^{3/2}\left(\frac{n \sqrt{12 H} B}{\sqrt{b}}\right) \right)
$$
\end{lemma}

\begin{proof}
In order to prove Lemma \ref{lem:key}, we actually move from
Rademacher complexity to covering numbers, use smoothness and Lemma
\ref{lemma:smooth} to obtain an $r$-dependent cover of the empirically
restricted class, and then return to the Rademacher complexity.  More
specifically:
\vspace{-0.18in}
\begin{itemize}
  \setlength{\topsep}{0pt}
  \setlength{\itemsep}{1pt}
  \setlength{\parskip}{0pt}
  \setlength{\parsep}{0pt}
\item We use a modified version of Dudley's integral to bound the
  Rademacher complexity of the empirically restricted class in terms
  of its $L_2$-covering numbers.
\item We use smoothness to get an $r$-dependent bound on the
  $L_2$-covering numbers of the empirically restricted loss class in
  terms of $L_{\infty}$-covering numbers of the unrestricted
  hypothesis class.
\item We bound the $L_{\infty}$-covering numbers of the unrestricted
  class in terms of its fat-shattering dimension, which in turn can be
  bounded in terms of its Rademacher complexity.
\end{itemize}
Before we proceed, recall the following definitions of covering numbers and fat shattering dimension.
For any $\epsilon >0$ and function class $\mathcal{F} \subset \mathbb{R}^{\mathcal{Z}}$:
\begin{description}
\item[ ] \vspace{-0.1in} The $L_2$ covering number $\mathcal{N}_2\left(\mathcal{F},\epsilon, n\right)$ is the supremum over samples $z_1,\ldots,z_n$ of the size of a minimal cover $\mathcal{C}_\epsilon$ such that
$\forall f \in \mathcal{F}$, $\exists f_\epsilon \in
\mathcal{C}_\epsilon$ s.t.~$\sqrt{\frac{1}{n}
  \sum_{i=1}^n (f(z_i) - f_\epsilon(z_i))^2} \le
\epsilon$.\\
\item[ ] \vspace{-0.15in} The $L_\infty$ covering number $\mathcal{N}_\infty\left(\mathcal{F},\epsilon, n\right)$  is the supremum over samples $z_1,\ldots,z_n$ of the size of a minimal cover $\mathcal{C}_\epsilon$ such that
$\forall f \in \mathcal{F}$,  $\exists f_\epsilon \in
\mathcal{C}_\epsilon$ s.t.~$ \max_{i \in [n]} |f(z_i) - f_\epsilon(z_i)| \le
\epsilon$.\\
\item[ ] \vspace{-0.15in} The fat-shattering dimension
  $\mathrm{fat}_{\epsilon}(\mathcal{F})$ at scale $\epsilon$ is the
  maximum number of points $\epsilon$-shattered by $\mathcal{F}$ (see
  e.g.~\cite{Mendelson02}).
\end{description}

\vspace{-0.2in}
\paragraph{Bounding $\mc{R}_n(\mc{L}_\phi(r))$ in terms of
  $\mathcal{N}_2(\mc{L}_\phi(r))$}
Dudley's integral bound lets us bound the Rademacher complexity of a
class in terms of its empirical $L_2$ covering number. Here we use a
more refined version of Dudley's integral bound due to Mendelson
\cite{Mendelson02} and more explicitly stated in \cite{SrebroSr10} and
included for completeness as Lemma \ref{lem:cover} in the Appendix:
\begin{align}\label{eq:dud}
  \mc{R}_n(\mc{L}_\phi(r))\le \inf_{\alpha > 0}\left\{ 4 \alpha + 10
    \int_{\alpha}^{\sqrt{b r}}
    \sqrt{\frac{\mathcal{N}_2\left(\mc{L}_\phi(r),\epsilon,
          n\right)}{n}} d \epsilon \right\}
\end{align}

\paragraph{Bounding $\mathcal{N}_2(\mc{L}_\phi(r))$ in terms of
  $\mathcal{N}_{\infty}(\mathcal{H})$}
In the Appendix we show that a corollary of Lemma \ref{lemma:smooth}
is that for a non-negative $H$-smooth $f(\cdot)$ we have $\left(f(t) -
  f(r)\right)^2 \le 6 H ( f(t) + f(r) )  (t - r)^2$ (Lemma
\ref{lem:smoothbnd}).  Using this inequality, for any sample
$(x_1,y_1),\ldots,(x_n,y_n)$:
\begin{align*}
\sqrt{\frac{1}{n} \sum_{i=1}^n (\phi(h(z_i),z_i) - \phi(h_\epsilon(z_i),z_i))^2}  & \le \sqrt{\frac{6 H}{n} \sum_{i=1}^n \left( \phi(h(z_i),z_i)  +  \phi(h_\epsilon(z_i),z_i)\right) (h(z_i) - h_\epsilon(z_i))^2} \\
& \le \sqrt{\frac{6 H}{n} \sum_{i=1}^n \left( \phi(h(z_i),z_i)  +  \phi(h_\epsilon(z_i),z_i)\right) } \sqrt{\max_{i \in [n]} (h(z_i) - h_\epsilon(z_i))^2}\\
& \le \sqrt{12 H  r}\ \max_{i \in [n]}| h(z_i) - h_\epsilon(z_i)|
\end{align*}
That is, an empirical $L_{\infty}$ cover of $\left\{h \in \H : \hat{L}(h)
  \le r \right\}$ at radius $\epsilon/\sqrt{12 H r}$ is also an
empirical $L_2$ cover of $\mc{L}_\phi(r)$ at radius $\epsilon$, and we
can conclude that:
\begin{equation}\label{eq:N2Ninf}
\mathcal{N}_2\left(\mc{L}_\phi(r),\epsilon, n\right) \le \mathcal{N}_\infty\left(\left\{h \in \H : \hat{L}(h) \le r\right\},\frac{\epsilon}{\sqrt{12 H r}}, n\right) \le \mathcal{N}_\infty\left(\H,\frac{\epsilon}{\sqrt{12 H r}}, n\right)
\end{equation}

\paragraph{Bounding $\mathcal{N}_{\infty}(\mathcal{H})$ in terms of
  $\mc{R}_n(\H)$}
The $L_{\infty}$ covering number at scale $\epsilon/\sqrt{12 H r}$ can be bounded
in terms of the fat shattering dimension at
that scale as \cite{Mendelson02}:
\begin{equation}\label{eq:NinfFat}
\mathcal{N}_\infty\left(\H,\frac{\epsilon}{\sqrt{12 H r}}, n\right) \le \left(\frac{n \ \sqrt{12 H r} B }{\epsilon}\right)^{\mathrm{fat}_\frac{\epsilon}{\sqrt{12 H r}}(\mathcal{H})}~.
\end{equation}

Hence by Equation \eqref{eq:dud} we have:
\begin{align}
\mc{R}_n(\mc{L}_\phi(r)) & \le \inf_{\alpha \ge 0} \left\{ 4 \alpha + 10 \int_{\alpha}^{\sqrt{b r}} \sqrt{\frac{\mathrm{fat}_\frac{\epsilon}{\sqrt{12 H r}}(\mathcal{H}) \log\left(\frac{n \ \sqrt{12 H r} B }{\epsilon}\right)}{n}} d \epsilon \right\}
\intertext{choosing $\alpha = \sqrt{12 H r} \mc{R}_n(\H)$:}
 & \le 4 \sqrt{12 H r}\ \mc{R}_n(\H) + 10 \int_{\sqrt{12 H r} \mc{R}_n(\H)}^{\sqrt{b r}} \sqrt{\frac{\mathrm{fat}_\frac{\epsilon}{\sqrt{12 H r}}(\mathcal{H}) \log\left(\frac{n \ \sqrt{12 H r} B }{\epsilon}\right)}{n}} d \epsilon
\intertext{after a change of integration variable:}
& \le 4 \sqrt{12 H r}\ \mc{R}_n(\H) + 10 \sqrt{12 H r} \int_{\mc{R}_n(\H)}^{\sqrt{\frac{b}{12 H}}} \sqrt{\frac{\mathrm{fat}_\epsilon(\mathcal{H}) \log\left(\frac{n B}{\epsilon}\right)}{n}} d \epsilon
\intertext{bounding the fat-shattering dimension in terms of the
  Rademacher complexity (Lemma \ref{lem:fatrad}):}
& \le 4 \sqrt{12 H r}\ \mc{R}_n(\H) + 20 \sqrt{12 H r}\ \mc{R}_n(\H) \int_{\mc{R}_n(\H)}^{\sqrt{\frac{b}{12 H }}} \frac{\sqrt{\log\left(\frac{n B}{\epsilon}\right)}}{\epsilon} d \epsilon \\
& \le \sqrt{12 H r}\ \mc{R}_n(\H) \left( 4 + 20 \left[- \frac{2}{3}
    \log^{3/2}(n B/\epsilon) \right]_{\mc{R}_n(\H)}^{\sqrt{b/12 H}}
\right) \notag \\
& \le \sqrt{12 H r}\ \mc{R}_n(\H) \left( 4 + 14 \left(
    \log^{3/2}\left(\frac{n B}{\mc{R}_n(\H)}\right) -
    \log^{3/2}\left(\frac{n B \sqrt{12 H}}{\sqrt{b}}\right) \right)
\right) \notag \\
& \le \sqrt{12 H r}\ \mc{R}_n(\H) \left( 18 \log^{3/2}\left(\frac{n
      B}{\mc{R}_n(\H)}\right) - 14 \log^{3/2}\left(\frac{n \sqrt{12 H}
      B}{\sqrt{b}}\right) \right) \qedhere \notag
\end{align}
\end{proof}

\begin{proof}[Proof of Theorem \ref{thm:mainRad}]
  Equipped with Lemma \ref{lem:key}, the proof follows standard Local
  Rademacher arguments.  Applying Theorem 6.1 of \cite{Bousquet02} to
  $\psi_n(r)=56 \sqrt{H r} \log^{1.5} n \hat{\mathcal{R}}_n(\H)$ we
  can show that:
  \begin{equation*}
\L{h} \le \hat{L}(h) + 106 r^*_n  + \frac{48 b}{n}  \left(\log\tfrac{1}{\delta} + \log \log\ n \right) + \sqrt{\hat{L}(h) \left(8 r^{*}_n + \frac{4 b}{n}  \left(\log\tfrac{1}{\delta} + \log \log\ n \right) \right)}
  \end{equation*}
where $r^*_n = 56^2 H \log^3\!n \mc{R}_n(\H)$ is the solution to
$\psi_n(r)=r$.  Further details can be found in the Appendix.
\end{proof}

\subsection{Related Results}

Rates faster than $1/\sqrt{n}$ have been previously explored under
various conditions, including when $L^*$ is small.

\paragraph{The Finite Dimensional Case}

Lee et al \cite{LeeBaWi98} showed faster rates for squared loss,
exploiting the strong convexity of this loss function, even when
$L^*>0$, but only with finite VC-subgraph-dimension.  Panchenko
\cite{Panchenko02} provides fast rate results for general Lipschitz
bounded loss functions, still in the finite VC-subgraph-dimension
case.  Bousquet \cite{Bousquet02} provided similar guarantees for
linear predictors in Hilbert spaces when the spectrum of the kernel
matrix (covariance of $X$) is exponentially decaying, making the
situation almost finite dimensional.  All these methods rely on
finiteness of effective dimension to provide fast rates.  In this
case, smoothness is not necessary.  Our method, on the other hand,
establishes fast rates, when $L^*=0$, for function classes that do
{\em not} have finite VC-subgraph-dimension.  We show how in this
non-parametric case, smoothness is necessary and plays an important
role (see also Table \ref{tab:tight}).

\paragraph{Aggregation}

Tsybakov \cite{Tsybakov04} studied learning rates for aggregation,
where a predictor is chosen from the convex hull of a finite set of
base predictors.  This is equivalent to an $\ell_1$ constraint where
each base predictor is viewed as a ``feature''.  As with
$\ell_1$-based analysis, since the bounds depend only logarithmically
on the number of base predictors (i.e. dimensionality), and rely on
the scale of change of the loss function, they are of ``scale
sensitive'' nature.  For such an aggregate classifier, Tsybakov
obtained a rate of $1/n$ when zero (or small) risk is achieve by one
of the base classifiers.  Using Tsybakov's result, it is not enough
for zero risk to be achieved by an aggregate (i.e.~bounded $ell_1$)
classifier in order to obtain the faster rate.  Tsybakov's core result
is thus in a sense more similar to the finite dimensional results,
since it allows for a rate of $1/n$ when zero error is achieved by a
finite cardinality (and hence finite dimension) class.

Tsybakov then used the approximation error of a small class of
base predictors w.r.t.~a large hypothesis class (i.e.~a covering) to
obtain learning rates for the large hypothesis class by considering
aggregation within the small class.  However these results only imply
fast learning rates for hypothesis classes with very low complexity.
Specifically to get learning rates better than $1/\sqrt{n}$ using
these results, the covering number of the hypothesis class at scale
$\epsilon$ needs to behave as $1/\epsilon^p$ for some $p < 2$.  But
typical classes, including the class of linear predictors with bounded
norm, have covering numbers that scale as $1/\epsilon^2$ and so these
methods do not imply fast rates for such function classes.  In fact,
to get rates of $1/n$ with these techniques, even when $L^*=0$,
requires covering numbers that do not increase with $\epsilon$ at all,
and so actually finite VC-subgraph-dimension.

Chesneau et al \cite{ChesneauLe06} extend Tsybakov's work also to
general losses, deriving similar results for Lipschitz loss function.
The same caveats hold: even when $L^*=0$, rates faster when
$1/\sqrt{n}$ require covering numbers that grow slower than
$1/\epsilon^2$, and rates of $1/n$ essentially require finite
VC-subgraph-dimension.  Our work, on the other hand, is applicable
whenever the Rademacher complexity (equivalently covering numbers) can
be controlled.  Although it uses some similar techniques, it is also
rather different from the work of Tsybakov and Chesneau et al, in that
it points out the importance of smoothness for obtaining fast rates in
the non-parametric case: Chesneau et al relied only on the Lipschitz
constant, which we show, in Section \ref{sec:tight}, is not enough for
obtaining fast rates in the non-parametric case, even when $L^*=0$.

\paragraph{Local Rademacher Complexities}

Bartlett et al \cite{BartlettBoMe05} developed a general machinery for
proving possible fast rates based on local Rademacher complexities.
However, it is important to note that the localized complexity term
typically dominates the rate and still needs to be controlled.  For
example, Steinwart \cite{Steinwart07} used Local Rademacher Complexity
to provide fast rate on the 0/1 loss of Support Vector Machines (SVMs)
($\ell_2$-regularized hinge-loss minimization) based on the so called
``geometric margin condition'' and Tsybakov's margin condition.
Steinwart's analysis is specific to SVMs.  We also use Local
Rademacher Complexities in order to obtain fast rates, but do so for
general hypothesis classes, based only on the standard Rademacher
complexity $\mc{R}_n(\H)$ of the hypothesis classes, as well as the
smoothness of the loss function and the magnitude of $L^*$, but
without any further assumptions on the hypothesis classes itself.

\paragraph{Non-Lipschitz Loss}

Beyond the strong connections between smoothness and fast rates which
we highlight, we are also not aware of prior work providing an
explicit and easy-to-use result for controlling a generic
non-Lipschitz loss (such as the squared loss) solely in terms of the
Rademacher complexity.

\section{Online and Stochastic Optimization of Smooth Convex Objectives}\label{sec:convex}

\iffalse
In the previous section we considered problems where we use smooth
loss and predict from hypothesis class $\H$ and the convergence rates
were provided uniformly for all $h \in \H$ in terms of Rademacher
complexity of the function class $\H$. However while this scenario is
apt for supervised learning problem where Rademacher complexity has to
be necessarily $o(1)$ for learnability, in the general setting for
learning this is not necessarily the case \cite{ShalevShSrSr09a}.
\fi

We now turn to online and stochastic convex optimization.  In these
settings a learner chooses $\ensuremath{\mathbf{w}} \in \ensuremath{\mathbf{W}}$, where $\ensuremath{\mathbf{W}}$ is a closed convex
set in a normed vector space, attempting to minimize an objective
(loss) $\ell(\ensuremath{\mathbf{w}},z)$ on instances $z \in \ensuremath{\mathbb{Z}}$, where
$\ell:\ensuremath{\mathbf{W}}\times\ensuremath{\mathbb{Z}}\rightarrow\ensuremath{\mathbb{R}}$ is an objective function which is
convex in $\ensuremath{\mathbf{w}}$.  This captures learning linear predictors w.r.t.~a
convex loss function $\phi(t,z)$, where $\ensuremath{\mathbb{Z}} = \mathcal{X} \times \mathcal{Y}$ and
$\ell(\ensuremath{\mathbf{w}},(x,y)) = \phi(\ip{\ensuremath{\mathbf{w}}}{x},y)$, and extends well beyond
supervised learning.

We consider the case where the objective $\ell(\ensuremath{\mathbf{w}},z)$ is $H$-smooth
w.r.t.~some norm $\norm{\ensuremath{\mathbf{w}}}$ (the reader may choose to think of $\ensuremath{\mathbf{W}}$
as a subset of a Euclidean or Hilbert space, and $\norm{\ensuremath{\mathbf{w}}}$ as the
$\ell_2$-norm): By this we mean that for any $z \in \ensuremath{\mathbb{Z}}$, and all $\ensuremath{\mathbf{w}},
\ensuremath{\mathbf{w}}' \in \ensuremath{\mathbf{W}}$
$$
\norm{\nabla \ell(\ensuremath{\mathbf{w}},z) - \nabla \ell(\ensuremath{\mathbf{w}}',z)}_* \le H \norm{\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}'}
$$
where $\|\cdot\|_*$ is the dual norm.  The key here is to generalize
Lemma \ref{lemma:smooth} to smoothness
w.r.t.~a vector $\ensuremath{\mathbf{w}}$, rather then scalar smoothness:
\begin{lemma}\label{lemma:gensmooth}
For an $H$-smooth non-negative $f: \ensuremath{\mathbf{W}} \rightarrow
\mathbb{R}$, for all $\ensuremath{\mathbf{w}} \in \ensuremath{\mathbf{W}}$:
$
\|\nabla f(\ensuremath{\mathbf{w}})\|_* \le \sqrt{4 H f(\ensuremath{\mathbf{w}})}
$
\end{lemma}
\begin{proof}
For any $\ensuremath{\mathbf{w}}_0$ such that $\|\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0\| \le 1$, let $g(t) = g(\ensuremath{\mathbf{w}}_0 + t (\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0))$.  For any $t , s \in \mathbb{R}$,
\begin{align*}
|g'(t) - g'(s)|&= |\ip{\nabla f(\ensuremath{\mathbf{w}}_0 + t (\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0)) - \nabla f(\ensuremath{\mathbf{w}}_0 + s (\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0)) }{\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0}|\\
& \le \|\nabla f(\ensuremath{\mathbf{w}}_0 + t (\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0)) - \nabla f(\ensuremath{\mathbf{w}}_0 + s (\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0))\|_* \ \|\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0\|\\
& \le H |t - s| \|\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0\|^2\\
& \le H |t-s|
\end{align*}
Hence $g$ is $H$-smooth and so by Lemma \ref{lemma:smooth}
$
|g'(t)| \le \sqrt{4 H g(t)}
$.
Setting $t = 1$ we have, $\ip{\nabla f(\ensuremath{\mathbf{w}})}{\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0} \le \sqrt{4 H f(\ensuremath{\mathbf{w}})}$.  Taking supremum over $\ensuremath{\mathbf{w}}_0$  such that $\|\ensuremath{\mathbf{w}}_0 - \ensuremath{\mathbf{w}}\| \le 1$ we conclude that
\begin{equation}
\|\nabla f(\ensuremath{\mathbf{w}})\|_* = \sup_{\ensuremath{\mathbf{w}}_0 : \|\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0\| \le 1} \ip{\nabla
  f(\ensuremath{\mathbf{w}})}{\ensuremath{\mathbf{w}} - \ensuremath{\mathbf{w}}_0} \le \sqrt{4 H f(\ensuremath{\mathbf{w}})} \qedhere \notag
\end{equation}
\end{proof}

In order to consider general norms, we will also need to rely on a
non-negative regularizer $F :\ensuremath{\mathbf{W}} \mapsto \mbb{R}$ that is a $1$-strongly
convex (see Definition in e.g.~\cite{Zalinescu02}) w.r.t.~to the norm $\norm{\ensuremath{\mathbf{w}}}$ for all $\ensuremath{\mathbf{w}}\in\ensuremath{\mathbf{W}}$.  For the
Euclidean norm we can use the squared Euclidean norm regularizer: $F(\ensuremath{\mathbf{w}})={\scriptstyle \frac{1}{2}}\norm{\ensuremath{\mathbf{w}}}^2$.

\subsection{Online Optimization Setting}

In the online convex optimization setting we consider an $n$ round
game played between a learner and an adversary (Nature) where at each
round $i$, the player chooses a $\ensuremath{\mathbf{w}}_i \in \ensuremath{\mathbf{W}}$ and then the adversary picks a
$z_i \in \ensuremath{\mathbb{Z}}$.  The player's choice $\ensuremath{\mathbf{w}}_i$ may only depend on the
adversary's choices in {\em previous} rounds.  The goal of the player
is to have low average objective value $\frac{1}{n}\sum_{i=1}^{n}
\ell(\ensuremath{\mathbf{w}}_{i},z_i)$ compared to the best single choice in hind sight \cite{CesaBianchiLu06}.

A classic algorithm for this setting is Mirror Descent
\cite{BeckTe03}, which starts at some arbitrary $\ensuremath{\mathbf{w}}_1\in\ensuremath{\mathbf{W}}$ and
updates $\ensuremath{\mathbf{w}}_{i+1}$ according to $z_i$ and a stepsize $\eta$ (to be discussed
later) as follows:
\begin{equation}
  \label{eq:md}
  \ensuremath{\mathbf{w}}_{i+1} \leftarrow \arg\min_{\ensuremath{\mathbf{w}}\in\ensuremath{\mathbf{W}}} \ip{\eta\nabla
    \ell(\ensuremath{\mathbf{w}}_i,z_i)-\nabla F(\ensuremath{\mathbf{w}}_i)}{\ensuremath{\mathbf{w}}}+F(\ensuremath{\mathbf{w}})
\end{equation}
For the Euclidean norm
with $F(\ensuremath{\mathbf{w}})={\scriptstyle \frac{1}{2}}\!\norm{\ensuremath{\mathbf{w}}}^2$, the update
\eqref{eq:md} becomes projected online gradient descent \cite{Zinkevich03}:
\begin{equation}
  \label{eq:sd}
  \ensuremath{\mathbf{w}}_{i+1} \leftarrow \Pi_{\ensuremath{\mathbf{W}}}(\ensuremath{\mathbf{w}}_i - \eta \nabla \ell(\ensuremath{\mathbf{w}}_i,z_i))
\end{equation}
where $\Pi_{\ensuremath{\mathbf{W}}}(\ensuremath{\mathbf{w}})=\arg\min_{\ensuremath{\mathbf{w}}'\in\ensuremath{\mathbf{W}}}\norm{\ensuremath{\mathbf{w}}-\ensuremath{\mathbf{w}}'}$ is the
projection onto $\ensuremath{\mathbf{W}}$.

\begin{theorem}\label{onlinesmooth}
For any $B \in \mathbb{R}$ and $\overline{L^*}$ if we use stepsize $\eta = \frac{1}{H B^2 + \sqrt{H^2 B^4 + H B^2 n \overline{L^*}}}$ for the Mirror Descent algorithm then for any instance sequence $z_1,\ldots,z_n \in \ensuremath{\mathbb{Z}}$, the average regret w.r.t.~any
  $\ensuremath{\mathbf{w}}^* \in \ensuremath{\mathbf{W}}$ s.t.~$F(\ensuremath{\mathbf{w}}^*)\leq B^2$ and $\tfrac{1}{n}\sum_{j=1}^n
  \ell(\ensuremath{\mathbf{w}}^*,z_i) \le \overline{L^*}$ is bounded by:
\begin{align*}
  \frac{1}{n}\sum_{i=1}^{n} \ell(\ensuremath{\mathbf{w}}_{i},z_i) - \frac{1}{n}\sum_{i=1}^{n}
  \ell(\ensuremath{\mathbf{w}}^*,z_i) \le \frac{4 H B^2}{n} + 2 \sqrt{\frac{H B^2 \overline{L^*}}{n}}
\end{align*}
\end{theorem}
Note that the stepsize depends on the bound $\overline{L^*}$ on the loss in hindsight.
\begin{proof}
  The proof follows from Lemma \ref{lemma:gensmooth} and Theorem 1 of
  \cite{Shalev07}, using $U_1 = B^2$ and $U_2 = n \overline{L^*}$ in
  the Theorem.
\end{proof}

\subsection{Stochastic Optimization I: Stochastic Mirror Descent}

An online algorithm can also serve as an efficient one-pass learning
algorithm in the stochastic setting.  Here, we again consider an
i.i.d.~sample $z_1,\ldots,z_n$ from some unknown distribution (as in
Section \ref{sec:rad}), and we would like to find $\ensuremath{\mathbf{w}}$ with low risk
$L(\ensuremath{\mathbf{w}})=\E{\ell(\ensuremath{\mathbf{w}},Z)}$.  When $z=(\ensuremath{\mathbf{x}},y)$ and
$\ell(\ensuremath{\mathbf{w}},z)=\phi(\ip{\ensuremath{\mathbf{w}}}{\ensuremath{\mathbf{x}}},y)$ this agrees with the supervised
learning risk discussed in the Introduction and analyzed in Section
\ref{sec:rad}.  But instead of focusing on the ERM, we run Mirror
Descent (or Projected Online Gradient Descent in case of a Euclidean
norm) on the sample, and then take $\tilde{\ensuremath{\mathbf{w}}}={\scriptstyle
  \frac{1}{n}}\!\sum_{i=1}^{n}\ensuremath{\mathbf{w}}_i$.  Standard arguments
\cite{CesaBianchiCoGe02} allow us to convert the online regret bound
of Theorem \ref{onlinesmooth} to a bound on the excess risk:
\begin{corollary}\label{cor:sgd}
For any $B \in \mathbb{R}$ and $\overline{L^*}$ if we run Mirror Descent on a random sample with stepsize $\eta =  \frac{1}{H B^2 + \sqrt{H^2 B^4 + H B^2 n \overline{L^*}}}$, then for any $\ensuremath{\mathbf{w}}^* \in \ensuremath{\mathbf{W}}$ with $F(\ensuremath{\mathbf{w}}^*)\leq B^2$ and $L(\ensuremath{\mathbf{w}}^*) \leq \overline{L^*}$, with expectation over the sample:
$$\L{\tilde{\ensuremath{\mathbf{w}}}_n} - \L{\wopt} \leq \frac{4 H B^2}{n} + 2 \sqrt{\frac{H B^2 \overline{L^*}}{n}} .
$$
\end{corollary}
Again, one must know a bound $\overline{L^*}$ on the risk in order to
choose the stepsize.

It is instructive to contrast this guarantee with similar looking
guarantees derived recently in the stochastic convex optimization
literature \cite{Lan09}.  There, the model is stochastic first-order
optimization, i.e. the learner gets to see an unbiased estimate
$\nabla l(\ensuremath{\mathbf{w}},z_i)$ of the gradient of $L(\ensuremath{\mathbf{w}})$. The variance of the
estimate is assumed to be bounded by $\sigma^2$. The expected accuracy
after $n$ gradient evaluations then has two terms: a ``accelerated''
term that is $O(H/n^2)$ and a slow $O(\sigma/\sqrt{n})$ term. While
this result is applicable more generally (since it doesn't require
non-negativity of $\ell$), it is not immediately clear if our
guarantees can be derived using it.  The main difficulty is that
$\sigma$ depends on the norm of the gradient estimates. Thus, it
cannot be bounded in advance even if we know that $L(\wopt)$ is small.
That said, it is intuitively clear that towards the end of the
optimization process, the gradient norms will typically be small if
$L(\wopt)$ is small because of the self bounding property
(Lemma~\ref{lemma:gensmooth}).  Exploring this connection can be
fruitful direction for further research.

\subsection{Stochastic Optimization II: Regularized Batch Optimization}

It is interesting to note that using stability arguments, a guarantee
very similar to Corollary \ref{cor:sgd}, avoiding the polylogarithmic
factors of Theorem \ref{thm:mainRad} as well as the dependence on the
bound on the loss ($b$ in Theorem \ref{thm:mainRad}), can be obtained
also for a ``batch'' learning rule similar to ERM, but incorporating
penalty-type regularization.  For a given regularization parameter
$\lambda > 0$ define the regularized empirical loss as
$$
\hat{L}_\lambda(\ensuremath{\mathbf{w}}) := \hat{L}(\ensuremath{\mathbf{w}}) + \lambda F(\ensuremath{\mathbf{w}})
$$
and consider the Regularized Empirical Risk Minimizer
\begin{equation}\label{eq:regerm}
\hat{\ensuremath{\mathbf{w}}}_{\lambda} = \arg\min_{\ensuremath{\mathbf{w}}\in\ensuremath{\mathbf{W}}} \hat{L}_\lambda(\ensuremath{\mathbf{w}})
\end{equation}
The following theorem provides a bound on excess risk similar to Corollary \ref{cor:sgd}:

\begin{theorem}\label{thm:regerm}
For any $B \in \mathbb{R}$ and $\overline{L^*}$ if we set $\lambda = \frac{128 H}{n} + \sqrt{\frac{128^2 H^2}{n^2} + \frac{128 H \overline{L^*}}{n B^2}}$
then for all $\wopt \in \ensuremath{\mathbf{W}}$ with $F(\wopt)\leq B^2$ and $L(\wopt)\leq
 \overline{L^*}$,  we have that in expectation over sample of
 size $n$:
\begin{equation*
\L{\hat{\ensuremath{\mathbf{w}}}_{\lambda}} - \L{\wopt} \leq \frac{256 H B^2}{n} + \sqrt{\frac{2048 H B^2 \overline{L^*}}{n}} .
\end{equation*}
\end{theorem}

To prove Theorem \ref{thm:regerm} we use stability arguments similar
to the ones used by Shalev-Shwartz et al~\cite{ShalevShSrSr09}, which
are in turn based on Bousquet and Elisseeff~\cite{BousquetEl02}.
However, while Shalev-Shwartz et al~\cite{ShalevShSrSr09} use the
notion of uniform stability, here it is necessary to look at stability
in expectation to get the faster rates (uniform stability does not
hold with the desired rate).

To use stability based arguments, for each $i \in [n]$ we consider a
perturbed sample where instance $z_i$ is replaced by instance $z'_i$
drawn independently from same distribution as $z_i$.  Let
$\hat{L}^{(i)}(\ensuremath{\mathbf{w}}) = {\scriptsize \frac{1}{n}}(\sum_{j\not=i}
\ell(\ensuremath{\mathbf{w}},z_j)+\ell(\ensuremath{\mathbf{w}},z'_i))$ be the empirical risk over the perturbed
sample, and consider the corresponding regularized empirical risk
minimizer $\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda = \arg\min_{\ensuremath{\mathbf{w}}} \hat{L}^{(i)}_\lambda(\ensuremath{\mathbf{w}})$,
where $\hat{L}^{(i)}_\lambda(\ensuremath{\mathbf{w}}) = \hat{L}^{(i)}(\ensuremath{\mathbf{w}}) +\lambda F(\ensuremath{\mathbf{w}})$.  We
first prove the following Lemma on the expected stability of the
regularized minimizer:

\begin{lemma}\label{lem:stability}
For any $i \in [n]$ we have that
$$
\Es{z_1,\ldots,z_n, z'_i}{\ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i) - \ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i) } \le \frac{32 H}{\lambda n} \Es{z_1,\ldots,z_n}{L(\hat{\ensuremath{\mathbf{w}}}_\lambda)}
$$
\end{lemma}
\begin{proof}
\begin{align*}
\hat{L}_\lambda(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda) - \hat{L}_\lambda(\hat{\ensuremath{\mathbf{w}}}_\lambda) & = \frac{\ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i) - \ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i) }{n} + \frac{\ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i') - \ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i') }{n} + \hat{L}^{(i)}_\lambda(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda) - \hat{L}^{(i)}_\lambda(\hat{\ensuremath{\mathbf{w}}}_\lambda)  \\
& \le \frac{\ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i) - \ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i) }{n} + \frac{\ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i') - \ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i') }{n} \\
& \le \frac{1}{n} \|\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda - \hat{\ensuremath{\mathbf{w}}}_\lambda \| \left(\|\nabla \ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i)\|_* + \|\nabla \ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i')\|_*\right) \\
& \le \frac{2 \sqrt{H}}{n} \|\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda - \hat{\ensuremath{\mathbf{w}}}_\lambda \| \left( \sqrt{\ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i)} + \sqrt{\ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i')}\right)
\end{align*}
where the last inequality follows from Lemma \ref{lemma:gensmooth}. By $\lambda$-strong convexity of $\hat{L}_\lambda$ we have that
$$
\hat{L}_\lambda(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda) - \hat{L}_\lambda(\hat{\ensuremath{\mathbf{w}}}_\lambda) \ge \frac{\lambda}{2} \|\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda - \hat{\ensuremath{\mathbf{w}}}_\lambda \|^2.
$$
We can conclude that
$$
\|\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda - \hat{\ensuremath{\mathbf{w}}}_\lambda \| \le \frac{4 \sqrt{H}}{\lambda n}  \left( \sqrt{\ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i)} + \sqrt{\ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i')}\right)
$$
This gives us:
\begin{align*}
\ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i) - \ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i)  &\le \|\nabla \ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i)\|_* \|\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda - \hat{\ensuremath{\mathbf{w}}}_\lambda \| \\
& \le \sqrt{4 H \ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i)} \left(\frac{4 \sqrt{H}}{\lambda}  \left( \sqrt{\ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i)} + \sqrt{\ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i')}\right) \right)\\
& \le \frac{16 H}{\lambda n} \left( \ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i) + \ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i')\right)
\end{align*}
Taking expectation:
\begin{multline}
\Es{z_1,\ldots,z_n, z'_i}{\ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i) - \ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i)}  \le \frac{16 H}{\lambda n} \Es{z_1,\ldots,z_n, z'_i}{ \ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i) + \ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i')} \\
= \frac{16 H}{\lambda n} \Es{z_1,\ldots,z_n, z'_i}{
  \L{\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda}+ \L{\hat{\ensuremath{\mathbf{w}}}_\lambda}} = \frac{32 H}{\lambda n}
\Es{z_1,\ldots,z_n}{ \L{\hat{\ensuremath{\mathbf{w}}}_\lambda}} \notag\qedhere
\end{multline}

\end{proof}

\begin{proof}[Proof of Theorem \ref{thm:regerm}]
By Lemma \ref{lem:stability} we have :
\begin{align*}
\Es{z_1,\ldots,z_n}{L_\lambda(\hat{\ensuremath{\mathbf{w}}}_\lambda) - L_\lambda(\wopt_\lambda)} & \le \Es{z_1,\ldots,z_n}{L_\lambda(\hat{\ensuremath{\mathbf{w}}}_\lambda) - \hat{L}_\lambda(\hat{\ensuremath{\mathbf{w}}}_\lambda)} = \Es{z_1,\ldots,z_n}{L(\hat{\ensuremath{\mathbf{w}}}_\lambda) - \hat{L}(\hat{\ensuremath{\mathbf{w}}}_\lambda)}\\
& = \frac{1}{n} \sum_{i=1}^n \Es{z_1,\ldots,z_n, z'_i}{\ell(\hat{\ensuremath{\mathbf{w}}}^{(i)}_\lambda,z_i) - \ell(\hat{\ensuremath{\mathbf{w}}}_\lambda,z_i) } \le \frac{32 H}{\lambda n} \Es{z_1,\ldots,z_n}{L(\hat{\ensuremath{\mathbf{w}}}_\lambda)}
\end{align*}
Noting the definition of $\hat{L}_\lambda(\ensuremath{\mathbf{w}})$ and rearranging we get
\begin{align*}
\Es{z_1,\ldots,z_n}{L(\hat{\ensuremath{\mathbf{w}}}_\lambda) - L(\wopt)} & \le \frac{32 H}{\lambda n} \Es{z_1,\ldots,z_n}{L(\hat{\ensuremath{\mathbf{w}}}_\lambda)} + \lambda F(\wopt) - \lambda F(\hat{\ensuremath{\mathbf{w}}}_\lambda) \le \frac{32 H}{\lambda n} \Es{z_1,\ldots,z_n}{L(\hat{\ensuremath{\mathbf{w}}}_\lambda)} + \lambda F(\wopt)
\end{align*}
Rearranging further we get
$$
\Es{z_1,\ldots,z_n}{\L{\hat{\ensuremath{\mathbf{w}}}_\lambda}} - \L{\wopt} \le \left(\frac{1}{1 - \frac{32 H}{\lambda n}} - 1\right) \L{\wopt} + \frac{\lambda}{1 - \frac{32 H}{\lambda n}} F(\wopt)
$$
plugging in the value of $\lambda$ gives the result.
\end{proof}

\section{Tightness}\label{sec:tight}

In this Section we return to the learning rates for the ERM for
parametric and for scale-sensitive hypothesis classes (i.e.~in terms
of the dimensionality and in terms of scale sensitive complexity
measures), discussed in the Introduction and analyzed in Section
\ref{sec:rad}.  We compare the guarantees on the learning rates in
different situations, identify differences between the parametric and
scale-sensitive cases and between the smooth and non-smooth cases, and
argue that these differences are real by showing that the
corresponding guarantees are tight.  Although we discuss the tightness
of the learning guarantees for ERM in the stochastic setting, similar
arguments can also be made for online learning.

Table \ref{tab:tight} summarizes the bounds on the excess risk of the
ERM implied by Theorem \ref{thm:mainRad} as well previous bounds for Lipschitz
loss on finite-dimensional \cite{Panchenko02} and scale-sensitive
\cite{BartlettMe02} classes, and a bound for squared-loss on
finite-dimensional classes \cite[Theorem 11.7]{CesaBianchiLu06} that
can be generalized to any smooth strongly convex loss.
\begin{table}[h]
\begin{center}
  \begin{tabular}{| l | c | c |}
    \hline
     & Parametric  & Scale-Sensitive \\
     Loss function is:& $\text{dim}(\H)\leq d$\;\;,\;\; $\abs{h}\leq 1$ & $R_n(\H) \leq \sqrt{R/n}$ \\ \hline
$D$-Lipschitz   & $\frac{d D}{n} + \sqrt{\frac{d D L^*}{n}}$ & $\sqrt{\frac{D^2 R}{n}}$ \\ \hline
$H$-smooth  & $\frac{d H}{n} + \sqrt{\frac{d H L^*}{n}}$ & $\frac{H R}{n} + \sqrt{\frac{H R L^*}{n}}$\\ \hline
$H$-smooth and $\lambda$-strongly Convex& $\frac{H}{\lambda} \frac{d H}{n}$ & $\frac{H R}{n} + \sqrt{\frac{H RL^*}{n}}$ \\
\hline
  \end{tabular}
\vspace{-0.1in}
{\small \caption{\small Bounds on the excess risk, up to
    polylogarithmic factors.}\label{tab:tight}}
\end{center}
\vspace{-0.1in}
\end{table}

We shall now show that the $1/\sqrt{n}$ dependencies in Table
\ref{tab:tight} are unavoidable.  To do so, we will consider the class
$\H = \left\{\ensuremath{\mathbf{x}} \mapsto \ip{\ensuremath{\mathbf{w}}}{\ensuremath{\mathbf{x}}} : \norm{\ensuremath{\mathbf{w}}} \le 1\right\}$ of
$\ell_2$-bounded linear predictors (all norms in this Section are
Euclidean), with different loss functions, and various specific
distributions over $\mathcal{X} \times \mathcal{Y}$, where $\mathcal{X} = \left\{\ensuremath{\mathbf{x}} \in \ensuremath{\mathbb{R}}^d :
  \|\ensuremath{\mathbf{x}}\| \le 1\right\}$ and $Y = [0,1]$.  For the non-parametric
lower-bounds, we will allow the dimensionality $d$ to grow with the
sample size $n$.

\noindent {\bf Infinite dimensional, Lipschitz (non-smooth), separable}\\
Consider the absolute difference loss $\phi(h(\ensuremath{\mathbf{x}}),y) = \abs{h(\ensuremath{\mathbf{x}}) -
  y}$, take $d=2n$ and consider the following distribution: $X$ is
uniformly distributed over the $d$ standard basis vectors $\mbf{e}_i$
and if $X=\mbf{e}_i$, then $Y = {\scriptstyle \frac{1}{\sqrt{n}}}
r_i$, where $r_1,\ldots,r_d \in \{\pm 1\}$ is an arbitrary sequence of
signs unknown to the learner (say drawn randomly beforehand).  Taking
$\wopt = {\scriptstyle \frac{1}{\sqrt{n}}} \sum_{i=1}^n r_i
\mbf{e}_i$, $\norm{\wopt}=1$ and $L^*=\L{\wopt} = 0$. However any
sample $(\ensuremath{\mathbf{x}}_1,y_1), \ldots,(\ensuremath{\mathbf{x}}_n,y_n)$ reveals at most $n$ of $2n$
signs $r_i$, and no information on the remaining $\geq n$ signs.  This
means that for any algorithm used by the learner, there exists a
choice of $r_i$'s such that on at least $n$ of the remaining points
not seen by the learner the learner has to suffer a loss of at least
${\scriptstyle 1/\sqrt{n}}$, yielding an overall risk of at least
$1/(2\sqrt{n})$.

\noindent {\bf  Infinite dimensional, smooth, non-separable, even if strongly convex}\\
Consider the squared loss $\phi(h(\ensuremath{\mathbf{x}}),y) = (h(\ensuremath{\mathbf{x}}) - y)^2$ which is
$2$-smooth and $2$-strongly convex. For any $\sigma \ge 0$ let $d=\sqrt{n}/\sigma$ and consider the following distribution: $X$ is uniform over $\mbf{e}_i$ as before, but
this time $Y|X$ is random, with $Y|(X=\mbf{e}_i) \sim
\mathcal{N}(\frac{r_i}{2\sqrt{d}},\sigma)$, where again $r_i$ are
pre-determined, unknown to the learner, random signs.  The minimizer
of the expected risk is $\wopt = \sum_{i=1}^d {\scriptstyle
  \frac{r_i}{2\sqrt{d}}} \mbf{e}_i$, with
  $\norm{\wopt}={\scriptstyle\frac{1}{2}}$ and $L^*=L(\wopt)=\sigma^2$.
  Furthermore, for any $\ensuremath{\mathbf{w}} \in \ensuremath{\mathbf{W}}$,
$$
\L{\ensuremath{\mathbf{w}}} -\L{\wopt} = \E{\ip{\ensuremath{\mathbf{w}} - \wopt}{\ensuremath{\mathbf{x}}}}^2 =
\frac{1}{d}\sum_{i=1}^d (\ensuremath{\mathbf{w}}[i] - \wopt[i])^2 = \frac{1}{d}\norm{\ensuremath{\mathbf{w}}-\wopt}^2
$$

If the norm constraint becomes tight, i.e. $\norm{\hat{\ensuremath{\mathbf{w}}}}=1$, then
$L(\hat{\ensuremath{\mathbf{w}}})-L({\wopt})\geq 1/(4d)=\sigma/(4\sqrt{n}) = \sqrt{L^*}/(4 \sqrt{n})$.  Otherwise, each
coordinate is a separate mean estimation problem, with $n_i$ samples,
where $n_i$ is the number of appearances of $\mbf{e}_i$ in the
sample.  We have $\E{(\hat{\ensuremath{\mathbf{w}}}[i]-\wopt[i])^2}=\sigma^2/n_i$ and so
$$L(\hat{\ensuremath{\mathbf{w}}})-L^* = \frac{1}{d} \norm{\hat{\ensuremath{\mathbf{w}}}-\wopt}^2 = \frac{1}{d}
\sum_{i=1}^d \frac{\sigma^2}{n_i} \geq \frac{\sigma^2}{d}\frac{d^2}{\sum_i n_i} =
\frac{\sigma^2 d}{n}= \frac{\sigma}{\sqrt{n}} = \sqrt{\frac{L^*}{n}}$$

\noindent {\bf  Finite dimensional, smooth, not strongly convex, non-separable:}\\
Take $d=1$, with $X=1$ with probability $q$ and $X = 0$ with
probability $1-q$. Conditioned $X=0$ let $Y=0$ deterministically,
while conditioned on $X = 1$ let $Y=+1$ with probability $p = \frac{1}{2} +
\frac{0.2}{\sqrt{q n}}$ and $Y=-1$ with probability $1 - p$.  Consider
the following $1$-smooth loss function, which is quadratic around the correct
prediction, but linear away from it:
\begin{align*}
\phi(h(\ensuremath{\mathbf{x}}),y) = \begin{cases}
(h(\ensuremath{\mathbf{x}})-y)^2 &\text{if $\abs{h(\ensuremath{\mathbf{x}})-y}\leq 1/2$} \\
\abs{h(\ensuremath{\mathbf{x}})-y}-1/4 &\text{if $\abs{h(\ensuremath{\mathbf{x}})-y}\geq 1/2$}
\end{cases}
\end{align*}
First note that irrespective of choice of $\ensuremath{\mathbf{w}}$, when $\ensuremath{\mathbf{x}}=0$ and so
$y=0$ we always have $h(\ensuremath{\mathbf{x}})=0$ and so suffer no loss.  This happens
with probability $1-q$.  Next observe that for $p>1/2$, the optimal
predictor is $\wopt \geq 1/2$.  However, for $n > 20$, with probability at
least $0.25$, $\sum_{i=1}^n y_i < 0$, and so the empirical minimizer
is $\hat{\ensuremath{\mathbf{w}}} \leq -1/2$.  We can now calculate
$$L(\hat{\ensuremath{\mathbf{w}}})-L^* > L(-1/2)-L(1/2) = q (2p-1) + (1 - q) 0 =  \frac{0.4\ q}{\sqrt{q n}} = \frac{0.4\ \sqrt{q}}{\sqrt{ n}}.$$
However note that for $p > 1/2$,  $\ensuremath{\mathbf{w}}^* = \frac{3}{2} - \frac{1}{2p}$ and so for $n > 20$:
$$
L^*  > \frac{q}{2}
$$
Hence we conclude that with probability $0.25$ over the sample,
$$L(\hat{\ensuremath{\mathbf{w}}})-L^* > \sqrt{\frac{0.32 L^*}{n}}.$$

\section{Implications}

We demonstrate the implications of our results in several settings.

\subsection{Improved Margin Bounds}

``Margin bounds'' provide a bound on the expected zero-one loss of a
classifiers based on the margin zero-one error on the training sample.
Koltchinskii and Panchenko \cite{KoltchinskiiPa02} provides margin
bounds for a generic class $\H$ based on the Rademacher complexity of
the class.  This is done by using a non-smooth Lipschitz ``ramp'' loss
that upper bounds the zero-one loss and is upper-bounded by the margin
zero-one loss.  However, such an analysis unavoidably leads to a
$1/\sqrt{n}$ rate even in the separable case, since as we discuss in
Section \ref{sec:tight}, it is not possible to get a faster rate for a
non-smooth loss.  Following the same idea we use the following smooth
``ramp'': {\small $$ \phi(t) = \left\{\begin{array}{lc}
      1 & t \le 0 \\
      \frac{1 + \mathrm{cos}(\pi t/\gamma)}{2} & 0 < t < \gamma \\
      0 & t \ge \gamma
\end{array} \right.
$$}
This loss function is $\frac{\pi^2}{4 \gamma^2}$-smooth and is lower
bounded by the zero-one loss and upper bounded by the $\gamma$ margin
loss. Using Theorem \ref{thm:mainRad} we can now provide improved
margin bounds for the zero-one loss of any classifier based on
empirical margin error.  Denote $\mathrm{err}(h) = \E{\ind{h(x) \ne y}}$ the
zero-one risk and for any $\gamma >
0$ and sample $(\ensuremath{\mathbf{x}}_1,y_1),\ldots,(\ensuremath{\mathbf{x}}_n,y_n) \in \mathcal{X} \times \{\pm1\}$
define the $\gamma$-margin empirical zero one loss as
$$
\widehat{\mathrm{err}}_\gamma(h) := \frac{1}{n} \sum_{i=1}^n \ind{y_i h(\ensuremath{\mathbf{x}}_i) < \gamma}
$$
\begin{theorem}\label{thm:margin}
  For any hypothesis class $\H$, with $\abs{h}\leq b$, and any $\delta
  > 0$, with probability at least $1 - \delta$, simultaneously for all
  margins $\gamma > 0$ and all $h \in\H $:
\begin{align*}
\mathrm{err}(h) \le \widehat{\mathrm{err}}_\gamma(h) + K\left( \sqrt{\widehat{\mathrm{err}}_\gamma(h)} \left(\tfrac{\log^{1.5} n }{\gamma}\mc{R}_n(\H)  + \sqrt{\tfrac{ \log(\log(\frac{4b}{\gamma})/\delta)}{n}}\right)+ \tfrac{\log^{3} n }{\gamma^2} \mc{R}_n^2(\H)  + \tfrac{\log(\log(\frac{4 b}{\gamma})/\delta)}{n}\right)
\end{align*}
where $K$ is a numeric constant from Theorem \ref{thm:mainRad}
\end{theorem}
In particular, the above bound implies:
\begin{align*}
\mathrm{err}(h) \le 1.01 \widehat{\mathrm{err}}_\gamma(h) + K\left( \frac{2 \log^{3} n }{\gamma^2} \mc{R}_n^2(\H)  + \frac{2  \log(\log(\frac{4 b}{\gamma})/\delta)}{n} \right)
\end{align*}
where $K$ is an appropriate numeric constant.

Improved margin bounds of the above form have been previously shown
specifically for linear prediction in a Hilbert space (as in Support
Vector Machines) based on the PAC Bayes theorem
\cite{McAllester03,LangfordSh03}.  However these PAC-Bayes based
results are specific to the linear function class.  Theorem
\ref{thm:margin} is, in contrast, a generic concentration-based result
that can be applied to any function class with and yields rates dominated
by $\mc{R}^2(\H)$.

\subsection{Interaction of Norm and Dimension}

Consider the problem of learning a low-norm linear predictor with
respect to the squared loss $\phi(t,z)=(t-z)^2$, where $\mathcal{X} \in \ensuremath{\mathbb{R}}^d$,
for finite but very large $d$, and where the expected norm of $X$ is low.
Specifically, let $X$ be Gaussian with $\E{\norm{X}^2}=B$,
$Y=\ip{\ensuremath{\mathbf{w}}^*}{X}+\mathcal{N}(0,\sigma^2)$ with $\norm{\ensuremath{\mathbf{w}}^*}=1$, and
consider learning a linear predictor using $\ell_2$ regularization.  What
determines the sample complexity?  How does the error decrease as the
sample size increases?

From a scale-sensitive statistical learning perspective, we expect
that the sample complexity, and the decrease of the error, should
depend on the norm $B$, especially if $d \gg B^2$.  However, for any
fixed $d$ and $B$, even if $d \gg B^2$, asymptotically as the number of
samples increase, the excess risk of norm-constrained or
norm-regularized regression actually behaves as $L(\hat{\ensuremath{\mathbf{w}}}) - L^*
\approx \frac{d}{n} \sigma^2$, and depends (to first order) only on
the dimensionality $d$ and not at all on $B$ \cite{LiangBaBoJo10}.
How does the scale sensitive complexity come into play?

The asymptotic dependence on the dimensionality alone can be
understood through Table \ref{tab:tight}.  In this non-separable situation,
parametric complexity controls can lead to a $1/n$ rate, ultimately
dominating the $1/\sqrt{n}$ rate resulting from $L^*>0$ when
considering the scale-sensitive, non-parametric complexity control
$B$.  (The dimension-dependent behavior here is actually a bit better
then in the generic situation---the well-posed Gaussian model allows
the bound to depend on $\sigma^2=L^*$ rather then on
$\sup(w'x-y)^2 \approx B^2+\sigma^2$).

Combining Theorem \ref{thm:regerm} with the asymptotic $\frac{d}{n}\sigma^2$
behavior, and noting that at the worst case we can predict using a
zero vector, yields the following overall picture on the expected
excess risk of ridge regression with an optimally chosen $\lambda$:
$$
L(\hat{\ensuremath{\mathbf{w}}}_{\lambda})-L^* \leq O \left( \min\left( B^2 , \frac{B^2}{n} +  \frac{B\sigma}{\sqrt{n}} , \frac{d \sigma^2}{n} \right) \right)
$$
Roughly speaking, each term above describes the
behavior in a different regime of the sample size:
\begin{itemize}
\item The first (``random'') regime until $n = \Theta(B^2)$ where the
  excess is is $B^2$.
\item The second (``low-noise'') regime, where the excess risk is dominated
  by the norm and behaves as $B^2/n$, until $n = \Theta(B^2/\sigma^2)$
  and $L(\hat{\ensuremath{\mathbf{w}}}) = \Theta(L^*)$.
\item The third (``slow'') regime, where the excess risk is controlled
  by the norm and the approximation error and behaves as
  $B\sigma/\sqrt{n}$, until $n = \Theta(d^2 \sigma^2/B^2)$ and
  $L(\hat{\ensuremath{\mathbf{w}}})=L^*+\Theta(B^2/d)$.
\item the fourth (``asymptotic'') regimes, where the excess risk is
  dominated by the dimensionality and behaves as $d/n$.
\end{itemize}
This sheds further light on recent work on this phenomena by Liang and
Srebro based on exact asymptotics of simplified situations
\cite{LiangSr10}.

\subsection{Sparse Prediction}

The use of the $\ell_1$ norm has become very popular for learning
sparse predictors in high dimensions, as in the LASSO.  The LASSO
estimator \cite{Tibshirani96b} $\hat{\ensuremath{\mathbf{w}}}$ is obtained by considering
the squared loss $\phi(z,y)=(z-y)^2$ and minimizing $\hat{L}(\ensuremath{\mathbf{w}})$
subject to $\|\ensuremath{\mathbf{w}}\|_1 \le B$.  Let us assume there is some (unknown)
sparse reference predictor $\ensuremath{\mathbf{w}}^0$ that has low expected loss and
sparsity (number of non-zeros) $\norm{\ensuremath{\mathbf{w}}^0}_0=k$, and that
$\|\ensuremath{\mathbf{x}}\|_\infty \le 1, y\le 1$.  In order to choose $B$ and apply
Theorem~\ref{thm:mainRad} in this setting, we need to bound
$\norm{\ensuremath{\mathbf{w}}^0}_1$.  This can be done by, e.g., assuming that the
features $\ensuremath{\mathbf{x}}[i]$ {\em in the support of $\ensuremath{\mathbf{w}}^0$} are mutually
uncorrelated.  Under such an assumption, we have: $\norm{\ensuremath{\mathbf{w}}^0}_1^2 \le
k\E{\ip{\ensuremath{\mathbf{w}}^0}{x}^2} \leq 2k(L(\ensuremath{\mathbf{w}}^0)+\E{y^2}) \le 4k$. Thus,
Theorem~\ref{thm:mainRad} along with Rademacher complexity bounds from
\cite{KakadeSrTe08} gives us,
\begin{equation}\label{eq:LASSObound}
L(\hat{\ensuremath{\mathbf{w}}}) \le L(\ensuremath{\mathbf{w}}^0) + \tilde{O}\left(
\frac{k\,\log(d)}{n} + \sqrt{\frac{ k\,L(\ensuremath{\mathbf{w}}^0)\,\log(d)}{n}} \right).
\end{equation}
It is possible to relax the no-correlation assumption to a bound on
the correlations, as in mutual incoherence, or to other weaker
conditions \cite{ShalevSrZh09_tech}.  But in any case, unlike typical analysis for
compressed sensing, where the goal is recovering $\ensuremath{\mathbf{w}}^0$ itself, here
we are only concerned with correlations {\em inside the support of
  $\ensuremath{\mathbf{w}}^0$}.  Furthermore, we do not need to require that the optimal
predictor is sparse or close to being sparse, or that the model is
well specified: only that there exists a good (low risk) predictor
using a small number of fairly uncorrelated features.

Bounds similar to \eqref{eq:LASSObound} have been derived using
specialized arguments
\cite{Koltchinskii09,VanDeGeer08,bickel2009simultaneous}---here we
demonstrate that a simple form of these bounds can be obtained under
very simple conditions, using the generic framework we suggest.

It is also interesting to note that the methods and results of Section
\ref{sec:convex} can also be applied to this setting.  But since
$\norm{\ensuremath{\mathbf{w}}}^2_1$ is {\em not} strongly convex with respect to
$\norm{\ensuremath{\mathbf{w}}}_1$, we must instead use the entropy regularizer
\begin{equation}
  \label{eq:Fent}
  F(\ensuremath{\mathbf{w}}) = B \sum_i \ensuremath{\mathbf{x}}[i] \log \left(\frac{\ensuremath{\mathbf{x}}[i]}{1/d}\right) + \frac{B^2}{e}
\end{equation}
which {\em is} non-negative and 1-strongly convex with respect to
$\norm{\ensuremath{\mathbf{w}}}_1$ on $\ensuremath{\mathbf{W}} = \left\{ \ensuremath{\mathbf{w}} \in \ensuremath{\mathbb{R}}^d \middle| \ensuremath{\mathbf{w}}[i]\geq 0,
  \norm{\ensuremath{\mathbf{w}}}_1 \leq B \right\}$, with $F(\ensuremath{\mathbf{w}}) \leq B^2 (1+\log d)$ (we
consider here only non-negative weights---in order to allow $\ensuremath{\mathbf{w}}[i]<0$
we can include also each features negation, doubling the
dimensionality).  Recalling that $\norm{\ensuremath{\mathbf{w}}^0}_1 \leq 2\sqrt{k}$ and
using $B=2\sqrt{k}$ in \eqref{eq:Fent}, we have from Theorem
\ref{thm:regerm} we that:
\begin{equation}\label{eq:entbound}
L(\hat{\ensuremath{\mathbf{w}}}_{\lambda}) \le L(\ensuremath{\mathbf{w}}^0) + O\left(
\frac{k\,\log(d)}{n} + \sqrt{\frac{ k\,L(\ensuremath{\mathbf{w}}^0)\,\log(d)}{n}} \right).
\end{equation}
where $\hat{\ensuremath{\mathbf{w}}}_{\lambda}$ is the regularized empirical minimizer
\eqref{eq:regerm} using the entropy regularizer \eqref{eq:Fent} with
$\lambda$ as in Theorem \ref{thm:regerm}.  The advantage here is that
using Theorem \ref{thm:regerm} instead of Theorem \ref{thm:mainRad}
avoids the extra logarithmic factors (yielding a clean big-$O$
dependence in \eqref{eq:entbound} as opposed to big-$\tilde{O}$ in
\eqref{eq:LASSObound}).

More interestingly, following Corollary \ref{cor:sgd}, one can use
stochastic mirror descent, taking steps of the form \eqref{eq:md} with
the entropy regularizer \eqref{eq:Fent}, to obtain the same performance
guarantee as in \eqref{eq:entbound}.  This provides an efficient,
single-pass optimization approach to sparse prediction as an
alternative to batch optimization with an $\ell_1$-norm constraint,
and yielding the same (if not somewhat better) guarantees.

\bibliographystyle{plain}

redpajama_set_name: RedPajamaArXiv


text:
class CreateSpreeMakerSearch < ActiveRecord::Migration
  def change
    create_table :spree_add_on_makers do |t|
      t.string     :name
      t.datetime   :deleted_at
      t.timestamps
    end
    create_table :spree_add_on_maker_products do |t|
      t.integer    :product_id
      t.integer    :add_on_maker_id
      t.integer    :position
      t.timestamps
    end
    create_table :spree_add_on_maker_translations do |t|
      t.integer    :add_on_maker_id
      t.string     :locale
      t.string     :name
      t.timestamps
    end
  end
end


We are delighted to report that after our recent appointment as the TV buying agency for Windsor Mint, their first TV campaign with us has been reported as their most successful ever.
Mostly Media planned and bought the Queen Elizabeth II 90th birthday campaign which ran in April, May and June showcasing Windsor Mint's commemorative strike's which celebrated the event.
The campaign ran across ITV, SKY, Channel 4 and Turner channels using a mixture of 30 and 60 second commercials.
"It's always nice to receive such great feedback from one of our campaigns but even more so for a new client. With our huge agency experience in DRTV we were confident that our suggested strategy for Windsor Mint would bear fruit and of course it's lovely when a media plan comes together.
The success of our campaign for Windsor Mint further enhances our reputation as a TV buying agency not only with a proud track record for consistently delivering ROI, but also for increasing return from existing TV budgets".