No matter what type of business you are starting, what matters most are the people you bring on board to execute your vision. When we see entrepreneurs come up with great ideas (and we see many), we hope that they also remember that while an idea is key, it won't go anywhere without the right team in place. So, today we'd like to highlight a few main points to consider when building your startup team.
As Malcolm Gladwell says, "nobody accomplishes success by themselves." As entrepreneurs we need to recognize the importance of having a solid support system. Large corporations have advisory teams to ensure that they stay on track; until you grow your empire, you need to create your own "advisory" team in-house. Here we list three aspects of building a startup team that are essential to the success of your organization: Team Characteristics, Team Roles, and Finding Your Team.
There are three essential attributes that members of your team must have: talent, passion and dedication.
Each team member needs to have talent – they all need to own their abilities and focus on their assets. Your team does not need to be comprised of jacks-of-all-trades. What you need are individual team members who are masters of their own craft. But while you are small and lean, take advantage of everyone skills; if someone on your team is a great writer, have them take charge of your press releases and marketing content. Just like in basketball, there are the various positions because each team member has their own set of skills. Find your forward and your point guard.
Passion: Each team member needs to be passionate about what he or she is doing and the goal of the project. If your developer knows ruby-on-rails but doesn't care about the application you are creating, then there is no guarantee that their work will be as precise or consistent as you need – or that they will go the extra mile. People do their best work when they care about what they are doing. The passion must be there to excel. Startup's can be stressful and there may be many times when people want to quit. If your team members are not passionate about the goal of the organization, what's going to keep them from leaving?
Dedication: At a typical organization, dedication of its employees is important. But at a startup, it's critical. Members of the team must be dedicated to the mission and be willing to give what it takes in order to succeed – and feel that each member is just as dedicated as they are. A new business hasn't proven itself to anyone yet – so things like passion and dedication must shine through to make up for lack of history. And just like passion, dedication ensures that your deliverables will be met and your startup has a better chance of being funded. No investor will support an entrepreneur who is not full-heartedly dedicated to his or her project.
There are six main roles that most new organizations need to consider in some capacity. In a new venture, it's typical that each team member will likely wear multiple hats (i.e. be the manager and product developer). Just make sure that at any given time, each person knows what they are responsible for. As you grow, think about the right timing for bringing on more people with specific roles, but remember – hire slow, fire fast. As Tony Hsieh of Zappos pointed out during his DCWEEK visit, bad hires have accounted for millions of lost dollars through the years. And for a startup, that lost time, money and energy can be fatal.
Leader/Project Manager – Someone in the organization must drive the plan and keep people focused on a daily basis. This person should be able to lead the group and ensure that all aspects of the project are covered.
Developer – Most likely your product requires some engineering. Make sure you have the right developer(s) on board in whatever capacity you need them, whether it's a part time HTML or WordPress developer or a full time PHP wiz.
Designer – Good design can make or break a product so make sure you have someone who can think about and drive all design aspects of your business – from brand/logos, the appearance of the product and marketing materials.
Legal Advisor – There are lots of things that come up on a regular basis that you'll want to run by someone with real legal knowledge. While you don't want to overspend on legal, it's important to protect your business and understand your risks.
Business/Marketer – Your business won't go very far if no one on the team is good at talking about it. Make sure someone is thinking about overarching plans around PR, marketing and sales and spends time crafting the right messages, understanding audiences and figuring out distribution channels.
Accountant – Someone needs to pay attention to the money. This may sound funny as many startups don't make much money at first, but you should have someone managing the books, paying attention to important tax dates and more. This solid foundation will make things much easier down the road.
In some cases, if you are looking to grow and be able to spread some of the responsibilities, finding a duplicate of yourself might make the most sense. So if you do find your duplicate, hire them as soon as possible.
This can be the most frustrating part of building a team – finding the right people. Unless you're already funded, you probably aren't going to hire a recruiter. We've seen teams come together in many unexpected ways, but here are a couple you should always consider.
Personal Network: Ask your friends and colleagues for suggestions. Send them a note about what you are looking for and ask them to pass it around to their network. This extends to your online social networks too – don't be afraid to Tweet out what you are looking for.
Job Boards: Post a job on various boards describing what you are looking for. Look at the job boards for people who are looking for work and hold interviews. Craigslist is a great place to start.
Networking: Get out there and meet people. Go to lots of events where the type of people you are looking for hang out. If you need a great Drupal developer, find a local Drupal meetup group. There are meetup groups for everything so look around and be creative. We've seen lots of people find their cohorts at our TECH cocktail startup mixer events, so we know this works.
Friends: If you hang with a tech crowd, you may be tempted to just hire your friends. If you consider this, just go into it with caution, understanding that a friend is not necessarily the best co-worker and they still need to fit all the criteria above. You don't want to lose a friendship over an opportunity gone bad.
Past teams: If you've worked within a successful team before, try to keep the magic alive. One of the best teams we've seen is a small group of guys who have built 4 startups together. They have amazing chemistry and already have deep trust in one another.
Paper isn't proof: The right team is worth more than just the sum of all its parts and how everyone works together, communicates, grows and supports each other is crucial.
We hope some of these tips are helpful -and if you have other words of advice from your own experience, please leave them in the comments below.


Wiccans can be polytheist, animist, pantheist, monist, duotheist, atheist/archetypalist, or "all of the above depending on the day". Most Pagans believe that the divine is, or deities are, immanent in the world; and that includes most Wiccans.
This theological diversity works in ritual settings as long as everyone can "translate in their head" and have a certain amount of flexibility as to practice and the wording of rituals.
That's enough apple pie metaphors… let's get down to brass tacks.
It's good to have descriptions of what a word means, so that labels are mutually comprehensible. It's also quite nice when the meaning of a word bears some vague resemblance to its etymology. But there's a conflict between creating a meaning that is inclusive enough to include the majority of people who want to identify as that label, and making a word completely meaningless.
A definition is a fairly precise meaning or set of meanings that are generally agreed usage(s) of a word and what it denotes.
However, language usage is fluid and changeable, and different groups of people use words differently in different contexts. That's why it is a good idea to examine the connotations of a word, so that we can describe how it is used in different contexts.
Examples of words that have highly fluid — and thus highly disputed — meanings: Pagan, polytheist, and Wiccan.
Part of the reason that these words are disputed is because the dictionary definitions are largely unhelpful and out of date.
Why are the meanings disputed?
If a group of people wants to describe its practice, beliefs, and values as distinct from those of another group, it becomes helpful to have a name that describes only that group, and is not in use by another group. This is why the various denominations of Christianity have created labels to distinguish themselves from each other. It's why there are umpteen different varieties of witchcraft, Druidry, and Heathenry. You can recognise some common factor that makes them fit in their respective categories; but there's enough difference between them that it is worth adding a qualifier to the label.
The word "Wiccan" has a fairly chequered history. Gerald Gardner referred to all witches as "the Wica". Charles Cardell described his group as "Wiccens". Gradually, in the USA, Wicca came to refer to any Wiccan or "Wiccanish" tradition. In the UK, it tends to refer to Gardnerian and Alexandrian Wiccans — but many people identify as Wiccan who have never been initiated into those traditions. (Part of the reason for this is that it became very difficult to identify as a witch during the late 1980s and early 1990s, and Wiccan became a handy euphemism for witch.) The word "Wicca" has become so broad and confusing that it may be impossible to restrict its meaning to Gardnerian and Alexandrian Wiccans.
There are also other witchcraft traditions (in the UK, Canada, Australia, and the USA) such as the 1734 tradition, the Clan of Tubal Cain, Feri, Reclaiming, and so on. Most of them are initiatory. Fortunately, hardly anyone disputes that the word "witch" applies to all these different traditions.
To my mind, polytheism just means "many gods" or "belief in many gods". It doesn't say anything about how you worship them, or what type of rituals you perform to get in touch with them. Some people want to define polytheism as "religious regard for many gods" (in order to exclude those who acknowledge that gods exist but don't have any truck with them – but I think that is redundant, as even if a Christian acknowledges that our gods exist in some way, they don't acknowledge them as gods, so their view is irrelevant to the definition of the term).
If you want to describe a particular way that people interact with the gods, or a particular concept of what they are, then I would argue that you need a qualifying adjective. Various qualifying adjectives have been suggested (hard polytheism, soft polytheism, devotional polytheism, relational polytheism, Jungian or archetypalist polytheism, monistic polytheism, henotheistic polytheism, mystical polytheism), not in order to split polytheism as a whole, but to provide more accurate descriptions of how people relate to the gods.
add a qualifying adjective to distinguish your usage of the term from theirs.
Should you have a specific patron deity?
Can you choose your patron deity, or should they choose you?
Do we serve the gods, or are they allies?
Can we ever know the true nature of the gods?
Who has the authority to answer these questions for others?
It is probably not possible to "win" one of these arguments, or answer the question to everyone's satisfaction. But there is always someone who wants their definition of a concept to be the only valid definition, and to be a gatekeeper of who gets to identify as a particular label. Being a gatekeeper or the person who gets to define a term is a position of power and control, potentially with authority attached to it.
Whoever gets to define or describe what polytheism is will have a huge influence on its future development. If it is a broad-brush movement with many different ways to be polytheist, it will become large, nebulous, and hard to control. If it is narrowly defined, it will be much smaller, but possibly easier to control. And it will end up excluding people whose insights, ideas, and practices might have been valuable to it.
My own position is that I don't want to control anything. I am inherently distrustful of authority (including any authority that I myself may accidentally have acquired). Any authority should have checks and balances with it. If you are the high priestess of a coven, or the leader of a religious tradition, there should be a process for consultation and establishing consensus, and in large groups, for democracy and accountability. For example, in the Inclusive Wicca Discussion Group that I founded on Facebook, I created a set of group guidelines and invited members to vote on them and add to them; and there is more than one moderator for the group. In my coven, we take it in turns to write rituals, so everyone gets the kind of ritual they like; and whoever has written the ritual is the facilitator for that ritual. Whilst I am the most experienced witch in the coven, so the buck does tend to stop with me, I do try to empower others. The process of teaching and learning that we use is all about sharing ideas.
If you want to create a sub-tradition of polytheism that has a set of beliefs, practices, and values that meet your expectations and requirements, that's fine. But don't try to label it as the One True Way of polytheism. You will need to give it a more specific name. Some have argued that if polytheism is seen as a catch-all term that includes soft polytheists, archetypalists, and so on, then it becomes a less useful term. Maybe so, but that's just how language and terminology work.
That's why Niki Whiting proposed the term 'relational polytheism', and others have proposed other qualifying adjectives: to be clear about how our polytheism works out in practice and in context. Similarly, there are different flavours of Wicca and witchcraft, each with their own label, to enable people to find a flavour of Wicca that's right for them.
People are confusing denotation with connotation, as often happens when the meaning of a term is contested. The term polytheism denotes 'many gods'. To a devotional polytheist, that has connotations of devotion, religious regard, and so on. To a relational polytheist, the connotations are forming relationships with the gods. In order for different groups of people to find the polytheists they want to hang out with, we need those qualifying adjectives so that everyone who honours many gods can call themselves polytheists, without insisting on a particular definition of a god, but if you want to be more precise about how you want to honour the gods, or what you think gods are, then use a qualifying adjective.
The alternative is that a tiny group of people get to define what polytheism is and who gets to call themselves polytheist, till the whole thing turns into a clique and everyone else loses interest.
As Bekah Evie Bel points out, polytheism isn't yours. And it's not mine either. It belongs to everyone.
The Brewery of the Gods!
It is well known from the lore that the gods actually have a brewery, not a bakery.
It's true that Iðunn's golden apples keep the Aesir ever-young… but on Mount Olympos, the tipple of choice is nectar and ambrosia. The devas of Hindu mythology drink Amrita, a related substance.
Norse mythology tells of Aegir, the ale brewer of the gods, who held a big party for honored guests every winter. The party was held inside a great hall whose floor was littered with glittering gold, providing enough light that no fires were necessary for illumination. The special beer for the event was brewed in a giant cauldron given to him by Thor and served in magical cups that refilled as soon as they were empty. He even had a couple of loyal servants who distributed food and otherwise cared for the guests' needs. The shindig was the highlight of the social season and all the gods attended. However, like so many off-campus college parties, alcohol and animosity could sometimes spoil a perfectly good evening.
In Greek mythology, Bacchus was the god of wine; in Roman mythology, Dionysos had that role.
Hellenistic mosaics discovered in 1962 close to the city of Paphos depicting Dionysos, god of wine. – Photo by Georgeg, Public Domain.
There was also the time when Oðinn stole the mead brewed from the blood of Kvasir from under the mountain where it had been hidden by Suttung.
Alcohol was clearly important and sacred to our ancestors. It's a shame that we merely abuse it, instead of using it in a sacred manner (like other drugs that ought to be used as a sacrament).
The Taplow drinking horns. Photo by unforth – Flickr.com, CC BY-SA 2.0.
The brewing of alcohol seems rather magical – taking some ingredients that are not intoxicating, mixing them together, leaving them to ferment, and thereby producing a drink that can transform your perceptions – if used carefully and sparingly.
It's quite exciting to put fruit, sugar, water, and yeast in a demijohn and watch it blooping and bubbling away as it ferments. There's nothing quite like the taste of home-made wine, especially if you gathered the ingredients yourself from the hedgerows. It's like alchemy!
Many people maintain that alcohol is sacred because it has yeast in it and has been fermented, which makes it alive somehow.
If you have some form of alcoholic sacrament at the end of your ritual (cakes and wine if you're Wiccan, or a sharing of mead if you're a Heathen), there may be some people who are recovering alcoholics who cannot partake. Maybe you could share some other fermented drink, like drinkable yoghurt? (Personally I don't think this is cultural appropriation unless you steal the ritual that goes with it. YMMV.) Other solutions to this that have been suggested are having a non-alcoholic alternative (but some people prefer the symbolism of everyone drinking from the same cup / horn / chalice).
Alcohol is sacred and powerful, and there are many different types and flavours of alcohol available. Some people prefer mead; others prefer wine, or cider, or beer, or lager. Some people say "I don't like beer", but do not realise the huge variety of beers that are available. Some people call alcoholic drinks by different names – cider in the UK, hard cider in the US; ale, beer, lager, porter, stout… If you ask me what my favourite drink is, I'd have to say, all of the above, depending on the day and my mood.
Religion is also sacred and powerful, and there are many different types and flavours of religion available. Some people prefer Heathenry; others prefer Wicca, or Druidry, or eclectic Paganism, or Asatru. Some people say "I don't like religion" but don't realise the huge variety of religious experiences that are available. And some people call religious experiences by different names – polytheism, hard polytheism, relational polytheism, devotional polytheism, pantheism, mysticism, and so on. Some people, when presented with a list of theological perspectives, will say, "all of the above, depending on the context, the day, and my mood".
As this excellent post by Bekah Evie Bel, Polytheism Isn't Yours, points out, you could just ask people what they mean by the label they are using, instead of assuming that they mean the same by it as you do, and then being disappointed when it turns out that they don't.
So if someone shares their surface with you and they say, "I am a polytheist" try not to make any assumptions. Ask for clarification if you are interested by this surface, move on if you aren't interested. Clarification (usually expanded labels) will tell you how much deeper you want to go. But if you don't ask for clarification, if you don't seek to go even a bit below the surface then it's not that persons fault if you make a stupid assumption.
Just like you read the list of ingredients on food and drink to make sure there's nothing that you're allergic to.
Thing is, it's in the nature of language that people will use words to mean something slightly different, or even wildly different – so labels should be used as a way to start a conversation, not as a substitute for a conversation.
This post was inspired by a comment by Bekah Evie Bel on my previous post which mentioned (hard) cider. The metaphorical aspect of it is entirely my fault.
The way things have been going lately, anyone would think that Eris had lobbed an apple pie into the middle of the Patheos Pagan channel. There are so many apple and apple pie related posts, it's hard to keep track of them all. But let's keep the discussion civil.
What Eris teaches us is that sometimes throwing all the pieces up in the air to see where they land is a good thing. It's very uncomfortable while it's happening, but it is necessary. At the moment, polytheists are going through a phase of throwing everything up in the air to see where it lands (or perhaps it's an awkward adolescence). Let's just take care that it doesn't land on someone and squash them when the dust settles.
Metaphors are sometimes useful. But there's a difference between a metaphor and an analogy.
A metaphor is applicable to a situation but can be interpreted in a number of different ways. A classic metaphor is "My love is like a red red rose" (Robert Burns). If you try to turn this into an analogy, it doesn't work. Robert Burns is not saying that his love has thorns and a stem and the petals fall off. He is saying that his love evokes the same feelings as a red rose (beautiful, sensuous, smells nice). So those qualities of the rose are transferable to the experience of his love; the rest of the rose's attributes are ignored.
An analogy is an exact mapping of one thing to another thing. For example, the solar system is often used as an analogy for atoms (it's not exactly how atoms work, but it's a good way to teach kids about atoms). The electrons orbit around the nucleus. The planets orbit around the sun. There's a direct mapping of all the features of the two systems being compared.
I see John Beckett's Bakery of the Gods as a metaphor, not as an analogy.
What's wrong with chocolate cake?
I am assuming that in John Beckett's Bakery of the Gods metaphor, the people selling chocolate cake were Wiccans and Wiccanish Pagans. I like chocolate cake and Wicca. I am not so keen on chocolate cake with not enough chocolate in it, but each to their own. This metaphor, however, implies that you can't mix Wicca and polytheism (or maybe I am reading too much into it). That's the problem with vague metaphors, they can mean all sorts of things that may not have been intended. I wouldn't mix chocolate cake with apples – but you most definitely can mix polytheism and Wicca.
If my view of polytheism is different from yours, that's a good thing – it means that more flavours of polytheism are available; and that's helpful. Some people like apple pie with cinnamon; others like it with shortcrust pastry, or puff pastry, or less sugary; others still don't even like apple pie; some people maintain that desserts are bad for you. There are many desserts available, and many flavours of polytheism (none of which are the One True Flavour).
None of us know objectively what the nature of the gods is; we only perceive them with our limited, local, and finite perspective. It is interesting to discuss their nature and how we interact with them, so that we can learn from each other's perspectives. But we can't be certain what the nature of the gods is.
The only way to discover whether one perspective is better than another is by observing its results in the world. If your perspective makes you feel closer to the gods, happy, fulfilled, and able to function effectively as a human being without harming others, then it is probably worth sharing. If your perspective makes you angry, bitter, jealous, and vengeful, then it probably isn't doing you or anyone else any good.
And, here's the rub: apple pie with cinnamon makes me say "Yuk!" but for someone else, it may be the only way to make apple pie. That's just fine, as long as I don't make them eat my recipe, and they don't make me eat their recipe.
John Beckett – The gods are like apples.
There are many different varieties of apple (Granny Smith, Bramley, Russet, etc) but they are all apples. They are also distinguishable from oranges.
However, deciding where the tree ends and the apple begins – or whether it's still an apple when you've eaten it – is difficult.
Eris's golden apple was inscribed "To the fairest" – and beauty is in the eye of the beholder. In other words, everyone's experience of the gods is unique.
Everyone has a different experience of trees and apples, but what matters is that there are many stories and many perspectives.
Various commenters have also pointed out that they are allergic to apple pie, or prefer rhubarb pie, or pear pie, or apple crumble.
Did I omit your apple / apple-pie post? Let me know in the comments.


text:
\subsubsection*{\bibname}}

\usepackage{float}
\usepackage{subcaption}
\usepackage{amsmath}
\usepackage{algorithmic}
\usepackage{algorithm}
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage{comment}
\usepackage{graphicx}
\usepackage{graphics}
\usepackage{color}
\usepackage{epstopdf}

\include{defs}

\begin{document}

\twocolumn[

\aistatstitle{Logarithmic Regret for Online Gradient Descent Beyond Strong Convexity}

\aistatsauthor{Dan Garber}

\aistatsaddress{Technion - Israel Institute of Technology}

]

\begin{abstract}
Hoffman's classical result gives a bound on the distance of a point from a convex and compact polytope in terms of the magnitude of violation of the constraints. Recently, several results showed that Hoffman's bound can be used to derive strongly-convex-like rates for first-order methods for \textit{offline} convex optimization of curved, though not strongly convex, functions, over polyhedral sets.
In this work, we use this classical result for the first time to obtain faster rates for \textit{online convex optimization} over polyhedral sets with curved convex, though not strongly convex, loss functions. We show that under several reasonable assumptions on the data, the standard \textit{Online Gradient Descent} algorithm guarantees logarithmic regret. To the best of our knowledge, the only previous algorithm to achieve logarithmic regret in the considered settings is the \textit{Online Newton Step} algorithm which requires quadratic (in the dimension) memory and at least quadratic runtime per iteration, which greatly limits its applicability to large-scale problems. In particular, our results hold for  \textit{semi-adversarial} settings in which the data is a combination of  an arbitrary  (adversarial) sequence and a stochastic sequence, which might provide reasonable approximation for many real-world sequences, or under a natural assumption that the data is low-rank. We demonstrate via experiments that the regret of OGD is indeed comparable to that of ONS (and even far better)  on curved though not strongly-convex losses.
\end{abstract}

\section{Introduction}
The celebrated \textit{Online Gradient Descent} algorithm (OGD), originally due to \cite{Zinkevich03}, is a natural adaptation of the classical projected (sub)gradient descent algorithm for \textit{offline} convex optimization, to the setting of \textit{Online Convex Optimization} \cite{Hazan16,SSS12}. The benefits of OGD are two folded: (i) in many problems of interest it performs very efficient iterations, which can often be executed in linear time (in the dimension), and (ii) it often guarantees optimal regret rates, mainly in terms of the length of the sequence $T$, e.g., $\sqrt{T}$ regret for arbitrary convex loss functions, and $\log{T}$ regret in case all loss functions are strongly-convex \cite{AHK07}.

However, there exists a highly-important and wide class of loss functions, known as \textit{exp-concave losses} \cite{AHK07}, for which OGD does not guarantee optimal regret (in terms of $T$). For instance, the family of exp-concave losses capture important problems such as online linear regression with the square loss and online LASSO, online logistic regression, online portfolio selection, and more.
While for exp-concave losses, OGD only guarantees regret that scales like $\sqrt{T}$, it is known that an online algorithm known as \textit{Online Newton Step} (ONS), originally due to \cite{AHK07}, guarantees $\log{T}$ regret (see also recent work \cite{Luo6} which gives an improved variant in terms of runtime and regret bound for low-rank data). On the downside, while OGD applies very efficient iterations in terms of runtime and memory requirements (e.g., when computing the gradient vector of the loss function and projecting onto the feasible set is computationally-cheap), ONS requires quadratic memory and to solve a linear system on each iteration (which requires at least quadratic runtime via efficient implementation). ONS also requires to compute a non-Euclidean projection on each iteration to enforce the constraints, which can be considerably more expensive than the Euclidean projection required by OGD (e.g., might require to use an iterative algorithm). Thus, despite the improved regret bound, ONS is often not applicable to large-scale problems.  This naturally motivates the following question:
\begin{center}
\textit{Can Online Gradient Descent be shown to enjoy a logarithmic regret bound for classes of loss functions beyond the class of strongly convex losses?}
\end{center}

In this paper we take a step forward towards understanding the conditions under-which OGD can guarantee logarithmic regret, hence yielding both an efficient algorithm and an improved convergence rate in such settings.
In particular, we focus on an important sub-class of the exp-concave losses: loss functions which can be written as a strongly-convex function applied to a linear transformation of the input variables. Such loss functions include important examples such as the square loss for linear regression, the online portfolio optimization loss, the logistic regression loss, and more. While such losses are not necessarily strongly convex in the entire space, they are strongly convex on a certain subspace, which corresponds to the row-span of the linear transformation.
Our main result shows that when all loss functions are of this form, with linear transformations that satisfy certain consistency conditions, and the feasible set is a convex and compact polytope, the vanilla OGD algorithm, with a suitable choice of learning rate, indeed guarantees logarithmic regret.
To the best of our knowledge, this is the first result to establish strongly-convex-like rates for OGD without strong convexity, on an important and wide class of applications.

Technically, at the heart of our result lies a classical result in convex analysis, originally due to Hoffman \cite{Hoffman52}, which roughly speaking, bounds the distance of a point from a convex and compact polytope in terms of the magnitude of violation of the constraints describing the polytope. In case the feasible set is a polytope and the loss function is as described above (i.e., strongly convex applied to linear transformation), it can be shown that Hoffman's bound implies a property known as \textit{quadratic growth}, which upper bounds the $\ell_2$ distance between any feasible point and a feasible optimal solution, in terms of the distance in function values - a property well known to enable faster convergence rates in convex optimization settings (this is often also the main consequence of strong convexity needed in order to achieve fast rates for strongly-convex optimization).

Indeed, several recent works have used this classical result by Hoffman \cite{Hoffman52}, to achieve fast rates without strong convexity for \textit{offline optimization} problems,  see for instance the recent works \cite{Necoara16, wang2014, Karimi16, Beck15,Xu17}. Importantly, all of these results consider only \textit{stationary} settings, in which the objective function is fixed. As we show in the sequel, obtaining such fast rates results in the \textit{online convex optimization} setting is considerably more challenging since, as opposed to strong convexity which is a property that holds in the entire space, and hence, given a sequence of strongly convex functions, this property holds throughout the sequence, Hoffman's bound on the other-hand, is related to a specific subspace (which corresponds to the row-span of the linear transformation in the losses discussed above), and thus, given a sequence of such losses with different corresponding subspaces, these subspaces need not be, informally speaking, consistent with each other. Hence, a main contribution of this work is to formalize and analyze conditions under which this property could indeed be leveraged towards obtaining fast rates in a non-stationary online setting.

In particular, we show that our logarithmic-regret result holds for sequences which can be expressed as a combination of an arbitrary (adversarial) sequence and a stochastic sequence with certain stationary characteristics, which may potentially serve as reasonable approximation to many real-world data-streams, or when the data enjoys a low-rank structure. We report preliminary experimental results on both synthetic and real-world datasets which indeed show that OGD can outperform the Online Netwon Step method, both in terms of the regret and computational efficiency, on \textit{non-strongly convex} sequences.

\section{Preliminaries}

Throughout this work we use $\Vert\cdot\Vert$ to denote the Euclidean norm for vectors and the spectral norm (i.e., largest singular value) for matrices. Also, for a compact set $\mathcal{P}\subset\mathbb{R}^d$ and a matrix ${\mathbf{C}}\in\mathbb{R}^{m\times d}$, we use the notation ${\mathbf{C}}\mathcal{P} := \{{\mathbf{C}}{\mathbf{x}} ~|~{\mathbf{x}}\in\mathcal{P}\}$.

\subsection{Convex optimization preliminaries}
\begin{definition}
Given a convex and compact set $\mathcal{K}\subset\mathbb{R}^d$ and a real-valued function $f$, differentiable over $\mathcal{K}$, we say $f$ is $G$-Lipschitz over $\mathcal{K}$ if $\forall {\mathbf{x}}\in\mathcal{K}:$ $\Vert{\nabla{}f({\mathbf{x}})}\Vert \leq G$.
\end{definition}
In particular, if $f$ is convex, differentiable and $G$-Lipschitz over a convex and compact $\mathcal{K}$, we have that $\forall {\mathbf{x}},{\mathbf{y}}\in\mathcal{K}$:
\begin{eqnarray}\label{eq:Lip}
f({\mathbf{x}})-f({\mathbf{y}}) \leq ({\mathbf{x}}-{\mathbf{y}})^{\top}\nabla{}f({\mathbf{x}}) \leq G\Vert{{\mathbf{x}}-{\mathbf{y}}}\Vert.
\end{eqnarray}

\begin{definition}
Given a convex and compact set $\mathcal{K}\subset\mathbb{R}^d$ and a real-valued function $f$ which is differentiable over $\mathcal{K}$, we say $f$ is $\alpha$-strongly convex over $\mathcal{K}$ if $\forall {\mathbf{x}},{\mathbf{y}}\in\mathcal{K}$: $f({\mathbf{x}}) \leq f({\mathbf{y}}) + ({\mathbf{x}}-{\mathbf{y}})^{\top}\nabla{}f({\mathbf{x}}) - \frac{\alpha}{2}\Vert{{\mathbf{x}}-{\mathbf{y}}}\Vert^2$.

\end{definition}

We recall the first-order optimality condition for convex differentiable functions (see for instance \cite{boyd2004convex}): for any $\mathcal{K}\subset\mathbb{R}^d$, convex and compact, and a real-valued function $f$, convex and differentiable over $\mathcal{K}$, we have that $\forall {\mathbf{x}}\in\mathcal{K},~{\mathbf{x}}^*\in\argmin_{{\mathbf{y}}\in\mathcal{K}}f({\mathbf{y}})$: $({\mathbf{x}}^*-{\mathbf{x}})^{\top}\nabla{}f({\mathbf{x}}^*) \leq 0$.

\subsection{Online convex optimization preliminaries}
We now briefly recall the setting of Online Convex Optimization (OCO). For a more in-depth introduction we refer the reader to \cite{Hazan16,SSS12}.

In the OCO problem, a decision maker (DM) is required to iteratively choose points in a fixed convex and compact set $\mathcal{K} \subset\mathbb{R}^d$. On each round $t$, after the DM makes his choice, i.e., chooses some ${\mathbf{x}}_t\in\mathcal{K}$, a convex function $f_t:\mathcal{K}\rightarrow\mathbb{R}$ is revealed, and the DM suffers the loss $f_t({\mathbf{x}}_t)$. This process continues for $T$ rounds, where $T$ is assumed to be known in advanced.
The goal is to design an algorithm for choosing the actions of the DM so to minimize a quantity called \textit{regret}, which is given by
\vspace{-10pt}
\begin{eqnarray*}
\textrm{regret}_T := \sum_{t=1}^Tf_t({\mathbf{x}}_t) - \min_{{\mathbf{x}}\in\mathcal{K}}\sum_{t=1}^Tf_t({\mathbf{x}}).
\end{eqnarray*}
It is well known than an algorithm known as Online Gradient Descent, see Algorithm \ref{alg:ogd} below, can guarantee a $O(GD\sqrt{T})$ bound on the regret, where $D$ is the $\ell_2$ diameter of $\mathcal{K}$ and $G$ is an $\ell_2$ upper bound on the gradients of the functions $f_1,\dots,f_T$ \cite{Zinkevich03}, which is in general optimal. It is also known that when all functions $f_1,\dots,f_T$ are $\alpha$-strongly convex, the same algorithm (though with different learning rate) guarantees $O((G^2/\alpha)\log{T})$ regret \cite{AHK07}, which is also optimal under this assumption \cite{Hazan14}.

\begin{algorithm}
\caption{Online (projected) Gradient Descent}
\label{alg:ogd}
\begin{algorithmic}[1]
\STATE ${\mathbf{x}}_1 \gets $ some arbitrary point in $\mathcal{K}$
\FOR{$t=1\dots T$}
\STATE ${\mathbf{y}}_{t+1} \gets {\mathbf{x}}_t - \eta_t\nabla{}f_t({\mathbf{x}}_t)$
\STATE ${\mathbf{x}}_{t+1} \gets \arg\min_{{\mathbf{x}}\in\mathcal{K}}\Vert{{\mathbf{x}}-{\mathbf{y}}_{t+1}}\Vert^2$
\ENDFOR
\end{algorithmic}
\end{algorithm}

\subsection{Hoffman's bound and the quadratic growth property}

\begin{definition}
We say a matrix ${\mathbf{C}}\in\mathbb{R}^{m\times d}$ is $\sigma$-Hoffman with respect to a convex and compact polytope $\mathcal{P}\subset\mathbb{R}^d$ for some $\sigma >0$, if for any vector ${\mathbf{c}}\in\mathbb{R}^m$ such that the set $\mathcal{P}({\mathbf{C}},{\mathbf{c}}) := \{{\mathbf{x}}\in\mathcal{P} ~ | ~ {\mathbf{C}}{\mathbf{x}} = {\mathbf{c}}\}$ is not empty, it holds that
$\forall{\mathbf{x}}\in\mathcal{P}$: $\textrm{dist}({\mathbf{x}},\mathcal{P}({\mathbf{C}},{\mathbf{c}}))^2 \leq \sigma^{-1}\Vert{{\mathbf{C}}{\mathbf{x}}-{\mathbf{c}}}\Vert^2$.
\end{definition}

The following Lemma, originally due to Hoffman \cite{Hoffman52}, shows that a Hoffman parameter bounded away from zero, always exists. Here we give the result in rephrased form. A proof is given in the appendix for completeness.

\begin{lemma}\label{lem:hoffman}
Let $\mathcal{P}:=\{{\mathbf{x}}\in\mathbb{R}^d ~ | ~ {\mathbf{A}}{\mathbf{x}} \leq {\mathbf{b}}\}$  be a compact and convex polytope and let ${\mathbf{C}}\in\mathbb{R}^{m\times d}$. Given a vector ${\mathbf{c}}\in\mathbb{R}^m$, define the set $\mathcal{P}({\mathbf{C}},{\mathbf{c}}) := \{{\mathbf{x}}\in\mathcal{P} ~ | ~ {\mathbf{C}}{\mathbf{x}}={\mathbf{c}}\}$. If $\mathcal{P}({\mathbf{C}},{\mathbf{c}}) \neq \emptyset$, then there exists $\sigma > 0$ such that
$\forall {\mathbf{x}}\in\mathcal{P}$: $\textrm{dist}({\mathbf{x}},\mathcal{P}({\mathbf{C}},{\mathbf{c}}))^2\leq \sigma^{-1}\Vert{{\mathbf{C}}{\mathbf{x}} - {\mathbf{c}}}\Vert^2$.
Moreover, we have the bound $\sigma \geq \min_{{\mathbf{Q}}\in\mathcal{M}}\lambda_{\min}\left({{\mathbf{Q}}\Q^{\top}}\right)$, where $\mathcal{M}$ is the set of all $d\times d$ matrices whose rows are linearly independent rows of the matrix ${\mathbf{M}} := ({\mathbf{A}}^{\top},~ {\mathbf{C}}^{\top})^{\top}$, and $\lambda_{\min}(\cdot)$ denotes the smallest non-zero eigenvalue.
\end{lemma}

\begin{definition}.
We say a function $f:\mathbb{R}^d\rightarrow\mathbb{R}$ has the quadratic growth property with parameter $\kappa$ with respect to a compact and convex set $\mathcal{K}\subset\mathbb{R}^d$, if it holds that
$\forall{\mathbf{x}}\in\mathcal{K}$: $\textrm{dist}({\mathbf{x}},\mathcal{X}^*)^2 \leq \frac{2}{\kappa}\left({f({\mathbf{x}}) - f^*}\right)$,
where $\mathcal{X}^* := \arg\min_{{\mathbf{y}}\in\mathcal{K}}f({\mathbf{y}})$ and $f^* := \min_{{\mathbf{y}}\in\mathcal{K}}f({\mathbf{y}})$.
\end{definition}

The following lemma, which will be instrumental in the proof of our main result, demonstrates the connection between Hoffman's bound and the quadratic growth property for (possibly stochastic) convex objectives.
\begin{lemma}[from Hoffman's bound to quadratic growth]\label{lem:hoffmanExpectation}
Let $\mathcal{P}\subset\mathbb{R}^d$ be a convex and compact polytope.
Let $\mathcal{D}$ be a distribution over pairs $(g(\cdot),{\mathbf{C}})\in(\mathbb{R}^m\rightarrow\mathbb{R})\times\mathbb{R}^{m\times d}$ satisfying:
\begin{enumerate}
\item for each pair $(g,{\mathbf{C}})$ in the support of $\mathcal{D}$, the function $g$ is differentiable, $G$-Lipschitz, and $\alpha$-strongly convex over ${\mathbf{C}}\mathcal{P}$.
\item the function $F({\mathbf{x}}):=\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}\left[{g({\mathbf{C}}{\mathbf{x}})}\right]$ is differentiable over $\mathcal{P}$.
\item  the expectation $\mathbb{E}_{(g(\cdot),{\mathbf{C}})\sim\mathcal{D}}[{\mathbf{C}}^{\top}{\mathbf{C}}]$ exists.
\end{enumerate}
Let ${\mathbf{C}}_{\mathcal{D}}\in\mathbb{R}^{k\times d}$ be such that  ${\mathbf{C}}_{\mathcal{D}}^{\top}{\mathbf{C}}_{\mathcal{D}} = \mathbb{E}_{(g(\cdot),{\mathbf{C}})\sim\mathcal{D}}[{\mathbf{C}}^{\top}{\mathbf{C}}]$, and denote by $\sigma$ the Hoffman constant of ${\mathbf{C}}_{\mathcal{D}}$ w.r.t. $\mathcal{P}$. Finally, define $\mathcal{X}^*:=\arg\min_{{\mathbf{y}}\in\mathcal{P}}\{F({\mathbf{y}}):=\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}\left[{g({\mathbf{C}}{\mathbf{y}})}\right]\}$. Then, there exists ${\mathbf{c}}_{\mathcal{D}}\in\mathbb{R}^k$ such that ${\mathbf{x}}\in\mathcal{X}^*$ $\Longleftrightarrow$ ${\mathbf{C}}_{\mathcal{D}}{\mathbf{x}} = {\mathbf{c}}_{\mathcal{D}}$. Moreover, $\forall {\mathbf{x}}\in\mathcal{P}$: $\textrm{dist}({\mathbf{x}},\mathcal{X}^*)^2 \leq \frac{1}{\sigma}\Vert{{\mathbf{C}}_{\mathcal{D}}{\mathbf{x}} - {\mathbf{c}}_{\mathcal{D}}}\Vert^2 \leq \frac{2}{\alpha\sigma}\left({F({\mathbf{x}}) - \min_{{\mathbf{y}}\in\mathcal{P}}F({\mathbf{y}})}\right)$.
\end{lemma}

\begin{proof}
For any ${\mathbf{x}}\in\mathcal{P}$ and ${\mathbf{x}}^*\in\mathcal{X}^*$ it holds that
{\small
\begin{align*}
F({\mathbf{x}}^*) - F({\mathbf{x}}) &= \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[g({\mathbf{C}}{\mathbf{x}}^*) - g({\mathbf{C}}{\mathbf{x}})] \\
&\underset{(a)}{\leq} \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}\Big[({\mathbf{x}}^*-{\mathbf{x}})^{\top}{\mathbf{C}}^{\top}\nabla{}g({\mathbf{C}}{\mathbf{x}}^*) \\
&- \frac{\alpha}{2}\Vert{{\mathbf{C}}({\mathbf{x}}^*-{\mathbf{x}})}\Vert^2\Big] \\
&= ({\mathbf{x}}^*-{\mathbf{x}})^{\top}\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}\left[{{\mathbf{C}}^{\top}\nabla{}g({\mathbf{C}}{\mathbf{x}}^*)}\right] \\
&- \frac{\alpha}{2}({\mathbf{x}}^*-{\mathbf{x}})^{\top}\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}\left[{{\mathbf{C}}^{\top}{\mathbf{C}}}\right]({\mathbf{x}}^*-{\mathbf{x}}) \\
&\underset{(b)}{=} ({\mathbf{x}}^*-{\mathbf{x}})^{\top}\nabla{}F({\mathbf{x}}^*)  -\frac{\alpha}{2}\Vert{{\mathbf{C}}_{\mathcal{D}}({\mathbf{x}}^*-{\mathbf{x}})}\Vert^2 \\
&\underset{(c)}{\leq}  -\frac{\alpha}{2}\Vert{{\mathbf{C}}_{\mathcal{D}}({\mathbf{x}}^*-{\mathbf{x}})}\Vert^2,
\end{align*}}
where (a) follows since each $g(\cdot)$ in the support of $\mathcal{D}$ is differentiable and $\alpha$-strongly convex over ${\mathbf{C}}\mathcal{P}$, (b) follows since $F({\mathbf{x}})$ is differentaible over $\mathcal{P}$ and hence it's gradient vector is given by $\nabla{}F({\mathbf{x}}) = \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[\frac{d}{d{\mathbf{x}}}g({\mathbf{C}}{\mathbf{x}})] = \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[{\mathbf{C}}^{\top}\nabla{}g({\mathbf{C}}{\mathbf{x}})]$, and (c) follows form the first-order optimality condition for $F(\cdot)$.
Thus, $\forall {\mathbf{x}}^*\in\mathcal{X}^*, {\mathbf{x}}\in\mathcal{P}$:
\begin{eqnarray}\label{eq:hoffExp:1}
\Vert{{\mathbf{C}}_{\mathcal{D}}({\mathbf{x}}^*-{\mathbf{x}})}\Vert^2 \leq \frac{2}{\alpha}\left({F({\mathbf{x}}) - F({\mathbf{x}}^*)}\right).
\end{eqnarray}
Thus, setting ${\mathbf{c}}_{\mathcal{D}} = {\mathbf{C}}_{\mathcal{D}}{\mathbf{x}}^*$ for some ${\mathbf{x}}^*\in\mathcal{X}^*$, directly gives the $\Longrightarrow$ direction of the first part of the lemma

To prove the $\Longleftarrow$ direction of the first part of the lemma, let ${\mathbf{x}}\in\mathcal{P}$ such that ${\mathbf{C}}_{\mathcal{D}}{\mathbf{x}}={\mathbf{c}}_{\mathcal{D}}$. Then we have that
\begin{eqnarray*}
0 &=& ({\mathbf{x}}-{\mathbf{x}}^*){\mathbf{C}}_{\mathcal{D}}^{\top}{\mathbf{C}}_{\mathcal{D}}({\mathbf{x}}-{\mathbf{x}}^*) \\
&=& \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[\Vert{{\mathbf{C}}{\mathbf{x}}-{\mathbf{C}}{\mathbf{x}}^*}\Vert^2] \\
&\geq &\left({\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[\Vert{{\mathbf{C}}{\mathbf{x}}-{\mathbf{C}}{\mathbf{x}}^*}\Vert]}\right)^2.
\end{eqnarray*}

Since for each pair $(g,{\mathbf{C}})$ in the support of $\mathcal{D}$, $g$ is convex and $G$-Lipschitz over ${\mathbf{C}}\mathcal{P}$, using Eq. \eqref{eq:Lip} we have that
\begin{eqnarray*}
F({\mathbf{x}}) - F({\mathbf{x}}^*) &=& \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[g({\mathbf{C}}{\mathbf{x}}) - g({\mathbf{C}}{\mathbf{x}}^*)] \\
&\leq &\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[G\Vert{{\mathbf{C}}{\mathbf{x}} - {\mathbf{C}}{\mathbf{x}}^*}\Vert] = 0,
\end{eqnarray*}
meaning ${\mathbf{x}}\in\mathcal{X}^*$, which completes the proof of the first part of the lemma. The second part of the lemma follows directly form combining the first part of the lemma with Hoffman's bound (Lemma \ref{lem:hoffman}) and Eq. \eqref{eq:hoffExp:1}.
\end{proof}

\section{Informal Statement of Results and Examples}

We now give an informal statement of our theoretical results, followed by several concrete examples to demonstrate possible applications. We then conclude the section by drawing a connection between our setting and \textit{online exp-concave optimization} and the Online Newton Step algorithm.

\subsection{Logarithmic regret for Online Gradient Descent without strong convexity}
Suppose that the feasible set $\mathcal{P}$ is a convex and compact polytope in $\mathbb{R}^d$ and suppose all loss functions are of the form $f_t({\mathbf{x}}) := g_t({\mathbf{C}}_t{\mathbf{x}})$, where $g_t(\cdot)$ is differentiable and $\alpha_1$-strongly convex. Suppose further, that there exists a matrix ${\mathbf{M}}\in\mathbb{R}^{k\times d}$ such that for all $t\in[T]$, ${\mathbf{M}}^{\top}{\mathbf{M}}\succeq \mathbb{E}[{\mathbf{C}}_t^{\top}{\mathbf{C}}_t] \succeq \alpha_2{\mathbf{M}}^{\top}{\mathbf{M}}$, where the expectation is with respect to possible randomness in the choice of ${\mathbf{C}}_t$. Then, we show there exists a choice of step-sizes $\{\eta_t\}_{t\in[T]}$ such that OGD guarantees $O(\log{T})$ regret (treating all other quantities as constants).

We note that while the requirement ${\mathbf{M}}^{\top}{\mathbf{M}}\succeq\mathbb{E}[{\mathbf{C}}_t^{\top}{\mathbf{C}}_t] \succeq \alpha_2{\mathbf{M}}^{\top}{\mathbf{M}}$ seems not standard at first glance, observe that when ${\mathbf{C}}_t$ is full-rank, and hence $f_t({\mathbf{x}})$ is in particular strongly convex, this requirement holds trivially with ${\mathbf{M}} ={\mathbf{I}}$. Hence, this condition is natural for dealing with loss functions that are strongly convex only on a restricted subspace of $\mathbb{R}^d$, requiring them all to be consistent, at least in expectation, with the same subspace. In Subsection \ref{sec:examples} we discuss several settings of interest in which it is reasonable to assume this requirement holds.

\subsection{Examples of relevant settings and loss functions}\label{sec:examples}

\textbf{Linear regression and Lasso:}
Consider the $\ell_p$ linear regression loss function  $f_t({\mathbf{x}}) := \frac{1}{2}\Vert{{\mathbf{A}}_t{\mathbf{x}} - {\mathbf{b}}_t}\Vert_p^2$, with $p\in(1,2]$. In particular, when $p=2$ and the feasible polytope is an $\ell_1$-ball, i.e., $\mathcal{P} := \{{\mathbf{x}}\in\mathbb{R}^d ~ | ~ \Vert{{\mathbf{x}}}\Vert_1 \leq k\}$, for some $k >0$, we get an online version of the famous LASSO problem \cite{Lasso96}.

For our log-regret result to hold for \textit{deterministic data}, i.e., deterministic choice of $({\mathbf{A}}_1,{\mathbf{b}}_1)\dots({\mathbf{A}}_t,{\mathbf{b}}_T)$, it must hold that $\textrm{row-span}({\mathbf{A}}_1) = \textrm{row-span}({\mathbf{A}}_2) = \dots = \textrm{row-span}({\mathbf{A}}_T)$. This is reasonable if ${\mathbf{A}}_t\in\mathbb{R}^{m\times d}$ for a large enough value of $m$, and the data, i.e., the rows of ${\mathbf{A}}_1,\dots,{\mathbf{A}}_T$ lie in a certain low-dimensional subspace \footnote{this may be natural to assume for instance, if the data is the output of some dimension reduction technique such as the wildly used \textit{principal component analysis} procedure}.

A different \textit{non-deterministic} setting of interest is a ``semi-adversarial" model in which ${\mathbf{A}}_t := \tilde{{\mathbf{A}}}_t + {\mathbf{N}}_t$, where the matrices $\tilde{{\mathbf{A}}}_1,\dots\tilde{{\mathbf{A}}}_T$ are arbitrary and ${\mathbf{N}}_t\sim\mathcal{D}$ i.i.d. for all $t\in[T]$, for some fixed (yet unknown) distribution $\mathcal{D}$. Then, a sufficient condition for our fast OGD rate  to hold (in expectation), is that $\mathbb{E}_{\mathcal{D}}[{\mathbf{N}}] = \mathbf{0}$ and $\mathbb{E}_{\mathcal{D}}[{\mathbf{N}}^{\top}{\mathbf{N}}] \succeq \alpha\tilde{{\mathbf{A}}}_t^{\top}\tilde{{\mathbf{A}}}_t$ for all $t\in[T]$, for some $\alpha > 0$. That is, the data can be faithfully modeled as a deterministic sequence perturbed by a well-conditioned stochastic noise. For instance, such a model underlies the problem of \textit{Universal Linear Filtering} studied in \cite{Moon2009, GarberH13}.

Finally, if we can treat the data as generated by a stochastic mechanism that on each time $t$ randomly samples ${\mathbf{A}}_t$ from a (unknown) distribution $\mathcal{D}_t$ (note we allow the distribution to change each round), then a sufficient condition for our log-regret result to hold is that there exists a  matrix ${\mathbf{M}}$ and $\alpha > 0$ such that for all $t\in[T]$: ${\mathbf{M}}^{\top}{\mathbf{M}} \succeq \mathbb{E}_{{\mathbf{A}}_t\sim\mathcal{D}_t}[{\mathbf{A}}_t^{\top}{\mathbf{A}}_t] \succeq \alpha{\mathbf{M}}^{\top}{\mathbf{M}}$.

\textbf{Logistic regression:}
In online logistic regression, the loss on $m$ data points organized in a matrix ${\mathbf{A}}_t\in\mathbb{R}^{m\times d}$, can be written as $f_t({\mathbf{x}}) := \sum_{i=1}^m\log\left({1+\exp({\mathbf{A}}_t^{(i)\top}{\mathbf{x}})}\right)$,
where ${\mathbf{A}}_t^{(i)}$ denotes the $i$th row of the matrix ${\mathbf{A}}_t$. Observe $f_t({\mathbf{x}})$ can be rewritten as $f_t({\mathbf{x}}) := g({\mathbf{A}}_t{\mathbf{x}})$ with
$g({\mathbf{y}}) := \sum_{i=1}^m\log\left({1+e^{{\mathbf{y}}_i}}\right)$. It not difficult to verify that for bounded ${\mathbf{y}}$, $g({\mathbf{y}})$ is indeed strongly convex, and hence this problem also falls into our setting. As in the linear regression case, if the feasible set is a polytope (e.g., standard selections are a $\ell_1$ or $\ell_{\infty}$ ball), then same assumptions on the matrices ${\mathbf{A}}_1,\dots,{\mathbf{A}}_T$ will allow to apply our log-regret result.

\textbf{Online portfolio selection:}
In the online portfolio selection problem \cite{Hazan16}, the loss of a \textit{rebalancing} portfolio ${\mathbf{x}}$ (a point in the unit simplex) on $m$ consecutive trading rounds is given by $f_t({\mathbf{x}}) = -\sum_{i=1}^m\log\left({{\mathbf{A}}_t^{(i)\top}{\mathbf{x}}}\right)$, where the rows of ${\mathbf{A}}_t\in\mathbb{R}^{m\times d}$, ${\mathbf{A}}_t > \mathbf{0}$  (entry-wise) encodes the asset prices on each round. Similarly to the logistic regression example, we can write $f_t({\mathbf{x}}) := g({\mathbf{A}}_t{\mathbf{x}})$ with $g({\mathbf{y}}) := -\sum_{i=1}^m\log\left({{\mathbf{y}}_i}\right)$. Again, it is not hard to verify that if ${\mathbf{A}}_t \geq r$ (entry-wise) for some $r>0$, then $g(\cdot)$ is indeed strongly convex over the transformed  simplex ${\mathbf{A}}_t\Delta_d := \{{\mathbf{A}}_t{\mathbf{x}} ~ |~ {\mathbf{x}}\in\mathbb{R}^d, ~{\mathbf{x}}\geq\mathbf{0},~ \sum_{i=1}^d{\mathbf{x}}_i = 1\}$. Again, our log-regret result holds under the same assumptions on the data ${\mathbf{A}}_1,\dots,{\mathbf{A}}_T$, as above.

\subsection{Connection with exp-concavity and the Online Newton Step algorithm}

A real-valued function $f$, twice-differentiable over a compact set $\mathcal{K}\subset\mathbb{R}^d$, is $\sigma$ exp-concave on $\mathcal{K}$ if and only if
$\forall {\mathbf{x}}\in\mathcal{K}:  \nabla^2{}f({\mathbf{x}}) \succeq \sigma\nabla{}f({\mathbf{x}})\nabla{}f({\mathbf{x}})^{\top}$ \cite{Hazan16}.

Note that in case $f({\mathbf{x}}) := g({\mathbf{C}}{\mathbf{x}})$, where $g$ is $\alpha$-strongly convex and twice-differentiable over ${\mathbf{C}}\mathcal{K}$, denoting $G = \sup_{{\mathbf{x}}\in{\mathbf{C}}\mathcal{K}}\Vert{\nabla{}g({\mathbf{x}})}\Vert$, we have that $\forall {\mathbf{x}}\in\mathcal{K}$: {\small
\begin{align*}
\nabla^2{}f({\mathbf{x}}) &= {\mathbf{C}}^{\top}\nabla^2g({\mathbf{C}}{\mathbf{x}}){\mathbf{C}} \underset{(a)}{\succeq} \alpha{\mathbf{C}}^{\top}{\mathbf{C}} \succeq \frac{\alpha}{G^2}\Vert{\nabla{}g({\mathbf{C}}{\mathbf{x}})}\Vert^2{\mathbf{C}}^{\top}{\mathbf{C}} \\
&\underset{(b)}{\succeq} \frac{\alpha}{G^2}{\mathbf{C}}^{\top}\nabla{}g({\mathbf{C}}{\mathbf{x}})\nabla{}g({\mathbf{C}}{\mathbf{x}})^{\top}{\mathbf{C}} = \frac{\alpha}{G^2}\nabla{}f({\mathbf{x}})\nabla{}f({\mathbf{x}})^{\top},
\end{align*}}
where (a) follows since $g(\cdot)$ is $\alpha$-strongly convex, and (b) follows since for any vector ${\mathbf{y}}$ we have that ${\mathbf{y}}^{\top}{\mathbf{C}}^{\top}\nabla{}g({\mathbf{C}}{\mathbf{x}})\nabla{}g({\mathbf{C}}{\mathbf{x}})^{\top}{\mathbf{C}}{\mathbf{y}} = (({\mathbf{C}}{\mathbf{y}})^{\top}\nabla{}g({\mathbf{C}}{\mathbf{x}}))^2 \leq \Vert{\nabla{}g({\mathbf{C}}{\mathbf{x}})}\Vert^2\cdot\Vert{{\mathbf{C}}{\mathbf{y}}}\Vert^2 =  \Vert{\nabla{}g({\mathbf{C}}{\mathbf{x}})}\Vert^2\cdot{\mathbf{y}}^{\top}{\mathbf{C}}^{\top}{\mathbf{C}}{\mathbf{y}}$. Hence, $f({\mathbf{x}})$ is $\alpha/G^2$-exp-concave over $\mathcal{K}$.

Thus, if all loss functions are as above, i.e., $f_t({\mathbf{x}}) := {\mathbf{g}}_t({\mathbf{C}}_t{\mathbf{x}})$, and we let $G$ be a uniform upper bound on the $\ell_2$ norm of the gradients of $g_t(\cdot)$ and $C$ be a uniform upper bound on the spectral norm of the matrices ${\mathbf{C}}_t$ , the Online Newton Step (ONS) algorithm \cite{Hazan16}, guarantees regret bound: $\textrm{regret}_T(ONS)  = O\left({\frac{G^2}{\alpha} + CGD}\right)d\log{T}$, where $D$ is the $\ell_2$ diameter of $\mathcal{K}$.

\section{Logarithmic Regret for OGD Without Strong Convexity}

In this section we present and prove our main result - a logarithmic regret bound for Online Gradient Descent (Algorithm \ref{alg:ogd}) without strong-convexity. As discussed, our result holds under certain conditions on the data which are captured in the following assumption. In the following Subsection \ref{subsec:app} we discuss several concrete examples in which this assumption holds.

\begin{assumption}\label{ass:ogd:dist}
Given a convex and compact polytope $\mathcal{P}\subset\mathbb{R}^d$,
a distribution $\mathcal{D}$ over pairs $(g,{\mathbf{C}})\in(\mathbb{R}^m\rightarrow\mathbb{R})\times\mathbb{R}^{m\times d}$, is said to satisfy Assumption \ref{ass:ogd:dist} with parameters $({\mathbf{M}}, \bar{G}, \alpha_1, \alpha_2)\in\mathbb{R}^{k\times d}\times\mathbb{R}_+^3$ w.r.t. $\mathcal{P}$, if it holds that
\begin{enumerate}
\item each function $g$, part of a pair $(g,{\mathbf{C}})$ in the support of $\mathcal{D}$, is differentiable, $\alpha_1$-strongly convex over ${\mathbf{C}}\mathcal{P} := \{{\mathbf{C}}{\mathbf{x}} ~ | ~{\mathbf{x}}\in\mathcal{P}\}$ and $G$-Lipschitz over ${\mathbf{C}}\mathcal{P}$, for some finite $G>0$
\item the function $F({\mathbf{x}}):=\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}\left[{g({\mathbf{C}}{\mathbf{x}})}\right]\}$ is differentiable over $\mathcal{P}$
\item $\forall {\mathbf{x}}\in\mathcal{P}$: $\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}\left[{\Vert{\frac{d}{d{\mathbf{x}}}g({\mathbf{C}}{\mathbf{x}})}\Vert^2}\right]= \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}\left[{\Vert{{\mathbf{C}}^{\top}\nabla{}g({\mathbf{C}}{\mathbf{x}})}\Vert^2}\right] \leq \bar{G}^2$
\item  the expectation $\mathbb{E}_{(g(\cdot),{\mathbf{C}})\sim\mathcal{D}}[{\mathbf{C}}^{\top}{\mathbf{C}}]$ exists and satisfies $\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[{\mathbf{C}}^{\top}{\mathbf{C}}] \succeq \alpha_2{\mathbf{M}}^{\top}{\mathbf{M}}$.
\end{enumerate}
\end{assumption}

We can now state our main theorem, Theorem \ref{thm:ogd:master}. While the theorem holds under quite general conditions, we refer the reader again to Section \ref{sec:examples} for discussion of concrete applications.

\begin{theorem}\label{thm:ogd:master}[OGD Master Theorem]
Fix a convex and compact polytope $\mathcal{P}\subset \mathbb{R}^d$.
Consider a sequence of $T$ distributions $\mathcal{D}_1,\dots\mathcal{D}_T$ over $(\mathbb{R}^m\rightarrow\mathbb{R})\times\mathbb{R}^{m\times d}$ which satisfy Assumption \ref{ass:ogd:dist} with parameters $(\bar{{\mathbf{C}}},\bar{G},\alpha_1,\alpha_2)$ w.r.t. $\mathcal{P}$, where $\bar{{\mathbf{C}}}$ is a matrix satisfying $\bar{{\mathbf{C}}}^{\top}\bar{{\mathbf{C}}} = \frac{1}{T}\sum_{t=1}^T\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[{\mathbf{C}}^{\top}{\mathbf{C}}]$. Suppose further that $\bar{{\mathbf{C}}}$ is $\sigma$-Hoffman w.r.t. $\mathcal{P}$.
Let $f_1({\mathbf{x}})\dots f_T({\mathbf{x}})$ be a sequence of loss functions such that $f_t({\mathbf{x}}) = g_t({\mathbf{C}}_t{\mathbf{x}})$, with $(g_t,{\mathbf{C}}_t)\sim\mathcal{D}_t$ independently of the functions $\{f_{\tau}\}_{\tau\in[T]\setminus\{t\}}$. Then, applying Algorithm \ref{alg:ogd} with step-size $\eta_t = \frac{1}{\alpha_1\alpha_2\sigma{}t}$, w.r.t. the losses $f_1\dots f_T$ and the polytope $\mathcal{P}$, guarantees that
\vspace{-6pt}
\begin{eqnarray*}
&&\max_{{\mathbf{x}}\in\mathcal{P}}\mathbb{E}_{f_1\sim\mathcal{D}_1\dots f_T\sim\mathcal{D}_T}\left[{\sum_{t=1}^Tf_t({\mathbf{x}}_t) - \sum_{t=1}^Tf_t({\mathbf{x}})}\right] \leq \\
&&\frac{\alpha_1\alpha_2\sigma{}D^2}{2} + \frac{\bar{G}^2}{2\alpha_1\alpha_2\sigma}(1 + \ln{T}).
\end{eqnarray*}
\end{theorem}

Before we can prove the theorem, we need the following technical lemma which extends Lemma \ref{lem:hoffmanExpectation} from a single stochastic objective to a sequence of stochastic objectives, and hence plays a key role in our regret analysis .

\begin{lemma}[from Hoffman's bound to quadratic growth of a sequence]\label{lem:uniformMinimizers}
Let $\mathcal{D}_1,\dots,\mathcal{D}_T$ be distributions over $(\mathbb{R}^m\rightarrow\mathbb{R})\times\mathbb{R}^{m\times d}$ satisfying Assumption \ref{ass:ogd:dist} with parameters $(\bar{{\mathbf{C}}}, \bar{G}, \alpha_1, \alpha_2)\in\mathbb{R}^{k\times d}\times\mathbb{R}_+^3$, where $\bar{{\mathbf{C}}}$ is a matrix satisfying $\bar{{\mathbf{C}}}^{\top}\bar{{\mathbf{C}}} = \frac{1}{T}\sum_{t=1}^T\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[{\mathbf{C}}_t^{\top}{\mathbf{C}}_t]$.
Consider the function
$F({\mathbf{x}}) := \frac{1}{T}\sum_{t=1}^T\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[g({\mathbf{C}}{\mathbf{x}})]$,
and define the set of feasible minimizers:
$\mathcal{X}^* := \argmin_{{\mathbf{x}}\in\mathcal{P}}F({\mathbf{x}})$.
Then, it holds that $\forall {\mathbf{x}}^*,{\mathbf{y}}^*\in\mathcal{X}^*, ~ t\in[T]$: $\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[g({\mathbf{C}}{\mathbf{x}}^*)] = \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[g({\mathbf{C}}{\mathbf{y}}^*)]$.
Moreover, letting  $\sigma$ denote the Hoffman constant of $\bar{{\mathbf{C}}}$ w.r.t. the polytope $\mathcal{P}$, we have that $\forall {\mathbf{x}}\in\mathcal{P}, ~{\mathbf{x}}^*\in\mathcal{X}^*: ~ \textrm{dist}({\mathbf{x}},\mathcal{X}^*)^2 \leq \sigma^{-1}\Vert{\bar{{\mathbf{C}}}({\mathbf{x}}-{\mathbf{x}}^*)}\Vert^2$.
\end{lemma}

\begin{proof}
Consider a distribution $\mathcal{D}$ over $(\mathbb{R}^m\rightarrow\mathbb{R})\times\mathbb{R}^{m\times d}$, described by the following sampling procedure: pick $t\in[T]$ uniformly at random, and then sample $(g,{\mathbf{C}})\sim\mathcal{D}_t$. Clearly, it holds that
\begin{eqnarray*}
\forall {\mathbf{x}}\in\mathcal{P}: ~ F({\mathbf{x}}) &=& \frac{1}{T}\sum_{t=1}^T\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[g({\mathbf{C}}{\mathbf{x}})] \\
&=& \mathbb{E}_{t\sim\textrm{Uni}[T]}[\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[g({\mathbf{C}}{\mathbf{x}})]] \\
&=& \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[g({\mathbf{C}}{\mathbf{x}})].
\end{eqnarray*}
Thus, it follows that
$\mathcal{X}^* = \argmin_{{\mathbf{x}}\in\mathcal{P}}\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[g({\mathbf{C}}{\mathbf{x}})]$.

Note that since each distribution $\mathcal{D}_t$ satisfies Assumption \ref{ass:ogd:dist}, it also satisfies  the assumptions of Lemma \ref{lem:hoffmanExpectation}.
It can be easily verified that as a consequence, the distribution $\mathcal{D}$ also satisfies the assumptions of Lemma  \ref{lem:hoffmanExpectation}, and thus there exists a matrix $\bar{{\mathbf{C}}}$, satisfying $\bar{{\mathbf{C}}}^{\top}\bar{{\mathbf{C}}} = \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[{\mathbf{C}}^{\top}{\mathbf{C}}]$, such that $\forall {\mathbf{x}}^*,{\mathbf{y}}^*\in\mathcal{X}^*$:
\begin{eqnarray*}
\Vert{\bar{{\mathbf{C}}}({\mathbf{x}}^*-{\mathbf{y}}^*)}\Vert^2 = ({\mathbf{x}}^*-{\mathbf{y}}^*)^{\top}\bar{{\mathbf{C}}}^{\top}\bar{{\mathbf{C}}}({\mathbf{x}}^*-{\mathbf{y}}^*) =  0.
\end{eqnarray*}
By the definitions of $\bar{{\mathbf{C}}}$ and the distribution $\mathcal{D}$, it holds that
\begin{eqnarray*}
\bar{{\mathbf{C}}}^{\top}\bar{{\mathbf{C}}} &=& \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}}[{\mathbf{C}}^{\top}{\mathbf{C}}] \\
&=& \mathbb{E}_{t\sim\textrm{Uni}[T]}\left[{\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[{\mathbf{C}}^{\top}{\mathbf{C}}]}\right] = \frac{1}{T}\sum_{t=1}^T\bar{{\mathbf{C}}}_t^{\top}\bar{{\mathbf{C}}}_t,
\end{eqnarray*}
where we define $\bar{{\mathbf{C}}}_t$ to be a matrix satisfying $\bar{{\mathbf{C}}}_t^{\top}\bar{{\mathbf{C}}}_t = \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[{\mathbf{C}}^{\top}{\mathbf{C}}]$.

Thus, we have that
\begin{align*}
\forall {\mathbf{x}}^*,{\mathbf{y}}^*\in\mathcal{X}^*: ~ 0 &= ({\mathbf{x}}^*-{\mathbf{y}}^*)^{\top}\bar{{\mathbf{C}}}^{\top}\bar{{\mathbf{C}}}({\mathbf{x}}^*-{\mathbf{y}}^*) \\
&= \frac{1}{T}\sum_{t=1}^T({\mathbf{x}}^*-{\mathbf{y}}^*)^{\top}\bar{{\mathbf{C}}}_t^{\top}\bar{{\mathbf{C}}}_t({\mathbf{x}}^*-{\mathbf{y}}^*),
\end{align*}
which, since each $\bar{{\mathbf{C}}}_t^{\top}\bar{{\mathbf{C}}}_t$ is positive semidefinite, implies that $\forall {\mathbf{x}}^*,{\mathbf{y}}^*\in\mathcal{X}^*,~t\in[T]$:
\begin{align}\label{eq:lem:uniformMinimizers:1}
  ({\mathbf{x}}^*-{\mathbf{y}}^*)^{\top}\bar{{\mathbf{C}}}_t^{\top}\bar{{\mathbf{C}}}_t({\mathbf{x}}^*-{\mathbf{y}}^*) = 0.
\end{align}

Thus, fixing some ${\mathbf{x}}^*,{\mathbf{y}}^*\in\mathcal{X}^*$ and $t\in[T]$, it holds that
\begin{align*}
&\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[g({\mathbf{C}}{\mathbf{x}}^*) - g({\mathbf{C}}{\mathbf{y}}^*)]  \underset{(a)}{\leq}\\
&  \mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[G\Vert{{\mathbf{C}}({\mathbf{x}}^*- {\mathbf{y}}^*)}\Vert]= \\
& G\sqrt{\left({\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[\Vert{{\mathbf{C}}({\mathbf{x}}^*- {\mathbf{y}}^*)}\Vert]}\right)^2} \leq \\
& G\sqrt{\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[\Vert{{\mathbf{C}}({\mathbf{x}}^*- {\mathbf{y}}^*)}\Vert^2]} =\\
& G\sqrt{\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[({\mathbf{x}}^*- {\mathbf{y}}^*)^{\top}{\mathbf{C}}^{\top}{\mathbf{C}}({\mathbf{x}}^*- {\mathbf{y}}^*)]} =\\
& G\sqrt{({\mathbf{x}}^*- {\mathbf{y}}^*)^{\top}\mathbb{E}_{(g,{\mathbf{C}})\sim\mathcal{D}_t}[{\mathbf{C}}^{\top}{\mathbf{C}}]({\mathbf{x}}^*- {\mathbf{y}}^*)} =\\
& G\sqrt{({\mathbf{x}}^*- {\mathbf{y}}^*)^{\top}\bar{{\mathbf{C}}_t}^{\top}\bar{{\mathbf{C}}}_t({\mathbf{x}}^*- {\mathbf{y}}^*)} \underset{(b)}{=} 0,
\end{align*}
where (a) follows via Eq. \eqref{eq:Lip} since each $g(\cdot)$ in the support of $\mathcal{D}_t$ is convex and $G$-Lipschitz for some finite $G>0$, and (b) follows from Eq. \eqref{eq:lem:uniformMinimizers:1}. Thus, the first part of the lemma follows.

The second part of the lemma is a straightforward consequence of Lemma \ref{lem:hoffmanExpectation}, when applied to the distribution $\mathcal{D}$, defined above.
\end{proof}

\begin{proof}[Proof of Theorem \ref{thm:ogd:master}]
Let us denote the set of of minimizers in hindsight: $\mathcal{X}^* = \argmin_{{\mathbf{x}}\in\mathcal{P}}\frac{1}{T}\sum_{t=1}^T\mathbb{E}_{f_t\sim\mathcal{D}_t}\left[{f_t({\mathbf{x}})}\right]$.

Given the sequence of points generated by Algorithm \ref{alg:ogd} $\{{\mathbf{x}}_t\}_{t=1}^T$,  we define the sequence $\{{\mathbf{x}}_t^*\}_{t=1}^T$ as follows:
$\forall t\geq 1$: ${\mathbf{x}}_t^* := \argmin_{{\mathbf{x}}\in\mathcal{X}^*}\Vert{{\mathbf{x}}-{\mathbf{x}}_t}\Vert^2$,
i.e., ${\mathbf{x}}_t^*$ is the projection of ${\mathbf{x}}_t$ onto the set of optimal plays in hindsight $\mathcal{X}^*$.

Let us fix some ${\mathbf{x}}^*\in\mathcal{X}^*$. By an application of Lemma \ref{lem:uniformMinimizers}, it holds that
{\small
\begin{align*}
\max_{{\mathbf{x}}\in\mathcal{P}}\mathbb{E}\left[{\sum_{t=1}^Tf_t({\mathbf{x}}_t) - f_t({\mathbf{x}})}\right] &= \mathbb{E}\left[{\sum_{t=1}^T f_t({\mathbf{x}}_t) - f_t({\mathbf{x}}^*)}\right] \\
&= \mathbb{E}\left[{\sum_{t=1}^T f_t({\mathbf{x}}_t) - f_t({\mathbf{x}}_t^*)}\right].
\end{align*}}

Thus, to prove the theorem, it suffices to upper bound $R_T := \mathbb{E}_{f_1\dots f_T}\left[{\sum_{t=1}^Tf_t({\mathbf{x}}_t) - \sum_{t=1}^Tf_t({\mathbf{x}}_t^*)}\right]$.

Let us also define the sequence $\{r_t := f_t({\mathbf{x}}_t)-f_t({\mathbf{x}}_t^*)\}_{t=1}^T$. Throughout the rest of the proof we write $\nabla_t$ as a short notation for $\nabla{}f_t({\mathbf{x}}_t)
$.

As standard in the analysis of Online Gradient Descent, for every $t\geq 1$, we have that
\vspace{-5pt}
{\small
\begin{align*}
\Vert{{\mathbf{x}}_{t+1} - {\mathbf{x}}_t^*}\Vert^2 &\leq  \Vert{{\mathbf{y}}_{t+1} - {\mathbf{x}}_t^*}\Vert^2 = \Vert{{\mathbf{x}}_t - \eta_t\nabla_t - {\mathbf{x}}_t^*}\Vert^2 \\
&= \Vert{{\mathbf{x}}_t-{\mathbf{x}}_t^*}\Vert^2 - 2\eta_t({\mathbf{x}}_t-{\mathbf{x}}_t^*)^{\top}\nabla_t + \eta_t^2\Vert{\nabla_t}\Vert^2,
\end{align*}}
where the first inequality holds since ${\mathbf{x}}_{t+1}$ is the orthogonal projection of ${\mathbf{y}}_{t+1}$ onto $\mathcal{P}$.
Rearranging and recalling that $f_t({\mathbf{x}}) = g_t({\mathbf{C}}_t{\mathbf{x}})$ for some $g_t,{\mathbf{C}}_t$, we have that
{\small
\begin{align*}
({\mathbf{C}}_t({\mathbf{x}}_t-{\mathbf{x}}_t^*))^{\top}\nabla{}g_t({\mathbf{C}}_t{\mathbf{x}}_t) =& ({\mathbf{x}}_t - {\mathbf{x}}_t^*)^{\top}\nabla_t  \leq  \frac{1}{2\eta_t}\Vert{{\mathbf{x}}_t-{\mathbf{x}}_t^*}\Vert^2 \\
&- \frac{1}{2\eta_t}\Vert{{\mathbf{x}}_{t+1}-{\mathbf{x}}_t^*}\Vert^2 + \frac{\eta_t}{2}\Vert{\nabla_t}\Vert^2.
\end{align*}
}
Since $g_t(\cdot)$ is $\alpha_1$-strongly convex, we have
{\small
\begin{align*}
r_t  = g_t({\mathbf{C}}_t{\mathbf{x}}_t) - g_t({\mathbf{C}}_t{\mathbf{x}}_t^*) \leq & \frac{1}{2\eta_t}\Vert{{\mathbf{x}}_t-{\mathbf{x}}_t^*}\Vert^2 - \frac{1}{2\eta_t}\Vert{{\mathbf{x}}_{t+1}-{\mathbf{x}}_t^*}\Vert^2  \\
&-\frac{\alpha_1}{2}\Vert{{\mathbf{C}}_t({\mathbf{x}}_t-{\mathbf{x}}_t^*)}\Vert^2  + \frac{\eta_t}{2}\Vert{\nabla_t}\Vert^2 .
\end{align*}}

By the definition of the sequence $\{{\mathbf{x}}_t^*\}_{t=1}^T$, we have
\begin{align*}
r_t \leq &\frac{1}{2\eta_t}\textrm{dist}({\mathbf{x}}_t,\mathcal{X}^*)^2 - \frac{1}{2\eta_t}\textrm{dist}({\mathbf{x}}_{t+1},\mathcal{X}^*)^2  \\
& - \frac{\alpha_1}{2}\Vert{{\mathbf{C}}_t({\mathbf{x}}_t-{\mathbf{x}}_t^*)}\Vert^2  + \frac{\eta_t\Vert{\nabla_t}\Vert^2}{2}.
\end{align*}

Summing over all $T$ iterations, rearranging and taking expectation on both sides, we have that {\small
\begin{align}\label{eq:ogd:master:1}
R_T &\leq  \mathbb{E}_{f_1\dots f_T}\Bigg[\frac{1}{2\eta_1}\textrm{dist}({\mathbf{x}}_1,\mathcal{X}^*)^2 + \frac{\bar{G}^2}{2}\sum_{t=1}^T\eta_t \nonumber \\
&+ \frac{1}{2}\sum_{t=2}^T\Big(\frac{1}{\eta_t}\textrm{dist}({\mathbf{x}}_t,\mathcal{X}^*)^2 - \frac{1}{\eta_{t-1}}\textrm{dist}({\mathbf{x}}_t,\mathcal{X}^*)^2 \nonumber \\
& -\alpha_1\Vert{{\mathbf{C}}_t({\mathbf{x}}_t - {\mathbf{x}}_t^*)}\Vert^2\Big)\Bigg]\nonumber \\
& \leq \frac{1}{2}\mathbb{E}_{f_1\dots f_t}\Bigg[\sum_{t=2}^T\left({\frac{1}{\eta_t} - \frac{1}{\eta_{t-1}}}\right)\textrm{dist}({\mathbf{x}}_t,\mathcal{X}^*)^2 \nonumber\\
&- \alpha_1\Vert{{\mathbf{C}}_t({\mathbf{x}}_t - {\mathbf{x}}_t^*)}\Vert^2\Bigg] + \frac{D^2}{2\eta_1}+ \frac{\bar{G}^2}{2}\sum_{t=1}^T\eta_t,
\end{align}}
where we have used the fact that $\forall t\in[T]:$ $\mathbb{E}_{\mathcal{D}_t}[\Vert{\nabla{}f_t({\mathbf{x}}_t)}\Vert^2] \leq \bar{G}^2$.

Note that since for all $t$, ${\mathbf{C}}_t$ is independent of ${\mathbf{x}}_t,{\mathbf{x}}_t^*$, we have that
{\small
\begin{align}\label{eq:ogd:master:2}
&\mathbb{E}_{f_1\dots f_t}\Bigg[\left({\frac{1}{\eta_t} - \frac{1}{\eta_{t-1}}}\right)\textrm{dist}({\mathbf{x}}_t,\mathcal{X}^*)^2 - \alpha_1\Vert{{\mathbf{C}}_t({\mathbf{x}}_t - {\mathbf{x}}_t^*)}\Vert^2\Bigg]  =\nonumber\\
&\mathbb{E}_{f_1\dots f_{t-1}}\Bigg[\left({\frac{1}{\eta_t} - \frac{1}{\eta_{t-1}}}\right)\textrm{dist}({\mathbf{x}}_t,\mathcal{X}^*)^2 \nonumber \\
&- \alpha_1({\mathbf{x}}_t - {\mathbf{x}}_t^*)^{\top}\mathbb{E}_{f_t}[{\mathbf{C}}_t^{\top}{\mathbf{C}}_t]({\mathbf{x}}_t - {\mathbf{x}}_t^*)\Bigg]= \nonumber\\
&\mathbb{E}_{f_1\dots f_{t-1}}\Bigg[\left({\frac{1}{\eta_t} - \frac{1}{\eta_{t-1}}}\right)\textrm{dist}({\mathbf{x}}_t,\mathcal{X}^*)^2 - \alpha_1\Vert{\bar{{\mathbf{C}}_t}({\mathbf{x}}_t - {\mathbf{x}}_t^*)}\Vert^2\Bigg],
\end{align}}
where we let $\bar{{\mathbf{C}}}_t$ be such that $\bar{{\mathbf{C}}}_t^{\top}\bar{{\mathbf{C}}}_t = \mathbb{E}_{(g,{\mathbf{C}}_t)\sim\mathcal{D}_t}[{\mathbf{C}}_t^{\top}{\mathbf{C}}_t]$.

By Lemma \ref{lem:uniformMinimizers} and the assumption of the theorem, it holds that
\begin{eqnarray*}
\forall t\in[T]: \quad \textrm{dist}({\mathbf{x}}_t,\mathcal{X}^*)^2 &\leq &\sigma^{-1}\Vert{\bar{{\mathbf{C}}}({\mathbf{x}}_t-{\mathbf{x}}_t^*)}\Vert^2 \\
&\leq &(\alpha_2\sigma)^{-1}\Vert{\bar{{\mathbf{C}}}_t({\mathbf{x}}_t-{\mathbf{x}}_t^*)}\Vert^2.
\end{eqnarray*}
Thus, since the step-size $\eta_t$ is monotonically non-increasing with $t$, we have that $\forall t\geq 2$:
{\small
\begin{eqnarray}\label{eq:ogd:master:3}
&&\left({\frac{1}{\eta_t} - \frac{1}{\eta_{t-1}}}\right)\textrm{dist}({\mathbf{x}}_t,\mathcal{X}^*)^2 - \alpha_1\Vert{\bar{{\mathbf{C}}_t}({\mathbf{x}}_t - {\mathbf{x}}_t^*)}\Vert^2 \nonumber \\
&& \leq \left({\left({\frac{1}{\eta_t} - \frac{1}{\eta_{t-1}}}\right)\frac{1}{\alpha_2\sigma} - \alpha_1}\right)\Vert{\bar{{\mathbf{C}}}_t({\mathbf{x}}_t - {\mathbf{x}}_t^*)}\Vert^2.
\end{eqnarray}
}

Combining Eq. \eqref{eq:ogd:master:1}, \eqref{eq:ogd:master:2}, \eqref{eq:ogd:master:3}, and plugging-in our choice of step-size $\eta_t = \frac{1}{\alpha_1\alpha_2\sigma{}t}$, we conclude that
{\small
\begin{eqnarray*}
R_T \leq \frac{D^2}{2\eta_1}+ \frac{\bar{G}^2}{2}\sum_{t=1}^T\eta_t
\leq \frac{\alpha_1\alpha_2\sigma{}D^2}{2} + \frac{\bar{G}^2}{2\alpha_1\alpha_2\sigma}(1+ \ln{T}).
\end{eqnarray*}}
\end{proof}

\begin{figure*}[h]
    \centering
          \begin{subfigure}{0.3\textwidth}
        \includegraphics[width=1.8in]{random_d085_log.eps}

        \caption{synthetic}
      \end{subfigure}
    \begin{subfigure}{0.3\textwidth}
        \includegraphics[width=1.8in]{mnist_m1_log.eps}

        \caption{MNIST}
     \end{subfigure}
    \begin{subfigure}{0.3\textwidth}
        \includegraphics[width=1.8in]{cifar10_m1_log.eps}

        \caption{CIFAR10}
     \end{subfigure}
    \caption{Comparing the (log) average loss vs. number of iterations for OGD and ONS.}\label{fig:1}
\end{figure*}

\subsection{Applications of Theorem \ref{thm:ogd:master}}\label{subsec:app}
\paragraph{Deterministic data:}
In case  $g_1,\dots g_T:\mathbb{R}^m\rightarrow\mathbb{R}$ are arbitrary $\alpha_1$-strongly convex and differentiable functions over $\mathbb{R}^m$, then a sufficient condition on the matrices ${\mathbf{C}}_1,\dots,{\mathbf{C}}_T$ for applying the result of Theorem \ref{thm:ogd:master} is that there exists a positive constant $\alpha_2$ such that
\begin{eqnarray}\label{eq:deterministicData}
\forall (i,j)\in[T]\times[T]: \qquad {\mathbf{C}}_i^{\top}{\mathbf{C}}_i \succeq \alpha_2{\mathbf{C}}_j^{\top}{\mathbf{C}}_j,
\end{eqnarray}
or in a different formulation: $\textrm{row-span}({\mathbf{C}}_1) = \textrm{row-span}({\mathbf{C}}_2) = \dots = \textrm{row-span}({\mathbf{C}}_T)$.

A simple application of Theorem \ref{thm:ogd:master} upper-bounds the regret by $\frac{\alpha_1\alpha_2\sigma{}D^2}{2} + \frac{G^2}{2\alpha_1\alpha_2\sigma}(1 + \ln{T})$,
where $\sigma$ is the Hoffman constant of $\frac{1}{T}\sum_{t=1}^T{\mathbf{C}}_t^{\top}{\mathbf{C}}_t$ w.r.t. the polytope $\mathcal{P}$.

\paragraph{Semi-adversarial data:}
 A way to circumvent the limitation of condition \eqref{eq:deterministicData}, is to consider slightly ``easier" data. In particular if we let $\{g_t(\cdot)\}_{t\in[T]}$ be as in the deterministic case, but we assume that the matrices ${\mathbf{C}}_1,\dots,{\mathbf{C}}_T$ are perturbed realizations of some underlying deterministic sequence $\tilde{{\mathbf{C}}}_1,\dots,\tilde{{\mathbf{C}}}_T$. That is, we let $\tilde{{\mathbf{C}}}_t$ be arbitrary, but the observed matrix ${\mathbf{C}}_t$ is a perturbed version given by ${\mathbf{C}}_t = \tilde{{\mathbf{C}}}_t + {\mathbf{N}}_t$, where ${\mathbf{N}}_t\sim\mathcal{D}$, where $\mathcal{D}$ is a fixed unknown distribution. Then, the condition in \eqref{eq:deterministicData} could be easily replaced by the requirement: $\forall t\in[T]$,  $\mathbb{E}_{\mathcal{D}}[{\mathbf{N}}] = \mathbf{0}$ and $\mathbb{E}_{\mathcal{D}}[{\mathbf{N}}^{\top}{\mathbf{N}}] \succeq \alpha_2\tilde{{\mathbf{C}}}_t^{\top}\tilde{{\mathbf{C}}}_t$, where $\alpha_2$, as before, is a positive constant.

An application of Theorem \ref{thm:ogd:master} upper-bounds the expected regret by $\frac{\alpha_1\alpha_2\sigma{}D^2}{2} + \frac{\bar{G}^2}{2\alpha_1\alpha_2\sigma}(1 + \ln{T})$,
where $\sigma$ is the Hoffman constant of the expected matrix $\frac{1}{T}\sum_{t=1}^T{\mathbf{C}}_t^{\top}{\mathbf{C}}_t + \mathbb{E}_{\mathcal{D}}[{\mathbf{N}}^{\top}{\mathbf{N}}]$, w.r.t. the polytope $\mathcal{P}$.
As mentioned in Section \ref{sec:examples}, such a setting underlies for instance the \textit{Universal Linear Filtering} problem studied in \cite{Moon2009, GarberH13}.

\paragraph{Shifting stochastic data:} Assuming $(g_t(\cdot),{\mathbf{C}}_t)$ is sampled out of a distribution $\mathcal{D}_t$ (possibly changing from round to round), under assumptions on $g_t(\cdot)$ as above, a sufficient condition on the stochastic matrices ${\mathbf{C}}_t$ for applying the result of Theorem \ref{thm:ogd:master} is that there exists some $\alpha_2>0$ such that for all $t_1,t_2\in[T]$: $\mathbb{E}_{{\mathbf{C}}_1\sim\mathcal{D}_{t_1}}[{\mathbf{C}}_1^{\top}{\mathbf{C}}_1] \succeq \alpha_2\mathbb{E}_{{\mathbf{C}}_2\sim\mathcal{D}_{t_2}}[{\mathbf{C}}_2^{\top}{\mathbf{C}}_2]$, in which case we get the same bound as in the ``semi-adversarial" case with $\sigma$ being the Hoffman constant of $\frac{1}{T}\sum_{t=1}^T\mathbb{E}_{{\mathbf{C}}\sim\mathcal{D}_t}[{\mathbf{C}}^{\top}{\mathbf{C}}]$ w.r.t. $\mathcal{P}$.

\section{Experiments}

In this section we provide empirical evidence for the performance of Online Gradient Descent on curved, though not strongly convex, losses. Since the computational advantage of OGD over competing methods is clear, we focus on demonstrating convergence in terms of the average loss.
We consider the LASSO optimization problem, i.e., the loss function on each round $t$ is $f_t({\mathbf{x}}) := \frac{1}{2}\Vert{{\mathbf{a}}_t^{\top}{\mathbf{x}}-b_t}\Vert^2$ and the feasible polytope is an $\ell_1$ ball. In all experiments we compare OGD with Online Newton Step (ONS).

\textbf{Synthetic data:} We compare OGD and ONS in an online stochastic setting.
We fix the dimension to $d=100$ and generate a random PSD matrix ${\mathbf{M}}\in\mathbb{R}^{d\times d}$ with rank = $50$ and with decaying eigenvalues given by $\lambda_i = 10\cdot{}0.8^{i-1}$ for all $i\in[50]$. We set ${\mathbf{a}}_t = {\mathbf{v}}_t^{\top}{\mathbf{M}}$ for a random unit vector ${\mathbf{v}}_t$, and $b_t := {\mathbf{a}}_t^{\top}{\mathbf{w}}^*+0.1n_t$, where ${\mathbf{w}}^*$ is a fixed sparse vector chosen at random, and $n_t\sim\mathcal{N}(0, 1)$. The radius of the feasible $\ell_1$ ball is set to $r=10$.
Since determining the Hoffman constant is difficult in general, we heuristically set $\sigma$ according the eigenvalue of the covariance matrix ${\mathbf{A}} = \frac{1}{T}\sum_{t=1}^T{\mathbf{a}}_t{\mathbf{a}}_t^{\top}$ which corresponds to the numerical rank of ${\mathbf{A}}$. We use this choice in all of our experiments which seems to work well.
 and set the step-size accordingly to $\eta_t = \frac{1}{\sigma{}t}$ (note $g_t(\cdot)$ in our case is $1$-strongly convex). ONS is implemented as suggested in \cite{Hazan16}. For both methods we plot the (log) average loss vs. number of iterations (we use $T=10000$).

\textbf{MNIST data:} Next we experiment with the MNIST handwritten digit recognition dataset \cite{Lecun1998}. Specifically, we use the training dataset, keeping only the data related to digits $3,5$. We  set $b_t$ by assigning value $1$ to instances corresponding to the digit $5$ and $-1$ to those corresponding to $3$. Finally, in order to increase the amount of data, we replicate the data and concatenate twice. We set the radius of the feasible $\ell_1$ ball to $r=5$.

\textbf{CIFAR10 data:} we use the CIFAR10 tiny image dataset \cite{cifar-10} which contains 50000 32x32 images in RGB format. We convert the images to grayscale and keep only the data related to the classes "automobile" and "truck", assigning the first the label $b_t = -1$ and the second the label $b_t = 1$. Here we also replicate the data twice and set $r=5$.

The results for all datasets are presented in Figure \ref{fig:1}. It is clearly observable that in all three cases OGD is comparable to ONS in terms of regret and even far better.

\section{Acknowledgments}
This research was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 1108/18).

redpajama_set_name: RedPajamaArXiv


text:
The Silk Railway: freight train from China pulls up in Madrid
En route it passed through Kazakhstan, Russia, Belarus, Poland, Germany and France before arriving at the Abroñigal freight terminal in Madrid. It consisted of 30 containers carrying 1,400 tonnes of cargo – mostly toys, stationery and other items for sale over Christmas across Europe. It is also Spain's third largest source of imports, after Germany and France. However, the cargo had to be transferred three times during the journey as a result of incompatible rail gauges. MORE
Spain Kazakhstan Belarus Germany 40
Beijing-Hamburg freight service completes maiden journey
Global Cargo Monitoring / Cargo Tracking. By having many service providers involved, which among each other are often not connected, the friction in moving goods from source to destination are very high. The "Beijing-Hamburg Container Express" left the Chinese capital on January 9 with its cargo of shoes, toys and electronic goods and covered the distance of 10,000 kilometres (6,200 miles) in 15 days, Germanys state-owned rail operator Deutsche Bahn said. MORE
Mongolia Belarus Germany Cargo 40
Belarus Cargo Sourcing Transportation
En route it passed through Kazakhstan, Russia, Belarus, Poland, Germany and France before arriving at the Abroñigal freight terminal in Madrid. It consisted of 30 containers carrying 1,400 tonnes of cargo – mostly toys, stationery and other items for sale over Christmas across Europe. It is also Spain's third largest source of imports, after Germany and France. However, the cargo had to be transferred three times during the journey as a result of incompatible rail gauges.
Global Cargo Monitoring / Cargo Tracking. By having many service providers involved, which among each other are often not connected, the friction in moving goods from source to destination are very high. The "Beijing-Hamburg Container Express" left the Chinese capital on January 9 with its cargo of shoes, toys and electronic goods and covered the distance of 10,000 kilometres (6,200 miles) in 15 days, Germanys state-owned rail operator Deutsche Bahn said.
Mongolia 40

redpajama_set_name: RedPajamaCommonCrawl


text:
The tossers agony download firefox
The current design of firefox for windows desktop notice the similarities with the. Ginger would have been 2yrs july 1509 she was gaurding our house killing bugs. Uninstall the quicktime player and itll allow you to download. Agony is the sixth studio album by chicago celtic punk band the tossers. Available in any file format including fbx, obj, max, 3ds, c4d. Use filters to find rigged, animated, lowpoly or free 3d models. The firefox os app manager is a new developer tool available in firefox 26 that greatly improves the process of building and debugging firefox os apps, either in the simulator or on a connected device. You the user need to be aware of the dangers and avoid trusting chrome and firefox for now. If i type in a url, it hesitates after you click on it. The new async and await keywordswhich make asynchronous code more concise, obvious, and maintainablehave arrived in firefox 52.
Mozilla firefox, or simply firefox, is a free and opensource web browser developed by the mozilla foundation and its subsidiary, mozilla corporation. We recommend you to use latest chrome, firefox or safari. If mystique means concealing that youre a tosser, it might already be too late. Free download firefox firefox mozilla sex porn videos. Tip enable hidden secret dark and light themes in mozilla. Mozillas firefox os phone might sell because its cheap. It was released on march 7, 2007 by victory records. Starting today, firefox works with a hesitation on everything i try to do. After a forced landing, the hijacked firefox lies entombed in a frozen lake 20 miles from the norwegian frontier. In 2017, firefox began incorporating new technology under the code name quantum to promote parallelism and a more intuitive user. They formed in the early 90s following in the wake of the pogues and have toured with lost city angels, murphys law, streetlight manifesto, catch 22, dropkick murphys, lagwagon, and the siderunners. Feb 01, 2007 released in november of 2004, firefox version 1. When im typing information in a message box, i have to wait for it to catch up.
After the success of firefox, craig thomas used a retcon an alteration of previously established facts to set up a sequel. Jul 10, 2015 firefox for windows 10 is likely to arrive in august, but it remains to be seen how this project develops. Rixsteps new agony aunt column by famous tech analyst rob enderle. Firefox 2 as it used to be, but now with the power of the latest firefox engine beneath it. Shop for vinyl, cds and more from the tossers at the discogs marketplace. The latest version of the mozilla foundations browser, released tuesday, is all about. Its pilot, mitchell gant, is running for his life from the tracker dogs and helicopter patrols of the kgb border guard. Mitchell baker and the firefox paradox mozilla open. Hi all, i have an aspx webpage with buttons on to download files. This record not only shows a band that is growing in their writing and playing, but in life in general. Since then, mozilla firefox has consistently featured in the top 3 most popular browsers globally and this is set to continue thanks to the release of firefox 30. I saw the recent feature they did on madonna, the firefox theme creator with over 17,000 themes. Writers centre in byron bay or may be downloaded at au. The worst abusers tried to send you to a page to download their app, apparently not.
This one doesnt block ads at all but it sure does a good job of blocking my whole photography website. The new version firefox 57 is now available to download for windows, macos, and linux. In all browsers other than firefox, these buttons work perfectly everytime. Mar 03, 2017 the flagship firefox team and supporting platform team had been complaining about a lack of resources for a while, and with firefox market share slipping the finger of blame was pointed at firefox os. In the end i had download chromium in order get to page with firefox installators.
Sep 14, 2016 since then, mozilla firefox has consistently featured in the top 3 most popular browsers globally and this is set to continue thanks to the release of firefox 30. Fionn regan fir bolg fire in the attic firefox ak firespawn firewind. I went to the same websites on chrome, opera and ie. Beautiful agony bustygirl cums and squeezes her tits. No other sex tube is more popular and features more firefox mozilla sex scenes than pornhub. How to open a new support ticket to open a new support ticket, first select the relevant forum and then click the new post button. The flagship firefox team and supporting platform team had been complaining about a lack of resources for a while, and with firefox market share.
Aug 03, 2008 plays video instead of downloading on firefox my friend made a really funny video that i want to download and it is in avi format. The 1982 movie by clint eastwood, based on the novel. Clint eastwood was the director, producer, and played mitchell gant. Watch download firefox firefox mozilla sex porn videos for free on pornhub page 2. Were not affiliated or endorsed by the mozilla corporation but we love them just the same. Jul 29, 2018 download dark theme for youtube for firefox. I am curious though, does she just take other peoples artwork and basically just crop it down and call it her own.
Watch firefox mozilla sex porn videos for free, here on. When using firefox 10 and 11, the first time you click a button, the the dialog box to save the file appears and the download works. Watch our impressive selection of porn videos in hd quality on any device you own. For every field that is filled out correctly, points will be rewarded, some fields are optional but the more you provide the more you will get rewarded.
She got one that was poisened,and had to be put to sleep. The cold war plot involves an attempt by the cia and mi6 to steal a highly advanced experimental soviet fighter aircraft. I run firefox with ublock on my phone because autoplay video ads are so. Introducing the firefox os app manager mozilla hacks the.
If you already got that covered, you may be interested in a real full dark theme for firefox one that turns all firefox about. Dark theme for youtube get this extension for firefox. On oldschool sites the end of a url is a string of meaningless query parameters, but on good restful sites its the important part of the url, often including a meaningful position in the fragment identifier, and i. Orthodox takes you back to the original netscape 4 look. The songs are tales of reflection, friendship and just living life to the fullest. Firefox was released on november 9, 2004, and challenged internet explorers dominance with 60 million downloads within nine months. At its public launch in 2004 mozilla firefox was the first browser to challenge microsoft internet explorers dominance. Currently available in the latest developer edition release, firefox 52 is scheduled for general release in march 2017 javascript owes its excellent singlethreaded performance and responsiveness on the web to its pervasively asynchronous design. Find the tossers discography, albums and singles on allmusic.
Formulating before irish mainstays flogging molly and dropkick murphys, the tossers may have all the credibility in the world. The key features that have made mozilla firefox so popular are the simple and effective ui, browser speed and strong security capabilities. I can download them in safari i am on a mac, but im tired of switiching browsers everytime i need to download a yousentit link or any. Introducing the firefox os app manager mozilla hacks. This was gradually accelerated further in late 2019, so that new major releases occur on fourweek cycles starting in 2020. Firefox is a thriller novel written by craig thomas and published in 1977. Nov 15, 2017 the new version firefox 57 is now available to download for windows, macos, and linux. Firefox uses the gecko layout engine to render web pages, which implements current and anticipated web standards. Sure let me just download the source, compile it which will take another 2 hour to meet all the dependecies. Firefox is the spiritual successor of netscape navigator, as the mozilla community was created by netscape in 1998 before their acquisition by aol.
May 12, 2020 new version releases of browsers dont get the buzz they used to get, but firefox quantum is an exception. This might seem as a silly problem but it is extremly frustrating when youre trying to get installator for different system. Ciociosan and the other waitresses smirked as, in an agony of politeness, i swallowed it all. Their most recent release, agony victory records came out in march 2007. Oct 15, 20 the firefox os app manager is a new developer tool available in firefox 26 that greatly improves the process of building and debugging firefox os apps, either in the simulator or on a connected device. That way you wont get whiteflashed by the addons or preferences pages as well as getting a dark scrollbar for all the darkened pages you would already see if you use. Sundays season opener pitted reigning champs the byron tossers against the left bankers from mullumbimby. Firefox quantum is over twice as fast as firefox from 6 months ago, built on a completely overhauled core engine with brand new technology stolen from our advanced research group, and graced with a beautiful new look designed to get out of the way and let you do what you do best. Firefox is created by a global nonprofit dedicated to putting individuals in control online.
New version releases of browsers dont get the buzz they used to get, but firefox quantum is an exception. Aug 06, 2007 through this method, we hope to be able to seed firefox throughout the facebook community. Quicktime for example, you click the link and it plays on firefox with the quicktime player. Firefox was created by dave hyatt and blake ross as an experimental branch of the mozilla browser, first released as firefox 1. The tossers the toten crackhuren im kofferraum the tourist the tower. Firefox was made into a film by warner brothers based on the novel and released in 1982. Lyrics for not alone from agony, released march 6, 2007. Mozilla firefox is a fast, light and tidy open source web browser. The abrupt ending of the first novel makes this retcon possible, if a little implausible, the. Dec 07, 2016 the new async and await keywordswhich make asynchronous code more concise, obvious, and maintainablehave arrived in firefox 52. Nov 14, 2017 check out the new firefox browser on windows, mac or linux. The book was subject to a 1982 film adaptation produced and directed by clint eastwood who also played the role of gant in the film.
Orgasm beautiful agony free videos watch, download and. Well worth a read if you enjoy cold war era thrillers. Firefox marketplace offers apps in categories like games, news and media, music and audio, business and productivity so you can find an app for. Discover the growing collection of high quality most relevant xxx movies and clips. Down below down by law down my throat downchild download downpilot. Currently available in the latest developer edition release, firefox 52 is scheduled for general release in march 2017. Firefox for windows 10 is likely to arrive in august, but it remains to be seen how this project develops. Mozilla firefox is periodically updated by our editors team but sometimes you may find out that software informations are outdated, please note that the publisher of mozilla firefox can update the software without notifying us. If you want to switch between dark, light and default firefox theme frequently and dont want to use about. Get firefox for windows, macos, linux, android and ios today. Try using an old ultrabook 2nd gen i5 as i did for some time while using chrome or firefox. Firefox wont let me download anything mozillazine forums.
I remember the rosary, the catechism, the stations of the cross, the plenary. Discover the growing collection of high quality download firefox firefox mozilla sex xxx movies and clips. The tossers are a sevenpiece celtic punk band from chicago. Hate meditation hatebreed hateful agony hater hatesphere haudegen. Theres more that could be said about all the amazing work that went into quantum, or about some of the exciting stuff in the very near future, but at this point you should stop reading and download firefox quantum. Having released a handful of albums on windy city label thick records since 2000, chicagos the tossers didnt break out to national. A few thousand volunteer marketers had joined the campaign. How to download firefox from firefox firefox support.
No other sex tube is more popular and features more download firefox firefox mozilla sex scenes than pornhub. Czilla firefox setu type choose setup options choose the type of setup you prefer, then click next. Asyncawait arrive in firefox mozilla hacks the web. The topic firefox is closed to new replies, this may be for inactivity, please open a new topic if you have a similar question.
The chief protagonist is fighter pilot turned spy mitchell gant. How to download firefox from firefox firefox support forum. Browse through our impressive selection of porn videos in hd quality on any device you own. Firefox is written in a very particular, almost staccato, style that can be a little hard to follow at first, but it has a way of instilling you with a sense of the frustration the characters are feeling. Through this method, we hope to be able to seed firefox throughout the facebook community. I spent more than 15 minutes trying to figure this out.
New firefox runs like a rabbit web apps technewsworld. Firefox for osx used to let me download these things, and this problem was happening even before i downloaded the newest version yesterday. Its because you have the codec installed of the script that it needs to run on. To read this page, please turn off your ad blocker. Agony coming from the predominantly irish neighborhoods of southside chicago, the tossers have been expanding the boundaries of contemporary irish music since the early 90s.
If a direct wick has led you here, please correct the link so that it points to. The novel firefox down is a continuation of the story of firefox, beginning at the moment at which the previous book had concluded. We use cookies for various purposes including analytics. Versions 2, 3, and 4 came out in 2006, 2008, and 2011, after which they changed the criteria for being a new version and started to roll out new releases faster. Strange file download behaviour only in firefox mozilla. By continuing to use pastebin, you agree to our use of cookies as described in the cookies policy. Mozilla website hides all firefox download buttons when you browse it from firefox. The new appearance will also be available on ios and android. The tossers return in 2007 with their new victory fulllength, agony. The band chose their name for its derogatory meaning of. Now firefox sits on top with 46%, smacking internet explorer down to second. If you think this addon violates mozillas addon policies or has security or privacy issues, please report these issues to mozilla using this form please dont use this form to report bugs or request addon features. If you restore firefox customization settings using firefox menu button customize restore defaults option, itll also restore default firefox theme. Click to download woman, girl, female, face, browser, person, firefox, customer, user, sexy icon from browser girl iconset by.
1289 780 1101 1239 285 999 1560 999 1130 786 1058 922 50 1639 1066 672 1414 1028 1159 165 719 460 918 238 796 715 500 1278 988 1251 128 64 603 1319 381 438 1262 1458 1300 270 5 959 518 1431