You know, at first I only meant this to be awkward on account of the fact that Tabitha's "one of those creepy science-y people" and Stilez still isn't really sure about her. But the more I think about it, Stilez would probably be weirded out by a show of affection from ANYBODY who isn't Tax. I mean, even Ichabod, one of the few people she openly likes, is still clearly afraid of her. But this? This is just a thing that people don't do, not to her, anyway.
Cue awkward comment from Tabitha about how soft Stilez fur is, or that she has amazing muscle tone, or something like that. At least she jumped up to hug Stilez around her neck instead at their relative standing levels - that would have just made it even more awkward. Fortunately, "embarrassed Stilez" is probably not a thing that exists, so Layla will have to go ahead and be embarrassed on her behalf, even though the remark wasn't directed at her this time.
"Embarassed Stilez" is actually a thing!
We could possibly infer from this that spontaneous displays of genuine affection (from anyone other than Tax) are actually capable of embarassing Stilez, so she actually could be embarassed by Tabitha's hug if she gets past the confused "What the heck is this weird girl doing??" stage.


The Reasons Why We Need Fingerprint Services.
Finger printing is a concept that have been used by many people especially them that are in the police so that they can be able to determine the people that were involved in some sought of criminal activities. These days, the fingerprinting services have gone to many other people and companies because of the need to personalize items and other things that are considered as personal properties.
They are important in the making of the doors that only uses the concept of the fingerprint so that the items that are locked in there are able to be safe from the external factors. This is very important because it is something that can be done so that people do not go into stealing other people's properties. In this article, we are going to look at the importance of the fingerprinting near me in the public and the private sector. t is something that is very important.
The other important thing is that people are in the need of these services so that they can be able to have the things that are really important in the whole idea of having the doors that are locked by the fingers. This is something that is very important to any given persona even the company in general. This is because, the doors that are locked by the use of the finger prints can only be here with us and no one can be able to force some of the entry into the building if he or she is not given the license to do this as nevada notary application. It is something that we should be able to do and even admit.
We can be able to tame the cyber crime if we have the biometric systems in our personal computers so that we can be able to prevent the effect of the people misusing our computers. This is important so that we can be able to protect our very own data from people that are not authorized and even causing the same data to go missing.


This blissful Crystal Therapy combines the the amazing healing power of crystal, with Prana Energy and the purest organic essential oils and crystal tinctures and oils which is done by dipping the gemstones in the oil.
EnergyThis therapy revitalises your energy centers or Chakras, re balances your internal organs while improve your psychological, spiritual and emotional health. The energy work combined with gemstones works on a deeper spiritual level releasing blockages embedded in your energy centres and any negative karma that affect your life negatively. This blissful Crystal Therapy combines the the amazing representative of Mother Nature of crystal, with Prana Energy and the purest organic essential oils and crystal tinctures or even crystal oils, which is done by dipping the gemstones in the oil.


The Landing 8A is a top floor 2 bedroom, 2 bathroom condo is located right on Lake Hamilton. It includes a King bed in the master and a Queen bed in the guest room. It also features a full size roll-a-way bed and sleeper sofa. This unit also has a sleeper sofa in the living room. Not only is it beautifully decorated, but it also comes with a wonderful view overlooking the water. This unit has a flat screen TV/DVD/VCR in the living room and a TV in each bedroom. The living room has a fireplace and while the weather is nice, there's a seasonal swimming pool as well as an electric grill on the balcony.
We loved the condo and the view of the lake.Very quiet complex and close to all shopping.
Debbie, thank you for your review on this property. It is a lovely place! Hope to have you back!
This condo has great views, everything you need to be comfortable, and close to town. The only thing I would say they need to improve on is the trash disposal...the dumpster outside the complex is really gross and unsightly.
Great condo! We were extremely happy with the condo and the location. We will definitely be back.
The condo was everything we expected. This is a well taken care condo that you can tell the owners take care of and keep updated.
Great location and a perfect view!
Very clean and close to many restaurants.
Great condo enjoyed our stay.


Mother is not only the reason why you are here; she is also invariably the most influential person in your life. Such is her personality and preponderance in your thought-process that you will have to feel for words while writing an essay on her.
Her indomitable spirit – Mothers have this quality to rear children into strong individuals, regardless of the hard times she may face during the upbringing. Her spirit never dies! You can write on that point of view.
Her caring nature – She is there for you as you need your late night coffee; she is there to feel you as you suddenly need to leave home before sunrise. Her bestowals and caring hand is ever on you. Wonders never cease as to how she does it.
Her selflessness – The relation that a mother engenders with her kids is the most selfless; every other relation may be fraught with tension, bigotry and other lows; but this relation rocks.
Her vision – Mothers have this extra-sensory perception of what her child should do in life and what he/she is good at. She will naturally mold the child in the particular way.
Her arrangements – Even when you feel that you are cash-strung, she will somehow make sure that all your needs are answered and catered to. Again, how she does it; only she can explain.
Her strength of character – She passes her strength of character into her kids with such aplomb that you can only appreciate it.
Her sacrifices – Just as you go out to play, you hardly realize that someone is there in the kitchen making food to feed you as you return. Her sacrifices are untold but great.
Her understanding – In cases where the father has to spend much if his time outside, mothers understand your need to get connected to parents and go on an overtime.
Her lack of demand – Even when you become a man of substance, you will hardly find your mother demanding anything from you. It is always a one-way street for her.
Her vitality – When you feel despondent, dull or insouciant, all you need is a session with your mother to revel in energy once again. She is better than any shrink!
This article asks forgiveness from fathers, whose contributions are also enormous in your upbringing. It is just that mothers' personality is so enormous that fathers tend to play second fiddle.


CBI is a mobile application that runs on Android and iOS devices.
CBI access data from comics.org and helps you build a detailes list of of your comic book inventory. You can track what issues you have, which artists you have in your collection and you can even find out which issues you are missing in your favorite titles.


Our Deodorant Stick uses a gentle blend of essential oils and natural ingredients like Aloe Vera, Tea Tree and Sage to help soothe skin. Combats odor-causing bacteria, giving you the long-lasting freshness you need to feel confident all day. This powerful deodorant is the perfect alternative to traditional antiperspirants that may contain triclosan, phthalates, synthetic preservatives and other harmful chemicals.
You need a long-lasting deodorant that will keep fighting odors all day, but you don't want to resort to using messy, toxic chemical-laden deodorants that can clog pores and irritate skin.
all day! Hydrates and keeps skin soft and smooth with its special formula, unlike deodorants made with potentially harmful ingredients.
Apply on underarm area for long-lasting freshness.
• Unique, biodegradable formula is made with natural ingredients.
• The human underarm is among the most consistently warm areas on the surface of the human body.
• Sweat itself is odorless. It's the bacteria on the skin that mingles with it that produces body odor.


text:
#ifndef WorkerPermissionClient_h
#define WorkerPermissionClient_h

#include "core/workers/WorkerClients.h"
#include "wtf/Forward.h"

namespace WebCore {
class ExecutionContext;
}

namespace blink {

class WebFrame;
class WebString;
class WebWorkerPermissionClientProxy;

class WorkerPermissionClient : public WebCore::Supplement<WebCore::WorkerClients> {
public:
    static PassOwnPtr<WorkerPermissionClient> create(PassOwnPtr<WebWorkerPermissionClientProxy>);

    virtual ~WorkerPermissionClient();

    bool allowDatabase(const WebString& name, const WebString& displayName, unsigned long estimatedSize);
    bool allowFileSystem();
    bool allowIndexedDB(const WebString& name);

    static const char* supplementName();
    static WorkerPermissionClient* from(WebCore::ExecutionContext*);

private:
    explicit WorkerPermissionClient(PassOwnPtr<WebWorkerPermissionClientProxy>);

    OwnPtr<WebWorkerPermissionClientProxy> m_proxy;
};

void providePermissionClientToWorker(WebCore::WorkerClients*, PassOwnPtr<WebWorkerPermissionClientProxy>);

} // namespace blink

#endif // WorkerPermissionClient_h

redpajama_set_name: RedPajamaGithub


text:
NCRC launches $5 million Field Empowerment Fund to support nonprofits and CDFIs that promote wealth in traditionally underserved communities
By NCRC / January 19, 2021 January 19, 2021 / Press Releases
Today, the National Community Reinvestment Coalition (NCRC) announced a new $5 million grant program funded by Morgan Stanley to help its member organizations recover and rebuild from the coronavirus pandemic.
The Field Empowerment Fund (FEF) will supply grants ranging in amount from $50,000 to $250,000 for a one-year period with the goal of scaling up innovative investments that promote wealth in traditionally underserved communities.
"Our member organizations are the lifeblood of the work we do to fight for a more just economy," said Jesse Van Tol, CEO of NCRC. "They are working on the frontlines of communities most in need all across the country. They understand the urgency of helping communities rebound from the coronavirus pandemic and they know the best way to do it. NCRC has always supported our members in many ways, but this fund gives us a new opportunity to engage and support them in one of the most meaningful ways – providing direct funds for their innovative programs."
Awardees must be current NCRC members and be able to prove that they are either a 501(c)(3) organization or a U.S. Treasury CDFI Fund certified CDFI. Selection criteria focuses on organizations that:
Operate in markets that have been significantly impacted by the pandemic
Are led by people of color and/or women
Serve low-income and low-wealth communities, and communities of color
Increase access to affordable housing and homeownership
Serve small businesses and micro-businesses.
All applications are to be submitted online no later than 11:59 pm ET on February 19, 2021.
For more information and to submit an application, visit: www.ncrc.org/fef/

redpajama_set_name: RedPajamaCommonCrawl


text:
Home/Dresses/Mehndi Dresses/Mehndi Dresses for Boys Best Outfits 2023
DressesMehndi Dresses
Ibtisam Ellahi Send an email 3 weeks ago
Mehndi is a component of Asian weddings. In Asia before getting married, a customary celebration is held where all families of both the bride and groom dress in yellow for the Mehndi celebration. This is why today we will share the gorgeous Mehndi Dresses for Guys 2023. Mehndi is a classic wedding ceremony and on the day of the wedding, the Bride and Groom also wear the traditional Yellow shade of the dress.
In this Mehndi celebration, relatives and members of the Bride and Groom mark the hands of the newlyweds by wearing a traditional brown Mehndi to send them best wishes. Today we present incredible pictures of Mehndi Dresses for Men and Groom.
Mehndi Clothing For Couple Wear – Stylish New Mehndi Outfits Ideas For Boys
The wedding day is the most significant and crucial in every couple's life. The young couple, can enjoy the occasion and then on the next day, they begin a new chapter in their lives with the promise of a whole life in the shape of a new couple. The wedding day is the most important event of the year and the guests' focus is always directed to the new couple. So they must appear perfect and elegant in fashionable Mehndi Dresses for Men.
The grooms of Pakistan as well as India grooms typically prefer traditional attire. For the occasion of weddings, and as well on other wedding events the traditional dress is always the best choice for groomsmen. This article also discussed the significance of traditional attire for the Mehndi ceremony, particularly for groomsmen. You may recall that men were not particularly at all a few years ago. In his appearance or dress even on wedding days. However, fashion has changed, and also men pay attention and attach importance to their dress. On the day of the wedding ceremony, men choose the most appropriate attire they could for themselves.
Traditional Mehndi Dresses for Boys New Designs 2023
It is a common practice in Pakistan and India that Mehndi-themed daytime clothes for men typically include Shalwar Kameez Kurta Pajama and Sherwani with a simple or Churidar Pajama. The most elegantly designed Kurtas with extravagant Shalwars are among the most popular traditional outfits. These are the dresses preferred by Pakistani and Indian guys on the day of the Mehndi celebrations. Many prefer high-end brands or the finest designer's traditional dresses for wedding ceremonies. In Pakistan the famous designers HSY, Deepak Perwani, Nomi Ansari, and Amir Adnan. They are gaining fame for their Mehndi Dresses for Boys 2023.
See More: Stylish Men Wear Mehndi Dresses 2023 Ideas
Ibtisam Ellahi
I am Ibtisam Ellahi and I write about fashion on WahtInFashion. I keep up with the latest news and trends in the industry so that my readers always have access to the latest information. I love all things fashion, from clothes to makeup to hair, and enjoy sharing my knowledge with others.
New & Stylish Mehndi Dresses For Pakistani Brides 2023


text:
Irwin Mitchell Expands Regulatory Team In The North
New Senior Associate Appointed
Law firm Irwin Mitchell has added to its renowned northern Regulatory & Criminal Investigations Group (RCG) by appointing a new Senior Associate to work across the Sheffield and Manchester offices.
Joanne Hall moves north from the London office to join the RCG North team at Irwin Mitchell as a qualified barrister and has spent time on secondment to both the Serious Fraud Office (SFO) and Financial Services Authority (FSA).
The team now comprises of 11 people in the North and also draws on expertise from other offices, including London when required to ensure the most appropriate expertise for each case. This includes the ability to put together multi-disciplinary legal teams to deal with matters involving, for example, a civil, or employment law element.
Paul Haycock, Associate Director in Irwin Mitchell's RCG team, said: "We are delighted to welcome Joanne to our specialist team. She has a wide range of experience working across a range of regulatory investigations.
"With the Bribery Act just around the corner it is important that businesses get up to speed with their due diligence processes and Joanne's experience with the SFO and FSA will be a real benefit to our clients".
The Irwin Mitchell RCG team has a strong reputation for defending complicated criminal investigations and prosecutions and is currently handling a caseload where the fraud is alleged to exceed £150m.
Irwin Mitchell has seen a rise in the number of instructions to defend in criminal fraud cases recently, including include those brought by HM Revenue and Customs, the Fraud Prosecution Teams of the Crown Prosecution Service and the Serious Fraud Office.
Irwin Mitchell Advises On Anglian Refinancing
ICO Announces Intention To Fine Marriott International £99m For Data Breach
First Major UK GDPR Fine: ICO Announces Intention To Fine British Airways £183m For Data Breach
Irwin Mitchell Advise JLL In First Letting at "Landmark", St Peter's Square Manchester
Irwin Mitchell Advise El Gato Negro on New Restaurant Space in Leeds


Free online dating and matchmaking service for singles. 3,000,000 Daily Active Online Dating Users. Start online dating with Match. Sign up and get access to our free dating trials as well as singles night and events near you.
In the beginning of the summer a very attractive, young lady mailed me. She had only one profile picture and I was in doubt because she could be a fake. The fastest and the easiest way to find your perfect match. Sign up for QuickFlirt and have crazy fun online.
The Netherlands have a tolerant approach to the drug abuse of soft drugs like this page as soon as the source (http:///drugs/) is Discover Amsterdam with this tourism and travel guide: things to do, accommodations, festivals, transportation, culture, nightlife, and restaurants.
One in ten Americans have used an online dating site or mobile dating app; 66% of them have gone on a date with someone they met through a dating site or app, Ranging from hellos and question, to funny and flirty messages that help you stand out; browse over 100 creative online dating first messages examples.
Plenty Of Fish Dating, The Bottom Line. Plenty Of Fish dating is a fun and interesting way to meet new people for romance, relationships or other singles looking for something more casual. If you have yet to try POF dating you may want to give it a go. It is a fun, free and exciting way to meet people and socialize. POF is the preferred singles dating app because you can view matches AND communicate with them for FREE (unlike the paid dating apps). Unlike smaller dating apps, POF has the most users and thus, the highest chance for you to find your relationship! The POF Dating App has the most users, generates the best results and is FREE.


Untuck and get comfy… it's time for another love story!
Director Ira Sach will debut his highly anticipated semi-autobiographic romantic drama 'Keep The Lights On' at the movies tomorrow (Sept. 7th). The film "chronicles an emotionally and sexually charged journey of two men in New York City through love, friendship, and addiction. " A closeted lawyer and a documentary film maker come together though a "casual encounter." After finding love between each other, they discover they love drugs and other people even more.
The relationship lasted ten years. So somebody must have been (getting their bills paid or) laying the pipe right if you ask me.
Will you be seeing 'Keep The Lights On' this weekend?I want to!
Gay Film 'You Are Not Alone' Premieres This Weekend!


We are happy to announce that Gi60 #NextGen is back again for a fifth round and play submissions are now open!
The concept stays the same: one minute plays performed in succession by a group of actors, to be recorded and posted on YouTube for your viewing pleasure. So what's different? The group of actors will be entirely comprised of the Next Generation: all of them ages 11-16. For this portion we've teamed up with Acting Out! Brooklyn's premier acting school, to bring 20 brand new one-minute plays to life. All we need are your words.
How To Apply: Gi60 is open to everybody; all you need is an idea and an internet connection. Since its inception in 2003, the Gi60 festival has premiered hundreds of new pieces of work written by people of all ages and backgrounds from around the world. So if you have and idea or something to say, why not write a play?
• Individuals may submit as many one minute plays as they wish (one submission only per email). PLEASE SEND EACH PLAY IN A SEPARATE FILE WITH THE TITLE OF THE PLAY IN THE SUBJECT LINE.
• Plays lasting longer than 60 seconds will not be considered.
• Final play selection will be announced sometime in early May, 2018. Playwrights will be contacted via email to be informed of the selections.
- Performances for Gi60 #NextGen 2018 will take place the On June 7th and June 9th with performances at Acting Out! and at Studio 307, Roosevelt Extension Brooklyn College (Both in Brooklyn, NY).
Gi60 is nothing without the imagination, skills and enthusiasm of our writers - and for this we thank you!!


Good composition can mean many things to different people. We think it's really important to educate the eye not just in terms of what 'good' design looks like but also from the point of the viewers or users of the final product. Great master artists used something called the Golden Ratio/ Section to help shape their work and this is a good place to start. Our clients are encouraged to give us an example of good compositions and designs they like in order to start a dialogue that shapes the final product whether it is an illustration, an interface or course materials. We also take our role seriously as advocates for the end users drawing on good practice such as those of Jakob Nielsen Another issue to consider is the composition of the project team, the right crew with the right skills combined with all the stakeholders is so important in designing a successful project.


Export Import Data Smart Chip Ltd, is a premier company that exports and imports goods from countries including Singapore, Hong Kong and host of other countries.
Smart Chip Ltd, is a premier company that exports and imports goods from countries including Singapore, Hong Kong and host of other countries.
Smart Chip Ltd is a leading exporter of RE -EXPORT OF DEFECTIVE PARTS FOR REPAIR S AND RETURN SWIFT SIM (RANIA), having HS Codes 99930002. The company has its head office at 7, 2Nd Floor Community Centre, New Delhi110065.
Smart Chip Ltd is also a leading importer of 1 KB SECURED MEMORY SMART CARD, SMART CARD EMOSYN TP40, SMART CARD READER SCR 331, SMART CARD TP62-8M V3.1 UNPERSONALISED, SMART CARD, TP-40 V2.0 (UNPERSONALISED ), having HS Codes 85421020.
The company primarily imports products to India from Hong Kong and mainly receives the consignments at Delhi Air.


text:
Marc Overmars Provides Update on Hakim Ziyech's Future Amid Bayern Munich Interest
Manchester United's Dean Henderson Emerges as Surprise Summer Target for Bayern and Juventus
HT Media Editorial 07 Feb 2019
Manchester United goalkeeper Dean Henderson has caught many eyes with his performances on loan at Sheffield United this season, as European heavyweights Bayern Munich and Juventus have been named among a host of sides interested in the 21-year-old shot-stopper.
The England Under-21 international has been one of the outstanding goalkeepers in the Championship so far, not missing a minute of action as the Blades maintain a surprise push for promotion.
 'HE'S ONE OF OUR OWN' 
What's been your favourite moment during Chris Wilder's 1000 days as Blades boss?#SUFC pic.twitter.com/VMMjv1Geil
— Sheffield United (@SUFC_tweets) February 6, 2019
It follows an impressive season out on loan at Shrewsbury, where he was named in the League One team of the year, and it seems as if his development has not gone unnoticed.
According to ESPN, both Bayern and Juve are keeping an eye on him with a view for a summer move when his loan at Brammall Lane comes to an end, with Arsenal and Tottenham also named as interested parties for his signature.
At present, Henderson is thought to be fourth choice at Manchester United, and with David de Gea being David de Gea, it doesn't appear as if there is too much in the way of first-team opportunities at Old Trafford on the immediate horizon.
That being the case, a permanent move away would not come as a surprise, but the feeling is that United are desperate to keep hold of the player, who signed a new two-year deal in the summer.
He is reportedly seen as a future number one provided he continues his development at his current rate, and his career will certainly be an interesting one to follow in the years to come.


Any meal is only as good as the quality of its ingredients. Cosmopolitan Catering produces great meals as a result of the ingredients we start with.
Cosmopolitan Catering provides high quality food from our breakfast menu to our desserts.
Cosmopolitan Catering provides high-quality food and strives to be recognized for producing delicious, healthy, and socially responsible meals. We start with whole, raw, single-ingredient food that is processed by hand from scratch. We go to market daily for the freshest produce available and use high-grade proteins in every menu we create. As all of our food is ordered only on demand, we have no need for a sitting inventory of shelf-stable products. Our food is designed to be enjoyed as we produce it, not weeks or months later. Our philosophy to menu creation and production lend itself to consistently producing delicious food. Experience Cosmopolitan Catering and you will taste the difference.
Great food starts with great ingredients and food always tastes the best when you eat from your backyard. Cosmopolitan Catering has spent over a decade developing strong relationships with ranchers and farmers throughout California to ensure that we deliver the highest quality meals to our guests without compromise. We are committed to making socially responsible buying decisions that support local business, reduce our carbon footprint, and that are economically sound.
Breakfast has been said to be the most important meal of the day, and we can't disagree. We offer everything from Continental and Country breakfasts to brunch a la carte. Cosmo Catering knows the importance of a breakfast that will help you stay focused on what is important.
Made fresh daily, our salads, sandwiches, and wraps are the perfect way to compliment a lunch buffet to add variety, complete your employee breakroom, or build a box lunch for quick service. Our menu features dozens of options each month with unique choices offered week over week. Click on the menus below and reach out to Cosmopolitan Catering to begin planning your next lunch service.
Cosmopolitan Catering's lunch menus are written to achieve three distinct characteristics – Quality, Variety, and Consistency. We produce menus that rival restaurants and leave your employees satisfied. Our menus are comprised of over 120 distinct meals that rotate monthly and with the season, making it possible to eat our food every day for months without seeing the same menu twice. With Cosmopolitan Catering, you can come to expect a service that can be relied upon day in and day out.
Cosmo Catering offers a wide arrange of appetizers that will leave you wanting more. We offer a variety of options from which to choose that will help appease the pickiest taste. We will sit down with you and learn what you are wanting to achieve. We ask the questions to help your event be a success.
Every meal needs to be delicious to the last bite! Cosmopolitan Catering offers delicious dessert options to compliment every menu. Whether it is a decadent chocolate fondue station, an artfully displayed fruit board, a variety of fresh baked cookies, or anything in between, we will satisfy your taste buds from start to finish.


Due to new regulations imposed by government and society, your systems must constantly comply with the latest requirements and rules. The improvement of the reliability and sustainability of your installations is thus a continuous process. You must combine new developments, such as the extraction of raw materials from waste water and the energy-neutral operation of your installation with controlling costs and managing a reliable installation. Furthermore, your employees should develop knowledge of these new elements of your company.
You want your installations to operate properly and reliably at minimal cost. Power consumption plays an important role in this. Thanks to the latest technology and our knowledge of the treatment steps, we will provide you with insight into and control over your installations. Our process control and visualisation will allow your employees to quickly detect malfunctions and variances in your production process and make the necessary adjustments. Even remotely, you can quickly respond to alarms. This is a necessity rather than a luxury if you process biogas at your site.
With a simple push of a button we will provide you with a clear overview of the Information on the consumption of your installation on sub or component level. You are in control.
We involve more and more partners and suppliers so that we can have the most sustainable and energy efficient components available for your installations. This will help you make your installations even more energy efficient, and thus more energy neutral, in addition to generating energy through combined heat and power plants and biogas plants.


text:
\section{Proof of the universality of $h_u$}\label{a:relsmooth_proof}

\begin{proposition}[restatement of Proposition \ref{prop:rel_smoothness}]
    Assume that $F:{\mathbb R}^{n \times n}\rightarrow {\mathbb R}$ is a continuously differentiable convex function, and that $\nabla F$ is Lipschitz continuous with constant $L_F$. Then the function $f:{\mathbb R}^{n\times r} \rightarrow {\mathbb R}$ defined by $f(X) = F(XX^T)$ is $1$-smooth relatively to the distance kernel $h_u$ for $\alpha \geq 6L_F$ and $\sigma \geq 2\|\nabla F(0)\|$.
\end{proposition}

\begin{proof}
Let us first prove the result for twice continuously differentiable $f$, so as to use the Hessian characterization \eqref{eq:hessian_cond}. For $X, U \in {\mathbb R}^{n \times r}$, the second derivative of $h_u$ is written
\begin{equation}\label{eq:hess_h}
\begin{split}
    \nabla^2 h_u(X)[U,U] &= \alpha \left(\|X\|^2 \|U\|^2 + 2 \la X, U \ra^2 \right)  + \sigma \|U\|^2 \\
         &\geq \alpha \|X\|^2 \|U\|^2 + \sigma \|U\|^2
\end{split}
\end{equation}
On the other hand, the second derivative of $f$ is
\begin{equation}
    \nabla^2 f(X)[U,U] = \nabla^2 F(XX^T)[UX^T + XU^T, UX^T + XU^T] + 2 \la \nabla F(XX^T), UU^T \ra
\end{equation}
Since $F$ has a Lipschitz continuous gradient, the standard second derivative inequality yields
\[
\nabla^2 F(XX^T)[UX^T + XU^T, UX^T + XU^T] \leq L_F \|UX^T + XU^T\|^2.
\]
Now, the second term can be bounded by using the triangle inequality, the Cauchy-Schwarz inequality and the gradient Lipschitz property, to get
\begin{equation}
    \begin{split}
      \la \nabla F(XX^T), UU^T \ra &= \la  \nabla F(0), UU^T \ra + \la \nabla F(XX^T) - \nabla F(0), UU^T \ra\\
        &\leq \|\nabla F(0)\| \, \|U\|^2 + \|\nabla F(XX^T) - \nabla F(0)\| \, \|UU^T\| \\
        &\leq \Big( \|\nabla F(0)\| + L_F\|XX^T\| \Big) \|U\|^2
    \end{split}
\end{equation}
hence
\begin{equation}\label{eq:hess_f_maj}
    \begin{split}
        \nabla^2 f(X)[U,U] &\leq L_F \|UX^T + XU^T\|^2 + 2 \left(L_F\|XX^T\| + \|\nabla F(0)\| \right) \|U\|^2 \\
        &\leq 2L_F \left( \|UX^T\|^2 + \|XU^T\|^2 \right) + 2 \left(L_F\|XX^T\| + \|\nabla F(0)\| \right) \|U\|^2 \\
        &\leq 6L_F \|X\|^2 \|U\|^2 + 2 \|\nabla F(0)\| \|U\|^2 \\
        &\leq \alpha \|X\|^2 \|U\|^2 + \sigma \|U\|^2
    \end{split}
\end{equation}
    Where we used the submultiplicative property of the Frobenius norm, and our choice of parameters $\alpha,\sigma$. Combining \eqref{eq:hess_h} and \eqref{eq:hess_f_maj} gives that $\nabla^2 f(X)[U,U] \leq \nabla^2 h_u(X)[U,U]$ for all $X, U \in {\mathbb R}^{n\times r}$, hence that $f$ is $1$-smooth relatively to $h$ \citep{Bauschke2017}.

    Now, in the case that $f$ is not twice differentiable, we can apply Alexandrov's theorem \cite{Alexandrov1939} which states that a convex function is twice differentiable almost everywhere. Therefore $\nabla^2 F$ is defined almost everywhere, hence $\nabla^2 f$ also, and the same proof applies.
\end{proof}

\section{NoLips with dynamical step size}
\label{a:dyn_nolips}

In this section, we analyze the variant of NoLips with a dynamical step size strategy, and establish the same convergence properties as for the standard version from \citet{Bolte2018}.

\textbf{Setup.} Recall that the minimization problem is

\begin{equation}\label{eq:dyn_min_problem_app}
    \min_{X \in {\mathbb R}^{n \times r}} \Psi(X) \triangleq f(X) + g(X)
\end{equation}

where $f:\rnr \rightarrow {\mathbb R}$ is a continuously differentiable function and $g: \rnr \rightarrow {\mathbb R} \cup \{+\infty\}$ a convex proper lower semicontinuous function. Note that, contrary to the nonconvex NoLips algorithm presented in \cite{Bolte2018}, we need the convexity of the penalty function $g$ in our analysis.
We assume in addition that the problem is bounded from below, i.e. $\inf_{X \in \rnr} \Psi(X) > - \infty$.

We are provided with a distance kernel $h:\rnr \rightarrow {\mathbb R}$, which we assume twice continuously differentiable on $\rnr$, $\sigma$-strongly convex, and defined everywhere. $h$ is chosen adapted to $f$, in the sense that the relative smoothness condition holds with constant $L$:

\begin{equation}\label{eq:dyn_rel_smooth}
        f(X) \leq f(Y) + \la \nabla f(Y), X-Y \ra + L D_h(X,Y)
\end{equation}

for all $X,Y \in \rnr$.

\textbf{Dyn-NoLips scheme.}
Algorithm \ref{algo:dyn_nolips} specifies the variant of NoLips that allows to take more aggressive step size values than $1/L$. The step size strategy is adapted from the one presented in \cite{Nesterov2007}: at each iteration, we try to increase the step size $\lambda$ and ensure that it satisfies the local decrease condition, writing $X^+ = T_\lambda(X)$:

\begin{equation}\label{eq:local_dec_cond}
            f(X^+) \leq f(X) + \la \nabla f(X), X^+-X \ra + \frac{1}{\lambda} D_h(X^+,X)
\end{equation}

\begin{algorithm}[H]
	\begin{algorithmic}
		\STATE {\bfseries Input:} A distance kernel $h$, initial step size $\lambda_0$, maximal step size $\lambda_{\rm max}$.
		\STATE Initialize $X \in {\mathbb R}^{n\times r}$ such that $\Psi(X) < \infty$
		\STATE Set $\lambda \leftarrow \lambda_0$
        \REPEAT
    		\REPEAT
    		    \STATE Compute $X^+ = T_{\lambda}(X)$
		        \IF{the decrease condition ~\eqref{eq:local_dec_cond} is not satisfied with $(X^+, X)$}
		            \STATE Set $\lambda \leftarrow \lambda / 2$
		        \ENDIF
    		\UNTIL{decrease condition ~\eqref{eq:local_dec_cond} is satisfied}
    		\STATE Set $X \leftarrow X^+ $
    		\STATE Set $\lambda \leftarrow \min(2\lambda, \lambda_{\rm max})$
		\UNTIL{convergence criterion is satisfied}
	\end{algorithmic}
	\caption{NoLips with dynamical step size}
	\label{algo:dyn_nolips}
\end{algorithm}

The first lemma guarantees that the sequence of step size values lies in a fixed interval.

\begin{lemma}\label{lemma:stepsize_bound}
    Let $\{ \lambda_k\}_{k \geq 0}$ be the sequence of step sizes chosen at each iteration of Algorithm \ref{algo:dyn_nolips}. Then for every $k \geq 0$ we have
    \begin{equation}\label{eq:stepsize_bound}
        \frac{1}{2L} \leq \lambda_k \leq \lambda_{\rm max}
    \end{equation}
\end{lemma}

\textit{Proof.}
    The upper bound holds by construction of the algorithm. The lower bound comes from the the relative smoothness property: the condition \eqref{eq:local_dec_cond} is true for every $\lambda \in (0, \frac{1}{L}]$, so the inner loop will stop whenever $\lambda$ gets below $1/L$.
~~\rule[-1pt]{6pt}{6pt}}\def\QED{\QED

\textbf{Number of calls to oracle per iteration.} From Lemma~\ref{lemma:stepsize_bound}, we  conclude that at each iteration, the number of inner loops is bounded by $\lfloor \log_2(L \lambda_{\rm max})\rfloor + 1$.

We now prove the following Lemma, that establishes a crucial inequality for proving the convergence.
\begin{lemma}\label{lemma:decrease_eq}
    Let $\{X^k\}_{k\geq 0}$ the sequence generated by Algorithm~\ref{algo:dyn_nolips}, and $\{\lambda_k\}_{k\geq 0}$ the corresponding sequence of step size values. Then for every $k \geq 0$,
    \begin{equation}\label{eq:decrease_eq}
        \Psi(X^{k+1}) - \Psi(X^k) \leq  -\frac{1}{\lambda_k} D_h(X^k, X^{k+1})
    \end{equation}
\end{lemma}

\textit{Proof.}
    Since the condition \eqref{eq:local_dec_cond} holds at each iteration $k$, we can write
    \begin{equation}\label{eq:dec_f}
            f(X^{k+1}) \leq f(X^k) + \la \nabla f(X^k), X^{k+1} - X^k \ra + \frac{1}{\lambda_k} D_h(X^{k+1}, X^k)
    \end{equation}
    On the other hand, the optimality condition characterizing $X^{k+1} = T_{\lambda_k}(X^k)$ writes
    \begin{equation}\label{eq:cond_optim}
        0 \in \lambda_k \left(\partial g(X^{k+1}) + \nabla f(X^k) \right) + \nabla h(X^{k+1}) - \nabla h(X^{k})
    \end{equation}
    Where $\partial g$ denotes the subdifferential of the convex function $g$. Combining \eqref{eq:cond_optim} with the subgradient inequality for $g$ yields
    \begin{equation}\label{eq:dec_g}
            g(X^{k+1})\leq g(X^k) +\frac{1}{\lambda_k} \la \nabla h(X^k) - \nabla h(X^{k+1}), X^{k+1} - X^k \ra - \la \nabla f(X^k), X^{k+1} - X^k \ra
    \end{equation}
    Summing \eqref{eq:dec_f} and \eqref{eq:dec_g} gives
    \begin{align*}
        \Psi(X^{k+1}) &\leq \Psi(X^k) + \frac{1}{\lambda_k} \big[ D_h(X^{k+1}, X^k)  + \la \nabla h(X^k) - \nabla h(X^{k+1}), X^{k+1} - X^k  \ra \big] \\
        &= \Psi(X^k) - \frac{1}{\lambda_k} D_h(X^{k}, X^{k+1})
    \end{align*}
~~\rule[-1pt]{6pt}{6pt}}\def\QED{\QED

We can now prove the same convergence properties as for the standard NoLips scheme:
\begin{theorem}[restatement of Theorem \ref{thm:nolips}]
Let $\{X^k\}_{k \geq 0}$ be the sequence generated by Algorithm \ref{algo:dyn_nolips}. Assume that
    \begin{enumerate}
        \item $f$ is $L$-smooth relatively to a distance kernel $h$, $h$ is strongly convex on ${\mathbb R}^{n\times r}$, and the penalty function $g$ is convex.
        \item The function $\Psi = f+g$ is coercive (meaning that $\Psi(X)\rightarrow + \infty$ when $\|X\|\rightarrow +\infty$) and semialgebraic.
    \end{enumerate}
Then the  sequence $\{\Psi(X^k)\}_{k \geq 0}$ is nonincreasing, and the sequence $\{X^k\}_{k \geq 0}$ converges towards a critical point $X^*$ of problem~\eqref{eq:nolips_min_problem}.
\end{theorem}

\textit{Proof.} The monotonicity of the sequence $\{\Psi(X^k)\}_{k \geq 0}$ is a direct consequence of Lemma \ref{lemma:decrease_eq} and the fact that $D_h(\cdot,\cdot)$ is nonnegative. Since $\lambda_k \leq \lambda_{\rm max}$, it follows from this lemma that at every iteration $k \geq 0$,
\begin{equation}
    \Psi(X^{k}) - \Psi(X^{k+1}) \geq \frac{1}{\lambda_{\rm max}} D_h(X^k, X^{k+1})
\end{equation}

    Now, this inequality is the same as the one needed to prove convergence in the case of the fixed step size in \cite{Bolte2018}.

    Thus, global convergence towards a critical point is a consequence of \citep[Th.~4.1]{Bolte2018}, since all the assumptions are met:
    the kernel $h$ is defined over the entire space ${\mathbb R}^{n \times r}$, it is strongly convex, and $\nabla h$ is Lipschitz continuous on bounded subsets of ${\mathbb R}^{n \times r}$ (because we assumed it $C^2$). We also need the fact that the sequence $\{X^k\}_{k \geq 0}$ is bounded, which is a consequence of the monotonicity of $\{\Psi(X^k)\}_{k \geq 0}$   and the fact that the function $\Psi$ is coercive.

~~\rule[-1pt]{6pt}{6pt}}\def\QED{\QED

\section{Gram kernel}\label{a:gram}

In this section, we study the Gram kernel

\begin{equation}
    h_g(X) = \frac{\alpha}{4}\|X\|^4 + \frac{\beta}{4}\|X^T X\|^2 + \frac{\sigma}{2}\|X\|^2
\end{equation}

\subsection{Proof of Proposition \ref{prop:gram_gradient_map}}
\label{ss:proof_gram_kernel}

We recall the expression for the Bregman iteration map of the Gram kernel $h_g$:

\begin{proposition}[restatement of Proposition \ref{prop:gram_gradient_map}]
Assume that there is no penalty term, i.e. $g \equiv 0$.
Let  $X \in {\mathbb R}^{n \times r}$, and form the matrix $V = \nabla h_g(X) - \lambda \nabla f(X)$. Write the eigendecomposition of $V^T V = P^T D P$ where $P \in \mathcal{O}_r$ and $D = \mathop{\bf diag}(\eta_1^2,\dots,\eta_r^2)$, then the Bregman iteration map of the Gram kernel $h_g$ is given by
    \begin{equation}
        T_\lambda(X) = V \left[\alpha \mathop{\bf Tr}(Z) I_r + \beta Z + \sigma I_r \right]^{-1}  \quad \text{where} \quad
        Z = P^T \mathop{\bf diag}\big[ \mu_1^2,\dots, \mu_r^2 \big] P
    \end{equation}
    and $\mu = (\mu_1, \dots,\mu_r)$ is the unique solution of the convex minimization problem
    \begin{equation}\label{eq:subproblem_a}
        \min_{x \in {\mathbb R}^r} \phi(x) \triangleq \frac{\alpha}{4}\|x\|^4 + \frac{\beta}{4}\sum_{i=1}^r x_i^4 + \frac{\sigma}{2}\|x\|^2 - \sum_{i=1}^r \eta_i x_i.
    \end{equation}
\end{proposition}

\begin{proof}
The Bregman proximal iteration map of $h_g$ writes, for $X \in {\mathbb R}^{n\times r}$

\begin{equation}\label{eq:breg_map_gram}
    \begin{split}
        T_\lambda(X) &= \mathop{\rm argmin}_{U \in {\mathbb R}^{n\times r}} \left\{\la \nabla f(X), U-X \ra + \frac{1}{\lambda} D_{h_g}(U,X) \right\} \\
        &= \mathop{\rm argmin}_{U \in {\mathbb R}^{n\times r}} \left\{ h_g(U) - \la V,U \ra \right\} \\
    \end{split}
\end{equation}

where we remove constant terms and defined $V \triangleq \nabla h_g(X) - \lambda \nabla f(X)$.

Write for the sake of clarity $U^\star \triangleq T_\lambda(X)$. The optimization problem \eqref{eq:breg_map_gram} is strictly convex and the unique solution $U^\star$ verifies $\nabla h_g(U^\star) = V$, meaning that

\begin{equation}\label{eq:Z_cond}
    U^\star \left(\alpha \|U^\star\|^2 I_r + \beta U^{\star T} U^\star + \sigma I_r \right) = V
\end{equation}

Define $Z \triangleq U^{\star T} U^\star \in {\mathbb R}^{r \times r}$. Then the knowledge of $Z$ determines $U^\star$, since $\|U^\star\|^2 = \mathop{\bf Tr}(Z)$ and therefore $U^\star = V (\alpha \mathop{\bf Tr}(Z) I_r + \beta Z + \sigma I_r)^{-1} $.

Now, taking \eqref{eq:Z_cond} and multiplying by its transpose implies that

\begin{equation}\label{eq:z_v_eq}
    (\beta Z + \alpha \|U^\star\|^2 I_r + \sigma I_r)^2 Z = V^T V
\end{equation}

This shows that $V^T V$ is a polynomial in $Z$, and therefore that they admit the same egeinvectors. Write the diagonalization $V^T V = P^T \mathop{\bf diag}(\eta_1^2,\dots,\eta_r^2) P$ and $Z = P^T \mathop{\bf diag}(\mu_1^2,\dots,\mu_r^2) P$ where $P \in \mathcal{O}_r$ and we assume $\mu_i,\eta_i \geq 0$ for $i = 1\dots r$. It follows from diagonalizing \eqref{eq:z_v_eq} and taking the square root that

\begin{equation}
    \left( \alpha (\sum_{j=1}^r \mu_j^2 ) + \beta \mu_i^2 + \sigma \right) \mu_i = \eta_i \quad \forall i = 1,\dots,r
\end{equation}

This is exactly the first-order optimality conditions on $\mu = (\mu_1,\dots,\mu_r)$ for

\begin{equation}
    \mu = \mathop{\rm argmin}_{x \in {\mathbb R}^r} \frac{\alpha}{4}\|x\|^4 + \frac{\beta}{4}\sum_{i=1}^r x_i^4 + \frac{\sigma}{2}\|x\|^2 - \sum_{i=1}^r \eta_i x_i.
\end{equation}

Note that we do not need to enforce the nonnegativity constraint on $x$, since we chose $\eta_i \geq 0$ and it is easy to see that the optimal solution will be nonnegative.
\end{proof}

\subsection{Computing the Bregman iteration map efficiently}
\label{ss:gram_kernel_algo}

While it seems that computing the Bregman iteration map of the Gram kernel involves solving another difficult quartic subproblem, it is actually of small size ($r$ is typically not larger than a few dozens) and can be solved efficiently with the universal NoLips scheme.

Indeed, the objective function $\phi$ of problem \eqref{eq:subproblem_a} is 1-smooth relatively to the universal kernel in ${\mathbb R}^r$ $h_u(x) = \frac{\alpha_u}{4}\|x\|^4 + \frac{\sigma_u}{2}\|x\|^2$ with a choice of parameters $\alpha_u = \alpha + 3 \beta$ and $\sigma_u = \sigma$.

Algorithm \ref{algo:gram_kernel} details the procedure. We initialize $\mu$ with the values for the previous iteration of the outer procedure. This proves to be efficient as the values will not vary much from one iteration to another. For the stopping criterion, we use the scaled gradient norm $\|\nabla \phi(v)\|/\|\eta\|$ and a tolerance value $\epsilon = 10^{-6}$.

\begin{algorithm}[H]
	\begin{algorithmic}
		\STATE {\bfseries Input:} Matrix $X \in {\mathbb R}^{n \times r}$, gradient of the objective $\nabla f(X)$, step size $\lambda > 0$, parameters $\alpha, \beta,\sigma > 0$, subproblem tolerance $\epsilon$, and (optionally), values $\mu^-$ of $\mu$ computed at the previous iteration.
		\STATE
		\STATE Form $V = \nabla h_g(X) - \lambda \nabla f(X) =  \left(\alpha \|X\|^2 I_r + \beta X^{T} X + \sigma I_r \right) X - \lambda \nabla f(X)$
		\STATE Compute $V^T V$
		\STATE Form the eigendecomposition of $V^T V = P^T D P$ where $P \in \mathcal{O}_r$ and $D = \mathop{\bf diag}(\eta_1^2,\dots,\eta_r^2)$
		\STATE Initialize $\mu$ as $\mu^-$ if provided, and as $(0,\dots,0)$ otherwise.
        \REPEAT
            \STATE Compute $\nabla \phi(\mu)$ where $\nabla \phi(\mu)_i = \alpha \|\mu\|^2 \mu_i + \beta \mu_i^3 + \sigma \mu_i - \eta_i$
            \STATE Compute $\nabla h_u(\mu)$ where $\nabla h_u(\mu)_i = (\alpha + 3 \beta) \|\mu\|^2 \mu_i + \sigma \mu_i$
            \STATE Form $v = \nabla h_u(\mu) - \nabla \phi(u)$
            \STATE Set $\mu \leftarrow \tau_\sigma\left[(\alpha + 3 \beta) \|v\|^2\right]^{-1} v$ where $\tau_\sigma$ has been defined in Proposition \ref{prop:mirror_norm}
        \UNTIL{stopping criterion has been satisfied, i.e. $\|\nabla \phi(v)\| / \|\eta\| < \epsilon$}
    \STATE Form $Z = P^T \mathop{\bf diag}(\mu_1^r,\dots,\mu_r^2) P$
    \STATE Compute $T_\lambda(X) = V \left[\alpha \mathop{\bf Tr}(Z) I_r + \beta Z + \sigma I_r \right]^{-1} $
    \STATE {\bfseries Output}: Bregman gradient iterate $T_\lambda(X)$
	\end{algorithmic}
	\caption{Computing the Bregman iteration map of the Gram kernel}
	\label{algo:gram_kernel}
\end{algorithm}

\textbf{Complexity.} Note that the order of multiplication is important: we only need to compute the eigendecomposition of $V^T V$, which is of size $r \times r$. The inner minimization steps are all linear in the subproblem dimension $r$, and the final computation is of complexity $O(r^3+nr^2)$.

The subproblem being very well conditionned, it is minimized easily; in numerical experiments, it usually convergences in no more than $20$ iterations.
Due to this, the complexity of computing the Bregman iteration map of $h_g$ is $O(nr^2 + r^3)$. The main computational bottleneck remains in most applications the computation of the gradient $\nabla f$.

\section{Symmetric Nonnegative Matrix Factorization}\label{a:symnmf}

\subsection{Fast evaluation of the objective function}

Note that in order to check the sufficient decrease condition in the \texttt{Dyn-NoLips} algorithm, we need to do several evaluations of the objective function per iteration. As pointed out by \cite{Vandaele2016}, the most efficient way to compute it is by using
    \begin{equation}\label{eq:efficient_objective_eval}
        \|M - XX^T\|^2 = \|M\|^2 - 2 \la MX, X \ra + \|X^T X\|^2
    \end{equation}
which has a $O(pr + nr^2)$ complexity, if $p$ notes the number of nonzeros entries of sparse matrix $M$ instead of $O(n^2 r)$ for the naive way.
\subsection{Datasets}

We used two image and two text datasets:

\begin{itemize}
    \item \textbf{Image.}
    \begin{itemize}
        \item \textbf{CBCL}\footnote{http://cbcl.mit.edu/software-da\-tasets/Face\-Data2.html}: 2,429 images of faces of size $19 \times 19$

        \item \textbf{ORL}\footnote{https://www.cl.cam.ac.uk/research/dtg/\-attarchive/\-face\-data\-base.html}: 400 images of size $92\times 112$ representing 40 subjects under different angles and facial expressions.

        \item \textbf{Coil-20}\footnote{http://www.cs.columbia.edu/CAVE/software/softlib/\-coil-20.php}: 1440 images of size $128 \times 128$ representing 20 objects under various angles.
    \end{itemize}

    \item \textbf{Text.}
        \begin{itemize}
            \item \textbf{TDT2}\footnote{http://www.cad.zju.edu.cn/home/dengcai/Data/TextData.html \label{footnote:tdt2}}: dataset of 11,201 news articles classified in 96 semantic categories. We used the version provided by Cai et al. \cite{CWH09,CMHZ08,CHZH07,CHH05}, which has been restricted to the largest 30 categories, leaving a total of 9,394 documents.
            \item \textbf{Reuters}\textsuperscript{\ref{footnote:tdt2}}: dataset of news articles, which we restricted to the largest 25 categories, leaving a total of 7,963 documents.
        \end{itemize}
\end{itemize}

For all image and text datasets, we construct a sparse similarity matrix $M$ following the procedure described in \citep[Section 7.1]{Kuang2015}. We begin by computing the similarity graph between data points, using cosine similarity on term frequency vectors for text, and a Gaussian kernel for image (with the self-tuning method for the scale). The graph obtained is \textit{sparsified} by keeping only the edges connecting the k-nearest neighbors, with $k = \lfloor \log_2 n \rfloor + 1$. Then, $M$ is taken as a normalized version of the graph adjacency matrix.

\section{Introduction}
Consider a low-rank semidefinite program, written
\begin{equation}\label{eq:semidef_problem}
\tag{SDP-r}
    \min F(Y) \quad \text{subject to } \, Y \succeq 0, \, \text{rank}(Y) \leq r
\end{equation}
in the variable $Y \in {\mathbb R}^{n \times n}$, where $F:{\mathbb R}^{n\times n} \rightarrow {\mathbb R}$ is typically a differentiable convex function with Lipschitz continuous gradient and $r \leq n$ is the target rank. Fundamental applications  of \eqref{eq:semidef_problem} arise in various areas including matrix completion \cite{Cand2008,Completion2010,Jain2012}, matrix sensing \cite{Recht2007}, Euclidean matrix completion \cite{Mishra2011,Fang2012}, phase retrieval \cite{Candes2015}, robust principal component analysis \cite{Chen2015}, to name a few.

A popular approach to solving~\eqref{eq:semidef_problem}, known as the Burer-Monteiro formulation \cite{Burer2005}, consists in representing $Y$ as $Y = XX^T$ to solve
\begin{equation}\label{eq:min_problem}
\tag{P}
    \min \Psi (X) \triangleq F(XX^T) + g(X)
\end{equation}
in the variable $X \in {\mathbb R}^{n\times r}$, where $g$ is a lower semicontinuous \textit{simple} function (in a sense that will be made clear later) that enforces additional constraints or penalties on the factor $X$. We will write $f:{\mathbb R}^{n \times r} \rightarrow {\mathbb R}$ the factorized function with $f(X) = F(XX^T)$.

This reformulation has been thoroughly studied for various low rank minimization problems \cite{Mishra2014,Tu2015,Candes2015,Zheng2016,Chen2015,Bhojanapalli2015,Zhao2015,Sun2015,Zheng2016,Park2016}. It has several benefits: first, the rank constraint is directly enforced  by the factorization itself $XX^T$. Second, it allows adding penalty functions $g$ to impose additional structural properties on the factor, like nonnegativity or sparsity. More importantly, the problem size is reduced from $n^2$ to $nr$ variables, which makes it far more efficient on large-scale problems, since $r$ is usually much smaller than $n$.

This formulation comes however with an important drawback: the objective function of \eqref{eq:min_problem} becomes nonconvex, even when $F$ is originally convex. Therefore, local optimization methods can generally only hope to find a stationary point, or at best a local minimum. Nevertheless, recent work shows convergence towards a global optimum for a close enough initialization \cite{Tu2015,Bhojanapalli2015,Park2016}, or under additional statistical assumptions about the problem \cite{Chen2015,Zheng2015,Ge2016}. Although these global optimality results often impose restrictive assumptions that may not be satisfied in practice, they help to explain why the use of local algorithms for solving \eqref{eq:min_problem} usually leads to satisfactory solutions in practice.

\textbf{Non-Euclidean Gradient Methods.}
Here, we focus on efficient algorithms for finding a stationary point of
problem \eqref{eq:min_problem}. We adopt an original approach  based on a recent line of work on non-Euclidean gradient methods   \citep{Bauschke2017,Bolte2018,van} and subsequent work \cite{Lu2016}.

The most commonly used algorithm for \eqref{eq:min_problem} is some variant of the proximal gradient method.
However, a critical issue with gradient schemes is the choice of step sizes which can dramatically impact performance.
This is directly related to smoothness of the objective: when a function $f$ has a Lipschitz continuous gradient with constant $L$, i.e.
$\|\nabla f(X) - \nabla f(Y)\| \leq L \|X-Y\|$,
this implies the so-called Euclidean descent lemma
\begin{equation}\label{eq:eucl_desc_lemma}
f(X) \leq f(Y) + \la \nabla f(Y), X-Y \ra + \frac{L}{2} \|X-Y\|^2
\end{equation}
which means that gradient methods work for any fixed step size in the interval $(0, \frac{1}{L}]$.
This smoothness assumption is used in the broad majority of theoretical analyses of gradient algorithms, yet there are many cases where it is not satisfied \cite{Bauschke2017,Bolte2018}. In particular, it does not generally hold for the function $f(X) = F(XX^T)$ considered in problem \eqref{eq:min_problem} above. Indeed, $F$ will typically be a quadratic function, therefore $f$ will be quartic, and its Hessian will be unbounded as $\|X\|\rightarrow +\infty$.

Of course, there is a way to circumvent this issue to apply classical Euclidean methods. It suffices to use an Armijo line search \citep{Lin2007}, arguing that the objective function is Lipschitz continuous on bounded subsets. However, in some cases, this naive line search strategy generates very small step sizes which in turn involve costly subroutines. Other approaches impose a step size that is only proven to be be valid in a small neighborhood of the optimum \cite{Bhojanapalli2015,Park2016}.

The NoLips algorithm by contrast, introduced by \citet{Bauschke2017} for convex functions and extended to the nonconvex setting in \cite{Bolte2018}, is a gradient method designed for minimizing functions that satisfy a more general condition than Lipschitz continuity of the gradient known as relative smoothness \cite{Lu2016}. It consists in replacing the Euclidean distance $1/2\|X-Y\|^2$ in \eqref{eq:eucl_desc_lemma} by a more general \textit{Bregman divergence} $D_h(X,Y) = h(X) - h(X) - \la \nabla h(Y), X-Y \ra$,   with $h$ a properly chosen convex ``kernel'' function, and successively minimizing the resulting upper approximation.
A good choice of kernel function $h$ may fit better the geometry of the objective, and result in more efficient first order methods.

\textbf{Contributions.} Our main contribution is to provide favorable quartic geometries for implementing efficient first order methods to solve \eqref{eq:min_problem}. We first study a \textit{universal} quartic kernel that can be used for a variety of penalty functions $g$.
In addition, we introduce a novel family of quartic kernels that we call \textit{Gram kernels}, which can be applied when $g\equiv 0$. They provide richer geometries which greatly improve convergence speed with little impact on the iteration complexity.
We also extend the NoLips scheme to Dyn-NoLips, allowing for aggressive step size strategies. The resulting algorithm provably converges to a stationary point of \eqref{eq:min_problem}.

To highlight the benefits of our approach, we study applications to symmetric nonnegative matrix factorization and Euclidean distance matrix completion and show competitive numerical performance compared to specialized algorithms for these problems.

\textbf{Notations.} For two matrices $X$ and $Y$ of same size, we denote the standard Euclidean inner product and norm by $\la X,Y \ra = \mathop{\bf Tr}(X^T Y)$ and $\|X\| = \sqrt{ \mathop{\bf Tr}(X^T X) }$. For a function $f:{\mathbb R}^{n \times r} \rightarrow {\mathbb R}$, we denote by $\nabla F(X)$ its gradient matrix $\nabla F(X)_{ij} = \frac{\partial F(X)}{\partial x_{ij}}$ and by $\nabla^2 F(X)[U,V]$ the second derivative at $X$ in the directions $U,V \in {\mathbb R}^{n\times r}$.
We write $X_1,\dots,X_n$ the rows of a matrix $X \in {\mathbb R}^{n \times r}$. $I_r$ denotes the identity matrix of size $r\times r$. For a square matrix $M$, we denote its trace $\mathop{\bf Tr} M = \sum_{i=1}^n M_{ii}$.

\section{A family of non-Euclidean geometries for low rank minimization}
\subsection{Relative smoothness and the Bregman iteration map}
We consider solving
\begin{equation}\label{eq:nolips_min_problem}
    \min \Psi(X) \triangleq f(X) + g(X)
\end{equation}
in the variable $X\in{\mathbb R}^{n \times r}$, where $f(X) = F(XX^T)$ in our case. We assume that $f$ is a continuously differentiable possibly nonconvex function and $g:{\mathbb R}^{n\times r} \rightarrow {\mathbb R} \cup \{+\infty\}$ a proper lower semicontinuous function, and that the $\Psi$ is bounded from below.

An essential step in designing non-Euclidean algorithms  is the choice of a \textit{distance kernel}. In our context we choose a differentiable strictly convex function $h:{\mathbb R}^{n \times r} \rightarrow {\mathbb R}$, with $\mathop{\bf dom} h = {\mathbb R}^{n \times r}$ (although more general distance kernels can be used). The distance kernel $h$ induces in turn a \textit{Bregman distance}
\begin{equation}\label{eq:bregman_distance}
    D_h(X,Y) = h(X) - h(Y) - \la \nabla h(Y), X-Y \ra
\end{equation}
Note that $D_h$ is  not a proper distance, it is sometimes referred to as a \textit{Bregman divergence}. However $D_h$ enjoys a distance-like separation property: $D_h(X,X) = 0$ and $D_h(X,Y) > 0$ for $X \neq Y$. The choice of a distance kernel suited to the function $f$ is guided by the following relative smoothness condition, also called generalized Lipschitz property.

\begin{definition}[Relative smoothness \cite{Bauschke2017}]
We say that a differentiable function $f : {\mathbb R}^{n \times r} \rightarrow {\mathbb R}$ is L-smooth relatively to the distance kernel $h$ if there exists $L > 0$ such that for every $X,Y \in \rnr$,
\begin{equation}\label{eq:relative_smoothness}\tag{RelSmooth}
    f(X) \leq f(Y) + \la \nabla f(Y), X-Y \ra + L\, D_h(X,Y)
\end{equation}
\end{definition}

For twice differentiable functions,  relative smoothness has an elementary characterization:  $f$ is $L$-smooth relatively to $h$ if and only if
\begin{equation}\label{eq:hessian_cond}
     \nabla^2 f(X)[U,U] \leq L\, \nabla^2 h(X)[U,U], \quad \forall X,U \in {\mathbb R}^{n \times r}
\end{equation}

Notice that if $h(X) = \frac{1}{2} \|X\|^2$, then $D_h(X,Y) = \frac{1}{2}\|X-Y\|^2$ and we recover the Euclidean descent lemma \eqref{eq:eucl_desc_lemma} implied by Lipschitz continuity of the gradient of $f$.

\paragraph{Bregman iteration map.} Now that we are equipped with a non-Euclidean geometry generated by $h$, we define the Bregman proximal iteration map with step size $\lambda$ as follows.
\begin{equation}\label{eq:bregman_prox}
T_\lambda(X) = \mathop{\rm argmin}_{U \in {\mathbb R}^{n\times r}} \left\{ g(U) + \la \nabla f(X), U-X \ra + \frac{1}{\lambda} D_h(U,X) \right\}
\end{equation}
The relative smoothness condition ensures that this operation decreases the objective $\Psi$.
The iteration map is the basic brick for non-Euclidean methods \`a la Bregman.  The simplest method is NoLips \cite{Bauschke2017} and its extension Dyn-NoLips, described in Algorithm \ref{algo:nolips}, but other possibilities exist using momentum and acceleration ideas \cite{Auslender2006,Hanzely2018,Li2019,Mukkamala2019}.

This approach being recalled, we now proceed to identify kernels for low rank minimization.
The choice of a kernel is guided by two imperatives: it should be \textit{adapted} to the smooth part of the objective (here $f$), meaning that {\em the relative smoothness condition holds} while {\em the Bregman proximal iteration map \eqref{eq:bregman_prox} should be easy to compute}.

\subsection{A universal quartic kernel}\label{s:kernels}

Let us define for any matrix $X \in {\mathbb R}^{n \times r}$, the quartic \textit{universal kernel} $h_u$, written
\begin{equation}\label{eq:norm_kernel}
    h_u(X) =  \frac{\alpha}{4}\|X\|^4 + \frac{\sigma}{2}\|X\|^2
\end{equation}

where $\alpha,\sigma > 0$ are fixed parameters. Note that this kernel is not new by itself, as it has been already studied in \cite{Bolte2018} for vectors in ${\mathbb R}^n$. Our first contribution is to show that it is a \textit{universal} kernel for low rank minimization in the sense that it is adapted to every function of our class. The proof is deferred to Appendix \ref{a:relsmooth_proof}.

\begin{proposition}[A universal kernel]\label{prop:rel_smoothness}
    Let $F:{\mathbb R}^{n \times n}\rightarrow {\mathbb R}$ be a continuously differentiable convex function, and that $\nabla F$ is Lipschitz continuous with constant $L_F$. Then the function $f:{\mathbb R}^{n\times r} \rightarrow {\mathbb R}$ defined by $f(X) = F(XX^T)$ is $1$-smooth relatively to the distance kernel $h_u$ for $\alpha \geq 6L_F$ and $\sigma \geq 2\|\nabla F(0)\|$.
\end{proposition}

The Bregman iteration map \eqref{eq:bregman_prox} associated to the kernel $h_u$ can be computed easily in closed form. We give its expression in the unconstrained case \cite{Bolte2018}.
\begin{proposition}[Bregman iteration map for $h_u$, unconstrained case]
    Assume that there is no penalty term, i.e. that $g \equiv 0$. The Bregman iteration map of the universal kernel $h_u$ is given by
    \begin{equation}
         T_\lambda(X) = \frac{1}{\tau_\sigma(\alpha \|U\|^2)} U
    \end{equation}
    where
    \begin{equation}
           U = \nabla h_u(X) - \lambda \nabla f(X) = (\alpha \|X\|^2 + \sigma)X - \lambda \nabla f(X)
    \end{equation}

    and $\tau_\sigma(c)$ denotes the unique real solution $z$ to the cubic equation $z^2(z-\sigma) = c$.
\label{prop:mirror_norm}
\end{proposition}

Note that $\tau_\sigma(c)$ can be computed in closed form using Cardano's method,
which yields
\begin{equation} \label{eq:cardano}
\tau_\sigma(c) = \frac{\alpha}{3} + \sqrt[3]{\frac{c + \sqrt{\Delta}}{2} + \frac{\alpha^3}{27}}
    + \sqrt[3]{\frac{c - \sqrt{\Delta}}{2} + \frac{\alpha^3}{27}}
    \,\,\text{where}\,\,
    \Delta =c^2 + \frac{4}{27} c \alpha^3
\end{equation}
Compared to a standard gradient iteration, the additional operations are elementary and have a minimal impact on the complexity.

\textbf{Constraints and regularization terms.} In our formalism, the latter are embodied in the function $g$. Following the ideas in \cite{Bolte2018}, the Bregman iteration map of $h_u$ can also be easily computed in closed form when $g$ is the $l_1$ norm or the $l_0$ pseudonorm. As we will show in Section \ref{ss:SymNMF}, this also elementary when $g$ is the indicator function of the nonnegative orthant.

\subsection{A family of kernels for unconstrained problems}

While the kernel $h_u$ is universal, its numerical performance can be improved by considering novel richer underlying geometries, as those given by the family that we call the \textit{Gram kernels}. These kernels $h_g$, for $X \in {\mathbb R}^{n \times r}$, write
\begin{equation}\label{eq:gram_kernel}
    h_g(X) = \frac{\alpha}{4}\|X\|^4 + \frac{\beta}{4}\|X^T X\|^2 + \frac{\sigma}{2}\|X\|^2
\end{equation}
where $\alpha, \beta \geq 0, \,\sigma > 0$ are given parameters.

We show now that, when there is no penalty term $g$, the Bregman iteration map of this kernel can be computed efficiently, as it involves solving an easy quartic minimization subproblem of size $r$.
\begin{proposition}[Gram's iteration map]
\label{prop:gram_gradient_map}
For  $X \in {\mathbb R}^{n \times r}$, the Bregman iteration map of $f$ for the Gram kernel $h_g$, called Gram's iteration map, is given by
    \begin{equation}
        T_\lambda(X) = V \left[\alpha \mathop{\bf Tr}(Z) I_r + \beta Z + \sigma I_r \right]^{-1}
    \end{equation}
 where the matrices $V,Z$ are computed through the routine:\begin{itemize}
\item  Set  $V = \nabla h_g(X) - \lambda \nabla f(X)$,
\item  Diagonalize $V^TV$ as $V^T V = P^T D P$ where $P \in \mathcal{O}_r$ and $D = \mathop{\bf diag}(\eta_1^2,\dots,\eta_r^2)$,
\item let $\mu = (\mu_1, \dots,\mu_r)$ be the unique solution of the convex minimization problem
    \begin{equation}\label{eq:subproblem}
        \min_{x \in {\mathbb R}^r} \phi(x) \triangleq \frac{\alpha}{4}\|x\|^4 + \frac{\beta}{4}\sum_{i=1}^r x_i^4 + \frac{\sigma}{2}\|x\|^2 - \sum_{i=1}^r \eta_i x_i,
    \end{equation}
\item finally set $Z = P^T \mathop{\bf diag}\big[ \mu_1^2,\dots, \mu_r^2 \big] P.$
\end{itemize}
\end{proposition}

\begin{proof}
 See Appendix \ref{ss:proof_gram_kernel}.
\end{proof}

\textbf{Complexity.} Note that the order of multiplication is important: we only need to compute the eigendecomposition of $V^T V$, which is of size $r \times r$. We additionally need to solve a small minimization problem of size $r$, which can be done efficiently using the quartic NoLips algorithm with universal kernel (see Appendix \ref{ss:gram_kernel_algo} for implementation details). Due to this, the complexity of computing the Bregman iteration map of $h_g$ is $O(nr^2 + r^3)$, which is efficient since $r$ is usually small. The main computational bottleneck remains in most applications the gradient $\nabla f(X)$; therefore, using the Gram geometry has a minimal impact on the iteration complexity, as it will be confirmed in numerical experiments.

\textbf{Relative smoothness.} Note that $h_g$ is obtained from the universal kernel by adding $X\mapsto \frac{\beta}{4}\|XX^T\|^2$, which is a convex function. Therefore, if $f$ is smooth relatively to $h_u$ with constants $(\alpha,\sigma)$, then it will also be smooth relatively to $h_g$ with $(\alpha,\beta,\sigma)$ for any $\beta \geq 0$. Hence, Proposition \ref{prop:rel_smoothness} also implies that every function of our class is relatively smooth w.r.t. $h_g$ with $\alpha$ and $\sigma$ large enough. The addition of the term $\|X^T X\|^2$ allows us to refine the geometry and improve performances, as we will see in Section \ref{ss:MC}.

\section{Algorithms for quartic low-rank minimization}

Now that we are equipped with a non-Euclidean geometry induced by one of the kernels $h_u$ and $h_g$, we are ready to define the minimization scheme Dyn-Nolips in Algorithm \ref{algo:nolips}. It extends the NoLips algorithm from \citet{Bolte2018} to allow step sizes larger than the theoretical value $1/L$.

\begin{algorithm}[H]
	\begin{algorithmic}
		\STATE {\bfseries Input:} A distance kernel $h$ such that $f$ is smooth relatively to $h$ and a maximal step size $\lambda_{\rm max}$
		\STATE Initialize $X^0 \in {\mathbb R}^{n\times r}$ such that $\Psi(X^0) < \infty$.
		\FOR{k = 1,2,\dots}
		\STATE Choose a step size $\lambda_k \leq \lambda_{\rm max}$ such that the sufficient decrease condition \eqref{eq:suff_decrease} holds
		\STATE
		    Set $X^k = T_{\lambda_k}(X^{k-1}) $
		\ENDFOR
	\end{algorithmic}
	\caption{Dyn-NoLips}
	\label{algo:nolips}
\end{algorithm}

\textbf{Step size choice.} The step size $\lambda_k$ is chosen so that the new iterate $X^k = T_{\lambda_k}(X^{k-1})$ satisfies
\begin{equation}\label{eq:suff_decrease}
f(X^k) \leq f(X^{k-1}) + \la \nabla f(X^{k-1}), X^k - X^{k-1} \ra + \frac{1}{\lambda_k} D_h(X^k, X^{k-1})
\end{equation}
There are two ways to ensure this condition holds.
\begin{itemize}
    \item \textbf{Fixed step size.} Since $f$ is $L$-smooth relatively to $h$, \eqref{eq:suff_decrease} holds as soon as $0 < \lambda \leq {1}/{L}$.
   \item \textbf{Dynamical step size.} In some cases, the relative Lipschitz constant might be too conservative, and better numerical performance can be achieved by taking larger steps. We therefore can use a dynamical strategy for extending the step size, ensuring that \eqref{eq:suff_decrease} is satisfied at each iteration. There are many strategies to efficiently adjust the step size; we adapt the one from \cite{Nesterov2007} (see Appendix \ref{a:dyn_nolips} for details).
\end{itemize}

\textbf{Local convergence guarantees.} We now extend the theoretical convergence results from \cite{Bolte2018} to handle the dynamical step size strategy. The proof is deferred to Appendix \ref{a:dyn_nolips}.

\begin{theorem}[Convergence results]\label{thm:nolips}
Let $\{X^k\}_{k \geq 0}$ be the sequence generated by Algorithm \ref{algo:nolips}. Assume that
    \begin{enumerate}
        \item $f$ is $L$-smooth relatively to a distance kernel $h$, $h$ is strongly convex on ${\mathbb R}^{n\times r}$, and the penalty function $g$ is convex.
        \item The function $\Psi = f+g$ is coercive (meaning that $\Psi(X)\rightarrow + \infty$ when $\|X\|\rightarrow +\infty$) and semialgebraic.
    \end{enumerate}
Then the  sequence $\{\Psi(X^k)\}_{k \geq 0}$ is nonincreasing, and the sequence $\{X^k\}_{k \geq 0}$ converges towards a critical point $X^*$ of problem~\eqref{eq:min_problem}.
\end{theorem}

The semialgebraicity assumption is needed to establish the crucial nonsmooth Lojasiewicz property \citep{Bolte2007}, required to show convergence to a critical point. It holds for all the applications we cited, since the class of semialgebraic functions includes polynomial functions, $l_1$ and $l_2$ norms, the $l_0$ semi-norm and indicators of polynomial sets.

\section{Applications}\label{s:applications}
We now illustrate applications of our methodology to two different low-rank problems, symmetric nonnegative matrix factorization and Euclidean distance matrix completion.
We show that good numerical performance can be reached using the dynamical step strategy, and that, for Euclidean matrix completion, it can be improved further by using the Gram kernel.

\subsection{Symmetric Nonnegative Matrix Factorization}\label{ss:SymNMF}
Symmetric Nonnegative Matrix Factorization (SymNMF) is the task of finding, given a symmetric nonnegative matrix $M \in {\mathbb R}^{n\times n}$, a nonnegative matrix $X\in {\mathbb R}^{n\times r}$ such that
$M \approx XX^T$. This is done by solving
\begin{equation}\label{eq:symnmf}\tag{SymNMF}
\begin{array}{ll}
\mbox{min} & \frac{1}{2} \|M - X X^T\|_F^2\\
\mbox{subject to} & X \geq 0
\end{array}\end{equation}
in the variable $X\in{\mathbb R}^{n \times r}$, where the inequality constraint is meant componentwise and $r \leq n$ is the target rank.

\ref{eq:symnmf} is used as a probabilistic clustering or graph clustering technique \citep{Ding2005,He2011}. Numerical experiments by \citet{Kuang2015} have shown that it achieves state of the art clustering accuracy on several text and image datasets.

\textbf{Solving SymNMF.} While \eqref{eq:symnmf} looks similar to the well-known asymmetric NMF problem $\min_{X,Y} \frac{1}{2}\|M - XY^T\|$, it is actually harder. This is because NMF has a favorable block structure that allows the application of efficient alternating algorithms \citep{Controller2013,Cichocki2009}. SymNMF, however, does not enjoy the same block structure. Current solvers fall into two categories:

{\em Direct solvers.} There have been several attempts at solving the original problem, including multiplicative update rules \cite{He2011}, projected gradient algorithm quasi-Newton schemes \cite{Kuang2015}, and coordinate descent \cite{Vandaele2016}.

{\em Nonsymmetric relaxations.} Another  idea is  to use a mere penalty method \citep{Kuang2015,Lu2017,zhu2018a}, relaxing thus \ref{eq:symnmf} into the following penalized nonsymmetric problem.
\begin{equation}\label{eq:pnmf}\tag{P-NMF}
\begin{array}{ll}
\mbox{minimize} & \frac{1}{2} \|M - X Y^T\|_F^2 + {\mu} \|X - Y\|_F^2\\
\mbox{subject to} & X,Y \geq 0
\end{array}
\end{equation}
in the variables $X, Y \in {\mathbb R}^{n \times r}$, with parameter $\mu \geq 0$. This formulation is very similar to asymmetric NMF and can be solved by the same fast alternating algorithms that exploit the block structure, such as Alternating Nonnegative Least Squares and Hierarchical Alternating Least Squares \cite{zhu2018a}, which are arguably the fastest SymNMF solvers.

\textbf{Applying NoLips.} We propose to apply NoLips for optimizing the original objective function.
Problem \eqref{eq:symnmf} falls within our framework with $F(Y) = \frac{1}{2}\|M-Y\|^2$, which has a Lipschitz gradient with constant $1$, and $g(X) = i_{\{X \geq 0\}}$ the indicator function of the nonnegative orthant. Therefore, Proposition \ref{prop:rel_smoothness} implies that $f(X) \triangleq  \frac{1}{2}\|M-XX^T\|^2$ is $1$-smooth relatively to the kernel $h_u$ with $\alpha = 6$ and $\sigma = 2\|\nabla F(0)\| = 2\|M\|$.
Since, in addition, $f$ is polynomial and $g$ is the indicator of a polynomial set, $f+g$ is semialgebraic, and it is also coercive, so Theorem \ref{thm:nolips} guarantees that NoLips will converge towards a stationary point of problem \eqref{eq:symnmf}.

It can easily be shown that the Bregman iteration map is given by simply adding a projection step:
\begin{equation}
 T_\lambda(X) = \frac{1}{\tau_\sigma(\alpha \|\Pi_+(U)\|^2)}\Pi_+(U)
\end{equation}

where $U = \nabla h_u(X) - \lambda \nabla f(X)$, $\tau_\sigma$ has been defined in Proposition \ref{prop:mirror_norm} and $\Pi_+$ denotes the projection on the nonnegative orthant: $\Pi_+(U) = \max(U, 0)$ (entrywise).

\textbf{Numerical experiments.}
We implemented the following algorithms: Algorithm \ref{algo:nolips} with dynamical step size and the universal kernel (\texttt{Dyn-NoLips}), the  \texttt{$\beta$-SNMF} scheme from \cite{He2011}, where we set $\beta = 0.99$ as advised by the authors, the projected gradient algorithm (\texttt{PG}) with Armijo line search from  \cite{Kuang2015}, where we use the line search parameters $\beta = 0.1$ and $\sigma = 0.01$, the coordinate descent scheme (\texttt{CD}) from \cite{Vandaele2016}, and the two fast algorithms from \cite{zhu2018a} for solving the penalized problem \eqref{eq:pnmf}: \texttt{SymANLS} and \texttt{SymHALS}. For the last two, we tuned the $\mu$ penalization parameter for best performance.  We left out the Quasi-Newton algorithm from \cite{Kuang2015} because of its prohibitive $O(n^3)$ complexity for large datasets.

All algorithms were implemented in Julia \citep{Bezanson2017} which is a highly-optimized numerical computing language.\footnote{Since our algorithms have different complexity per iteration, it is essential to compare them in terms of running time, and Julia provides a fairly accurate way to do so as there is little interpreter overhead in loops. Tests were run on a PC Intel CORE i7-4910MQ CPU @ 2.90 GHz x 8 with 32 Go RAM.} We used four standard text and image datasets (links are provided in Appendix \ref{a:symnmf}). We construct a sparse similarity matrix $M$ following the procedure described in \citep[Section 7.1]{Kuang2015}.

\begin{figure}

  \centering
  \subfigure[COIL-20 (image) $n = 1440$, $r = 20$]{
    \includegraphics[width = 0.43\columnwidth]{figs/coil20.png}
  }
  \quad
  \subfigure[CBCL (image) $n = 2429$, $r = 20$]{
    \includegraphics[width = 0.43\columnwidth]{figs/faces.png}
  }

  \centering
  \subfigure[TDT2 (text) $n = 9394$, $r = 30$]{
    \includegraphics[width = 0.43\columnwidth]{figs/tdt2.png}
    \label{fig:tdt2}
  }
  \quad
  \subfigure[Reuters (text) $n = 7963$, $r = 25$]{
    \includegraphics[width = 0.43\columnwidth]{figs/reuters.png}
    \label{fig:reuters}
  }

\caption{SymNMF objective gap $f(X^k) - f_{\rm min}$ averaged over 20 random initializations, for various sparse similarity matrices $M \in {\mathbb R}^{n \times n}$. Hyperparameters for SymHALS, SymANLS were tuned for best performance.}
\label{fig:symnmf}
\end{figure}

Figure \ref{fig:symnmf} reports the performance of the different algorithms in terms of objective gap, averaged over 20 random initializations.
Overall, the algorithms that show the best convergence speed are \texttt{SymHALS} and \texttt{Dyn-NoLips}. Even if \texttt{SymHALS} performs better on some datasets, it has the disadvantage of requiring tuning of the penalization parameter $\mu$ for an optimal performance. In our experiments,  $\mu = 10^{-2}$ for image and $\mu = 10^{-1}$ for text showed the best convergence speed; the smallness of these values contrast with a penalization theory advising for large values.

\subsection{Euclidean distance matrix completion}\label{ss:MC}

\begin{figure}[H]
  \centering
  \subfigure[$n= 2000,r=3$]{
    \includegraphics[width = 0.4\columnwidth]{figs/helix2000_10runs_z.png}
  }
  \quad
  \subfigure[$n= 5000, r = 3$]{
    \includegraphics[width = 0.4\columnwidth]{figs/helix5000_10runs_z.png}
  }

\caption{Euclidean matrix completion problems on the \texttt{Helix} dataset, with 10\% known distances and two different sizes. We present the normalized RMSE over the full distance matrix versus CPU time. The results are averaged over 10 random initializations.}
\label{fig:mc}
\end{figure}

Euclidean distance matrix completion (EDMC) is the task of recovering the position of $n$ points $x_1^*,\dots,x_n^* \in {\mathbb R}^r$, given the knowledge of a partial set of pairwise distances $d_{ij} = \|x_i^* - x_j^*\|^2$ for $(i,j) \in \Omega$, where $\Omega \subset [1,n]\times[1,n]$. It is a fundamental problem with applications in sensor network localization and the study of conformation of molecules; see \cite{Fang2012,Qi2013,Dokmanic2015} and references therein.

The Burer-Monteiro nonconvex formulation for solving this problem writes

\newcommand\sumom{\sum_{i,j \in \Omega}}

\begin{equation}\label{eq:edmc}\tag{EDMC}
 \min f(X) \triangleq \frac{1}{2}\sum_{(i,j)\in \Omega} \left( \|X_i - X_j\|^2 - d_{ij} \right)^2
\end{equation}

in the variable $X\in {\mathbb R}^{n \times r}$. It can be rewritten $f(X) = \frac{1}{2} \|\mathcal{P}_\Omega(\kappa(XX^T)-D) \|^2$ where $D$ is the matrix of known distances, $\mathcal{P}_\Omega$ denotes the projection operator such that $\mathcal{P}_\Omega(Y)_{ij} = Y_{ij}$ if $(i,j) \in \Omega$, and $\mathcal{P}_\Omega(Y)_{ij} = 0$ elsewhere, and $\kappa$ is the linear operator defined for $Y \in {\mathbb R}^{n\times n}$ by
\begin{equation}
    \kappa(Y)_{ij} = Y_{ii} + Y_{jj} - 2 Y_{ij} \,\, \text{for} \,\, 1\leq i,j \leq n
\end{equation}
\textbf{Applying NoLips with the universal kernel.}
Problem \eqref{eq:edmc} falls within our framework with $F(Y) = \frac{1}{2}\|\mathcal{P}_\Omega(\kappa(Y)-D)\|^2$, which can be shown to have a Lipschitz gradient with constant $L_{EDM} \triangleq  9 \max_{i=1\dots n} |\{j| (i,j) \in \Omega\}|$.
Therefore, as in the case of SymNMF, the universal kernel $h_u$ can be used with a initial step size $1$ and parameters $\alpha = 6L_{EDM}$ and $\sigma = \frac{1}{3}\|\nabla F(0)\| = 2\|\mathcal{P}_\Omega(D)\|$.

\textbf{Using the Gram kernel.} As the problem is unconstrained, we can also apply minimization using the Gram kernel $h_g$. We use the parameters $\alpha = \beta = 6 L_{EDM}$ and $\sigma = 2 \|\mathcal{P}_\Omega(D)\|$, which ensure that $f$ is $1$-smooth relatively to $h_g$. While $\beta$ could be set to any positive value, taking it equal to $\alpha$ proves to be a simple and efficient choice in practice.

\textbf{Numerical experiments.}  We implement the following algorithms: NoLips with a dynamical step size and the universal kernel (\texttt{Dyn-NoLips}), NoLips with a dynamical step size and the Gram kernel (\texttt{Dyn-NoLips-Gram}), gradient descent with Armijo line search (\texttt{GD}), the Riemannian trust region algorithm from \cite{Mishra2011} (\texttt{TR}). We leave out semidefinite relaxations because of their $O(n^2)$ memory requirement which is prohibitive on large data.
As the implementation for \texttt{TR} is provided in Matlab, we run our experiments on Matlab as well, with the same setup as in Section \ref{ss:SymNMF}.

 We try the algorithms on a standard EDMC problem, the 3-dimensional \texttt{Helix} dataset \cite{Mishra2011} which is generated as $X_i = (\cos(3t_i),\sin(3t_i),2t_i)$ where $\{t_i\}_{i=1}^n$ are sampled uniformly in $[0,2\pi]$. We randomly keep only 10 \% on the pairwise distances, and test on two different problem sizes: $n = 2000$ and $n= 50000$. Figure \ref{fig:mc} reports the normalized root mean squared error (RMSE) over \textit{all distances} (known and unknown) averaged on $10$ random initializations. All the algorithms manage to recover the ground truth; the \texttt{Dyn-NoLips-Gram} algorithm shows the best numerical performance, which demonstrates the advantage of using the novel Gram geometry.

\section{Conclusion}
We proposed a generic approach for solving Burer-Monteiro formulations of low rank minimization problems using the methodology of Bregman gradient methods and relative smoothness.
We provided two quartic kernels, including the novel Gram kernel, and demonstrated their benefit on numerical experiments. In future work, performance could be improved further by studying inertial variants \cite{Mukkamala2019,Hanzely2018}.
New kernels could also be explored beyond the class of quartic functions to tackle other problems with inherent non-Euclidean geometries.

\section*{Acknowledgements}
Radu-Alexandru Dragomir would like to acknowledge support from an AMX fellowship, the Air Force Office of Scientific Research, Air Force Material Command, USAF, under grant number FA9550-18-1-0226, as well as from Sébastien Gadat.

J\'{e}r\^{o}me Bolte was partially supported by ANR-3IA Artificial and Natural Intelligence Toulouse Institute, and Air Force Office of Scientific Research, and Air Force Material Command, USAF, under grant numbers FA9550-18-1-0226 \& FA9550-19-1-7026.

Alexandre d'Aspremont is at CNRS \& d\'epartement d'informatique, \'Ecole normale sup\'erieure, UMR CNRS 8548, 45 rue d'Ulm 75005 Paris, France, INRIA and PSL Research University. The authors would like to acknowledge support from the {\em ML \& Optimisation} joint research initiative with the {\em fonds AXA pour la recherche} and Kamet Ventures, as well as a Google focused award.

\newpage

\subsection{Low-rank matrix completion}\label{ss:MC}

Matrix completion is the task of recovering a low rank matrix $M \in {\mathbb R}^{n \times n}$, given only the knowledge of a small set $k$ of entries. Let us denote $\Omega \subset [1,n]\times [1,n]$ the subset of known indices. Then the Burer-Monteiro formulation for low-rank matrix completion writes

\begin{equation}\label{eq:matrix_comp}
 \min  \frac{1}{2}\|\mathcal{P}_\Omega(XX^T - M)\|^2
\end{equation}

in the variable $X\in {\mathbb R}^{n \times r}$ where $r$ is the target rank, and $\mathcal{P}_\Omega:{\mathbb R}^{n\times n} \rightarrow {\mathbb R}^{n \times n}$ denotes the projection operator such that $\mathcal{P}_\Omega(Y)_{ij} = Y_{ij}$ if $(i,j) \in \Omega$, and $\mathcal{P}_\Omega(Y)_{ij} = 0$ otherwise.

While this formulation seems to be applicable solely to semidefinite matrices, it can also be used for general rectangular low-rank matrix completion using the so-called semidefinite lifting technique; see \cite{Tu2015,Park2016,Zheng2016} and references therein.

\textbf{Applying NoLips with the universal kernel.}
Problem \eqref{eq:matrix_comp} falls in our framework with $F(Y) = \frac{1}{2}\|\mathcal{P}_\Omega(Y-M)\|^2$, which has a Lipschitz gradient with constant $1$. Therefore, as in the case of SymNMF, the universal kernel $h_u$ can be used with a relative Lipschitz constant $6$ and the parameter $\alpha = \frac{1}{3}\|\nabla F(0)\|_2 = \frac{1}{3}\|\mathcal{P}_\Omega(M)\|_2$.

\textbf{Using the Gram kernel.} Unlike the universal kernel $h_u$, the relative smoothness condition cannot be established exactly for the matrix completion objective function and the Gram kernel $h_g$. We justify however the choice of this kernel by showing that the condition still holds in expectation with respect to the set of observed entries $\Omega$. Combined with a dynamical step strategy to ensure sufficient decrease, the convergence guarantees will still hold in this case (see Appendix for more details).

\begin{proposition}[Relative smoothnes in expectation for the Gram kernel] Assume that the indices of observed entries $\Omega$ are drawn indepentently at random.
Let $f(X) = \frac{1}{2}\|\mathcal{P}_\Omega(XX^T - M)\|$ the objective function of the matrix completion problem. Then the relative smoothness condition with the Gram kernel holds in expectation, i.e.

\begin{equation}
    \mathbb{E}\big[ \nabla^2 f(X)[U,U] \big] \leq \mathbb{E}\big[ \nabla^2 h_g(X)[U,U] \big]
\end{equation}

for $\alpha \geq $.

\end{proposition}\textbf{Numerical experiments.} We implemented Algorithm \ref{algo:nolips} with the universal kernel and a dynamical step size strategy (\textbf{Dyn-NoLips}), Algorithm \ref{algo:nolips} with the new Gram kernel and a dynamical step size strategy (\textbf{Dyn-NoLips-Gram}). We compare the performance of our schemes against a Euclidean gradient algorithm with backtracking (\textbf{GD}), and the Singular Value Projection algorithm from \cite{Meka2009} (\textbf{SVP}).

\textbf{Notes: what baseline and datasets ?}

\begin{itemize}
    \item Singular value tresholding (reference needed): popular algorithm for nuclear norm minimization.
    \item Keshavan 2010 (reference needed): Riemannian algorithm.
    \item \citet{Boumal2011}: Riemannian trust region method. Test on (quite large) square, rectangular matrices as well as the Jester dataset.
    \item \citet{Jain2012} studies alternating minimization, theoretical guarantees, no simulations.
    \item \citet{Hardt2014} studies an alternating algorithm, solely for square symmetric matrices. They provide \textbf{Julia code (!)} that can be used, but experiment only on synthetic symmetric data. The problem would be that it needs resampling ?
    \item \citet{Zheng2016}: gradient descent for rectangular matrix completion, experiments on synthetic data. Trust region, GD and SVP are the fastest.
\end{itemize}

redpajama_set_name: RedPajamaArXiv


text:
About Royal Caribbean International
Royal Caribbean International is an award-winning global cruise brand with a 46-year legacy of innovation and introducing industry "firsts" never before seen at sea. The cruise line features an expansive and unmatched array of features and amenities only found on Royal Caribbean including, jaw-dropping, Broadway-style entertainment and industry-acclaimed programming that appeals to families and adventurous vacationers alike. Onboard, guests are catered to with the cruise line's world-renowned friendly and engaging Gold Anchor Service by every staff and crew member. Royal Caribbean has been voted "Best Cruise Line Overall" for 12 consecutive years in the Travel Weekly Readers Choice Awards.
The cruise line sails 23 of the world's most innovative cruise ships calling on more than 288 destinations in 77 countries across six continent visiting the most popular destinations in Bermuda and the Caribbean, Europe, Canada and New England, Alaska, South America, Asia, and Australia and New Zealand.
Royal Caribbean : 7 Nights Mediterranean Cruise
7 Nights Mediterranean Cruise
mur 26 000 /Adult
SHIP : Brilliance of the Seas
Brilliance of the Seas is the ultimate escape for backroads ramblers — distinct cultures to discover ashore, and unique experiences onboard. Explore one-of-a-kind sights, sounds and tastes from the Mediterranean.
With the distinct cruise you will be immersed in new experiences onboard. Whether it's picking up Spanish while sailing from Spain or witnessing unrivaled views of South of France and Italy.
Onboard Brilliance of the Seas, you don't have to go far to be amazed. With acres of glass offering stunning views out to sea and the golden glow of sunrise and sunset every day, plus fantastic onboard features for guests of all ages, it's easy to see how this ship got its name. So sit back, relax, and let us take you on the most brilliant holiday you've ever had!
Cruise: 7 Nights Mediterranean
Leaving from: Barcelona
Ship: Brilliance of the Seas
As from MUR 26 000 per Adult - Double Share (Interior Stateroom)
Fare includes:
- Cruise fare
- All Inclusive Meal Plan
- Taxes & Port Charges
Fare excludes:
- Any related Visa & Travel Insurance
- Gratuity (Payable on Board)
- Item of Personal expense
SAILINGS :
Nice (Villefranche) – France
Portofino – Italy
Florence/ Pisa (Livorno) – Italy
Rome (Civitavecchia) – Italy
Naples (Salerno) – Italy
Cruise Sailing Date: From Barcelona
September: 22, 29
October: 13, 20, 27
EXCLUDES: AIRFARES | For Airfare quotes please contact us or you may use our FLIGHT BOOKING SYSTEM
Subject to availability when booking is made as limited space. Please CONTACT US for a quote combined with flights & packages.


text:
Showing posts with label Robert O'Hara. Show all posts
OPENING: CHICAGO PREMIERE OF BARBECUE BY ROBERT O'HARA VIA STRAWDOG THEATRE AT STEPPENWOLF'S 1700 THEATRE
Chi, IL LIVE Shows On Our Radar:
STRAWDOG THEATRE COMPANY ANNOUNCES THE CHICAGO PREMIERE OF
BY ROBERT O'HARA,
AUGUST 17 – SEPTEMBER 30, DIRECTED BY DAMON KIELY
A LOOKOUT VISITING COMPANY PRESENTED AT STEPPENWOLF'S 1700 THEATRE
What better way to bring summer to a close than with an emotional bomb shell of a BBQ/family intervention?! We're elated that one of our favorite storefront companies, Strawdog Theatre Company, will be playing one of our fav new cabaret spaces, Steppenwolf Theatre Company's 1700 Theatre. Not to mention, the author is the acclaimed and multi award winning Robert O'Hara. We adored his hilarious play, BootyCandy, that just ran for months at Windy City Playhouse. This one's sure to be hot! The intimate 1700 space fills fast so don't miss out.
Strawdog Theatre Company and Artistic Directors Michael Dailey, Heath Hays and Anderson Lawfer are proud to announce the first production in their 2017 – 2018 season, Barbecue by Robert O'Hara, August 17 – September 30, with direction by Damon Kiely at Steppenwolf Theatre Company's 1700 Theatre, 1700 N. Halsted Street as a LookOut Visiting Company.
Opening night performance is Monday Aug. 28 at 8 p.m. Previews are Thursday, Aug. 17 and Aug. 24, Friday, Aug. 18 and Aug. 25, Saturday, Aug. 19 and Aug. 26 at 8 p.m. and Sunday, August 27 at 3:30 p.m.
Regular run performances are Fridays at 8 p.m., Saturdays at 3:30 and 8 p.m. and Sundays at 3:30 p.m. Preview tickets are $35, regular run tickets are $45. Strawdog subscribers enjoy 25% off all ticket prices.
This production is also available to Steppenwolf's Red and Black Card holders. Group, senior, student, industry, and rush discounts are also available. To purchase subscriptions to Strawdog's 2017-18 Season, visit www.strawdog.org.
To purchase single tickets to Barbecue, please visit HERE.
A modern American family uses a summer barbecue as a pretext to ambush sister Barbara with an intervention. If you think "all families are crazy" is just a cliché, you've not spent time under the influence of the O'Mallerys. An afternoon in the park with these raucous siblings and you'll be challenging your own assumptions about family, race, and reality. The Chicago premiere of Barbecue by Obie and Helen Hayes Award winner Robert O'Hara will have you laughing out loud and questioning how it's true that in America, you sometimes taze the ones you love.
Since its world premiere, Barbecue has received critically acclaim. The Los Angeles Times said, "Ferociously funny! Barbecue raucously sends up the sociological pantomime — life as an award-seeking existential burlesque," and "Wickedly delightful!" by TheaterMania.com. Variety's review included, "Barbecue, shrewdly turns the formula for the American domestic comedy on its head."
The Barbecue cast includes Strawdog Ensemble Members John Henry Roberts and Kamille Dawkins with guest artists Kristin Collins, Celeste Cooper, Anita Deely, Barbara Figgins, Deanna Reed Foster, Abby Pierce, Terence Simms and Ginneh Thomas.
The Barbecue design team includes Strawdog Ensemble Members Costume Designer Aly Greaves Amidei and Sound Designer Heath Hayes with guest artists: Director Damon Kiely, Assistant Director Michael Burke, Set Designer Joanna Iwanicka, Lighting Designer Jared Gooding, Props Designer Leah Hummel and Dramaturg Taylor Barfield.
ABOUT DAMON KIELY, DIRECTOR
Damon Kiely is chair of Performance at DePaul's Theatre School and the author of the book How to Read a Play (Routledge) and the plays Thieves Like Us and The Revel. His production of Hank Williams: Lost Highway won the Jeff Award for Best Musical Midsized. He has directed for American Theatre Company, TimeLine, Route 66, A Red Orchid and many others. He lives in Edgewater with his wife Jennifer and their children, Bella and Finn.
ABOUT ROBERT O'HARA. PLAYWRIGHT
Robert O'Hara has received the NAACP Best Director Award, the Helen Hayes Award for Outstanding New Play, two OBIEs and the Oppenheimer Award. He directed the world premieres of Nikkole Salter and Dania Guiria's In the Continuum, Tarell McCraney's The Brother/Sister Plays (Part 2), Colman Domingo's Wild with Happy as well as his own plays, BootyCandy and Insurrection: Holding History. His new plays Zombie: The American and Barbecue, world premiered at Woolly Mammoth Theater and New York Shakespeare Festival, respectively. He is currently directing a new musical, Uni/son inspired from the poetry of August Wilson at Oregon Shakespeare Festival and a new musical, Bella by Kirsten Childs at Playwrights Horizons. He will direct the world premiere of his play, Mankind, at Playwrights Horizons this fall.
ABOUT STRAWDOG THEATRE COMPANY
Since its founding in 1988, Strawdog Theatre Company has offered Chicagoland the premiere storefront theatre experience and garnered numerous Non-Equity Jeff Awards with its commitment to ensemble acting and an immersive design approach. The celebrated Company develops new work, re-imagines the classics, melds music with theatre, asks provocative questions and delivers their audience the unexpected.
Strawdog Theatre Company is supported in part by The Richard H. Driehaus Foundation, the Gaylord and Dorothy Donnelley Foundation, the Alphawood Foundation, Chicago Department of Cultural Affairs, and the annual support of businesses and individuals.
Labels: Barbecue, BBQ, LookOut Series, opening, Robert O'Hara, shows on our radar, Steppenwolf 1700 Theatre, Strawdog Theatre Company


text:
Wildcat Events Board brings ice skating event to UA Mall
Published Dec 8, 2013 9:21pm
By Alison Dorf
Photo Courtesy of Wildcats Event Board
Students will get the chance to ice skate into finals week on the UA Mall Wednesday evening as part of an annual holiday-themed event.
Hosted by the Wildcat Events Board, WinterFest will include an ice skating rink, snow flurries, an opportunity to decorate a gingerbread house and a performance by Amplified Acapella, the newest co-ed a cappella group on campus, among other winter-related activities.
"You just feel like you're in a little winter wonderland," said Kayla Fiore, a general studies sophomore and coordinator of the event. "It really makes you get into the holiday spirit."
Wednesday also marks dead day eve, offering students a chance to de-stress before finals begin next week.
"It's kind of stressful with finals and everything going on around school, especially right now," Fiore said. "[The event is] just to have like one last fun thing to do before we all go back home and get us in the holiday spirit before we go back and celebrate with our families."
The ice skating rink is made of synthetic ice and is provided by BH Skating Parks International. To install the rink, synthetic ice sheets made of a non-toxic, recyclable polymer compound are hammered together like giant puzzle pieces, according to the company's website.
A non-toxic glide enhancer is sprayed onto the surface of the ice to reduce the friction created by a skater's blades. Unlike a typical ice skating rink, the rink is not refrigerated. It is also 10 to 15 percent slower than a normal rink, and feels more like skating on a frozen pond than an actual rink.
The Wildcat Events Board will provide free skates for students. While students wait for their turn on the ice, they can participate in a range of activities, including a snowman-making contest using marshmallows, a basketball free-throw machine, hot cocoa and s'mores and decorating gingerbread houses.
Pauline Lam, a pre-business sophomore and an arts director for the Wildcat Events Board is in charge of the gingerbread house activities. The activity was a hit for students last year, prompting its return this year, she said, adding the fact that the event is on campus and is free gives students easy access to join the festivities.
"They don't have to drive far and pay lots of money to do something fun," Lam said.
When the event began last year, around 400 students attended, a number Fiore said she hopes doubles this year.
Amplified Acapella will perform holiday songs such as "Let It Snow" and "Baby, It's Cold Outside."
Sean McCann, a music sophomore and director of Amplified Acapella, said it was exciting to take the time to learn some holiday songs and do something different for the group.
It's also important to have events such as WinterFest on dead day eve because although it's so close to finals week, students shouldn't have to be constantly stressed, McCann said.
"I think WinterFest is a very good break to hang out with friends," McCann said, "watch a performance, have a couple things to do before you really have to buckle down on Thursday and get all this studying out of the way."
- Follow Alison Dorf @AlisonRaeDorf


In the last video, you annotated interesting information by bolding it.
Then, you added comments about why you thought it was interesting.
The next step to annotating is to identify main ideas.
The main idea contains the most important part of a piece of writing and tells you what the author's point or purpose is.
In this video, you will annotate by underlining what you think are the main ideas of each paragraph.
As you read each paragraph, look for a sentence or sentences that make an important point that is then explained and supported by the rest of the paragraph.
The main idea often includes who, what, where, when, or why details.
Another way to figure out the main idea is to ask these questions as you read: What's this paragraph about?
or What point is the author trying to make?
Read each paragraph and underline what you think the main idea is.
Then, add this new category to your annotation key.
If you're discussing this article in class, you can now visually scan for the underlined sentences to help remind you of the most important ideas.
All of your annotations can help you prepare to speak or write about what you've read.
Once you have underlined the main idea for each paragraph and added the category to your annotation key, move on to the next video.
Now, it's your turn: underline the main idea of each paragraph, and add the new category to your annotation key.
Underline the main idea of each paragraph.
Add the new category to your annotation key.