text:
We found 419 matching items.
Media Types Images Geographic Location Colborne Item type(s) Correspondence or Postcards or Bookmarks
Page 1 of 21 123 ... 2021 → Next
Book postcard of Beaver River, Cannington, Ont.
Black lettering on front: "Beaver River, Cannington, Ont." Addressed to Miss Gertrude Mikel. Stamp removed. Posted from Cannington, Ontario, removal of stamp has removed some information - possibly posted in 1911. Initials appear in right hand corner: "J.P."
Black lettering on front: "Beaver River, Cannington, Ont." Addressed to Miss Gertrude Mikel. Stamp removed. Posted from Cannington, Ontario, removal of stamp has removed some information - possibly posted in ...
Cramahe Township Public Library
Book postcard of The Cove at Presqu'isle
Black lettering on front: "The Cove at Presqu'isle" Addressed to Miss Gertrude Mikel. Not stamped or posted. The note reads: "I send this a warm day just went to work it was lovely down to the bay"
Black lettering on front: "The Cove at Presqu'isle" Addressed to Miss Gertrude Mikel. Not stamped or posted. The note reads: "I send this a warm day just went to work ...
Book postcard of Smith's Falls Ont. Canoe Club
White lettering on front: "Smith's Falls. Ont. Canoe Club" Not stamped, posted or used.
Postcard of the Otonabee steamer at night
1907 Handwritten in ink on front: "The Steame Otonabee waiting while we were at launch" Addressed to Miss Gertie Mikle. Posted from Peterboro 1907. The note reads: "Gertie I reced. yours this morning but I did not get an opportunity to answer it this afternoon so will write on Sunday. T.B.P." ...
1907 Handwritten in ink on front: "The Steame Otonabee waiting while we were at launch" Addressed to Miss Gertie Mikle. Posted from Peterboro 1907. The note reads: "Gertie I reced. yours ...
Book postcard of King Charles dogs
White lettering on front: "We are seven" "9639 A Rotary Photo E.C." "Thos. Fall" Not stamped or posted, but addressed to Miss Gertrude Mikel. The note reads: "Dear G. - Told Ridley I wanted him to take me out but I don't know as yet whether he will or not ...
White lettering on front: "We are seven" "9639 A Rotary Photo E.C." "Thos. Fall" Not stamped or posted, but addressed to Miss Gertrude Mikel. The note reads: "Dear G. - ...
Letter and postcard from Albert M. Waller to Eliza J. Padginton.
9 July 1944, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
9 July 1944, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Postcard from Charles S. Rutherford to Eliza J. Padginton
31 August 1942 Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
31 August 1942 Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Postcard from Albert M. Waller to Eliza J. Padginton.
2 September 1940 Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
2 September 1940 Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from Pte. C.A. Burleigh to Eliza J. Padginton
17 May 1941, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
17 May 1941, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from Cpl. Jim Moore to Eliza J. Padginton
23 July 1945, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
23 July 1945, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from Cpl. G. C. Shier to Eliza J. Padginton
17 October 1942, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
17 October 1942, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from Pte. Kenneth Palmateer to Eliza J. Padginton
9 October 1942, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
9 October 1942, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from G. Robert Lindle to Eliza J. Padginton
194(?), Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
194(?), Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from Jim Moore to Eliza J. Padginton
17 November 194(?), Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
17 November 194(?), Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from C.R. Miller to Eliza J. Padginton
2 January 1942, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
2 January 1942, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from Doug McCracken to Eliza J. Padginton
19 January 1943, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
19 January 1943, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from Archie Kemp to Eliza J. Padginton
14 December 1943, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
14 December 1943, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from Alex Kells to Eliza J. Padginton
12 November 1941, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
12 November 1941, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from Francis Kemp to Eliza J. Padginton
1 March 1943, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
1 March 1943, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Letter from William Roy Knight to Eliza J. Padginton
22 September 1943, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen Victoria's) Royal Mail. She sent newspapers to enlisted Cramahe men and women during the Boer War (1899-1902), World War I ...
22 September 1943, Eliza J. Padginton (1862-1961) served the Colborne community as postmistress for over 70 years, beginning her career in January 1883 as an assistant to the Postmaster of Her Majesty's (Queen ...
Cramahe Township Public Library (419)
Colborne (419)
Eliza J. Padginton WWII Letters (4)
Photograph albums (53)
Clippings (15)
Has Comment?: 5


text:
Q: WebRTC on Raspberry pi will no longer work I installed the WebRTC/uv4l client from ADVANCED PROJECTS on my Raspberry pi 3A.  It worked fine.
But I wanted to remove the watermark on the video, so followed the instructions I found on Stack Overflow. Most mentioned using a different video driver (video0) so that is the procedure I used.  I rebooted and when I used my browser to check operation, the standard web page with the various options (this is even before I get to the video) would not load. I went back to uv4l-raspicam.conf to confirm that nothing had changed.  It had not.  I updated the firmware. I reloaded webRTC.  Nothing works. I supposed I could reload Raspbian from scratch, but that seems like overkill. I get video when I issue the command  raspivid -t 0 .How do I get webRTC working again? It seems the Rpi's web server itself no longer works.

redpajama_set_name: RedPajamaStackExchange


Khao Soi is something between a noodle and a curry dish. It's often served with noodles already in the dish and with a plate of fried noodles next to it that you can put in at your own pleasure. The curry of Khao Soi is coconut based and rich in flavor and spices. Of course, you can ask for 'not so spicy' but we recommend you at least try the original! Khao Soi comes in vegetarian, chicken (typically, a nice drumstick, but sometimes other parts), pork and sometimes beef or shrimp. We recommend the chicken or vegetarian. Khao Soi is typically served with a separate dish with some freshly diced red onion, lime and some pickled Chinese mustard root that you can add for taste.
For some really good Khao Soi, make sure you try one of the following. Because Khao Soi is considered a lunch dish, most of these places are only open in the morning and during lunch hours.
Located in the lovely Santitham neighborhood, anyone living around there will recommend this place. Open from 8am to 4pm.
Location: 29/1 Ratchaphuek Alley, ตำบล ช้างเผือก Amphoe Mueang Chiang Mai, Chang Wat Chiang Mai 50300.
Khun Yaay (Khun means Mister or Madam) is easy to find, as it's right on the northside of the old city along the moat (on the inside of the moat). It's only open from 10am to 2pm and is closed on Sundays.
Location: ซอย ศรีภูมิ 8 ถนน ศรีภูมิ Tambon Si Phum, Amphoe Mueang Chiang Mai, Chang Wat Chiang Mai 50200.
You can find this place right across the Ping river, north of Ratanakosin bridge. It's open from 9am to 4:30pm.
Samer Jai is just up the street from Khao Soi Lam Duan. Like Lam Duan, it offers a variety of other Northern Thai foods besides Khao Soi, if you want to mix it up. Samer Jai is open from 8am to 3pm.
Location: ถนน เจริญราษฏร์ Tambon Wat Ket, Amphoe Mueang Chiang Mai, Chang Wat Chiang Mai 50000.
This little shop doesn't have a Google Maps location but serves some of the best Khao Soi we had the pleasure of tasting. If you get the big portion for only 50 Bath, you get two (!) chicken drumsticks with your dish. If you are ever in the Central Festival neighborhood, make sure to check it out. The shop is part of a little row of food shops, located next to Tham Sang Thai Food and has a clear sign that says 'Khao Soi' in front of it.
Location: Chiang Mai Business Park Soi 1.CBP, Tambon Nong Pa Khrang, Amphoe Mueang Chiang Mai, Chang Wat Chiang Mai 50000.
If you're in Nimman or Su Thep and like a little bit of a food adventure, don't to go the touristy Khao Soi Nimman, but go to this place instead. Order the Khao Soi chicken and chances are you won't get a neat chicken drumstick with your bowl, but perhaps a hip-bone, perhaps something else. Nevertheless, it's extremely tasty and a place where the Thais eat Khao Soi (and other dishes). You'll see some military personnel eating here as well, as it's located right north of the Wing 41 air force base.
There is a great little Thai-style bakery right across the street, where you can buy original Thai 'candy', which is quite tasty!
Location: Mu Ban Khumkhon Phing 1 Alley, Su Thep, Mueang Chiang Mai District, Chiang Mai 50200.
Do you want to learn how to make Khao Soi yourself? Join one of our cooking classes and learn the tricks of a perfect Khao Soi.


text:
from datetime import timedelta

def job_callback():
    return "result"

def periodic_job(*args, **kwargs):
    """This periodic job will successfully load"""
    return {
        "func": job_callback,
        "timeout": 60,
        "interval": timedelta(minutes=1),
    }

redpajama_set_name: RedPajamaGithub


The strictly enforced dress code at Ruidoso Middle School draws complaints every year when the entering 6th grade class gets their first look at it.
The strictly enforced dress code at Ruidoso Middle School draws complaints every year when the entering 6th grade class gets their first look at it. But RMS Assistant Principal Mike Speck was still surprised at the earful he got at an orientation recently for graduating White Mountain 5th graders.
Most of the questions were demands to know why they couldn't continue wearing what they felt like to their new school in the fall. Speck said he told them the reasons behind the policy. But then he went home and did a little Googling on the topic.
"I wanted to be sure we were correct in our assumptions," he said.
Those assumptions are that a dress code promotes student safety, reduces social pressure on students who can't afford the latest styles or don't look good in them and eliminates distractions from class work. Speck found plenty to reassure him.
Speck's research produced a summary of results from a national survey in 2015 of all public and private schools in which 23 percent reported dress code or uniform policies and the statistics showed some striking benefits.
Crimes and vandalism dropped sharply in those schools, according to the survey. Discipline problems were reduced by lesser but still significant margins.
On the other hand, while teachers in schools with dress codes overwhelmingly approved of them, parents tended to be skeptical about the benefits. That appears to be true in Ruidoso too.
"You'd be surprised how many enabling parents we have who make even more excuses than their kids," Speck said. He added that the RMS policy was developed in conjunction with the school's parent advisory committee and has its support.
In a nutshell, the RMS dress code requires students to dress in conventionally styled clothing, with shirts limited to blue or white and pants or skirts either dark blue or khaki.
Speck said students and parents often object here, as they do around the country, that the narrow range of choices constrains student freedom of expression and imposes a financial burden on parents.
But he said students have ample opportunity for self-expression in the classrooms and hallways. As for cost, Speck noted that the school recycles "lightly used" outfits free to any parents who dread having to keep their growing children in compliant school clothes.
Principal Anna Addis conceded this can be a problem.
There are tables in the RMS entry hallway this week piled with clean used clothing to which parents can help themselves. If there's a mid-year need for larger sizes, they can return and trade up. "We want them to take enough to get their student through the whole week," Addis said.
Some parents who prefer to buy new outfits also say it's very difficult to find clothes in the stores that meet the detailed requirements of the code, such as the prohibition on very tight jeans or leggings. Speck is dealing with that objection too.
Speck is often the one who spots infractions, including clothing that meets the code but is being worn improperly. He's especially insistent that boys wear pants that fit correctly at the waist and that they tuck in their shirts.
To show what can happen when loose clothing is tolerated, Speck cued up a YouTube video in which a boy pulls an astonishing arsenal of weaponry, including a shotgun, out of his jeans. It's comical to watch, but Speck isn't laughing.
"I was attacked by a kid who was carrying two knives," he said, and showed a photo of the pair of large, vicious looking blades that was confiscated from the student after Speck subdued him in February of last year.
Incidents involving weapons or other contraband at RMS are extremely rare, Speck conceded, but he wants to keep it that way.
"When students complain to me about the rules, I tell them the rules are part of the way we keep them safe," he said.


text:
namespace Manta {

  class CuttingPlane : public Object {
    BBox bounds;             // Bounds of the internal object.
    Object *internal_object; // Object to be cut.

    Real   movement_scale;
    Vector initial_point;

    Vector plane_point;
    Vector plane_normal;

  public:
    CuttingPlane( const Vector &point_, const Vector &normal_,
                  Object *internal_object_ );

    // Preprocess the internal object and compute its bounds.
    void preprocess( const PreprocessContext &context );

    // Return the bounds of the object.
    void computeBounds(const PreprocessContext& context, BBox& bbox) const
    {
      bbox.extendByBox( bounds );
    }

    // Intersection method.
    void intersect(const RenderContext& context, RayPacket& rays) const;

    // Accessors.
    const BBox &getBounds() { return bounds; };
    Object     *getObject() { return internal_object; };
    void setObject( Object *object_ ) { internal_object = object_; };

    void  getPlanePoint ( Vector& result ) { result = plane_point; };
    void  getPlaneNormal( Vector& result ) { result = plane_normal; };

    void setPlanePoint ( const Vector &plane_point_ )
    {
      plane_point = plane_point_;
    }
    void setPlaneNormal( const Vector &plane_normal_ )
    {
      plane_normal = plane_normal_;
    }

    // Move the plane point a signed distance along the plane normal.
    void movePlaneAlongNormal( Real distance );
  };
}

#endif


The world's first known coffeehouse opened in Istanbul in the 16th century. With more than 500 years of coffee history, the city has hundreds of coffee shops varying from third wave coffee brewers to experimental labs and roasteries. We have selected some of the best coffee shops in Istanbul for this article. Next time you find yourself in the center of the ancient world, you'll want to check these out.
Definitely the most popular coffee shop in Istanbul, considering the number of Instagrammers inside. Ministry of Coffee distillates the roaster for 24 hours and serves as 'cold brew' with ice. The shop offers coffee from over 12 origins including Colombia, Costa Rica, Mexico, Peru, Panama, and Kenya. The venue has two floors and is spacious. Make sure you give the Chilli Mocha a try.
Kronotrop is a boutique coffee shop offering single origins, blends, hot and cold brews, filter coffees, decaf offerings, and whole beans. Kronotrop started as a small coffee shop on a side street of Beyoğlu. In a short time, its coffee became an indispensable haunt of passion. Kronotrop is aiming to carry the "Specialty Coffee" stream in the world to the highest level in Istanbul. You will love Kyoto Style cold brew coffee.
Dare to have a real Turkish coffee? Mandabatmaz is definitely the place to go! It literally is the answer to how dark, strong, and tasty a Turkish coffee can be. They cook and serve freshly grounded classic Turkish coffee beans and bake it in a cezve (a traditional small brass pot). Mandabatmaz is located in a humble street in Istiklal Caddesi yet its coffee is legendary and there is always a queue for coffee. Just follow the smell!
Another staple coffee shop for a well-brewed classic Turkish coffee is İstisna Tatlar, located on the Anatolian side. The coffee shop serves exceptionally flavorful coffee, as its name implies in Turkish. You will be amazed especially with the strawberry and mandarin flavored Turkish coffee. This sweet little coffee shop will put a smile on your face with its cozy and friendly atmosphere.
Situated at the colorful and overwhelmingly crowded Eminonu, Brew CoffeeWorks is a great stopover for a shopping break. Just below the Legacy Ottoman Hotel, the shop is teeming with excited patrons and a large, varied menu. Try the flat white if you are more of a light and creamy coffee lover, otherwise give a Bongo a try. From the moment you walk in to the moment you exit, its a rich sensory experience.
Galata is the home of Federal Coffee Company, known for their international expertise and bringing a new dimension to coffee culture in the form of 3rd Wave Coffee, which places immense value on the coffee's naturally occurring flavors. The coffee shop offers a warm and intimate atmosphere and consistently educates new baristas for the industry. It's almost certain that you'll be around Galata when you visit, so make sure you step into Federal Coffee Company to give Australian style roasted beans a try.
Right in the middle of Besiktas, Ayıbedenler is a cozy but super entertaining coffee house. The founders of the shop host their own online radio show with the same name, Ayıbedenler, broadcasting from the ground floor of their cafe. They explain their philosophy as "having good coffee while having fun". Their coffee is really delicious and we assure you will not want to leave once are trapped in the rhythm of the music. Look further if you are into good music and a little laughter to accompany a delicious cup of coffee.
Walter's Coffee Roastery, the first coffee laboratory of Istanbul, opened in 2015. The roastery was a huge success and now it is serving daily roasted specialty coffee in Brooklyn, NY and Dubai as well. The roastery specializes in daily roasted coffee and continuously running experiments, from extraction and brew temperatures to espresso machine pressure and grinder configuration. The staff aren't just baristas – they're craftsmen.
If we had a "most cool coffee shops" category, Petra might be the winner. The design of the shop and atmosphere is classic and charming. This coffee shop is located in Muse Istanbul in Gayrettepe. The venue is multifunctional – you can grab your coffee and pay a visit to the museum, work on your laptop as if you are in an open workspace, or shop for some souvenirs for your loved ones. Do not miss the cold Myrrha coffee served in whiskey glass before the summer is over. If you are innovative and you like to take risks, try Sbagliato.
Mambocino has few branches in Istanbul but it's Moda branch located on the Anatolian side is coffee lover's favorite. Mambocino is the name of the English coffee brand and they use syrups from Torani of San Francisco. Their latte art is over the top impressive. Lavender ice lattes and Affagatos are the way to go for cold coffee lovers. Guatemala filter coffee is also one of our favorites along with homemade cookies. With its cute decor and smooth music, Mambocino is a great down-to-earth alternative for delicious coffee lovers.
Ebru is a freelance journalist and content creator focused on music, travel and festivals. She is also the editor-in-chief at in Festival Soul TV, an online platform focusing on music festivals in Europe. Her articles have appeared in various local and international platforms online. Along with her editorial work, she also produces video content about music, travel and festivals and an accomplished producer in Balcony TV network. Ebru lives in Istanbul.


text:
Kościejów is een plaats in het Poolse district Miechowski, woiwodschap Klein-Polen. De plaats maakt deel uit van de gemeente Racławice en telt 420 inwoners.

Plaats in Klein-Polen

redpajama_set_name: RedPajamaWikipedia


CHICAGO, May 21, 2015 – Whether it's keeping candidates guessing as to where they are in the application process or simply neglecting to acknowledge their application, some employers are unwittingly leaving candidates with a bad impression – and it's taking a toll on their business.
The 2015 Candidate Behavior study, conducted by Inavero on behalf of CareerBuilder of more than 5,013 workers ages 18 and over and 2,002 hiring decision makers between February 3 and February 18, 2015, sheds light on the differences between what candidates expect from potential employers during the job application process and what employers actually deliver. View full results and the executive summary here.
According to the study, 82 percent of employers think there's little to no negative impact on the company when a candidate has a bad experience during the hiring process. The reality, however, is that the majority of candidates do not take poor treatment lying down: 58 percent are less likely to buy from a company to which they've applied if they don't get a response to their application; 69 percent are less likely if they have a bad experience in the interview; and the same is true of 65 percent if they didn't hear back after an interview.
Conversely, a good candidate experience can have the reverse effect: 69 percent of candidates are more likely to buy from a company to which they've applied if they're treated with respect throughout the application process, and 67 percent are likely to do the same if they receive consistent updates throughout the recruitment process.
Job seekers may be searching for jobs in a lot of places where employers don't have a presence. Candidates consult up to 18 resources throughout their job search – including job boards, social networking sites, search engines and online referrals – but the majority of employers (58 percent) don't use tracking or coding technology to learn where candidates are coming from and ensure they are making efficient use of their recruitment marketing efforts. Without any data on where their candidates are coming from, employers may be missing opportunities to connect with candidates where they are actually searching.
For some candidates, the myth of the infamous application "black hole" is all too real. More than half of employers (52 percent) respond to less than half of the candidates who apply. What these employers may not realize, however, is that not only do most candidates expect an automated reply that acknowledges their application, the majority (84 percent) also expect a personal email response, and 52 percent anticipate a phone call. Even when the news isn't what they hope to receive, candidates expect a response: 1 in 4 candidates (25 percent) expect to hear if the employer will not be bringing them in for an interview.
When it comes to candidate communication, employers seem to be falling short of candidates' expectations. Thirty-six percent of candidates expect to be updated throughout the application process, and 41 percent expect to be notified if they weren't chosen after they interviewed with the company. Yet only 26 percent of employers proactively communicate with candidates what stage of the hiring process they're in. Even when they've made it as far into the process as an interview, many candidates are still left in the dark: Nearly three in four (73 percent) candidates who interviewed with companies said they were never given an explanation for why they didn't get the job.
When it comes to keeping candidates engaged and interested in their opportunities, a company's application process can be its own worst enemy. Forty percent of candidates feel the application process has become more difficult in the last five years. Of those, 57 percent complain the process is too automated and lacks personalization, 51 percent are frustrated they have no idea where they are in the process, and 50 percent say the process has so many more steps than it used to have. A separate CareerBuilder survey found that 3 in 5 candidates do not complete the application process if they feel it is too long1. Employers can reduce these frustrations by taking the time to respond to candidates, by keeping the lines of communication going and by minimizing the number of steps candidates must go through during the application process.
As noted earlier, treating candidates well is good for the bottom line. More than 3 in 4 candidates (77 percent) are willing to accept a salary that is 5 percent lower than their expected offer if the employer created a great impression through the hiring process; even more (83 percent) would do the same if the company had a reputation as a great employer. Candidates would also accept a lower salary if the company had a lot of positive press recently (69 percent) and had great online reviews (73 percent).
These findings underscore the importance of having a strong employment brand; however, more than half of employers (52 percent) do not have a clearly defined employment brand – giving the other 48 percent a distinct edge when it comes to capturing in-demand candidates at competitive prices.
1Nationwide survey, conducted online by Harris Poll on behalf of CareerBuilder from June 2 to June 25, 2014, which included a representative sample of 374 HR professionals and 319 job seekers.
In partnership with Inavero, CareerBuilder surveyed 5,013 workers ages 18 and over, and 2,002 hiring managers, between February 3 and February 18, 2015 in the United States in an effort to understand the factors that influence candidates' job search behavior. With a pure probability sample of 5,013 job seekers, one could say with a 95 confidence interval that the job seeker results have a sampling error of +/- 1.4 percent. With a pure probability sample of 2,002 hiring managers, one could say with a 95 confidence interval that the hiring manager results have a sampling error of +/- 2.1 percent. Sampling error for data from sub-samples is higher and varies.


Simon Mackintosh is a Trustee of the Ray and Diana Harryhausen Foundation. Simon is a solicitor at Turcan Connell, where he is Chairman of the Partnership. He has a long-standing involvement with the Harryhausen family and the Ray and Diana Harryhausen Foundation, and brings specialist knowledge of charity law and related matters to the Board of Trustees.
Simon is accredited by the Law Society of Scotland as a Charity Law specialist and was a member of the Scottish Charity Law Review Commission (the McFadden Commission) whose report led to the introduction of the Charities & Trustee Investment (Scotland) Act 2005. He sits on the Executive Committee of the UK-wide Charity Law Association and was sometime chair of The Law Society of Scotland's Charity Law Sub-Committee.
Simon works with charities particularly on issues of governance, regulation, formation and mergers, including tax aspects. His experience as a charity trustee includes serving on the board of the Edinburgh International Book Festival and chairing the Court of Directors of the Edinburgh Academy. He also recently completed a term as Chairman of the Scottish Episcopal Church Pension Trustees.


text:
\section{Submission of conference papers to ICLR 2022}

ICLR requires electronic submissions, processed by
\url{https://openreview.net/}. See ICLR's website for more instructions.

If your paper is ultimately accepted, the statement {\tt
  {\textbackslash}iclrfinalcopy} should be inserted to adjust the
format to the camera ready requirements.

The format for the submissions is a variant of the NeurIPS format.
Please read carefully the instructions below, and follow them
faithfully.

\subsection{Style}

Papers to be submitted to ICLR 2022 must be prepared according to the
instructions presented here.

Authors are required to use the ICLR \LaTeX{} style files obtainable at the
ICLR website. Please make sure you use the current files and
not previous versions. Tweaking the style files may be grounds for rejection.

\subsection{Retrieval of style files}

The style files for ICLR and other conference information are available online at:
\begin{center}
   \url{http://www.iclr.cc/}
\end{center}
The file \verb+iclr2022_conference.pdf+ contains these
instructions and illustrates the
various formatting requirements your ICLR paper must satisfy.
Submissions must be made using \LaTeX{} and the style files
\verb+iclr2022_conference.sty+ and \verb+iclr2022_conference.bst+ (to be used with \LaTeX{}2e). The file
\verb+iclr2022_conference.tex+ may be used as a ``shell'' for writing your paper. All you
have to do is replace the author, title, abstract, and text of the paper with
your own.

The formatting instructions contained in these style files are summarized in
sections \ref{gen_inst}, \ref{headings}, and \ref{others} below.

\section{General formatting instructions}
\label{gen_inst}

The text must be confined within a rectangle 5.5~inches (33~picas) wide and
9~inches (54~picas) long. The left margin is 1.5~inch (9~picas).
Use 10~point type with a vertical spacing of 11~points. Times New Roman is the
preferred typeface throughout. Paragraphs are separated by 1/2~line space,
with no indentation.

Paper title is 17~point, in small caps and left-aligned.
All pages should start at 1~inch (6~picas) from the top of the page.

Authors' names are
set in boldface, and each name is placed above its corresponding
address. The lead author's name is to be listed first, and
the co-authors' names are set to follow. Authors sharing the
same address can be on the same line.

Please pay special attention to the instructions in section \ref{others}
regarding figures, tables, acknowledgments, and references.

There will be a strict upper limit of 9 pages for the main text of the initial submission, with unlimited additional pages for citations.

\section{Headings: first level}
\label{headings}

First level headings are in small caps,
flush left and in point size 12. One line space before the first level
heading and 1/2~line space after the first level heading.

\subsection{Headings: second level}

Second level headings are in small caps,
flush left and in point size 10. One line space before the second level
heading and 1/2~line space after the second level heading.

\subsubsection{Headings: third level}

Third level headings are in small caps,
flush left and in point size 10. One line space before the third level
heading and 1/2~line space after the third level heading.

\section{Citations, figures, tables, references}
\label{others}

These instructions apply to everyone, regardless of the formatter being used.

\subsection{Citations within the text}

Citations within the text should be based on the \texttt{natbib} package
and include the authors' last names and year (with the ``et~al.'' construct
for more than two authors). When the authors or the publication are
included in the sentence, the citation should not be in parenthesis using \verb|\citet{}| (as
in ``See \citet{Hinton06} for more information.''). Otherwise, the citation
should be in parenthesis using \verb|\citep{}| (as in ``Deep learning shows promise to make progress
towards AI~\citep{Bengio+chapter2007}.'').

The corresponding references are to be listed in alphabetical order of
authors, in the \textsc{References} section. As to the format of the
references themselves, any style is acceptable as long as it is used
consistently.

\subsection{Footnotes}

Indicate footnotes with a number\footnote{Sample of the first footnote} in the
text. Place the footnotes at the bottom of the page on which they appear.
Precede the footnote with a horizontal rule of 2~inches
(12~picas).\footnote{Sample of the second footnote}

\subsection{Figures}

All artwork must be neat, clean, and legible. Lines should be dark
enough for purposes of reproduction; art work should not be
hand-drawn. The figure number and caption always appear after the
figure. Place one line space before the figure caption, and one line
space after the figure. The figure caption is lower case (except for
first word and proper nouns); figures are numbered consecutively.

Make sure the figure caption does not get separated from the figure.
Leave sufficient space to avoid splitting the figure and figure caption.

You may use color figures.
However, it is best for the
figure captions and the paper body to make sense if the paper is printed
either in black/white or in color.
\begin{figure}[h]
\begin{center}
\fbox{\rule[-.5cm]{0cm}{4cm} \rule[-.5cm]{4cm}{0cm}}
\end{center}
\caption{Sample figure caption.}
\end{figure}

\subsection{Tables}

All tables must be centered, neat, clean and legible. Do not use hand-drawn
tables. The table number and title always appear before the table. See
Table~\ref{sample-table}.

Place one line space before the table title, one line space after the table
title, and one line space after the table. The table title must be lower case
(except for first word and proper nouns); tables are numbered consecutively.

\begin{table}[t]
\caption{Sample table title}
\label{sample-table}
\begin{center}
\begin{tabular}{ll}
\multicolumn{1}{c}{\bf PART}  &\multicolumn{1}{c}{\bf DESCRIPTION}
\\ \hline \\
Dendrite         &Input terminal \\
Axon             &Output terminal \\
Soma             &Cell body (contains cell nucleus) \\
\end{tabular}
\end{center}
\end{table}

\section{Default Notation}

In an attempt to encourage standardized notation, we have included the
notation file from the textbook, \textit{Deep Learning}
\cite{goodfellow2016deep} available at
\url{https://github.com/goodfeli/dlbook_notation/}.  Use of this style
is not required and can be disabled by commenting out
\texttt{math\_commands.tex}.

\centerline{\bf Numbers and Arrays}
\bgroup
\def1.5{1.5}
\begin{tabular}{p{1in}p{3.25in}}
$\displaystyle a$ & A scalar (integer or real)\\
$\displaystyle \va$ & A vector\\
$\displaystyle \mA$ & A matrix\\
$\displaystyle \tA$ & A tensor\\
$\displaystyle \mI_n$ & Identity matrix with $n$ rows and $n$ columns\\
$\displaystyle \mI$ & Identity matrix with dimensionality implied by context\\
$\displaystyle \ve^{(i)}$ & Standard basis vector $[0,\dots,0,1,0,\dots,0]$ with a 1 at position $i$\\
$\displaystyle \text{diag}(\va)$ & A square, diagonal matrix with diagonal entries given by $\va$\\
$\displaystyle \ra$ & A scalar random variable\\
$\displaystyle \rva$ & A vector-valued random variable\\
$\displaystyle \rmA$ & A matrix-valued random variable\\
\end{tabular}
\egroup
\vspace{0.25cm}

\centerline{\bf Sets and Graphs}
\bgroup
\def1.5{1.5}

\begin{tabular}{p{1.25in}p{3.25in}}
$\displaystyle \sA$ & A set\\
$\displaystyle \R$ & The set of real numbers \\
$\displaystyle \{0, 1\}$ & The set containing 0 and 1 \\
$\displaystyle \{0, 1, \dots, n \}$ & The set of all integers between $0$ and $n$\\
$\displaystyle [a, b]$ & The real interval including $a$ and $b$\\
$\displaystyle (a, b]$ & The real interval excluding $a$ but including $b$\\
$\displaystyle \sA \backslash \sB$ & Set subtraction, i.e., the set containing the elements of $\sA$ that are not in $\sB$\\
$\displaystyle \gG$ & A graph\\
$\displaystyle \parents_\gG(\ervx_i)$ & The parents of $\ervx_i$ in $\gG$
\end{tabular}
\vspace{0.25cm}

\centerline{\bf Indexing}
\bgroup
\def1.5{1.5}

\begin{tabular}{p{1.25in}p{3.25in}}
$\displaystyle \eva_i$ & Element $i$ of vector $\va$, with indexing starting at 1 \\
$\displaystyle \eva_{-i}$ & All elements of vector $\va$ except for element $i$ \\
$\displaystyle \emA_{i,j}$ & Element $i, j$ of matrix $\mA$ \\
$\displaystyle \mA_{i, :}$ & Row $i$ of matrix $\mA$ \\
$\displaystyle \mA_{:, i}$ & Column $i$ of matrix $\mA$ \\
$\displaystyle \etA_{i, j, k}$ & Element $(i, j, k)$ of a 3-D tensor $\tA$\\
$\displaystyle \tA_{:, :, i}$ & 2-D slice of a 3-D tensor\\
$\displaystyle \erva_i$ & Element $i$ of the random vector $\rva$ \\
\end{tabular}
\egroup
\vspace{0.25cm}

\centerline{\bf Calculus}
\bgroup
\def1.5{1.5}
\begin{tabular}{p{1.25in}p{3.25in}}
$\displaystyle\frac{d y} {d x}$ & Derivative of $y$ with respect to $x$\\ [2ex]
$\displaystyle \frac{\partial y} {\partial x} $ & Partial derivative of $y$ with respect to $x$ \\
$\displaystyle \nabla_\vx y $ & Gradient of $y$ with respect to $\vx$ \\
$\displaystyle \nabla_\mX y $ & Matrix derivatives of $y$ with respect to $\mX$ \\
$\displaystyle \nabla_\tX y $ & Tensor containing derivatives of $y$ with respect to $\tX$ \\
$\displaystyle \frac{\partial f}{\partial \vx} $ & Jacobian matrix $\mJ \in \R^{m\times n}$ of $f: \R^n \rightarrow \R^m$\\
$\displaystyle \nabla_\vx^2 f(\vx)\text{ or }\mH( f)(\vx)$ & The Hessian matrix of $f$ at input point $\vx$\\
$\displaystyle \int f(\vx) d\vx $ & Definite integral over the entire domain of $\vx$ \\
$\displaystyle \int_\sS f(\vx) d\vx$ & Definite integral with respect to $\vx$ over the set $\sS$ \\
\end{tabular}
\egroup
\vspace{0.25cm}

\centerline{\bf Probability and Information Theory}
\bgroup
\def1.5{1.5}
\begin{tabular}{p{1.25in}p{3.25in}}
$\displaystyle P(\ra)$ & A probability distribution over a discrete variable\\
$\displaystyle p(\ra)$ & A probability distribution over a continuous variable, or over
a variable whose type has not been specified\\
$\displaystyle \ra \sim P$ & Random variable $\ra$ has distribution $P$\\% so thing on left of \sim should always be a random variable, with name beginning with \r
$\displaystyle  \E_{\rx\sim P} [ f(x) ]\text{ or } \E f(x)$ & Expectation of $f(x)$ with respect to $P(\rx)$ \\
$\displaystyle \Var(f(x)) $ &  Variance of $f(x)$ under $P(\rx)$ \\
$\displaystyle \Cov(f(x),g(x)) $ & Covariance of $f(x)$ and $g(x)$ under $P(\rx)$\\
$\displaystyle H(\rx) $ & Shannon entropy of the random variable $\rx$\\
$\displaystyle \KL ( P \Vert Q ) $ & Kullback-Leibler divergence of P and Q \\
$\displaystyle \mathcal{N} ( \vx ; \vmu , \mSigma)$ & Gaussian distribution %
over $\vx$ with mean $\vmu$ and covariance $\mSigma$ \\
\end{tabular}
\egroup
\vspace{0.25cm}

\centerline{\bf Functions}
\bgroup
\def1.5{1.5}
\begin{tabular}{p{1.25in}p{3.25in}}
$\displaystyle f: \sA \rightarrow \sB$ & The function $f$ with domain $\sA$ and range $\sB$\\
$\displaystyle f \circ g $ & Composition of the functions $f$ and $g$ \\
  $\displaystyle f(\vx ; \vtheta) $ & A function of $\vx$ parametrized by $\vtheta$.
  (Sometimes we write $f(\vx)$ and omit the argument $\vtheta$ to lighten notation) \\
$\displaystyle \log x$ & Natural logarithm of $x$ \\
$\displaystyle \sigma(x)$ & Logistic sigmoid, $\displaystyle \frac{1} {1 + \exp(-x)}$ \\
$\displaystyle \zeta(x)$ & Softplus, $\log(1 + \exp(x))$ \\
$\displaystyle || \vx ||_p $ & $\normlp$ norm of $\vx$ \\
$\displaystyle || \vx || $ & $\normltwo$ norm of $\vx$ \\
$\displaystyle x^+$ & Positive part of $x$, i.e., $\max(0,x)$\\
$\displaystyle \1_\mathrm{condition}$ & is 1 if the condition is true, 0 otherwise\\
\end{tabular}
\egroup
\vspace{0.25cm}

\section{Final instructions}
Do not change any aspects of the formatting parameters in the style files.
In particular, do not modify the width or length of the rectangle the text
should fit into, and do not change font sizes (except perhaps in the
\textsc{References} section; see below). Please note that pages should be
numbered.

\section{Preparing PostScript or PDF files}

Please prepare PostScript or PDF files with paper size ``US Letter'', and
not, for example, ``A4''. The -t
letter option on dvips will produce US Letter files.

Consider directly generating PDF files using \verb+pdflatex+
(especially if you are a MiKTeX user).
PDF figures must be substituted for EPS figures, however.

Otherwise, please generate your PostScript and PDF files with the following commands:
\begin{verbatim}
dvips mypaper.dvi -t letter -Ppdf -G0 -o mypaper.ps
ps2pdf mypaper.ps mypaper.pdf
\end{verbatim}

\subsection{Margins in LaTeX}

Most of the margin problems come from figures positioned by hand using
\verb+\special+ or other commands. We suggest using the command
\verb+\includegraphics+
from the graphicx package. Always specify the figure width as a multiple of
the line width as in the example below using .eps graphics
\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]{myfile.eps}
\end{verbatim}
or
\begin{verbatim}
   \usepackage[pdftex]{graphicx} ...
   \includegraphics[width=0.8\linewidth]{myfile.pdf}
\end{verbatim}
for .pdf graphics.
See section~4.4 in the graphics bundle documentation (\url{http://www.ctan.org/tex-archive/macros/latex/required/graphics/grfguide.ps})

A number of width problems arise when LaTeX cannot properly hyphenate a
line. Please give LaTeX hyphenation hints using the \verb+\-+ command.

\subsubsection*{Author Contributions}
If you'd like to, you may include  a section for author contributions as is done
in many journals. This is optional and at the discretion of the authors.

\subsubsection*{Acknowledgments}
Use unnumbered third level headings for the acknowledgments. All
acknowledgments, including those to funding agencies, go at the end of the paper.

\section{Introduction}
\label{intro}
Interest in non-convex optimization has grown in recent years, driven by applications such as
training deep neural networks.
Often, one seeks convergence to a local minimizer in such problems because finding global minima is known to
be NP complete~\cite{murty87}. To ensure convergence to minimizers, one research direction
in non-convex optimization has been the identification of problem properties for which particular algorithms escape saddle points. One such property, which has become common in the non-convex optimization literature since its introduction in~\cite{ge2015escaping}, is the \textit{strict saddle property} (SSP), which states that the Hessian of every saddle point of a function has at least one negative eigenvalue. It was later shown that gradient descent and other first order methods almost always escape saddle points of objective functions
that satisfy the SSP (and other mild assumptions)~\cite{lee2016gradient,panageas2017gradient,lee2017first}.

Because of this behavior, a growing body of non-convex optimization research has
either focused on problems for which the SSP is known to hold, or simply assumed the SSP holds for a
generic problem and derived convergence guarantees that result from it.
However, verification of the SSP for a general, unstructured problem is difficult in practice,
and there exist problems in machine learning for which the SSP does not hold, such as training a neural
network with at least two hidden layers~\cite{kawaguchi2016deep}.

Motivated by these challenges, we develop an algorithmic method that will escape saddle points that are not
strict. We still wish to leverage the
current body of knowledge on problems with the SSP, rather than developing entirely new convergence analyses, and
thus our approach is to \textit{enforce} the SSP
by regularizing problems when in the vicinity of a non-strict saddle point, rather than simply assuming that the SSP holds.
We show that this can be done with a linear regularizer,
motivated by John Milnor's proof that almost all choices of such a term will
render a function Morse (and therefore
guarantee that the regularized problem has the SSP)~\yrcite{milnor2015lectures}.
We are also motivated by the success of regularization techniques in convex optimization, where quadratic
perturbations are used to provide strong convexity to objective functions~\cite{facchinei07}, and we believe that
the linear regularizers we present are their natural counterparts in the non-convex setting.

\subsection{Related Work}
\label{intro.relatedwork}

A large body of work exists on the convergence properties of gradient descent and other first-order methods on problems with the SSP, including algorithms that
consider deterministic gradient descent~\cite{dixit2020exit,schaeffer2019extending}, and those that incorporate noise into their updates~\cite{xu2017first,daneshmand2018escaping,yang2017fast,ge2015escaping}. These methods are shown to escape strict saddles, but have not been shown to escape non-strict saddles, and therefore rely on the SSP.

While these methods are shown to escape strict saddles in the limit, they can get stuck near strict saddles for exponential time, which can cause numerical slowdowns~\cite{du2017gradient}. Some attempts have been made to accelerate the escape near strict saddle points~\cite{jin2017escape,agarwal2017finding,jin2018accelerated}. However, when starting near non-strict saddles, first-order methods may actually converge to them, and such accelerated methods do not escape.

Current research into escaping non-strict saddle points requires higher-order information and/or algorithms.
Perhaps the best known is~\cite{anandkumar2016efficient}, which guarantees convergence to a third-order optimal critical point. That paper replaces the SSP, which is a property of the Hessian, with a condition on the third-order derivative of the problem. Work in~\cite{zhu2020adaptive} expands on these results and includes numerical simulations on a function that does not satisfy the SSP. Subsequent work in~\cite{chen2021high} provides a method to converge to $p^{th}$-order critical points using $p^{th}$-order information, while also demonstrating that doing so for $p \geq 4$ is NP-hard. Recent work in~\cite{truong2021new} examines the behavior of a second-order method on common toy examples of non-strict saddle points, and~\cite{nguyen2017loss} develop a weaker form of the SSP that guarantees escape from saddle points for the training of a particular neural network. In contrast, in this paper we require only first-order information
and provably escape from non-strict saddles using linear regularizers under
weak assumptions.

Previous research has shown that regularizing with quadratic or sums of squares (SOS) terms will make a function Morse, which is sufficient to ensure the SSP is satisfied~\cite{lerario2011plenty,nicolaescu2011invitation}. However, no convergence or bifurcation analysis was performed on the regularized function, and
indeed these
results originate outside the non-convex optimization literature.
We show in Example~\ref{e.quadregbad} that quadratic and SOS
regularizers can actually convert a non-strict saddle point into a local minimum,
and thus we do not use them.

\subsection{Contributions}
\label{intro.contributions}

The contributions of this paper are the following:
\begin{itemize}
    \item A local linear regularization method using only first-order information (i.e., $\nabla f(x)$) that enforces satisfaction of the SSP around critical points
    \item A proof that this regularization method escapes non-strict saddle points that satisfy a condition much
    weaker than the SSP, and examples of non-strict saddle points that satisfy this weaker condition
    \item Conditions under which linearly regularized gradient descent is guaranteed to converge to a local minimum
    \item Bounds on the regularization error seen at minima that is
    induced by linear regularizers
\end{itemize}

The remainder of this paper is organized as follows. In Section~\ref{linearregularizers}, we prove that almost all linear regularizers
enforce the SSP, and we
demonstrate the need to define this regularizer locally rather than globally. In Section~\ref{bifurcations}, we analyze the bifurcations that may occur when regularizing and
justify a particular choice of regularizer
that avoids problematic bifurcations.
In Section~\ref{exitcondition}, we describe the condition weaker than the SSP for which this regularization
method escapes saddle points and demonstrate this escape on examples of non-strict saddle points taken from the literature.
Then, in Section~\ref{convergencecondition} we give conditions
under which we converge to local minima. In Section~\ref{numericalresults}, we present
numerical results on a neural network with four hidden layers, and in Section~\ref{concludingremarks} we provide concluding remarks.

\section{Local Linear Regularization}
\label{linearregularizers}

Throughout this paper, $f$ denotes a function in $C^2$, the space of twice-continuously differentiable functions, with $L$-Lipschitz gradient $\nabla f$.
The symbol $g$ denotes the gradient descent map $g(x) = x - \gamma \nabla f(x)$, with $\gamma \in (0, 1/L)$. The following definition regards the \textit{critical points} of $f$:

\begin{definition}
\begin{enumerate}
    \item A point $x^*$ is a \textit{critical point} of $f$ if $\nabla f(x^*) = 0$ or, equivalently, $g(x^*) = x^*$.
    \item A critical point $x^*$ is \textit{isolated} if there is a neighborhood $U$ around $x^*$ and $x^*$ is the only critical point in $U$. Otherwise it is called \textit{non-isolated}.
    \item A critical point of $f$ is a \textit{local minimum} if there is a neighborhood $U$ around $x^*$ such that $f(x^*) \leq f(x)$ for all $x \in U$, and a \textit{local maximum} if $f(x^*) \geq f(x)$.
    \item A critical point of $f$ is a \textit{saddle point} if for all neighborhoods $U$ around $x^*$, there are $y,z \in U$ such that $f(y) \leq f(x^*) \leq f(z)$.
    \item A critical point of $f$ is a \textit{strict saddle} if $\lambda_{min}(\nabla^2f(x^*)) < 0$.
    \item The \textit{local stable set} $W^{s}(x^*)$ defined on some neighborhood $U$
    of a critical point $x^*$ is the set of initial conditions of gradient descent in $U$ that converge to $x^*$, i.e.,
    $W^s(x^*) = \{x \in U:\lim_{k \to \infty} g^k(x)=x^*\}$.
    The \textit{local unstable set} is given by $W^u(x^*) = \{x \in U:\lim_{k \to \infty} g^k(x)\neq x^*\}$.
    If $U = \mathbb{R}^n$, then $W^s(x^*)$ ($W^u(x^*)$) is the \textit{global stable (unstable) set}.
\end{enumerate}
\end{definition}

Here~$\lambda_{min}(\cdot)$ denotes the minimum eigenvalue of a square matrix.
Lemma~\ref{l.gdavoidsstrictsaddles} states that, for almost all initial conditions,
gradient descent does
not converge to a strict saddle:

\begin{lemma} \label{l.gdavoidsstrictsaddles}
Let $f : \mathbb{R}^n \rightarrow \mathbb{R}$ be a $C^2$ function with $L$-Lipschitz gradient. The set of initial conditions $x \in \mathbb{R}^n$ such that $g^k(x)$ converges to a strict saddle point of $f$ is of (Lebesgue) measure zero.
\end{lemma}
\textit{Proof:} See Theorem 2 in~\cite{panageas2017gradient}. $\hfill\square$

The underlying principle is that,
for a saddle~$x^*$,
a single negative eigenvalue
of $\nabla^2 f(x^*)$ renders $W^s(x^*)$ measure zero. This is the motivating
principle behind the study of the strict saddle property:

\begin{definition}
A function $f$ satisfies the \textit{strict saddle property (SSP)} if every saddle point of $f$ is strict.
\end{definition}

From Lemma~\ref{l.gdavoidsstrictsaddles}, gradient descent will almost always avoid every strict saddle point of an objective function $f$.
Therefore, if $f$ satisfies the SSP, then gradient descent will almost always avoid \textit{all} saddle points of $f$. Provided gradient descent converges (i.e., $\lim_{k \to \infty} g^k(x)$ exists), it must then
almost always
converge to a local minimum. We note that $g^k(x)$ is guaranteed to converge in a variety of settings, including when $f$ is analytic or coercive,
and we will proceed with the assumption that $f$ satisfies one of these properties.
However, verifying that a general, unstructured function satisfies the
SSP is difficult in practice, and functions of interest exist that
are known not to satisfy the SSP, such as the loss function of a neural network with at least two hidden layers~\cite{kawaguchi2016deep}. These functions may have \textit{non-strict saddles}:

\begin{definition}
A saddle point $x^*$ of $f$ is a \textit{non-strict saddle} if $\lambda_{min}(\nabla^2f(x^*)) = 0$.
\end{definition}

We make a brief point on terminology here. The definition of a degenerate saddle
varies between the dynamical systems and computer science literature, so to avoid
confusion in this paper a \textit{degenerate saddle} is any saddle point $x^*$
whose Hessian has at least one zero eigenvalue (i.e., $\nabla^2 f (x^*)$ is singular),
while a \textit{non-strict saddle} is a saddle whose minimum eigenvalue is zero. Using this terminology, any non-strict saddle is necessarily degenerate. We note that
the SSP is \textit{not} a non-degeneracy condition, as strict saddles may be degenerate, as long as they have at least one negative eigenvalue. Example~\ref{e.degeneratesaddle} illustrates the key problem with non-strict saddle points, which is
that their stable sets are not necessarily measure zero.

\begin{example} \label{e.degeneratesaddle}
\textit{(Non-Measure Zero $W^s$ for non-Strict Saddle)}

Consider the function $f(x,y) = \frac{1}{3}x^3 + \frac{1}{2}y^2$, with negative gradient field plotted in Figure~\ref{fig:degeneratesaddle}. Here, $(0,0)$ is a non-strict saddle of $f$, with $\nabla^2 f(0,0)$ having $1$ and $0$ as eigenvalues. Here, we see $W^s(0,0) = \{(x,y) : x > 0\}$, depicted by the red region. That is, the set of initial conditions for which $g^k(x)$ converges to $(0,0)$ is not measure zero and is in fact a closed halfspace of $\mathbb{R}^2$.
\end{example}

\begin{figure}[htp]
    \centering
    \includegraphics[width=6cm]{Degenerate_Saddle.pdf}
    \caption{The negative gradient field of $f(x,y) = \frac{1}{3}x^3+\frac{1}{2}y^2$. The blue dot at $(0,0)$ denotes the non-strict saddle point, $W^u(0,0)$ is denoted by the green region, and $W^s(0,0)$ by red.}
    \label{fig:degeneratesaddle}
\end{figure}

Instead of modifying our gradient descent algorithm to somehow accommodate non-strict saddles, we instead wish to modify the \textit{problem} itself in such a way that the modified function satisfies the SSP, either by making non-strict saddles strict or eliminating them altogether. That is, we wish to find a regularization scheme that enforces satisfaction of the SSP and thus
ensures the escape of non-strict saddles. While quadratic and sum of squares regularization terms are popular in convex optimization, they are not advisable in this context as they
change the positive semi-definite Hessian of a non-strict saddle into a positive definite one, turning such a saddle into a local minimum of the regularized function.

\begin{example} \label{e.quadregbad}
Consider again the function $f(x,y) = \frac{1}{3}x^3 + \frac{1}{2}y^2$, which has a non-strict saddle at $(0,0)$ with eigenvalues $1$ and $0$. If a sum of squares regularization term $\frac{1}{2}\alpha_x x^2 + \frac{1}{2}\alpha_y y^2$  is added to $f$, then $(0,0)$ remains a critical point of the regularized function, but the eigenvalues of the regularized Hessian become $\alpha_x$ and $1+\alpha_y$, rendering $(0,0)$ a local minimum for all $\alpha_x,\alpha_y > 0$.
\end{example}

Instead, the following lemma from~\cite{milnor2015lectures} provides motivation for using a linear regularization term.

\begin{lemma} \label{l.milnor}
If $f : \mathbb{R}^n \rightarrow \mathbb{R}$ is a $C^2$ function, then for almost all $l \in \mathbb{R}^n$,
the critical points of the function $f_l(x) = f(x) + l^T x$ have only non-singular Hessians.
\end{lemma}
\textit{Proof:} See Appendix~\ref{apx.l.milnor}. $\hfill\square$

This lemma states that for almost any choice of $l$, the regularized function $f_l$ will have only non-degenerate critical points.  This immediately gives us the following result:

\begin{theorem} \label{t.enforcessp}
If $f : \mathbb{R}^n \rightarrow \mathbb{R}$ is a $C^2$ function, then for almost all $l \in \mathbb{R}^n$, the function
$f_l(x) = f(x) + l^T x$ satisfies the SSP.
\end{theorem}
\textit{Proof:} This follows from Lemma~\ref{l.milnor} and the fact that non-degenerate saddle points must be strict. $\hfill\square$

This regularization method does not affect the Hessian (i.e., $\nabla^2 f(x) = \nabla^2 f_l(x)$), avoiding the problems
caused by sums of squares and quadratic regularizers and resulting in simpler analysis.
However, this regularization method enforces satisfaction of the SSP by creating bifurcations of degenerate critical points of $f$, and we must carefully analyze these bifurcations to ensure
that we attain the desired convergence properties.

\section{Bifurcations}
\label{bifurcations}

Regularization of a function naturally perturbs non-degenerate critical points, which can be limited by a judicious choice
of regularizer. However, more seriously, regularization can also cause a \textit{local bifurcation}
at a degenerate critical point:
\begin{definition}
Let $f: \mathbb{R}^n \times \mathbb{R}^k \rightarrow \mathbb{R}$ be a $C^2$ function. Let $(x^*,\mu^*)$ be a point for which $\nabla_x f(x^*,\mu^*) = 0$ and $\nabla_x^2 f(x^*,\mu^*)$ is singular. A \textit{local bifurcation} of this gradient system occurs at $x^*$ when a smooth change in $\mu$ away from $\mu^*$ induces a sudden change in the stability properties of the equilibrium at $x^*$.
\end{definition}

A ``sudden change in stability properties'' could mean $W^s(x^*)$ grows or shrinks in dimension, the critical point at $x^*$ splits into two or more critical points, or the critical point at $x^*$ is eliminated. A local bifurcation induced by linear regularization is illustrated in Example~\ref{e.bifurcation}:

\begin{example} \label{e.bifurcation}
Consider again the function ~$f(x, y) = \frac{1}{3}x^3 + \frac{1}{2}y^2$ and consider two regularizations that add terms
of the form $l_x x + l_y y$. The first sets~$l_x = 1$ and~$l_y = 0$ and
the second sets~$l_x = -1$ and~$l_y = 0$, and we
plot the trajectory behavior of gradient descent under
these two regularizations in Figure~\ref{fig:degeneratesaddlebifurcation}.
\end{example}

\begin{figure}[htp]
    \centering
   \includegraphics[width=8cm]{Bifurcation.pdf}
    \caption{When we choose $l_x = -1$ we create a local minimum \textit{and} a strict saddle, and the escape region actually shifts (left).
    When we choose $l_x = 1$, the critical point is destroyed and gradient descent escapes the saddle from every initial condition (right).}
    \label{fig:degeneratesaddlebifurcation}
\end{figure}

Observe that when $l_x = -1$, the original non-strict saddle point bifurcates
into a strict saddle at $(-1,0)$ and a local minimum at $(1,0)$. Both of these
points are non-degenerate, satisfying the SSP as ensured by Theorem~\ref{t.enforcessp}.
However, we can see that the stable set
(now defined for both of the resulting critical points) has actually expanded. This is because the resulting
bifurcation has created a \textit{false minimum} at $(1,0)$:

\begin{definition}
A \textit{false minimum} is a local minimum of $f_l$ that resulted from a bifurcation of a degenerate saddle point of $f$
caused by the linear regularizer~$l^Tx$.
\end{definition}

In Example~\ref{e.bifurcation}, one can see that for any $l_x < 0$, a \textit{saddle-node bifurcation} occurs, which
splits a degenerate critical
point into two non-degenerate critical points.
We also observe that when $l_x = 1$ (and in fact whenever $l_x > 0$) the critical point
at $(0,0)$ is destroyed and all trajectories
of gradient descent
escape the neighborhood of $(0,0)$.

This example demonstrates
that while almost all choices of $l$ will enforce the SSP, only some enforce the trajectory behavior
we want. Thus, $l$ should be chosen deliberately, based on some information from the function being minimized,
rather than arbitrarily. However, a choice of $l$ that enforces desired behavior around one non-strict saddle
point may not do so around another, as the following theorem states.

\begin{theorem} \label{t.timevaryingbad}
There exist~$C^2$ objective functions such that every choice of regularizer~$l \neq 0$ induces a saddle-node bifurcation of a non-strict saddle point which creates a false minimum.
\end{theorem}
\textit{Proof:} See Appendix~\ref{apx.t.timevaryingbad}. $\hfill\square$

\begin{figure}[htp]
    \centering
    \includegraphics[width=8cm]{Inverted_Wine_Bottle.pdf}
    \caption{
    Plots of the function
    $(x-1)^3(x+1)^3 + lx$ for~$l = 0$, $l > 0$, and~$l < 0$.
    Regardless of the sign of $l$, one of the original degenerate critical points is bifurcated into a false minimum and a local maximum, and the other eliminated
    for every regularizer~$l \neq 0$.
}
    \label{fig:timevarying}
\end{figure}

Theorem~\ref{t.timevaryingbad} is illustrated in Figure~\ref{fig:timevarying}, which shows a
function
for which any
non-zero
choice of regularizer~$l$ results in a saddle-node bifurcation of a non-strict saddle point.
A natural consequence of Theorem~\ref{t.timevaryingbad} is that it is infeasible to use a constant, global linear regularizer.
Instead, we should regularize ``locally'', changing our choice of $l$ when approaching
different critical points. We take inspiration from~\cite{jin2017escape} and define a ``small gradient region'', outside of which $l = 0$ and inside of which $l$ will be chosen according to some selection rule:

\begin{definition}
Fix~$\theta > 0$ and
let $L_{\theta}= \{x \in \mathbb{R}^n : \|\nabla f(x) \|_{2} \leq \theta\}$. That is, the \textit{small-gradient region}
$L_{\theta}$ is the subset of $\mathbb{R}^n$ for which the norm of the gradient of $f$ is less than or equal to $\theta$.
\end{definition}

As long as $\theta$ is chosen small enough, a point in $L_{\theta}$ must be near a critical point of $f$.
Given our knowledge of $\nabla f$, we now wish to choose $l$ such that, if the nearby critical point is a degenerate saddle,
we then induce the desired bifurcation,
which
either eliminates the critical point or ensures the resulting critical points are strict saddles.
The following theorem gives us motivation for one particular choice of $l$:

\begin{theorem} \label{t.3.4.1}
Consider the function $f(x) + \mu l^Tx$ with $\mu \in \mathbb{R}$ and $l,x \in \mathbb{R}^n$. Assume that for $\mu = 0$ there exists an equilibrium $x^*$ such that:
\begin{enumerate}
    \item $\nabla^2 f(x^*)$ has $n-1$ positive eigenvalues, and a simple eigenvalue 0 with eigenvector $v$.
    \item $v^Tl \neq 0$.
    \item $v^T \nabla^3 f(x^*)(v,v) \neq 0$.
\end{enumerate}
Then there is a smooth curve of equilibria in $\mathbb{R}^n \times \mathbb{R}$ passing
through $(x^*,0)$ tangent to the hyperplane $\mathbb{R}^n \times \{0\}$ with no equilibrium
on one side of the hyperplane for each value $\mu$ and two equilibria on the other side of
the non-degenerate for each $\mu$. The two equilibria are hyperbolic and have stable manifolds
of dimensions $n-1$ and $n$ respectively.
\end{theorem}
\textit{Proof:} This theorem simply restates Theorem 3.4.1 from~\cite{guckenheimer2013nonlinear} using the specific notation of our problem. $\hfill\square$

Theorem~\ref{t.3.4.1} considers a non-strict saddle point $x^*$ of $f$ whose Hessian has a single zero eigenvalue (1) and satisfies a mild third-order condition (3). It states that for almost any vector $r \in \mathbb{R}^n$, if the choice $l = r$ induces a saddle-node bifurcation at $x^*$ that creates a false minimum, then the choice $l = -r$ will instead eliminate the critical point~$x^*$. We now combine Theorem~\ref{t.3.4.1} with a concept that appears trivial at first: for some point $x_0$, the choice $l = -\nabla f(x_0)$ will create a critical point of $f_l$ at $x_0$. Qualitatively, if
the degenerate critical point
$x^*$ is the only critical point near $x_0$, then the
resulting
critical point at $x_0$ must be the result of a bifurcation of~$x^*$. If this is the case, then Theorem~\ref{t.3.4.1} implies that
instead choosing~$l = \nabla f(x_0)$ will eliminate the degenerate critical point.

\begin{algorithm}[tb]
   \caption{Locally Linearly Regularized Gradient Descent}
   \label{alg1}
\begin{algorithmic}
   \STATE {\textbf{Input}: Stepsize~$\gamma > 0$, Small gradient parameter~$\theta > 0$}
   \FOR{k = 0,1...}
   \IF{$\| \nabla f(x_k)\| > \theta$}
   \STATE {$x_{k+1} \gets x_k - \gamma \nabla f(x_k)$}
   \ELSIF {$\| \nabla f(x_k) \| \leq \theta \And \| \nabla f(x_{k-1})\| > \theta$}
   \STATE {$l \gets \nabla f(x_k)$}
   \STATE {$x_{k+1} \gets x_k - \gamma (\nabla f(x_k) + l)$}
   \ELSE
   \STATE {$x_{k+1} \gets x_k - \gamma (\nabla f(x_k) + l)$}
   \ENDIF
   \ENDFOR
\end{algorithmic}
\end{algorithm}

Theorem~\ref{t.3.4.1} and the above discussion
imply that the choice $l = \nabla f(x_0)$ for some point $x_0$ near a particular
degenerate critical point may be a good candidate for our regularization selection rule. This proposed rule is presented in Algorithm~\ref{alg1}, where the update law is
$g(x)$ when the gradient is large. When this algorithm enters the small-gradient region $L_{\theta}$ at some point $x_0$, $l$ is set to be $\nabla f(x_0)$ and the update law is switched to $g_l(x) = x-\gamma (\nabla f(x) + l)$ until the algorithm leaves $L_{\theta}$. Note that because linear regularization does not affect the Hessian, and by extension the Lipschitz constant $L$, $\gamma$ can remain unchanged between $g$ and $g_l$.

We note that, as stated, Theorem~\ref{t.3.4.1}
only guarantees that Algorithm~\ref{alg1} will escape non-strict saddles
that satisfy restrictive conditions.
To extend to more general saddles,
we must
determine more general cases for which Algorithm~\ref{alg1} works. One key assumption in our derivation of this
proposed algorithm is that $x^*$ and $x_0$ are sufficiently separated from other critical points. The definition of~$L_{\theta}$
and our choice of $\theta$ provide the means to ensure this assumption is formally satisfied. First, for a
point $x \in L_{\theta}$, we define its \textit{local small-gradient neighborhood}:

\begin{definition}
For a particular $x \in L_{\theta}$, let $\Theta (x)$ be the largest connected subset of $L_{\theta}$ that contains x.
\end{definition}

Thus, $\Theta(x)$ is simply the portion of $L_{\theta}$ that surrounds $x$. Consider a point $x_o \in L_{\theta}$. If $l = \nabla f(x_0)$, then the update law $x_{k+1} = g_l(x_k)$ will be used
at each $x_k \in \Theta(x_0)$. Recall from
Theorem~\ref{t.timevaryingbad} that a choice of $l$ that destroys one degenerate critical
point may induce a saddle-node bifurcation at another. Therefore,
to help avoid a saddle node bifurcation within~$\Theta(x_0)$,
we want $\Theta(x_0)$ to contain only one critical
point or connected critical subspace. We formalize this idea with the following definition and assumption:

\begin{definition}
Let $X^* = \{x^* \in \mathbb{R}^n : \nabla f(x^*) = 0\}$. That is, $X^*$ is the set of all isolated or non-isolated critical points of $f$. For a particular $x^* \in X^*$, let $\Phi (x^*)$ be the largest connected subset of $X^*$ such that $x^* \in \Phi (x^*)$. If $x^*$ is an isolated critical point, then $\Phi (x^*) = \{x^*\}$. If $x^*$ is non-isolated, then $\Phi (x^*)$ is the connected critical subset that contains $x^*$.
\end{definition}

\begin{assumption} \label{a.thetaseparate}
$\theta$ is chosen such that for every $x^* \in X^*$, $\Theta (x^*) \cap X^* = \Phi (x^*)$.
\end{assumption}

Note that, trivially, $X^* \subset L_{\theta}$ for any $\theta > 0$.
Assumption~\ref{a.thetaseparate} simply states that $\theta$ is chosen small enough that any critical point $x^*$ is isolated in $\Theta(x^*)$ from all other critical points it is not connected to. Let $x_0 \in \Theta(x^*)$ and $l = \nabla f(x_0)$. Recall from Theorem~\ref{t.timevaryingbad} that this choice of $l$ may induce saddle-node bifurcations at degenerate critical points of $f$ other than $x^*$ and create false minima at those other points.
We want to ensure that these false minima, or indeed any critical points that result from a bifurcation or perturbation of a critical point other than $x^*$, do not end up in the set $\Theta(x^*)$.
This is guaranteed by the following theorem:

\begin{theorem} \label{t.isolatedbifurcations}
Let $x^*$ be a critical point of $f$, and let $\|l\|_{2} \leq \theta$. Let $y^*_l$ be a critical point of the regularized function $f_l$ that resulted as a bifurcation or a perturbation of $x^*$. Then $x^*_l \in \Theta(x^*)$.
\end{theorem}
\textit{Proof:} See Appendix~\ref{apx.t.isolatedbifurcations}. $\hfill\square$

Because $l = \nabla f(x_0)$ for some $x_0 \in \Theta(x^*)$, this ensures that, even if a particular
choice of $l$ induces a bifurcation at another degenerate critical point $y^* \in X^*$, the critical
points that result from that bifurcation are contained within $\Theta(y^*)$, which, under
Assumption~\ref{a.thetaseparate}, is disjoint from $\Theta(x^*)$.
Thus, while the iterates of gradient descent are in~$\Theta(x^*)$, regularizing can create
false minima somewhere, but not in~$\Theta(x^*)$, and there is no risk of converging to false minima.
In fact, Theorem~\ref{t.isolatedbifurcations}
implies that the topology of $\Theta(x^*)$ after regularization depends \textit{only} on the topology
of $\Theta(x^*)$ prior to regularization. In the following section, we identify exactly where any critical
points of $f_l$ will lie and what their convergence properties will be.

\section{Exit Condition of $\Theta(x^*)$}
\label{exitcondition}

By construction, a point $x^*_l \in \Theta(x^*)$ is a critical point of $f_l$ if and only if $\nabla f(x^*_l) = -l$. Because a linear regularizer does not affect the Hessian, $\nabla^2 f_l (x^*_l) = \nabla^2 f (x^*_l)$. That is, if $x^*_l$ is a critical point of $f_l$, its convergence behavior is determined by the Hessian of $f$ at $x^*_l$. In order to analyze this, let us stratify $\Theta(x^*)$ based on the properties of its Hessian:

\begin{definition}
For a $C^2$ function $f: \mathbb{R}^n \rightarrow \mathbb{R}$:
\begin{itemize}
    \item $\Lambda^+ = \{x \in \mathbb{R}^n: \lambda_{min}\nabla^2 f(x) > 0\}$
    \item $\Lambda^0 = \{x \in \mathbb{R}^n: \lambda_{min}\nabla^2 f(x) = 0\}$
    \item $\Lambda^- = \{x \in \mathbb{R}^n: \lambda_{min}\nabla^2 f(x) < 0\}$
\end{itemize}
\end{definition}

Note that $\mathbb{R}^n = \Lambda^+ \cup \Lambda^0 \cup \Lambda^-$.
According to Lemma~\ref{l.milnor}, $\nabla^2 f(x^*_l)$ is non-singular, implying $x^*_l \notin \Lambda^0$.
If $x^*_l \in \Lambda^-$, then it is a strict saddle, and $g^k(x)$ will not converge to $x^*_l$, as shown by the following lemma:

\begin{lemma} \label{l.escapesaddles}
Let $x_0 \in \Theta(x^*)$ for some $x^* \in X^*$ and let $l = \nabla f(x_0)$. Let $Y^*_l = \Theta(x^*) \cap \Lambda^- \cap X^*_l$, where $X^*_l$ is the critical set of $f_l$. Then
\begin{equation}
    \Pr\left(\lim_{k \rightarrow \infty}g_l^k(x_0) \in Y^*_l\right) = 0.
\end{equation}
\end{lemma}
\textit{Proof:} All elements of $Y^*_l$ are strict saddle points of the function $f_l$. The update law $g_l(x)$ is equivalent to gradient descent on the function $f_l$. Using this information, Corollary 9 in~\cite{lee2016gradient} provides the result. $\hfill\square$

Because $x^*_l \notin \Lambda^0$ and $g_l(x)$ does not converge to any $x^*_l \in \Lambda^- \cap \Theta(x^*)$, then the only way $g_l(x)$ can fail to escape $\Theta(x^*)$ is if $x^*_l \in \Lambda^+ \cap \Theta(x^*)$, which is precisely when $x^*_l$ is a local minimum. Recall that if $x^*$ is not a local minimum of $f$,
then such a point~$x^*_l$ is a \textit{false minimum}. Therefore, in order to guarantee $g_l(x)$ escapes $\Theta(x^*)$ when $x^*$ is a saddle point, we wish to
show that the choice $l = \nabla f(x_0)$ for $x_0 \in \Theta(x^*)$ almost never creates a false minimum in $\Theta(x^*)$. We formalize this notion with the following definition and assumption:

\begin{definition}
Let~$\Psi(\Theta(x^*)) = \{x \in \Theta(x^*) : \exists y^* \in \Theta(x^*) \textit{ such that } \nabla f(y^*) = -\nabla f(x^*) \textit{ and } y^* \notin \Lambda^-\}$.
\end{definition}

\begin{assumption} \label{a.nomin}
For the function $f$, for all of its saddle points~$x^*$,
the set~$\Psi(\Theta(x^*))$ has Lebesgue measure zero.
\end{assumption}

If $x_0 \in \Psi(\Theta(x^*))$, then the choice $l = \nabla f(x_0)$ creates a false minimum or degenerate point in $\Theta(x^*)$. Assumption~\ref{a.nomin} therefore implies that for any saddle point $x^*$ of $f$ and for almost any point $x_0 \in \Theta(x^*)$, the choice $l = \nabla f(x_0)$ will not create a false minimum or degenerate point in $\Theta(x^*)$. This leads to the following theorem on the ability of linearly regularized
gradient descent to exit the small-gradient neighborhood of non-strict saddle points in finite time:

\begin{theorem} \label{t.escapetheta}
Let $x^* \in X^*$ be a saddle point of $f$, and let Assumptions~\ref{a.thetaseparate} and~\ref{a.nomin} hold.
Let $l = \nabla f(x_0)$ for some $x_0 \in \Theta(x^*) \setminus \Psi(\Theta(x^*))$ and define the sequence $\{g^k_l(x_0)\}_{k}$.
Then there almost always exists a finite integer $k_p$ such that $g^{k_{p}}_l(x_0) \notin \Theta(x^*)$.
\end{theorem}
\textit{Proof:} See Appendix~\ref{apx.t.escapetheta}. $\hfill\square$

Theorem~\ref{t.escapetheta} states that under Assumption~\ref{a.nomin}, $g^k_l(x_0)$ exits $\Theta(x^*)$ for any saddle point $x^*$ in finite time. Note that Assumption~\ref{a.nomin} only applies to saddle points, as we do not wish to escape $\Theta(x^*)$ if $x^*$ is a local minimum of $f$. We now examine two properties non-strict saddles may have that are sufficient to satisfy Assumption~\ref{a.nomin}.

\begin{corollary} \label{r.halfspace}
Let~$\nabla f(\Theta(x^*))$ denote the gradient image of $\Theta(x^*)$, i.e., the set of all gradients that exist on $\Theta(x^*)$. If $\nabla f(\Theta(x^*))$ lies on an open half-space of $\mathbb{R}^n$, then $\Psi(\Theta(x^*)) = \emptyset$.
\end{corollary}

Trivially, if $x_0 \in \Theta(x^*)$, then $\nabla f(x_0) \in \nabla f(\Theta(x^*))$. If $f$ satisfies the
condition in Corollary~\ref{r.halfspace} on $\Theta(x^*)$, then $-\nabla f(x_0) \notin \nabla f(\Theta(x^*))$.
That is, for~$l = \nabla f(x_0)$ no point $x^*_l \in \Theta(x^*)$ exists such that $\nabla f(x^*_l) = -l$,
which implies that there are no critical points of $f_l$ in $\Theta(x^*)$. Clearly, if no critical points
remain after regularization, then the original degenerate critical point has been eliminated, there are
no false minima in $\Theta(x^*)$, and $g_l^k(x_0)$ escapes~$\Theta(x^*)$ according to Theorem~\ref{t.escapetheta}.
Generally, this condition is satisfied by objectives with isolated non-strict saddle points, such as in Example~\ref{e.halfspace}.

\begin{example} \label{e.halfspace}
\textit{(No Critical Points After Regularization)}

Consider the function $f(x,y) = \frac{1}{3}x^3+xy^2$, which has a non-strict saddle point at $(0,0)$ that satisfies the condition in
Corollary~\ref{r.halfspace}. This is because $\nabla_x f(x,y) = x^2+y^2$, which is non-negative everywhere. $W^s(0,0)$ is represented by the
red region in Figure~\ref{fig:halfspace}, and for every~$x_0 \in W^s(0,0)$,
we see that the regularzer $l = \nabla f(x_0)$
results in no critical points of $f_l$ in $\Theta(0,0)$,
and $g^k_l(x_0)$ escapes $\Theta(0,0)$.
\end{example}

\begin{figure}[htp]
    \centering
    \includegraphics[width=8cm]{Halfspace.pdf}
    \caption{The point $x_0 = (1.5,0.5)$ lies in $W^s(0,0)$ for the function $f(x,y) = \frac{1}{3}x^3+xy^2$, so $g^k(x_0)$ converges to $(0,0)$ (left). With $l = \nabla f(x_0)$ the critical point at $(0,0)$ is eliminated, and $g_l^k(x_0)$ escapes $\Theta(x_0)$ for $\theta = 3$ in 7 iterations, and enters $W^u(0,0)$ (right).}
    \label{fig:halfspace}
\end{figure}

\begin{corollary}
If $\Lambda^+ \cap \Theta(p) = \emptyset$ then $\Psi(\partial\Theta(p)) = \emptyset$.
\end{corollary}

From Theorem~\ref{t.escapetheta}, if there are no false minima in $\Theta(x^*)$ after regularization, then $g^k(x_0)$ escapes~$\Theta(x^*)$. False minima must lie in $\Lambda^+$ and if $\Lambda^+ \cap \Theta(x^*)$ is empty (i.e., the Hessian is not positive definite anywhere on $\Theta(x^*)$), then no false minima can exist for any choice of $l$. Any critical points of $f_l$ must be strict saddles. Generally, this condition is satisfied by objectives with non-isolated non-strict saddle points, such as in Example~\ref{e.nonpositive}.

\begin{example} \label{e.nonpositive}
\textit{(No Minima After Regularization)}

Consider the function $f(x,y) = \frac{1}{3}xy^3$, which has a non-strict critical subspace on the $x$-axis. For this function $-\nabla f(x,y) = \nabla f(-x,-y)$, meaning choosing $l = \nabla f(x_0,y_0)$ for \textit{any} $(x_0,y_0)$ will create a critical point of $f_l$ at $(-x_0,-y_0)$. However, $\nabla^2 f (x,y)$ has a negative eigenvalue at every point with $y \neq 0$, meaning this critical point will be a strict saddle, and $g^k_l(x_0)$ escapes $\Theta(0,0)$, as shown in Figure~\ref{fig:nonpositive}.
\end{example}

\begin{figure}[htp]
    \centering
    \includegraphics[width=8cm]{Nonpositive.pdf}
    \caption{ The function $f(x,y) = \frac{1}{3}x^3+xy^2$ has a critical subset at the line $y = 0$. The point $x_0 = (1.5,1)$ lies in $W^s(y=0)$, so $g^k(x_0)$ converges to $y=0$ (left). With $l = \nabla f(x_0)$, the critical subset at $y = 0$ is eliminated and a strict saddle point of $f_l$ is created at $(-1.5,-1)$, and $g_l^k(x_0)$ escapes $\Theta(x_0)$ for $\theta = 4.7$ in 15 iterations, and enters $W^u(y=0)$ (right).}
    \label{fig:nonpositive}
\end{figure}

\section{Convergence to Local Mimima} \label{convergencecondition}

Theorem~\ref{t.escapetheta} has shown that, under Assumptions~\ref{a.thetaseparate} and~\ref{a.nomin}, Algorithm~\ref{alg1} will exit $\Theta(x^*)$ for any saddle point $x^*$ of $f$. In order to show that Algorithm~\ref{alg1} eventually converges to a minimum, it must be shown that after exiting $\Theta(x^*)$ and reverting to non-regularized dynamics, Algorithm~\ref{alg1} does not immediately re-enter $\Theta(x^*)$. In order to examine the possibility of this behavior, we examine how the dynamics of $g_l$ inside $\Theta(x^*)$ and the dynamics of $g$ outside $\Theta(x^*)$ interact on the \textit{boundary} of $\Theta(x^*)$.
The boundary of $\Theta (x^*)$ is
denoted~$\partial \Theta (x^*) = \{x \in \Theta (x^*) : \|\nabla f(x)\|_{2} = \theta\}$.

Suppose that~$x_{k_p} = g^{k_p}_l(x_0)$ is the iteration at which
Algorithm~\ref{alg1} exits $\Theta(x^*)$ (i.e., $x_{k_{p}-1} \in \Theta(x^*)$
and $x_{k_{p}} \notin \Theta(x^*)$). Then
the regularized gradient~$-\nabla f_{l}$ pointed ``out'' of~$\Theta(x^*)$ to make
this exit possible, and, because~$x_{k_p}$ is the first iterate computed
after the exit,
$x_{k_{p}}$ will lie near the region
of $\partial\Theta(x^*)$ for which $-\nabla f_l$ points ``out'' of $\Theta(x^*)$.
Since Algorithm~\ref{alg1} reverts to non-regularized dynamics at $x_{k_p}$, we
want $g(x_{k_p})$ to move away from $\Theta(x^*)$. Intuitively, this means $x_{k_p}$
should lie near the region of $\partial\Theta(x^*)$ for which $- \nabla f$ also
points ``out'' of $\Theta(x^*)$. That is, we want the ``exit'' region
of $\partial\Theta(x^*)$ defined by $-\nabla f_l$ to lie inside the ``escape''
region defined by $-\nabla f$.

Borrowing notation from~\cite{katz2019morse}, we formalize this
condition by explicitly defining the regions of $\partial\Theta(x^*)$ for
which $-\nabla f_l$ and $- \nabla f$ point outside $\Theta(x^*)$. By definition,
$\partial\Theta(x^*)$ is a subset of a level set of the function
$h(x) = \frac{1}{2}\|\nabla f(x)\|^2_2$. Therefore $\nabla h(x)$ is normal
to $\partial\Theta(x^*)$ for all $x \in \partial\Theta(x^*)$. If the
product $\nabla f^T(x) \nabla h(x)$ is negative for some $x \in \partial\Theta(x^*)$,
then $-\nabla f(x)$ points ``out'' of $\Theta(p)$.
Because $\nabla h(x) = \nabla ^2 f(x) \nabla f(x)$, we have the following definition:

\begin{definition} \label{d.boundaryoutside}
For $\partial\Theta(x^*)$ and $l$, $\partial^-_l\Theta(x^*) = \{x \in \partial\Theta(x^*) : (\nabla f(x) + l)^T\nabla ^2 f(x) \nabla f(x) < 0\}$.
\end{definition}

Note that if $l = 0$, then $\partial^-_0\Theta(x^*)$ defines the ``escape'' region dependent on $-\nabla f$. Also note that equivalent regions where the negative gradient points into $\Theta(x^*)$ ($\partial^+_l\Theta(x^*)$) and is tangent to $\partial\Theta(x^*)$ ($\partial^0_l\Theta(x^*)$) can be defined when the product in Definition~\ref{d.boundaryoutside} is positive or zero. Intuitively, Algorithm~\ref{alg1} will pass through $\partial_l^-\Theta(x^*)$ when it exits $\Theta(x^*)$, and will escape if $x_{k_p}$ lies near $\partial^0_l\Theta(x^*)$. This leads us to the following assumption:

\begin{assumption} \label{a.boundaryescape}
For a saddle point $x^*$ and regularizer $l$, $\partial_l^-\Theta(x^*) \subseteq \partial_0^-\Theta(x^*)$.
\end{assumption}

Under Assumption~\ref{a.boundaryescape}, when Algorithm~\ref{alg1} exits $\Theta(x^*)$,
it will exit into a region that takes it away from $\Theta(x^*)$.
While determining other sufficient conditions for which Assumption~\ref{a.boundaryescape} holds is a subject of
future work, it has been observed to hold in all examples presented in this paper.

If Algorithm~\ref{alg1} escapes all saddle points, then it must eventually enter $\Theta(x^*)$ for a local minimum $x^*$.
If this minimum is non-degenerate (i.e., $x^* \in \Lambda^+$), then there will be an open neighborhood $U$ of $x^*$ for which $U \subset \Lambda^+$. We assume that $\theta$ is chosen small enough such that $\Theta(x^*)$ is restricted to this region.

\begin{assumption} \label{a.allpd}
$\theta$ is chosen small enough such that, for every critical point of $f$ $x^* \in \Lambda^+$, $\Theta(x^*) \subset \Lambda^+$.
\end{assumption}

Under Assumption~\ref{a.allpd}, linear regularization with $\|l\|_2 < \theta$ will only perturb the minimum (i.e., it will not be eliminated or bifurcated into two critical points) and the perturbed minimum $x^*_l$ will remain in $\Theta(x^*)$. Furthermore, if $f$ satisfies the Polyak-Lojasiewicz Inequality on $\Theta(x^*)$, which is a weaker condition than strong convexity~\cite{karimi2016linear}, the error due to regularization can be bounded in terms of $\theta$.

\begin{theorem} \label{t.error}
Let Assumption~\ref{a.allpd} hold for a function $f$, let $\|l\|_2 < \theta$, and let $x^*$ be a non-degenerate minimum of $f$. Then $f_l$ will have exactly one critical point $x^*_l$ in $\Theta(x^*)$, and $x^*_l$ will be a non-degenerate minimum. Additionally, if $f$ satisfies the Polyak-Lojasiewicz Inequality with constant $c$ on $\Theta(x^*)$, then
the error in cost between~$x^*_l$ and~$x^*$ that is induced by
regularizing is bounded by
$f(x^*_l) - f(x^*) \leq \frac{\theta^2}{2c}$.
\end{theorem}
\textit{Proof:} See Appendix~\ref{apx.t.error}. $\hfill\square$

\section{Numerical Results} \label{numericalresults}
To isolate the impact of linear regularizers, we consider a simple
neural network model that sees only a single regularization event
occur during its training, and we quantify the impact that a single
regularization has. Specifically,
we compare the performance of Algorithm~\ref{alg1} (Regularized GD) with full batch non-regularized gradient descent (GD) on the training of a fully connected
feedforward deep neural network on the MNIST database, implemented in PyTorch. The model has four hidden layers of width 1,000 with ReLU activation functions. The cross entropy loss function of this model has non-strict saddle points according to Corollary~3.2 of~\cite{kawaguchi2016deep}. Regularized GD uses the gradient threshold $\theta = 5$, and both algorithms use learning rate $\gamma = 0.00005$. Both algorithms are initialized with the same random weights and run for 1,000 epochs. This is repeated for 100 trials.

\begin{figure}[htp]
    \centering
    \includegraphics[width=8cm]{ubl_fig.pdf}
    \caption{Loss at each epoch for GD and Regularized GD}
    \label{fig:cost}
\end{figure}

The average loss at epoch $1,000$ achieved by GD is~$0.4575$, which is 32.8\% greater than the average loss achieved by Regularized GD in the same number of epochs, which is~$0.3446$.
Figure~\ref{fig:cost} presents the change in loss for a typical trial, in which both algorithms are identical until the gradient threshold $\theta = 5$ is crossed at epoch 312.
That is, a single regularization event
during training is responsible for a nearly one-third difference
in the final value of the loss function.

\section{Concluding Remarks}
\label{concludingremarks}

We have presented a local linear regularization scheme for non-convex optimization that enforces satisfaction of the SSP. This scheme is proven to escape a broad class of isolated and non-isolated non-strict saddle points, and demonstrated superior performance to full batch gradient descent on the training of a neural network model with non-strict saddle points. Future work will address identifying sufficient conditions that ensure the satisfaction of Assumptions~\ref{a.nomin} and~\ref{a.boundaryescape}. We note that the results presented in this paper are not restricted to gradient descent; they can be applied to any optimization method that avoids strict saddles.

redpajama_set_name: RedPajamaArXiv


text:
You are here: Home → Entertainment → 2023 01 23 → Article 623786
Archived Entertainment News
Nigerian Famous People
Entertainment of Monday, 23 January 2023
I was scared while coming to Ghana – American citizen
Yaa Baby
Chloe, popularly known as Yaa Baby was born and raised in the United States of America and had to journey to Ghana together with her family at age ten.
Her father was a missionary and was tasked to oversee the affairs of a Baptist Church in the West African country.
The preparation was, however, shrouded by fear due to the narratives they had heard about the country and being oblivious to how the journey would be.
"We were anxious because, for instance, I hadn't sat in a plane before," she said in her interview on The Delay Show. "I had heard about Ghana but I really didn't know the country and the things people said about the country were not encouraging. It didn't sound like a place you'd be happy to stay in there so I was a bit scared."
"Both the whites and blacks abroad said it's not interesting to stay in Ghana; there is erratic power supply, scarcity of food. These were Ghanaians and whites who had experienced Ghana. They say seeing is believing, I hadn't seen Ghana so I came with an open mind. Regardless, I was jittery," Yaa Baby added.
The situation was not the same for her parents because as missionaries, they left their fate in the hands of God.
Sharing her experience in Ghana, Yaa Baby, now married to a Ghanaian video director, Kofi Awuah, indicated that the weather was enough to make her recognize she was in another country.
"My first memory was the heat that welcomed me as soon as they opened the aircraft doors. I noticed there's a different vibe and different scent," she said. "It's normal if you go to a place for the first time. It was a bit different from summer. I think we came in October so we were just about to experience harmattan so it was slightly different from summer."
According to Yaa Baby who has maintained her American accent but speaks impeccable Twi language, they spent a week in Accra before heading to Kumasi.
"We lived at Kwadaso and moved to other places," the actress and social media influencer recalled.
Having been homeschooled by her mother who holds a master's degree, Yaa Baby said she wanted to be a nurse "but for one reason or the other, I just decided I wasn't ready for that."
Her parents stayed in Ghana until 2016 when they decided to return home. This was after living in Ghana for over a decade. She mentioned that she decided to not go with her parents because of her husband, Kofi Awuah.
"We married before they left," she said.
Yaa Baby now says Ghana is her home.
'If you disrespect my husband, I'll come for you' – Anita Joseph warns colleagues, fans
'Davido is coming to pick me' - Tweeps says as Uber driver turns out to be David Adeleke, Nigerians react
'Take care of your mental health' - Reaction as UNIABUJA student sets her room on fire
Burna Boy reacts as South African rapper, Gigi Lamayne professes love to him
There is a successful organ harvesting ring in Nigeria – Former BBNaija housemate raises alarm
Enzo Fernandez, Hakim Ziyech, Moises Caicedo, Jorginho: Top transfers that could be completed on deadline day

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: LineageOS stuck in boot animation I got a new FairPhone 2 today, and I followed this guide
I got the latest TWRP (twrp-3.2.3-0-FP2.img) - and I can get into recovery mode fine.
I sideloaded the latest build for Fairphone 2 (lineage-15.1-20190327-nightly-FP2-signed.zip), and immediately after I sideloaded MindTheGapps-8.1.0-arm-20180808_153837.zip.
I then rooted by installing addonsu-15.1-arm-signed.zip.
I rebooted and I get the bubble going from right to left forever. Can someone help me with this?