!!!Upgraded From Top To Bottom!!! New Kitchens With Quartz Countertop, New Washroom, New Flooring, New Baseboard, New Light Fixtures, New Appliances!!! Upgraded Windows And Doors!!! Pot Lights!! Very Clean And Well Kept Home. Large Professionally Landscaped Yard With Pattern Concrete Patio Perfect For Entertaining. 3 Bright And Spacious Bedrooms. Quiet Crescent Street. Just Steps Away From School, Park, Transit, And Shopping. A Must See Home In The Area!!!


Wall Cabinets Bedroom Design Grand Bedroom Wall Cabinets Bedroom Ideas was posted in November 30, 2017 at 6:05 am. Wall Cabinets Bedroom Design Grand Bedroom Wall Cabinets Bedroom Ideas has been viewed by 81 users. If you want to use this wallpaper, please click the download link below to go to the download page. Right click the image and select "Save Image As" to download the Wall Cabinets Bedroom Design Grand Bedroom Wall Cabinets Bedroom Ideas to your computer or select "Set Desktop Background As" if your browser has that capability.


Oakland Symphony Chorus Chamber Singers is a newly formed ensemble led by Lynne Morrow. Members of the group are auditioned and selected from the Symphony Chorus. The group's mission is to explore and present a wide variety of musical genres from Bach to Broadway.


Saakshi Tanwar, who is best known for her role of Parvati in the popular Kaahani Ghar Ghar Ki, has a lot in her kitty, starting with her role as Ganga in the mythological serial Kahaani Hamaaray Mahaabhaarat Ki.
Earlier too, Saakshi played a spiritually inclined woman with a strong connection to goddess Durga in the serial Devi. The actress will also be seen in a few movies later in the year like Coffee House and Ekta Kapoor's film C Kompany.
Rajul Hegde caught up with the busy actress to know more about Ganga.
How was the experience of playing Ganga?
It was a wonderful experience. Something I will always cherish. The aura and the mystique surrounding Ganga is what made the character such a challenge to portray. A short yet powerful role, I loved playing Ganga.
How much do you know about Ganga?
My basic knowledge about Ganga was acquired from whatever I had seen in B R Chopra's Mahabharat. I was a big fan of the show and watched it religiously. But I was amazed to know that in our mythology there are so many other stories about Ganga apart from Mahabarat. All this was very enriching for me.
What kind of research have you done for this character?
I was given detailed narrations about the character and then I just followed my instinct to portray it.
Let's talk about your look and costume.
Ekta [producer] knew exactly how she wanted Ganga to look like. I appeared for a 'look' test with Bharat and Dorris. It was instantly approved. My costume has been designed by Manish Malhotra, who has always performed magic for me. He made me look like a dream. That's the response I got from everyone, who saw the hoardings and promos.
How long you have been shooting for this show?
I shot for just 10-12 days, as it is a brief role.
I really enjoyed shooting for Mahabharat. With every different show, an actor learns and grows as a performer, and the same has been with me on Mahabharat. The canvas on which it was shot was huge -- the sets, the costumes, treatment of scenes -- the entire feel was so mesmerising.
It was reported that you fell sick during the shooting. What happened?
Nothing much. It was just a little acclimatisation problem in Ladakh due to lack of oxygen. I was fine in a couple of hours and resumed shooting the next day as scheduled.
After that, you didn't resume shooting for sometime. Are you trying to work harder now to keep episode bank?
Not really, since my role as Ganga was very brief and everything was managed very well.
How was it to share screen space again with Kiran Karmarkar who play's Shantanu, your husband?
Having worked with Kiran for the last many years, there is a comfort level and a great on-screen rapport which helped both of us in portraying our respective roles with so much ease.
What is the kind of response you have got from people, who have already seen it?
Well, everyone who has seen it liked the look of my character. Now the performance remains to be judged. I have my fingers crossed.
Are you working hard because the show is being aired before the scheduled time?
My work was done even before the telecast date was decided so there are no pressures on that end.
Will you be able to evoke the same kind respect which you got for playing Parvati?
I have performed with full earnestness, honesty and 100 per cent dedication. Now it's for the viewers to decide. It's their verdict that finally counts.
Does it mean that you will be seen less as Parvati in Kahaani Ghar Ghar Ki?
(Laughs) No, I am still there in 14 out of 15 scenes in Kahaani.
I have recently done a film directed by Gurbir Grewal titled Coffee House opposite Ashutosh Rana. I also have a comic cameo in Balaji Motion Pictures film C Kompany directed by Sachin Yardi. As an actor, I am always greedy for good roles. So far, I have been lucky to play strong and powerful roles on television. Now, I am looking at doing good cinema.
Catch Kahaani Hamaaray Mahaabhaarat Ki every Monday to Thursday on 9X at 9 pm.


August 21, 2010: Tasco Sales has announced local availability of the latest model in Ricoh's highly-rated CX series of digital compact cameras, the CX4.
The new model introduces new creative shooting modes, a new subject tracking auto-focus system, improved image stabilisation technology and host of other sophisticated features – all encapsulated in a redesigned chassis with sleek curves. Ricoh's new CX4 has been designed for creative photographers. New Soft Focus, Cross-process and Toy Camera shooting modes join the popular Miniaturise, high contrast black-and-white and high dynamic range modes first introduced by the CX2.
In addition, Ricoh has included a 'night landscape multi-shot mode', allowing CX4 users to experience the joys of handheld, tripod-free photography after dark. Using this feature, the camera reduces the image noise normally associated with pictures taken under low light by automatically and precisely combining four exposures shot at high sensitivities into a single optimally-exposed photograph.
Crisp, clear photos are further assured with the evolution of Ricoh's image stabilisation technology. Whether shooting under dim indoor lighting or while making the most of the 10.7x wide angle zoom lens (28-300 mm equivalent), the improved image stabiliser reduces blur by an average of about 3.7 shutter-speed stops.
The CX4 also offers a new subject-tracking auto-focus system that automatically tracks the subject, ensuring that photographs are in focus and correctly exposed whenever the shutter is released. Ideal for shots of active children and pets, this feature is also perfect for close-up photos of flowers and other subjects that demand careful composition.
Boasting a 10 megapixel back-illuminated CMOS image sensor to more efficiently capture light coming through the lens, the CX4 also sees the return of multi-pattern auto white balance to faithfully recreate the detail seen by the naked eye in scenes which include mixed lighting. When combined with the Smooth Imaging Engine IV, the result is dramatic – beautiful images that have maintained their original resolution, tone characteristics and colour saturation.
– a 920,000-dot, 3-inch high resolution LCD monitor featuring an anti-reflection coating to ensure clear visibility even in bright sunlight.
Available from leading camera and electronics retailers from September, the Ricoh CX4 is available in black, champagne silver and bubblegum pink.
The camera will be available from the first week in September, and carries an RRP of $499.
Contact Ricoh's Australian distributor, Tasco on 02 9938 3244 or visit the new Australian Ricoh website: www.ricohcameras.com.au.


Come support and celebrate a @bklynburro 's one year anniversary of providing the absolute best burritos to New York residents. $1 Burros and a party? We'll be there, you should be too.
All Wolves No Sheep (AWNS) is a custom tattoo shop focused on giving each client top-notch service and amazing tattoos to wear for life.


text:
SCOR publishes a net income of EUR 512 million along with a 15% increase in shareholders' equity, and proposes a dividend of EUR 1.40, up 8%
March 5, 2015 — 00:00
SCOR's annual results of 2014.
Over the course of 2014, the Group has continuously and consistently delivered high quality results, improving its competitive position. Thanks to a well-diversified global franchise, a strong worldwide presence and a solid financial performance, SCOR once again delivers a strong level of profitability and achieves a solvency level in line with its strategic objectives in 2014.
Gross written premiums reach EUR 11,316 million, up 10.4%1 compared to 2013 (10.8% at constant exchange rates). This is driven by the Generali US contribution following its acquisition by the Group in October 2013, and by the growth of both SCOR Global Life in Asia and Financial Solutions & Longevity, and SCOR Global P&C:
SCOR Global P&C gross written premiums increase by 2.7% at constant exchange rates (+1.8% at current exchange rates) to EUR 4,935 million;
SCOR Global Life gross written premiums reach EUR 6,381 million, up by 5.5% at both constant and current exchange rates on a pro-forma basis.
SCOR Global P&C delivers excellent 2014 technical profitability with a net combined ratio of 91.4%, compared to 93.9% in 2013.
SCOR Global Life's 2014 technical margin stands at 7.1%, compared to 7.4% on pro-forma basis in 2013, reflecting the ongoing evolution of the underlying mix, as disclosed in "Optimal Dynamics".
SCOR Global Investments achieves a 2.9% return on invested assets thanks to its active portfolio management and the evolution of its investment portfolio.
In 2014, the Group generates a strong operating cashflow of EUR 894 million, with strong contributions from both SCOR Global P&C and SCOR Global Life.
Net income stands at EUR 512 million in 2014, an increase of 40% compared to 2013, excluding an exceptional gain on purchase of EUR 183 million linked to Generali US. Return on equity (ROE) stands at 9.9%2.
Shareholders' equity reaches EUR 5,729 million at 31 December 2014, compared to EUR 4,980 million at 31 December 2013. After distribution of EUR 243 million in cash dividends, book value per share increases by 15% to EUR 30.60 as at 31 December 2014, compared to EUR 26.64 as at 31 December 2013.
SCOR's solvency ratio, as defined by the 2014 internal model3, remains marginally above the optimal range.
SCOR's financial leverage stands at 23.1% as at 31 December 2014. The placement of two perpetual subordinated debts of EUR 250 million and CHF 125 million4 in 2014 demonstrates the Group's active management of its liabilities.
SCOR will propose to the Annual General Meeting an increased dividend of EUR 1.40 per share for 2014, up from EUR 1.30 for 2013, representing a payout ratio of 51%. The proposed ex-dividend for 2014 will be set at EUR 1.40 on 5 May 2015 and the dividend will be paid on 7 May 2015.
Denis Kessler, Chairman & CEO of SCOR, comments: "The Group delivers high quality results in 2014. It has continued to grow and expand its franchise. The technical profitability of SCOR Global P&C – which delivers a combined ratio of 91.4 % –, and that of SCOR Global Life – which records a technical margin of 7.1 % -, are highly satisfactory. Despite the weakness of interest rates, SCOR Global Investments records a return on assets of 2.9 %. SCOR once again delivers solid profitability, and achieves a solvency level in line with its strategic targets. Having invested in new underwriting and risk modelling tools, the SCOR group has prepared for the new prudential regime, Solvency II, which will come into force on 1 January 2016. It is confident in its ability to meet the challenges of a difficult financial environment, a heightened competitive situation and a demanding new prudential regime."
1 On a published basis; on a pro-forma basis gross written premium growth is 3.8% (4.2% at constant exchange rates).
2 The ROE calculation method was adjusted to take into account material foreign exchange rate movements that do not occur evenly through the reporting period. A daily weighted average is applied for the currency or currencies that experienced such movements and a simple weighted average is applied for the other currencies.
3 This estimate is based on the 2014 internal model taking into account the estimated available capital at year-end 2014 divided by the estimated SCR as at that date, allowing for planned business in 2015. The internal model will be subject to a review and approval process conducted by the ACPR over the coming months. The estimate based on the 2015 model will be provided with the Q1 2015 publication.
4 See press releases of 25 September 2014 and 24 September 2014 respectively.
SCOR Group ...
SCOR's annu ...
Résultats a ...

redpajama_set_name: RedPajamaCommonCrawl


Timelessly beautiful, it's hard to believe the Eileen Gray E1027 Coffee Table was designed in the 1920s. Its sleek glass table top and stainless steel frame still look modern today. Our VOGA version stays true to the original both in terms of quality and functionality. The strengthened glass top is stronger and safer than regular glass and can be raised to different heights to suit your needs. The result is a stylish, versatile table that fits perfectly next to any chair or bedside.
Eileen Gray designed the E1027 Coffee Table in 1927 for the guesthouse bedroom of a summerhouse in France, which she built with her lover, Jean Badovici. The 'E' stands for Eileen. The numbers represent the positions of letter of the alphabet. So the 'J' for Jean (as it's the 10th letter of the alphabet), the 2 is the 'B' for Badovici and the '7' is the 'G' for Gray. This contemporary coffee table was designed for a specific need, but now you can enjoy its stunning design in your home.


text:
James McCloskey would love your feedback! Got a few minutes to write a review?
A Better Alternative
James McCloskey 1 story
I'm going to kill the main character because I'm evil like that. Luke is a mysterious magician and time traveller who lives for the simple things like having relationships with fictional characters.
Fantasy / Scifi
James McCloskey
To write a story is to create life. I have always thought of God as an author, who allows us, the characters, our own free will, to move with the story as we do, as every great authors do. Authors do not wish their characters harm (well, usually they don't), but it's their own personalities (to be specific, their hamartia) that leads them to their own untimely destruction. I would bet that Shakespeare himself wept at the demise of Juliet, or King Lear. There is also the fact that the story must send a message. A death of a character you get to know and love will get an emotional response from you, if required.
Luke had no hamartia as far as he was concerned. He had witnessed far too many people who were too innocent or too self consumed to fall into that pit. Luke was what you would call "street wise", but the term "magic time portal wise" would probably be a more accurate description, due to the fact that, yes, he spent his life time travelling through a magic portal. I say "magic" due to the fact that it is it's official name, according to the Society of Time Travellers, which Luke was a part time member of.
The rules of the Society of Time Travellers are, were, have been, and always will be extremely complicated and something I will describe throughout the story in order to accelerate the pace of the narrative.
Luke was not far from home at this point. He was on Earth-42 (basically, the one you live on. Boring, yes, but don't worry, Earth -314 is far better) in 1945 (universal date; 890.3.6212), several miles out from Nagasaki. He reflected upon the idiocy of humanity at this time, as the atomic bomb dropped and exploded killing thousands of people. This was, in Luke's opinion, one of the most idiotic decisions throughout all of humanity. All of those innocent people to end a war that was basically already over, which would in turn strike fear in to the heart of everyone for over fourty years afterwards. Luke was born during what historians called the "Cold War". It was awful. The teachers had to tell everyone to hide under the table if a bomb were ever to drop, as if that would do a difference. At this moment he was also in Hiroshima.
"What a waste," he said, aloud.
And he waved his wand (which was the classic magic wand, a polished, black, smooth stick, with two white ends, on Earth-42 it would be considered somewhat ridiculous looking).
"Earth-57, universal date; 27.535.897" Luke says in a clear manner (especially the Earth part, as he had previously ended up on Mars, which had on undesirably thin and deadly atmosphere).
A dark purple and black swirly hole of mist was created in front of Luke, and he stepped though. The portal disappeared behind him.
"Luke, how does a time traveller manage to be late so often?" said a voice from behind him, which Luke immediately recognised as the voice of his girlfriend, Princess Guinevere. And yes, the legendary wife of the more legendary King Arthur. Luckily this wasn't on the Earth that Luke originated in so he could mess up the timeline without causing a paradox. Also, he was Merlin's best friend. Luke really enjoyed this version of Earth. If only it weren't infested with dragons.
"A wizard is never late, he-"
"If you quote that wretched Earth-42 book one more time, I swear, we're over."
"It's a classic, and it's pure genius. And I have explained time-travel to you before, you-"
"Can't remain out of sync with another person's timeline if you wish for your history with that person to remain linear, and if your close relationships do not remain linear, the other magicians will be angry."
"Actually, we're not all magicians, some of them are scientists who have cracked the code to time travel via Physics, and so-"
"The way you talk still makes no sense to me," Guinevere laughed.
"I wish you'd let me show you the multiverse," said Luke
"Your black magic is dangerous. I cannot allow myself to be consumed by it."
Luke shrugged. Guin was one of the prettiest and most compassionate women he had ever known, but she still had the ignorance that was common in the seemingly fictitious Middle Age.
They were in a large, impressive castle. It was made of just stone. The stone which would turn dark grey in a hundred years or so was at this point a nice pale colour which one could mistake for a silvery white. He looked outside as the low burning sun seemed to sink below the horizon. This is why Luke had chosen this life. It was perfect and nothing was going to go wrong (that's just a lie, actually, but I need to lure you into a false sense of security).
And so, Luke and Guin had a perfectly boring date and slept within the same bed, Guin's. Luke lived for the simple things.
Are you enjoying my ongoing story? Please let me know what you think by leaving a review! Thanks, James McCloskey
Different by katiehines1998
heather: I liked that the story followed and made me want to keep reading. I could put it down. I read it all in just a few hours.
mohammed: I recommend this book to my sister
kaybogart1953: I loved loved this story well written and l hope to read more from you,I know some will pick about some errors but I just enjoyed the story
The Cat Whisperer [Shifters #1] by Nikki Larousse
rumemakilolo: Well I like the boundless love between Catherine and Wyatt, honestly love is meant to break all bounds. I love the author skeptical and unique way of writing. Awesome
The Shifter Bar (Shifter World - Book Three) (Series of 13 Short Stories) by Koko Aranck
Wendy Iwaro: Excellent read
Jacheri: Good story line. Enjoyed this book
Alpha Xander by Holly Prange
dervece: I looked the plot, the suspense had me !!!!
tonyavankuren: This was really good. I can't wait to read more of your books.
Destined Alpha by Ct521
Jennifer Leigh Anne Ciliska: Awesome read thank you for sharing your story with me
giotakiosse: It is definitely one of the most well-written werewolf stories I have ever read. It triggers all kinds of feelings; shock, compassion, relief, love, worship, pride. Lucy deserved all the best and she overcame great difficulties in her life. She is both innocent and fierce. The end intelligently g...
Killian (sample) by CanadianMomof2
gogihabtu24: Beautiful story ever!!
Tulip's Ghost - Book two of the Torian Warriors series. by Natalie Le Roux
shelbybelbs: I love this one even more than the first!!
Books Similar to 'It'
Books About Aliens
Books Similar to Dan Brown's
Books Similar to 'Pans Labyrinth'
Books About Demons
Books Similar to 'A Thousand Splendid Suns'
Books Similar to 'Criminal Minds'
Books About Hate
Books Similar to 'The Mortal Instruments'


Podcast 17: Techstars President David Brown — After Nines Inc.
Good evening, I.T. entrepreneurs. In tonight's podcast conversation (Episode 17), we explore how I.T. entrepreneurs can partner with accelerators to potentially grow their businesses more rapidly. But how exactly does an accelerator relationship work? Techstars President and Managing Partner David Brown offers insights and answers.


About us - Aspira-Vac Inc.
With the aim to offer the best possible service to his customers, Mr. Desrosiers, who counted already more than 10 years of experiment in the field at that time, founded in 1976 Distributions N.D. Inc who later became part of Aspira-Vac Inc.
Thanks to the experience aquired in the repair of FilterQueen® vacuum cleaner, the company of Mr. Desrosiers quickly become a reference in the area of Montreal and the surrounding area for the quality of its service.
Being a specialized service center in the brand FilterQueen®, Aspira-Vac Inc. uses only genuine parts at the time of its repairs. You can thus count on the expertise of our technicians as well as the quality of the FilterQueen® parts, when we carry out a repair on your vacuum cleaner.


How a highly productive collaboration among a trio of creative Angelenas—and a good dose of Barragán—turned a dark and beleaguered mid-century house into a family home for the ages.
A good dose of Barragán—turned a dark and beleaguered mid-century house into a family home for the ages. The paint colors chosen by the residents and architect Linda Taalman are American Cheese 2019-40 and Blushing Bride 2086-50, both by Benjamin Moore.
The living room has a Dadone sofa by Antonio Citterio for B&B Italia and a Safari chair by Arne Norell.
The master bedroom features a Case Study bed from Modernica.
Devis works in the kitchen.
Purdy, Eva Luna, and Simon play cards under a Louvre Light by Established & Sons.
The cork stairs with a rope railing lead down to the kids' level.
Simon and Eva Luna play in front of a wall covered in Daks wallpaper from Walnut.
The pair's art covers a cork wall where Eva Luna reads in a vintage Danish lounge chair.
The first thing landscape designer Laura Cooper asked Devis and Purdy was to recall childhood gardens and outdoor play. In that spirit, she designed their backyard, integrating the high ground with the low just outside the "kids' wing." The resulting series of outdoor rooms on this quarter-acre is full of memory and play.
Near the house, Cooper clustered lush, big-leafed plants philodendron, purple heart, and flowering canna. Farther out, vegetable gardens sprout in raised beds, and thyme and mint thrive beneath citrus trees. Native California silver wild rye and matilija poppies bind the slope, bordering the low "snake wall" that winds around the yard, narrowly ducking beneath a big ficus tree with just enough clearance for a sprinting kid. Wide stadium steps connect the garden levels, turning the steep hillside into amphitheater seating.
Some 35 percent of the materials from the renovation were resurrected in the yard: Redwood panels became fencing. Blocks cut from original concrete patios form the snake wall. And Purdy plans to crochet colorful cozies over various branches and stumps.
When experimental filmmaker Laura Purdy describes the transformation of her house in Los Angeles's Los Feliz neighborhood, she speaks glowingly of a "perfect storm" among the women who collaborated on the renovation. For about a year, architect Linda Taalman, landscape designer Laura Cooper, and Purdy met for creative sessions, exchanging ideas as they converged on a detail or spatial sequence. "In a field so often dominated by men, it was unusual to find myself in these intensely fruitful all-female meetings," recalls Taalman. As the women met—getting critical input from Purdy's husband Juan Devis and Taalman's husband and former codirector at Taalman Koch Architecture (TKA), Alan Koch—their children often played together in the background.
And like those great conversations, the resulting design acquired its own flow, full of colorful narrative, spirited counterpoint, and anecdote. Now, in place of dark, disconnected spaces, outdoor rooms echo luminous indoor ones, and Purdy and her family's eclectic collections of art and personal artifacts share space with flashes of pattern and interior planes of saffron and pink stucco.
Vibrant as that infusion of color appears, the renovation strikes a gentle balance. Taalman extracted the latent design intent of this quiet, mid-century-modern house while imbuing it with the warm character and spirited life stories of Purdy, Devis, a Colombian-born filmmaker and public-television producer, and their two young children. Their 2,000-square-foot home was the 1952 work of 23-year-old Les Guthrie (who later developed Redondo Beach's King Harbor Marina). Tucked behind a carport on a cul-de-sac, the white-stuccoed building steps down a slope, largely hidden from the street.
Once in their new home, they made few changes, pulling up carpet to reveal existing parquet floors and sandblasting green paint from interior paneling to expose redwood. The main living level provided three side-by-side bedrooms—fully occupied following Simon's birth, in 2006—and the lower floor a home office and playroom. But, by 2009, they were outgrowing the arrangement and, much as they loved the place, wearying of its dark interiors.
Among the architects approached for remodeling, only TKA proposed adding no additional square footage. "The house had good bones," Taalman says. "We could solve the shortcomings within the existing envelope by creating a master suite, a grown-up retreat upstairs, and a kids' zone with bedrooms and a playroom below." The clients leapt for it.
Good as the bones were, Taalman remembers, many rooms and connections seemed spatially "trapped or dead-ended." A laundry room came between the kitchen and backyard. The lower floor lacked flow to the outdoors. And the yard's two levels—paved in concrete and separated by a steep slope—barely communicated with each other. TKA soon enlisted Cooper, an unconventional, self-taught landscape designer who trained as an artist and teaches at Art Center College of Design in Pasadena.
The process ultimately involved stripping the building to its studs to modify openings and achieve an even finish. Now, from the street, the house appears virtually unchanged—but just over the threshold, an animated dialogue begins. A new floor-to-ceiling window, facing an existing one, transforms the entry hall into a luminous, glass-walled connector, introducing glimpses of color and a visual link between indoor and outdoor spaces. Cork—replacing an eye-stalling patchwork of parquet—gives the entire floor continuity and a soft bounce underfoot, and even runs down steps and up some of the kids' walls, morphing into a pinboard. Its flecked rhythm echoes the back patio's round eucalyptus "pavers," salvaged from a moribund tree. "For every indoor space, I wanted to create an outdoor counterpart," says Cooper. Through big living-area windows, a black-and-cream-tiled wading pool appears, reading almost like a patterned rug just outside the glass.
Nearby, jungly plantings evoke the tropical warmth of Devis's native Latin America and the era in which the house was built— as do the living room's raked-stucco walls in pink and yellow, inspired by architect Luis Barragán's Mexican modernism. These vivid, rustic planes—some extending from inside to out—contribute to the living area's newfound architectural clarity (inherent but partially trapped in Guthrie's design). Freed from compartmentalizing partitions, a fluid living-dining-kitchen area now rises to the existing pitched ceiling. Taalman created a glass doorway between the kitchen and patio that seems as if it's always been there. Downstairs, a big glass panel, hung from a barn-door track, opens the playroom to the outdoors. Here, as in the master bedroom, a single wall of artfully patterned wallpaper animates the space—lightly echoing the living room's energizing stucco. And a bit of improvisation during construction yielded such playful touches as a large porthole between the kids' zone and a stairway with a rope handrail.
Though Taalman and Purdy ardently sought out just the right materials and finishes, the results are remarkably laid-back, with children scampering about and artwork, ranging from playful to provocative, displayed in thoughtful yet unfussy ways.
Just as the stucco reveals the raked process of its creation, the Devis-Purdy House and landscape conveys, most of all, the quality of collaboration that went into their making—and, in the end, the family that brings it all to life.


Metro, bus and ropeway operator in city Tbilisi - Tbilisi Transport Company offers official public transport application for Tbilisi bus and metro. Application offers online and offline features.
TBILISI PASS – a great way to guide you through Tbilisi, capital of Georgia. TBILISI PASS will accompany your Georgian Voyage with exclusive offers, things to do and places to visit during your trip, local mobile sim card support and etc. If you tend to get stuck about where to even start, the Tbilisi Pass App will help you to easily decide what attractions suit your interests as a traveler. Application contains information about most interesting places in Tbilisi, including information about discounts, contact information, descriptions, exact location data and reviews submitted by other application users. All places can be reached by navigating lists of categories, using search button or by exploring the map. Application also contains information about mobile internet, GSM support, low cost Taxi service, personal assistant service, and explains ways how to use these services.
Tbilisi.LOVES YOU application – the most extensive mobile source for Tbilisi city. Application contains information about most of the Tbilisi places. Including contact information, descriptions, exact location data and reviews submitted by other application users. All places can be reached by navigating lists of categories, using search or by exploring the map. You can then bookmark and share your favorite places. Worried about getting lost? Just mark the place you are staying at or just create new place if you are staying in a place which is not listed in the app. Use "My Tbilisi" to access the place you're staying at and all your bookmarked places. If it's your first experience in Tbilisi, we encourage you to take a look at the "Suggestions", where you can find some of the most interesting things around. Data is stored on your phone, so please don't forget to refresh and upload latest data updates.


I have clearly been on holiday too long as I opened Illustrator the other day, made the above pattern almost on autopilot and totally love it. Hopefully I can learn how to do that while sleeping too.
I posted some end of year lists over at diskant i you want to know my favourite records, films, books, zines and events of 2011. The other lists are a good read too.
All my shops are open again and shipping is back to normal as of tomorrow.
Speaking of diaries, they are all sold out, hurray! I hope you're all enjoying filling them in. There will be another edition in the Autumn to start in 2013, but if anyone needs a custom one starting on another month this year, get in touch and I can work out a price for that.
My Sew Cute rainboots are also starting to sell out worldwide so if you fancy a pair you're running out of time. UK sizes 5 and 6 are all gone everywhere. I've got a few in size 7 and 8 and there are some size 4s available from Plueys.
In other shop news, my Folksy store now only accepts orders from UK buyers and has free shipping. My Etsy and DaWanda stores have free shipping within the EU and everyone else will pay a small shipping fee. This has made things so much easier for me and since most of my orders from those shops are for one item, hopefully it makes things easier for you too. I think I'm keeping shipping separate on my main shop though, so if you're planning a big order, that's probably the best value for combined shipping. Wow, that was boring.
Don't forget, the end of year sale is still running, and so is the giveaway!


The harmonized sales tax is going up on P.E.I., to 15 per cent in October.
The increase was announced in the annual budget speech in the legislature Tuesday, presented by Finance Minister Allen Roach.
In his budget speech, Roach said an increase in revenue was necessary to maintain core services in the province, and an increase in the HST was the best option.
"Experts agree that increasing consumption taxes like the HST are less harmful to economic growth than increasing income taxes for individuals," said Roach.
The tax, currently 14 per cent, will go up effective Oct. 1.
The province estimates the increase will provide a further $11 million in revenue over the half year it is in effect in 2016-17, and an extra $22 million in future years.
Roach told the media in the budget lockup that the province was not influenced in its HST decision by increases in the tax in Nova Scotia and New Brunswick.
"Not at all," said Roach.
"We have our own budget process on Prince Edward Island."
During that budget process, said Roach, government officials heard Islanders were concerned about cuts to services, and decided to move to increase revenues rather than reduce expenditures.
P.E.I. budget 2016-17: should the government try to balance it?


Broken Social Scene are releasing a new EP, Let's Try the After - Vol. 1, tomorrow (February 15) via Arts & Crafts. Previously they shared its first single, "All I Want," which was one of our Songs of the Week. Now on the eve of the EP's release the Canadian band have shared another track from it, "Boyfriends," via a fun video made up entirely of GIFs. Watch it below. The band also have Valentine's Day cards inspired by the song that you can email to friends from here.
Frontman Kevin Drew had this to say about the EP in a previous press release: "The point is to keep going. We have more to give. These songs have lived beside us and become our commencement party while continuing together. We hope you enjoy this EP for you and me."
It doesn't really surprise me that broken social scene partnered with Plus1. They're good.


text:
Q: Search on multiple terms with Elastic Search I'm new to ES and want to setup a query to search on multiple terms.
My SQL looks like:
"SELECT field4 FROM tablename WHERE (field1 = 'A' OR field1 = 'AA') AND (field2 = 'B' OR fields2 = 'BB') AND (field3 = 'C' OR field3 = 'CC')"

In ES I'm trying
{
"query": {
"bool": {
  "must": [
    {
      "term": {
        "field1": {"A","AA"}
      }
    },
    {
      "term": {
        "field2": {"B","BB"}
      }
    },
    {
    "term": {
      "field3": {"C","CC"}
    }
   }
  }
 }
}

A: What you most likely want is a terms query. You can see the documentation for it here: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-query.html
An example from that site is:
{
    "terms" : { "user" : ["kimchy", "elasticsearch"]}
}

Also, your query right now isn't valid json, you'll need to correct it.

redpajama_set_name: RedPajamaStackExchange


Argentina's friendly against Venezuela on Friday night was supposed to be a celebratory return for star player Lionel Messi. But at the end of the disappointing defeat, another inquest had begun in the nation about how terrible the team played in the game.
It showed that the team malaise is far from over despite recent victories in friendlies for the Abiceleste. Goals from Salomon Rondon, John Murillo and Josef Martinez condemned Argentina to another defeat despite Lutaro Martinez scoring.
Lionel Messi would definitely be having second thoughts about the national team that looks a shadow of itself. Argentina not quite long ago played in three consecutive finals in a run that started with the 2014 World Cup and ended in the Centenary Copa America in 2016.
Most of the members of that run have either been banished from the team due to the poor World Cup in Russia last year or are out of form. The Argentine first eleven that played against Venezuela had no recognizable star player apart from Messi.
It is appalling that the country has to turn to the likes of Gabriel Mercado, Le Celso and Gonzalo Martinez. This is similar to just before the 2010 World Cup were players like Fernando Cavenaghi played for the National team.
Lionel Scaloni has to find a way to incorporate the likes of Paulo Dybala, Mauro Icardi and Angel Di Maria to the squad. TheArgentina FA has also the responsibility of keeping its house in order as this led to the first retirement of Lionel Messi in 2016.
They have to decide on keeping Lionel Scaloni or hiring a new coach ahead of the Copa America tournament in June. Scaloni has been the interim coach for over 9 months and this is not a good scenario for a football giant like Argentina.


Francis Ford Coppola, now turning 80, wrote film history with a handful of blockbusters, but he also faced disastrous financial flops. Fans celebrated the eclectic career director of The Godfather through a look back at his films.
Coppola, who was born in Detroit on 7 April 1939, had a career that could have easily been the topic of a film. He was for years one of the most influential and commercially successful directors, but also faced disastrous box office flops that brought him financial ruin. At 80, he is a veteran US cinema legend.
You're a Big Boy Now (1966) was Coppola's third feature film after debuting as co-director three years earlier with the horror movie The Terror. The comedy is about a man who finally frees himself from his mother's clutches and discovers life. At the time, it wasn't quite clear where the 27-year-old director was headed.
Two films later, Coppola was offered the film adaptation of Mario Puzo's crime novel The Godfather. The 30-year-old was not the studio's first choice, and the project was not really close to the director's heart, but the film was a huge success, artistically and at the box office, making Coppola one of the top directors in the US.
Coppola followed The Godfather with the 1974 thriller The Conversation starring Gene Hackman as an introverted surveillance expert. The film was not very successful back then, but today it is considered one of the best US films of the 1970s. The Conversation reflects the social and political climate of the country at the time of the Watergate scandal.
In 1974, The Godfather II was a huge success as well. Next, the director turned to the Vietnam War. Inspired by Heart of Darkness, the 19th century British novelist Joseph Conrad, and the narration by US war correspondent Michael Herr, the director told an epic apocalyptic story. Shooting took a long time, but when it was released in 1979, Apocalypse Now was an overwhelming success.
Three years later, Coppola, one of the country's most influential directors, shot the melodrama One from the Heart, starring Nastassja Kinski. It flopped terribly and led to the director's financial ruin. Coppola, who wanted to be independent from the big Hollywood studios, had taken his own studio to the brink.
Coppola basically had to start all over again from scratch, and also work for other producers. With talented new young actors and two movies about youth gangs, he managed a comeback: The Outsiders and Rumble Fish.
Far from humble, Coppola wanted his next film to be big again. Cotton Club (1984), set in 1930's New York, was a wild mixture of a music and gangster film about a roaring period. But the film's box office results didn't cover its production costs and Coppola was struggling once more.
For years afterwards, Coppola was making films that did not meet his artistic expectations, but he needed the money. The 1988 film Tucker, starring Jeff Bridges in the role of the American entrepreneur who rebelled against the mainstream automobile industry of that day, is one of the better films from that era, and it echoes the director's own moves toward independence.
After releasing The Godfather Saga in 1990, the director moved on two years later to shoot the classic story of the famous bloodsucker, Dracula. Reviews of the film adaptation of Bram Stoker's novel were not overwhelmingly positive, but critics praised the film's setting and opulence.
Over the past two decades, the director has made more headlines with the winery he runs in California than with his films. Some critics say he gets better results with his wines than with his last cinematic works, but arguably, his 1997 legal drama The Rainmaker, starring a very young Matt Damon, was the best film adaptation of the John Grisham novel ever made.


The Rodman Hall Alliance calls on Brock University to withdraw its recently announced plans to remove Rodman Hall's art collection to the Marilyn I. Walker School of Fine and Performing Art (MIWSFPA) and repurpose Rodman Hall for commercial use.
Brock University acquired the building, grounds and art collection in 2003 for two dollars, and in a signed agreement with the Rodman Hall board of directors, committed to operating the venue as an art gallery for at 20 years.
"Brock's new plans do not fulfill the covenants of the 2003 agreement between the community and Brock University." says alliance member Sandy Middleton. The agreement includes important restrictions such as those that limit the Rodman Hall property to use as a public art gallery and for charitable purposes.
Rodman Hall Art Centre is an award-winning regional public art museum that has operated for 60 years as the principal art museum for Niagara, providing excellence in visual arts programming and education. It is nationally certified as a 'category A' institution under the Canadian Cultural Property Export Review Board. This status is designated to museum facilities with the professional standards and staff needed to maintain, preserve and exhibit a significant art collection. The majority of Rodman Hall's art collection has been gifted to the institute by members of the local community.
The Rodman Hall Alliance was formed in 2016 when Brock first floated the option of selling Rodman Hall during public consultations. The public outcry that resulted led the university to bring together three distinct ad hoc entities connected to Rodman Hall, including the newly formed Rodman Hall Alliance. The university worked with this group to conduct community consultations, evaluate options and ultimately hire consultants to undertake a feasibility study outlining plans to transition Rodman Hall back to community ownership and management.
This study was finalized in the fall of 2018 and has not been publicly released by the university's administration.
Rodman Hall Alliance is a volunteer group whose aim is to support Rodman Hall Arts Centre in its current location as a vital and multifaceted community asset. In light of Brock University's stated desire to end its stewardship of Rodman Hall in 2023, the Alliance supports renewed community partnerships to share Rodman Hall's assets and take responsibility for its management and financial obligations.
Brock's Plan: "Locate Rodman Hall's permanent collection into class-A certified storage and display space at Brock's Marilyn I. Walker School of Fine and Performing Arts (MIWSFPA) in downtown St. Catharines."
What are the expected costs and timeline for creation of storage, preparation, exhibition and administrative space necessary to support a regional public art museum?
Has the University undertaken the necessary application for category A status for transfer of the art collection into new facilities?
Will the University maintain appropriate staffing requirements for operation of a class-A art museum?
Brock's plan: "offer 2,000 sq. ft. of gallery space at the MIWSFPA"
Given that Rodman Hall added 4,000 sq. ft. of gallery space to its facility in 1973 to receive national gallery designation, how will Brock preserve Category A status with 2,000 sq. ft.?
How will the new art museum requirements impact MIWSFPA facilities which were built to carefully identified specifications?
Will 2,000 plus sq. ft. be renovated within the existing footprint of the school or will Brock undertake to build new space?
Brock's Plan: "transfer the Rodman Hall art collection to a community transition group, who would display the collection throughout Niagara"
What does Brock mean by "throughout Niagara"? We are not aware of other category A exhibition spaces in Niagara that could meet loan conditions required as set by the Canadian Cultural Properties Review and Export Board.
Brock's Plan: "Create a new downtown 'culture commons' with up to 7,000 sq. ft. for arts organizations and community partners"
Is Brock offering to lead the planning, development and build of a multipurpose facility?
Was Brock aware that other local arts organizations have been pursuing a similar initiative for more than a year?
Has Brock consulted with the arts organizations currently investigating this idea?
What funds will Brock contribute to this suggested partnership?
Why won't Brock use those funds to preserve Rodman Hall as an art museum for the next four years?
How does this offer mitigate the loss of Rodman Hall to the arts community and, more importantly, to the visual art-going community?
Brock's Plan: "negotiate with a heritage property company to operate Rodman Hall under the restrictions that protect the historical integrity of its architecture and grounds, and with the potential to include a dedicated art gallery space… Brock envisions a commercially sustainable Rodman Hall . . ."
How much income will Brock generate from their new use of Rodman Hall?
Rodman Hall is a charitable cultural property that was transferred to Brock as a charitable institution. How does Brock reconcile the commercial use of a charitable asset?
Is Brock planning to sell Rodman Hall in 2023 to this undisclosed private company?
Brock's Statement: "since 2014 has paid about $300,000 for studies to develop a divestment plan for Rodman Hall."
Brock has spent funds on studies by expert museum and cultural consultants whose recommendations – the Lundholm Report (2010), the Barlow Report (2015) and the Bogusky/Pappert Report (2018) – have not been implemented. Why?
In these reports, partnerships have been identified as essential to the future of Rodman Hall. What efforts has Brock made to identify and approach possible community partners?


text:
Magyarország bukaresti nagykövetsége 1921 óta működik Románia fővárosában (1954-ig követségként). A nagykövetséghez tartozik az ugyancsak nagymúltú kolozsvári és a 2006-ban nyílt csíkszeredai főkonzulátus. A nagykövetség 2019-ben a strada Jean-Louis Calderon 63-65 szám alatt van, nagykövetünk 2013 óta Zákonyi Botond.

Feladata, hatásköre
A nagykövetségek általános feladatkörének megfelelően a bukaresti misszió a két ország kapcsolatainak egyik legfontosabb intézménye, feladata a mindenkori magyar kormányok politikájának képviselete a fogadó országban. A konzuli szolgálat – mely a romániai nagykövetségétől eltérő helyszínen működik – feladata az állampolgárok ügyeinek intézése: útlevelek pótlása, beutazási engedélyek, haláleseti intézkedés stb.

Története
1918 előtt az Osztrák–Magyar Monarchia diplomatái képviselték a magyar érdekeket Romániában, azonban a két ország egyre romló viszonya, majd Románia hadbalépése az antant oldalán a képviselet felszámolásához, és a diplomaták kiutasításához vezettek. A monarchia utolsó nagykövete a cseh származású Ottokar Czernin (később a monarchia külügyminisztere), első titkára a magyar Csáky Imre volt. A diplomaták 1916 őszén távoztak Romániából. A Magyar Királyság 1921 januárjában nyitotta meg követségét, első magyar követi kinevezésre 1921 nyarán került sor, amikor Rubidó-Zichy Iván lett a bukaresti magyar misszió vezetője (követi címét már 1920 végén megkapta). A követség a strada Boteanu 4. szám alatt volt. 1921 decemberében érkezett Bukarestbe Aichhorn Richárd hivatalvezető, hogy a repatriálási ügyeket segítse az engedélyek kiadásával a kolozsvári útlevélhivatalban. Ő ilymódon a későbbi kolozsvári konzulátus első vezetője volt.

A két világháború közti időszak meghatározó követe volt Bárdossy László, a háborús bűneiért kivégzett későbbi miniszterelnök, akit 1934. október 24-én neveztek ki Magyarország bukaresti követévé, ahol nagy tudása és korrekt viselkedése miatt a románok is tisztelettel és elismeréssel kezelték. A pozíciót külügyminiszteri tisztségbe lépéséig, 1941-ig töltötte be. A követség 1936-ban költözött át (akkori nevén) a strada Gogu Cantacuzino 65-be. Az utca neve időközben Jean-Louis Calderonra változott, de a követség mindmáig ugyanott van. A román kiugrás előtti utolsó nagykövetünk Hertelendy Andor volt, akit 1944 elején neveztek ki és augusztus 14-én mentettek fel.

A második világháborút követően az Ideiglenes Nemzeti Kormány már 1945 augusztusában szorgalmazta, hogy Romániában követséget nyissunk. A követi poszt betöltésére a Magyar Kommunista Párt is bejelentette igényét. A követség végül 1947. november 11-én nyílt meg, első vezetője Gyöngyössy István lett mint ügyvivő, első követünk pedig a kommunista Széll Jenő 1948. március 3-tól. Nagykövetséggé 1954. január 18-án emelték az intézményt, első nagykövetünk Pataki László, a rendszerváltás utáni első nagykövetünk pedig Rudas Ernő volt.

A 20.-21. század során hullámzóan alakuló magyar-román kapcsolatok hatásának mindig is szimbolikus helyszíne volt nagykövetségünk – hasonlóképpen Románia budapesti nagykövetsége is. Az évtizedek alatt számos tüntetés, bombariadó, de demonstratív szimpátiatüntetés is volt már az épület előtt. Így például 1990-ben a román média által felheccelt ezres nagyságrendű tömeg követelte, hogy a román-magyar határok ne változzanak meg. Szűts Pál nagykövet fogadta a tüntetők képviselőit és tájékoztatta őket, hogy határmódosításról nincs szó a két ország kapcsolatában, mire maguk a tüntetők hívták ki a román televíziót, hogy adjon hírt a helyzetet tisztázó magyar álláspontról. 2011-ben a Magyarországon akkor bevezetni tervezett új médiatörvény ellen demonstráltak – magyar kollégáikkal szolidaritást vállalva – román újságírók, egy évvel később azonban már román szélsőségesek tüntettek Füzes Oszkár akkori nagykövetnek a székely autonómiatörekvéseket támogató megszólalása miatt.

Magyarország bukaresti követei és nagykövetei a második világháborútól a rendszerváltásig
Gyöngyössy István dr. ideiglenes ügyvivő 1947-7-31; 1948-4-2
Széll  Jenő követ 1948-3-3 (állomáshely elfoglalása); 1948-4-2 (megbízólevél átadása); 1950-2-8 (távozás)
Kálló Iván követ 1950-2-8; ...; 1951-5-7
Pataki László követ 1951-5-7 ...; nagykövet 1954-6-22; 1954-7-9; 1956-7-13
Keleti Ferenc nagykövet 1956-7-13; ...; 1959-11-29
Némety Béla nagykövet 1959-11-28; 1960-1-14; 1962-5-31
Kuti  Jenő nagykövet 1962-5-31; 1962-7-11; 1965-2-24
Vince József nagykövet 1965-2-25; 1965-3-17; 1969-11-6
Martin Ferenc nagykövet 1969-11-6; 1969-12-10; 1974-10-18
Biczó György dr. nagykövet 1974-10-18; 1974-11-5; 1978-10-5
Rajnai Sándor nagykövet 1978-10-5; 1978-10-31; 1982-7-9
Barity Miklós nagykövet 1982-7-9; 1982-9-17; 1985-10-1
Szűts Pál nagykövet 1985-10-1; 1985-11-6; 1990-7-11

Jegyzetek

Források

Kapcsolódó szócikk
Magyarország és Románia kapcsolatai

Bukarest
Magyar–román kapcsolatok
Bukarest
Diplomáciai missziók Romániában

redpajama_set_name: RedPajamaWikipedia


Here are actual analogies and metaphors found in High School essays.
8. The revelation that his marriage of 30 years had disintegrated came as a rude shock, like a surcharge at a formerly surcharge-free ATM.
14. Long separated by cruel fate, the star-crossed lovers raced across the grassy field toward each other like two freight trains, one having left Cleveland at 6:36 p.m.traveling at 55 mph, the other from Topeka at 4:19p.m.at a speed of 35 mph.


Breast lipofilling, is also known as breast fat transfer augmentation or fat grafting. It involves harvesting excess fat from an area with a prominence, like the lower abdomen, "saddlebags" on the outer thigh, or "love handles" on the flanks. This fat can be processed then injected into the breasts. The operation is performed as a adjunctive procedure for patients who have had lumpectomy, reconstructive surgery after mastectomy, or cosmetically for improved shape and size.
This operation is commonly done under general anaesthetic. The region where the fat will be obtained is infiltrated with a solution that facilitates harvesting the fat, and decreases postoperative pain and bleeding. Small incisions are then made in the skin, and a thin metal tube (cannula) is used to aspirate the fat into a sterile collection chamber. The fat is then processed and separated from the blood and other fluids that have been removed. The purified fat is then injected into the breast. Tiny incisions are made around the breast and the fat is injected using tiny cannulas.
Fat transfer to the breasts is commonly done as an outpatient procedure. Showering is permitted 48 hours after surgery, and most patients return to work within 1-2 weeks. Care must be taken to avoid excessive pressure to the breasts or vigorous exercise for at least 4-6 weeks after surgery.
Fat transfer has several advantages including using your own tissue, no need for a breast implant or the longer incision required for implant insertion, and improving the contour of the area where the fat is harvested. However, a significant amount of transferred fat will eventually resorb, so often multiple grafting sessions may be required to gain the desired result. There can be complications like infection, bleeding, asymmetry, contour irregularities or other problems, either on the breasts or the site where the fat was harvested. Transferred fat can form cysts and calcifications in the breast, which may affect mammogram results. When fat is transferred to a breast to improve the appearance after breast cancer, there is a potential risk of stimulating a recurrence of breast cancer (studies are ongoing). During a consultation, Dr. Andrade will discuss the potential risks and benefits of surgery in more detail.


Is there any possibility to persist the process filter list (for profiling type .NET Process) in dotTrace 10.0.2? Or is this a bug that the filters are cleared when restarting dotTrace?
It is not a bug, process filter list is saved in the corresponding launch session. If you select launch session, you will see that there is a process list (please see screenshot).


8 years & up. Have fun using the scientific method with microbiology experiments!
Bacteria are everywhere and can be responsible for things like decay and fermentation or used for good things like medicine and yogurt. This hands-on Mighty Microbes Lab allows your child to grow and observe harmless, small-scale bacteria and learn how microbes affect our everyday lives.


text:
package com.tq.requisition.infrastructure.Specifications.orgMgt;

import java.util.UUID;

import com.tq.requisition.domain.Specification.Specification;
import com.tq.requisition.domain.Specification.expression.IHqlExpression;
import com.tq.requisition.domain.Specification.expression.OperationType;
import com.tq.requisition.domain.model.organization.Organization;
import com.tq.requisition.infrastructure.Specifications.Expression.HqlExpression;

public class OrgIdExistsSpecificator  extends Specification<Organization> {

	private String orgId;

	public OrgIdExistsSpecificator(Class<Organization> _t,UUID orgId) {
		super(_t);
		this.orgId = orgId.toString();
	}

	@Override
	public IHqlExpression getHqlExpression() {
		IHqlExpression expression = new HqlExpression();
		expression.setSql("select count(1) from tb_org where id=?");
		expression.setType(OperationType.SQL);
		expression.setParameters(orgId);
		return expression;
	}

}

redpajama_set_name: RedPajamaGithub


text:
The FOX 2 Giving Guide makes giving back easy!
FOX 2 is working for you all year long, but we know we're part of a bigger community -- a giving community. That's why we go to work every year to help connect you to the people and agencies who need your help.
At the heart of that effort is The Giving Guide - a book we put together with Gardner-White Furniture...
Our Giving Guide makes it easy for you to make a difference. We create this guide to be used as tool to help you get more involved. The book features the needs of more than 100 local charities – what they do, what they need, and how you can help!
Pick up a Giving Guide at any Gardner-White Furniture Location or get it online, click here.
As part of this effort we produce a special show called "Holiday Connection" hosted by Sherry Margolis and Roop Raj, spotlighting some of the charities in the book and featuring stories of amazing people in our community who are giving back in big and small ways. Every person can make a difference!
Watch the complete show on FOX 2 throughout the holiday season
We hope you'll be inspired to reach out to one of the agencies featured in our show!
FOX 2, and our partners Gardner-White, and 100.3 WNIC invite to you make a holiday connection! Make a difference today!


text:
\section{Introduction}
Generalizability is crucial to deep learning, and many efforts have been made to improve the training and generalization accuracy of deep neural nets (DNNs) \cite{GreedyTraining:2007,DBN:2006}. Advances in network architectures such as VGG networks \cite{VGG:2014}, deep residual networks (ResNets)\cite{DRN:2016,IdentityMap:2016} and more recently DenseNets \cite{Huang:2017CVPR} and many others \cite{Chen:2017NIPS}, together with powerful hardware make the training of very deep networks with good generalization capabilities possible. Effective regularization techniques such as dropout and maxout \cite{hinton2012improving, wan2013regularization, goodfellow2013maxout}, as well as data augmentation methods \cite{krizhevsky2012imagenet,VGG:2014, Zhu:2017DeepLearning} have also explicitly improved generalization for DNNs.

A key component of neural nets is the activation function. Improvements in designing of activation functions such as the rectified linear unit (ReLU) \cite{glorot2011deep}, have led to huge improvements in performance in computer vision tasks \cite{nair2010rectified, krizhevsky2012imagenet}. More recently, activation functions adaptively trained to the data such as the adaptive piecewise linear unit (APLU) \cite{agostinelli2014learning} and parametric rectified linear unit (PReLU) \cite{he2015delving} have lead to further improvements in performance of DNNs. For output activation, support vector machine (SVM) has also been successfuly applied in place of softmax\cite{Tang:2013}. Though training DNNs with softmax or SVM as output activation is effective in many tasks, it is possible that alternative activations that consider manifold structure of data by interpolating the output based on both training and testing data can boost performance of the network. In particular, ResNets can be reformulated as solving control problems of a class of transport equations in the continuum limit \cite{ResNet:PDE, chang2017multi}. Transport theory suggests that by using an interpolating function that interpolates terminal values from initial values can dramatically simplify the control problem compared to an ad-hoc choice. This further suggests that a fixed and data-agnostic activation for the output layer may be suboptimal.

To this end, based on the ideas from manifold learning, we propose a novel output layer named weighted nonlocal Laplacian (WNLL) layer for DNNs. The resulted DNNs achieve better generalization and are more robust for problems with a small number of training examples. On CIFAR10/CIFAR100, we achieve on average a 30\%/20\% reduction in terms of test error on a wide variety of networks. These include VGGs, ResNets, and pre-activated ResNets. The performance boost is even more pronounced when the model is trained on a random subset of CIFAR with a low number of training examples. We also present an efficient algorithm to train the WNLL layer via an auxiliary network. Theoretical motivation for the WNLL layer is also given from the viewpoint of both game theory and terminal value problems for transport equations.

This paper is structured as follows: In Section \ref{sec:Network-Architecture}, we introduce the motivation and practice of using the WNLL interpolating function in DNNs. In Section \ref{sec:Train-Test}, we explain in detail the algorithms for training and testing DNNs with WNLL as output layer. Section \ref{sec:Theory} provides insight of using an interpolating function as output layer from the angle of terminal value problems of transport equations and game theory. Section \ref{Experiments} demonstrates the effectiveness of our method on a variety of numerical examples.

\section{Network Architecture}
\label{sec:Network-Architecture}
In coarse grained representation, training and testing DNNs with softmax layer as output are illustrated in Fig. \ref{fig:DNN-Structure} (a) and (b), respectively. In $k$th iteration of training, given a mini-batch training data $(\mathbf{X}, \mathbf{Y})$, we perform:

{\it Forward propagation:} Transform $\mathbf{X}$ into deep features by DNN block (ensemble of conv layers, nonlinearities and others), and then activated by softmax function to obtain the predicted labels $\tilde{\mathbf{Y}}$:
$$
\tilde{\mathbf{Y}} = {\rm Softmax}({\rm DNN}(\mathbf{X}, \Theta^{k-1}), \mathbf{W}^{k-1}).
$$
Then compute loss (e.g., cross entropy) between $\mathbf{Y}$ and $\tilde{\mathbf{Y}}$:
$\mathcal{L} = {\rm Loss}(\mathbf{Y}, \tilde{\mathbf{Y}})$.

{\it Backpropagation:} Update weights ($\Theta^{k-1}$, $\mathbf{W}^{k-1}$) by gradient descent (learning rate $\gamma$):
$$\mathbf{W}^{k} = \mathbf{W}^{k-1} - \gamma \frac{\partial \mathcal{L}}{\partial \tilde{\mathbf{Y}}}\cdot \frac{\partial \tilde{\mathbf{Y}}}{\partial \mathbf{W}}, \ \ \Theta^{k} = \Theta^{k-1} - \gamma \frac{\partial \mathcal{L}}{\partial \tilde{\mathbf{Y}}}\cdot \frac{\partial \tilde{\mathbf{Y}}}{\partial \tilde{\mathbf{X}}}\cdot \frac{\partial \tilde{\mathbf{X}}}{\partial \Theta}.$$

\begin{figure}[h]
\centering
\begin{tabular}{cc}
\includegraphics[width=0.105\columnwidth]{DNN_training.png}&
\includegraphics[width=0.15\columnwidth]{DNN_testing.png}\\
(a)&(b)\\
\end{tabular}
\caption{Training (a) and testing (b) procedures of DNNs with softmax as output activation layer.}
\label{fig:DNN-Structure}
\end{figure}

Once the model is optimized, for testing data $\mathbf{X}$, the predicted labels are:
$$
\tilde{\mathbf{X}} = {\rm Softmax}({\rm DNN}(\mathbf{X}, \Theta), \mathbf{W}),
$$
for notational simplicity, we still denote the test set and optimized weights as $\mathbf{X}$, $\Theta$, and $\mathbf{W}$, respectively.
In essence the softmax layer acts as a linear model on the space of deep features $\tilde{\mathbf{X}}$, which does not take into consideration the underlying manifold structure of $\tilde{\mathbf{X}}$. The WNLL interpolating function, which will be introduced in the following subsection, is an approach to alleviate this deficiency.

\subsection{Manifold Interpolation - An Harmonic Extension Approach}
Let $\mathbf{X} = \{\mathbf{x}_1, \mathbf{x}_2, \cdots, \mathbf{x}_n\}$ be a set of points in a high dimensional manifold $\mathcal{M}\subset\mathbb{R}^d$ and $\mathbf{X}^{\rm te} = \{\mathbf{x}^{\rm te}_1, \mathbf{x}^{\rm te}_2, \cdots, \mathbf{x}^{\rm te}_m\}$ be a subset of $\mathbf{X}$. Suppose we have a (possibly vector valued) label function $g(\mathbf{x})$ defined on $\mathbf{X}^{\rm te}$, and we want to interpolate a function $u$ that is defined on the entire manifold and can be used to label the entire dataset $\mathbf{X}$. Interpolation by using basis function in high dimensional space suffers from the curse of dimensionality. Instead, an harmonic extension is a natural and elegant approach to find such an interpolating function, which is defined by minimizing the following Dirichlet energy functional:
\begin{equation}
\label{DirichletEnergy}
\mathcal{E}(u)=\frac{1}{2}\sum_{\mathbf{x}, \mathbf{y}\in \mathbf{X}} w(\mathbf{x}, \mathbf{y})\left(u(\mathbf{x})-u(\mathbf{y})\right)^2,
\end{equation}
with the boundary condition:
$$
u(\mathbf{x})=g(\mathbf{x}), \ \mathbf{x}\in \mathbf{X}^{\rm te},
$$
where $w(\mathbf{x}, \mathbf{y})$ is a weight function, typically chosen to be Gaussian: $w(\mathbf{x}, \mathbf{y})=\exp(-\frac{||\mathbf{x}-\mathbf{y}||^2}{\sigma^2})$ with $\sigma$ a scaling parameter. The Euler-Lagrange equation for Eq.\eqref{DirichletEnergy} is:
\begin{equation}
\label{EL-Equation}
\begin{cases}
\sum_{\mathbf{y}\in \mathbf{X}} \left(w(\mathbf{x}, \mathbf{y})+w(\mathbf{y}, \mathbf{x})\right)\left(u(\mathbf{x})-u(\mathbf{y})\right)=0 & \mathbf{x}\in \mathbf{X}/\mathbf{X}^{\rm te}\\
u(\mathbf{x})=g(\mathbf{x}) &\mathbf{x}\in \mathbf{X}^{\rm te}.
\end{cases}
\end{equation}
By solving the linear system Eq.(\ref{EL-Equation}), we get the interpolated labels $u(\mathbf{x})$ for unlabeled data $\mathbf{x}\in \mathbf{X}/\mathbf{X}^{\rm te}$. This interpolation becomes invalid when labeled data is tiny, i.e., $|\mathbf{X}^{\rm te}|\ll |\mathbf{X}
/\mathbf{X}^{\rm te}|$. There are two solutions to resolve this issue: one is to replace the $2$-Laplacian in Eq.(\ref{DirichletEnergy}) by a $p$-Laplacian \cite{Calder:2018}; the other is to increase
the weights of the labeled data in the Euler-Lagrange equation \cite{WNLL:2017}, which gives the following weighted nonlocal Laplacian (WNLL) interpolating function:
\begin{equation}
\label{WNLL}
\begin{cases}
\sum_{\mathbf{y}\in \mathbf{X}} \left(w(\mathbf{x}, \mathbf{y})+w(\mathbf{y}, \mathbf{x})\right)\left(u(\mathbf{x})-u(\mathbf{y})\right)+\\
\left(\frac{|\mathbf{X}|}{|\mathbf{X}^{\rm te}|}-1\right)\sum_{\mathbf{y}\in \mathbf{X}^{\rm te}}w(\mathbf{y}, \mathbf{x})\left(u(\mathbf{x})-u(\mathbf{y})\right)=0 & \mathbf{x}\in \mathbf{X}/\mathbf{X}^{\rm te}\\
u(\mathbf{x})=g(\mathbf{x}) &\mathbf{x}\in \mathbf{X}^{\rm te}.
\end{cases}
\end{equation}
For notational simplicity, we name the solution $u(\mathbf{x})$ to Eq.\ref{WNLL} as ${\rm WNLL}(\mathbf{X}, \mathbf{X}^{\rm te}, \mathbf{Y}^{\rm te})$.
For classification tasks, $g(\mathbf{x})$ is the one-hot labels for the example $\mathbf{x}$. To ensure accuracy of WNLL, the labeled data should cover all classes of data in $\mathbf{X}$. We give a necessary condition in Theorem \ref{Rep-Thm}.
\begin{thm}\label{Rep-Thm}
Suppose we have a data pool formed by $N$ classes of data uniformly, with the number of instances of each class be sufficiently large. If we want all classes of data to be sampled at least once, on average at least $N\left(1+\frac{1}{2}+\frac{1}{3}+\cdots +\frac{1}{N}\right)$ data is need to be sampled from the data pool. In this case, the number of data sampled, in expectation for each class, is $1+\frac{1}{2}+\frac{1}{3}+\cdots +\frac{1}{N}$.
\end{thm}

\subsection{WNLL Activated DNNs and Algorithms} \label{sec:Train-Test}
In both training and testing of the WNLL activated DNNs, we need to reserve a small portion of data/label pairs denoted as $(\mathbf{X}^{\rm te}, \mathbf{Y}^{\rm te})$, to interpolate the label $Y$ for new data. We name $(\mathbf{X}^{\rm te}, \mathbf{Y}^{\rm te})$ as the preserved template. Directly replacing softmax by WNLL (Fig. \ref{fig:WNLL-DNN-Structure}(a)) has difficulties in back propagation, namely, the true gradient $\frac{\partial \mathcal{L}}{\partial \Theta}$ is difficult to compute since WNLL defines a very complex implicit function. Instead, to train WNLL activated DNNs, we propose a proxy via an auxiliary neural nets (Fig. \ref{fig:WNLL-DNN-Structure}(b)). On top of the original DNNs, we add a buffer block (a fully connected layer followed by a ReLU), and followed by two parallel layers, WNLL and the linear (fully connected) layers. The auxiliary DNNs can be trained by alternating between the following two steps (training DNNs with linear and WNLL activations, respectively):

{\bf Train DNNs with linear activation: } Run $N_1$ steps of the following forward and back propagation, where in $k$th iteration, we have:

{\it Forward propagation: } The training data $\mathbf{X}$ is transformed, respectively, by DNN, Buffer and Linear blocks to the predicted labels $\tilde{\mathbf{Y}}$:
$$\tilde{\mathbf{Y}} = {\rm Linear}({\rm Buffer}({\rm DNN}(\mathbf{X}, \Theta^{k-1}), \mathbf{W}_B^{k-1}), \mathbf{W}_L^{k-1}).$$
Then compute loss between the ground truth labels $\mathbf{Y}$ and predicted ones $\tilde{\mathbf{Y}}$, denoted as $\mathcal{L}^{\rm Linear}$ (e.g., cross entropy loss, and the same as following $\mathcal{L}^{\rm WNLL}$).

{\it Backpropagation: } Update weights ($\Theta^{k-1}$, $\mathbf{W}_B^{k-1}$, $\mathbf{W}_L^{k-1}$) by gradient descent:
$$
\mathbf{W}_L^k = \mathbf{W}_L^{k-1} - \gamma \frac{\partial \mathcal{L}^{\rm Linear}}{\partial \tilde{\mathbf{Y}}}\cdot \frac{\partial \tilde{\mathbf{Y}}}{\partial \mathbf{W}_L},\ \ \mathbf{W}_B^k = \mathbf{W}_B^{k-1} - \gamma \frac{\partial \mathcal{L}^{\rm Linear}}{\partial \tilde{\mathbf{Y}}}\cdot \frac{\partial \tilde{\mathbf{Y}}}{\partial \hat{\mathbf{X}}}\cdot \frac{\partial \hat{\mathbf{X}}}{\partial \mathbf{W}_B},
$$
$$
\Theta^k = \Theta^{k-1} - \gamma \frac{\partial \mathcal{L}^{\rm Linear}}{\partial \tilde{\mathbf{Y}}}\cdot \frac{\partial \tilde{\mathbf{Y}}}{\partial \hat{\mathbf{X}}}\cdot \frac{\partial \hat{\mathbf{X}}}{\partial \tilde{\mathbf{X}}}\cdot \frac{\partial \tilde{\mathbf{X}}}{\partial \Theta}.
$$

{\bf Train DNNs with WNLL activation: } Run $N_2$ steps of the following forward and back propagation, where in $k$th iteration, we have:

{\it Forward propagation: } The training data $\mathbf{X}$, template $\mathbf{X}^{\rm te}$ and $\mathbf{Y}^{\rm te}$ are transformed, respectively, by DNN, Buffer, and WNLL blocks to get predicted labels $\hat{\mathbf{Y}}$:
$$\hat{\mathbf{Y}} = {\rm WNLL}({\rm Buffer}({\rm DNN}(\mathbf{X}, \Theta^{k-1}), \mathbf{W}_B^{k-1}), \hat{\mathbf{X}}^{\rm te}, \mathbf{Y}^{\rm te}).$$
Then compute loss, $\mathcal{L}^{\rm WNLL}$, between the ground truth labels $\mathbf{Y}$ and predicted ones $\hat{\mathbf{Y}}$.

{\it Backpropagation: } Update weights $\mathbf{W}_B^{k-1}$ only, $\mathbf{W}_L^{k-1}$ and $\Theta^{k-1}$ will be tuned in the next iteration in training DNNs with linear activation, by gradient descent.

\begin{equation}
\mathbf{W}_B^{k} =  \mathbf{W}_B^{k-1} - \gamma \frac{\partial \mathcal{L}^{\rm WNLL}}{\partial \hat{\mathbf{Y}}}\cdot \frac{\partial \hat{\mathbf{Y}}}{\partial \hat{\mathbf{X}}}\cdot \frac{\partial \hat{\mathbf{X}}}{\partial \mathbf{W}_B} \approx \mathbf{W}_B^{k-1} - \gamma \frac{\partial \mathcal{L}^{\rm Linear}}{\partial \tilde{\mathbf{Y}}}\cdot \frac{\partial \tilde{\mathbf{Y}}}{\partial \hat{\mathbf{X}}}\cdot \frac{\partial \hat{\mathbf{X}}}{\partial \mathbf{W}_B}.
\label{eq:bp-wnll}
\end{equation}

Here we use the computational graph of the left branch (linear layer) to retrieval the approximated gradients for WNLL. For a given loss value of $\mathcal{L}^{\rm WNLL}$, we adopt the approximation
$\frac{\partial \mathcal{L}^{\rm WNLL}}{\partial \hat{\mathbf{Y}}}\cdot \frac{\partial \hat{\mathbf{Y}}}{\partial \hat{\mathbf{X}}} \approx \frac{\partial \mathcal{L}^{\rm Linear}}{\partial \tilde{\mathbf{Y}}}\cdot \frac{\partial \tilde{\mathbf{Y}}}{\partial \hat{\mathbf{X}}}$ where the right hand side is also evaluated at this value.
The main heuristic behind this approximation is the following: WNLL defines a harmonic function implicitly, and a linear function is the simplest nontrivial explicit harmonic function. Empirically, we observe this simple approximation works well in training the network. The reason why we freeze the network in the DNN block is mainly due to stability concerns.

\begin{figure}[h]
\centering
\begin{tabular}{ccc}
\includegraphics[width=0.18\columnwidth]{WNLL_DNN_direct_training.png}&
\includegraphics[width=0.29\columnwidth]{WNLL_DNN_training_2.pdf}&
\includegraphics[width=0.25\columnwidth]{WNLL_DNN_testing.png}\\
(a)&(b)&(c)\\
\end{tabular}
\caption{Training and testing procedure of the deep neural nets with WNLL as the last activation layer.(a): Direct replacement of the softmax by WNLL, (b): An alternating training procedure. (c): Testing.}
\label{fig:WNLL-DNN-Structure}
\end{figure}

The above alternating scheme is an algorithm of a greedy fashion. During training, WNLL activation plays two roles: on one hand, the alternating between linear and WNLL activations benefits each other which enables the neural nets to learn features that is appropriate for both linear classification and WNLL based manifold interpolation. On the other hand, in the case where we lack sufficient training data, the training of DNNs usually gets stuck at some bad local minima which cannot generalize well on new data. We use WNLL interpolation which provides a perturbation to the trained sub-optimal weights and can help to arrive at a local minima with better generalizability. At test time, we remove the linear classifier from the neural nets and use the DNN block together with WNLL to predict new data (Fig. \ref{fig:WNLL-DNN-Structure} (c)). The reason for using WNLL instead of a linear layer is because WNLL is superior to the linear classifier and this superiority is preserved when applied to deep features (which will be shown in Section. \ref{Experiments}). Moreover, WNLL utilizes both the learned DNNs and the preserved template at test time which seems to be more stable to perturbations on the input data.

We summarize the training and testing procedures for the WNLL activated DNNs in Algorithms \ref{alg-Train} and \ref{alg-Test}, respectively. In each round of the alternating procedure i.e., each outer loop in Algorithm. \ref{alg-Train}, the entire training set $(\mathbf{X}, \mathbf{Y})$ is first used to train the DNNs with linear activation. We randomly separate a template, e.g., half of the entire data, from the training set which will be used to perform WNLL interpolation in training WNLL activated DNNs. In practice, for both training and testing, we use minibatches for both the template and the interpolated points when the entire dataset is too large. The final predicted labels are obtained by a majority voted across interpolation results from all the template minibatches.

\begin{remark}
In Algorithm. \ref{alg-Train}, the WNLL interpolation is also performed in mini-batch manner (as shown in the inner iteration). Based on our experiments, this does not reduce the interpolation accuracy significantly.
\end{remark}

\begin{algorithm}
\caption{DNNs with WNLL as Output Activation: Training Procedure.}\label{alg-Train}
\begin{algorithmic}
\State \textbf{Input: } Training set: (data, label) pairs $(\mathbf{X}, \mathbf{Y})$.
\State \textbf{Output: } An optimized DNNs with WNLL as output activation, denoted as ${\rm DNN}_{\rm WNLL}$.
\For {${\rm iter} = 1, $\dots$, N$ (where $N$ is the number of alternating steps.)}
\State //Train the left branch: DNNs with linear activation.
\State Train DNN $+$ Linear blocks, and denote the learned model as ${\rm DNN}_{\rm Linear}$.
\State //Train the right branch: DNNs with WNLL activation.
\State Split $(\mathbf{X}, \mathbf{Y})$ into training data and template, i.e., $(\mathbf{X}, \mathbf{Y}) \doteq (\mathbf{X}^{\rm tr}, \mathbf{Y}^{\rm tr}) \bigcup (\mathbf{X}^{\rm te}, \mathbf{Y}^{\rm te})$.
\State Partition the training data into $M$ mini-batches, i.e., $(\mathbf{X}^{\rm tr}, \mathbf{Y}^{\rm tr}) = \bigcup_{i=1}^M (\mathbf{X}_i^{\rm tr}, \mathbf{Y}_i^{\rm tr})$.
\For {$i = 1, 2, \cdots, M$}
\State Transform $\mathbf{X}_i^{\rm tr}\bigcup \mathbf{X}^{\rm te}$ by ${\rm DNN}_{\rm Linear}$, i.e.,  $\tilde{\mathbf{X}}^{\rm tr} \bigcup \tilde{\mathbf{X}}^{\rm te} = {\rm DNN}_{\rm Linear}(\mathbf{X}_i^{\rm tr}\bigcup \mathbf{X}^{\rm te})$.
\State Apply WNLL (Eq.(\ref{WNLL})) on $\{\tilde{\mathbf{X}}^{\rm tr} \bigcup \tilde{\mathbf{X}}^{\rm te}, \mathbf{Y}^{\rm te}\}$ to interpolate label $\tilde{\mathbf{Y}}^{\rm tr}$.
\State Backpropagate the error between $\mathbf{Y}^{\rm tr}$ and $\tilde{\mathbf{Y}}^{\rm tr}$ via Eq.(\ref{eq:bp-wnll}) to update $\mathbf{W}_B$ only.
\EndFor
\EndFor
\end{algorithmic}
\end{algorithm}

\begin{algorithm}
\caption{DNNs with WNLL as Output Activation: Testing Procedure.}\label{alg-Test}
\begin{algorithmic}
\State \textbf{Input: } Testing data $\mathbf{X}$, template $(\mathbf{X}^{\rm te}, \mathbf{Y}^{\rm te})$. Optimized model ${\rm DNN}_{\rm WNLL}$.
\State \textbf{Output: } Predicted label $\tilde{\mathbf{Y}}$ for $\mathbf{X}$.
\State Apply the DNN block of ${\rm DNN}_{\rm WNLL}$ to $\mathbf{X}\bigcup \mathbf{X}^{\rm te}$ to get the representation $\tilde{\mathbf{X}}\bigcup \tilde{\mathbf{X}}^{\rm te}$.
\State Apply WNLL (Eq.(\ref{WNLL})) on $\{\tilde{\mathbf{X}} \bigcup \tilde{\mathbf{X}}^{\rm te}, \mathbf{Y}^{\rm te}\}$ to interpolate label $\tilde{\mathbf{Y}}$.
\end{algorithmic}
\end{algorithm}

\section{Theoretical Explanation}
\label{sec:Theory}
In training WNLL activated DNNs, the two output activation functions in the auxiliary networks are, in a sense, each competing to minimize its own objective where, in equilibrium, the neural nets can learn better features for both linear and interpolation-based activations. This in flavor is similar to generative adversarial nets (GAN) \cite{GAN}.
Another interpretation of our model is the following: As noted in  \cite{ResNet:PDE}, in the continuum limit, ResNet can be modeled as the following control problem for a transport equation:
\begin{equation}
\label{Linear-Transport}
\begin{cases}
\frac{\partial u(\mathbf{x}, t)}{\partial t}+\mathbf{v}(\mathbf{x}, t)\cdot \nabla u(\mathbf{x}, t)=0 & \mathbf{x}\in \mathbf{X}, t\geq 0\\
u(\mathbf{x}, 1)=f\left(\mathbf{x}\right) &\mathbf{x}\in \mathbf{X}.
\end{cases}
\end{equation}
Here $u(\cdot, 0)$ is the input of the continuum version of ResNet, which maps the training data to the corresponding label. $f(\cdot)$ is the terminal value which analogous to the output activation function in ResNet which maps deep features to the predicted label. Training ResNet is equivalent to tuning $\mathbf{v}(\cdot, t)$, i.e., continuous version of the weights, s.t. the predicted label $f(\cdot)$ matches that of the training data. If $f(\cdot)$ is a harmonic extension of $u(\cdot, 0)$, the corresponding weights $v(\mathbf{x}, t)$ would be close to zero. This results in a simpler model and may generalize better from a model selection point of view.

\section{Numerical Results} \label{Experiments}
To validate the classification accuracy, efficiency and robustness of the proposed framework, we test the new architecture and algorithm on CIFAR10, CIFAR100 \cite{Cifar:2009}, MNIST\cite{MNIST:1998} and SVHN datasets \cite{SVHN:2011}.
In all experiments, we apply standard data augmentation that is widely used for the CIFAR datasets \cite{DRN:2016,Huang:2017CVPR,Zagoruyko:2016}. For MNIST and SVHN, we use the raw data without any augmentation. We implement our algorithm on the PyTorch platform \cite{paszke2017automatic}. All computations are carried out on a machine with a single Nvidia Titan Xp graphics card.

Before diving into the performance of DNNs with different output activation functions, we first compare the performance of WNLL with softmax on the raw input images for various datasets. The training sets are used to train the softmax models and interpolate labels for testing set in softmax and WNLL, respectively. Table \ref{Simple-Classifiers} lists the classification accuracies of WNLL and softmax on three datasets.
For WNLL interpolation, in order to speed up the computation, we only use 15 nearest neighbors to ensure sparsity of the weight matrix, and the 8th neighbor's distance is used to normalize the weight matrix. The nearest neighbors are searched via the approximate nearest neighbor (ANN) algorithm \cite{ANN:2014}. WNLL outperforms softmax significantly in all three tasks. These results show the potential of using WNLL instead of softmax as the output activation function in DNNs.

\begin{table}[!h]
\renewcommand{\arraystretch}{1.3}
\centering
\caption{Accuracies of softmax and WNLL in classifying some classical datasets.}
\label{Simple-Classifiers}
\centering
\begin{tabular}{cccc}
\hline
\footnotesize{Dataset} & \footnotesize{CIFAR10} & \footnotesize{MNIST} & \footnotesize{SVHN}\\
\hline
softmax & 39.91\%  & 92.65\%   & 24.66\% \\
WNLL    & 40.73\%  & 97.74\%   & 56.17\% \\
\hline
\end{tabular}
\end{table}

For the deep learning experiments below: We take two passes alternating steps, i.e., $N=2$ in Algorithm. \ref{alg-Train}. For the linear activation stage (Stage 1), we train the network for $n=400$ epochs. For the WNLL stage, we train for $n=5$ epochs. In the first pass, the initial learning rate is 0.05 and halved after every 50 epochs in training linear activated DNNs, and 0.0005 when training the WNLL activation. The same Nesterov momentum and weight decay as used in \cite{ResNet,Huang:2016ECCV} are used for CIFAR and SVHN experiments, respectively. In the second pass, the learning rate is set to be one fifth of the corresponding epochs in the first pass. The batch sizes are 128 and 2000 when training softmax/linear and WNLL activated DNNs, respectively. For fair comparison, we train the vanilla DNNs with softmax output activation for 810 epochs with the same optimizers used in WNLL activated ones. All final test errors reported for the WNLL method are done using WNLL activations for prediction on  the test set. In the rest of this section, we show that the proposed framework resolves the issue of lacking big training data and boosts the generalization accuracies of DNNs via numerical results on CIFAR10/CIFAR100. The numerical results on SVHN are provided in the appendix.

\subsection{Resolving the Challenge of Insufficient Training Data}
When we do not have sufficient training data, the generalization accuracy typically degrades as the network goes deeper, as illustrated in Fig.\ref{Degenerate}. The WNLL activated DNNs, with its superior regularization of the parameters and perturbation on bad local minima, are able to overcome this degradation.  The left and right panels plot the cases when the first 1000 and 10000 data in the training set of CIFAR10 are used to train the vanilla and WNLL DNNs.
As shown in Fig. \ref{Degenerate}, by using WNLL activation, the generalization error rates decay consistently as the network goes deeper, in contrast to the degradation for vanilla DNNs. The generalization accuracy between the vanilla and WNLL DNNs can differ up to 10 percent within our testing regime.

\begin{figure}[h]
\centering
\begin{tabular}{cc}
\includegraphics[width=0.25\columnwidth]{PreactResNet1000_Degenerate.png}&
\includegraphics[width=0.25\columnwidth]{PreactResNet10000_Degenerate.png}\\
(a)&(b)\\
\end{tabular}
\caption{Resolving the degradation problem of vanilla DNNs by WNLL activation. Panels (a) and (b) plot the generation errors when 1000 and 10000 training data are used to train the vanilla and the WNLL activated DNNs, respectively. In each plot, we test three different networks: PreActResNet18, PreActResNet34, and PreActResNet50. All tests are done on the CIFAR10 dataset.}
\label{Degenerate}
\end{figure}

Figure.\ref{Generation-Acc-Evolution} plots the evolution of generalization accuracy during training. We compute the test accuracy per epoch.
Panels (a) and (b) plot the test accuracies for ResNet50 with softmax and WNLL activations (1-400 and 406-805 epochs corresponds to linear activation), respectively, with only the first 1000 examples as training data from CIFAR10. Charts (c) and (d) are the corresponding plots with 10000 training instances, using a pre-activated ResNet50. After around 300 epochs, the accuracies of the vanilla DNNs plateau and cannot improve any more. In comparison, the test accuracy for WNLL jumps at the beginning of Stage 2 in first pass; during Stage 1 of the second pass, even though initially there is an accuracy reduction, the accuracy continues to climb and eventually surpasses that of the WNLL activation in Stage 2 of first pass.
The jumps in accuracy at epoch 400 and 800 are due to switching from linear activation to WNLL for predictions on the test set. The initial decay when alternating back to softmax is caused partially by the final layer $W_L$ not being tuned with respect to the deep features $\tilde{\mathbf{X}}$, and partially due to predictions on the test set being made by softmax instead of WNLL.  Nevertheless, the perturbation via the WNLL activation quickly results in the accuracy increasing beyond the linear stage in the previous pass.

\begin{figure}[h]
\centering
\begin{tabular}{cccc}
\includegraphics[width=0.23\columnwidth]{ResNet50_1000.png}&
\includegraphics[width=0.23\columnwidth]{ResNet50_1000_Acc_WNLL.png}&
\includegraphics[width=0.23\columnwidth]{PreActResNet50_10000.png}&
\includegraphics[width=0.23\columnwidth]{PreResNet50_10000_Acc_WNLL.png}\\
(a)&(b)&(c)&(d)\\
\end{tabular}
\caption{The evolution of the generation accuracies over the training procedure. Charts (a) and (b) are the accuracy plots for ResNet50 with 1000 training data, where (a) and (b) are plots for the epoch v.s. accuracy of the vanilla and the WNLL activated DNNs. Panels (c) and (d) correspond to the case of 10000 training data for PreActResNet50. All tests are done on the CIFAR10 dataset.}
\label{Generation-Acc-Evolution}
\end{figure}

\subsection{Improving Generalization Accuracy}

We next show the superiority of WNLL activated DNNs in terms of generalization accuracies when compared to their surrogates with softmax or SVM output activations. Besides ResNets, we also test the WNLL surrogate on the VGG networks. In table \ref{Cifar10}, we list the generalization errors for 15 different DNNs from VGG, ResNet, Pre-activated ResNet families on the entire, first 10000 and first 1000 instances of the CIFAR10 training set. We observe that WNLL in general improves more for ResNets and pre-activated ResNets, with less but still significant improvements for the VGGs. Except for VGGs, we can achieve relatively 20$\%$ to $30\%$ testing error rate reduction across all neural nets. All results presented here and in the rest of this paper are the median of 5 independent trials. We also compare with SVM as an alternative output activation, and observe that the results are still inferior to WNLL. Note that the bigger batch-size is to ensure the interpolation quality of WNLL. A reasonable concern is that the performance increase comes from the variance reduction due to increasing the batch size. However, experiments done with a batch size of 2000 for vanilla networks actually deteriorates the test accuracy.

\begin{table*}[!htbp]
\centering
\caption{Generalization error rates over the test set of vanilla DNNs, SVM and WNLL activated ones trained over the entire, the first 10000, and the first 1000 instances of training set of CIFAR10. (Median of 5 independent trials)}
\label{Cifar10}
\begin{tabular}{cccccccc}
\toprule
\footnotesize{Network} &  \multicolumn{3}{c}{\footnotesize{Whole}} &  \multicolumn{2}{c}{\footnotesize{10000}} & \multicolumn{2}{c}{\footnotesize{1000}}\\
\midrule
{}  & \textbf{\footnotesize{Vanilla}} &\textbf{\footnotesize{WNLL}} &\textbf{\footnotesize{SVM}} &\textbf{\footnotesize{Vanilla}}   &\textbf{\footnotesize{WNLL}}   &\textbf{\footnotesize{Vanilla}}   &\textbf{\footnotesize{WNLL}}  \\
\footnotesize{VGG11} 			&\footnotesize{9.23\%} & \footnotesize{{\bf 7.35\%}}  &\footnotesize{9.28\%}  &\footnotesize{10.37\%}&\footnotesize{{\bf 8.88\%}}   &\footnotesize{26.75\%} &\footnotesize{{\bf 24.10\%}} \\
\footnotesize{VGG13} 			&\footnotesize{6.66\%} & \footnotesize{{\bf 5.58\%}}  &\footnotesize{7.47\%}  &\footnotesize{9.12\%} &\footnotesize{{\bf 7.64\%}}   &\footnotesize{24.85\%} &\footnotesize{{\bf 22.56\%}} \\
\footnotesize{VGG16} 			&\footnotesize{6.72\%} & \footnotesize{{\bf 5.69\%}}  &\footnotesize{7.29\%}  &\footnotesize{9.01\%} &\footnotesize{{\bf 7.54\%}}   &\footnotesize{25.41\%} &\footnotesize{{\bf 22.23\%}} \\
\footnotesize{VGG19} 			&\footnotesize{6.95\%} & \footnotesize{{\bf 5.92\%}}  &\footnotesize{7.99\%}  &\footnotesize{9.62\%} &\footnotesize{{\bf 8.09\%}}   &\footnotesize{25.70\%} &\footnotesize{{\bf 22.87\%}} \\
\footnotesize{ResNet20} 		&\footnotesize{9.06\%} \footnotesize{(8.75\%\cite{ResNet})} & \footnotesize{{\bf 7.09\%}}  &\footnotesize{9.60\%}  &\footnotesize{12.83\%}&\footnotesize{{\bf 9.96\%}}   &\footnotesize{34.90\%} &\footnotesize{{\bf 29.91\%}} \\
\footnotesize{ResNet32} 		&\footnotesize{7.99\%} \footnotesize{(7.51\%\cite{ResNet})} & \footnotesize{{\bf 5.95\%}}  &\footnotesize{8.73\%}  &\footnotesize{11.18\%}&\footnotesize{{\bf 8.15\%}}   &\footnotesize{33.41\%} &\footnotesize{{\bf 28.78\%}} \\
\footnotesize{ResNet44} 		&\footnotesize{7.31\%} \footnotesize{(7.17\%\cite{ResNet})} & \footnotesize{{\bf 5.70\%}}  &\footnotesize{8.67\%}  &\footnotesize{10.66\%}&\footnotesize{{\bf 7.96\%}}   &\footnotesize{34.58\%} &\footnotesize{{\bf 27.94\%}} \\
\footnotesize{ResNet56} 		&\footnotesize{7.24\%} \footnotesize{(6.97\%\cite{ResNet})} & \footnotesize{{\bf 5.61\%}}  &\footnotesize{8.58\%}  &\footnotesize{ 9.83\%}&\footnotesize{{\bf 7.61\%}}   &\footnotesize{37.83\%} &\footnotesize{{\bf 28.18\%}} \\
\footnotesize{ResNet110} 		&\footnotesize{6.41\%} \footnotesize{(6.43\%\cite{ResNet})} & \footnotesize{{\bf 4.98\%}}  &\footnotesize{8.06\%}  &\footnotesize{ 8.91\%}&\footnotesize{{\bf 7.13\%}}   &\footnotesize{42.94\%} &\footnotesize{{\bf 28.29\%}} \\
\footnotesize{ResNet18} 		&\footnotesize{6.16\%} & \footnotesize{{\bf 4.65\%}}  &\footnotesize{6.00\%}  &\footnotesize{8.26\%} &\footnotesize{{\bf 6.29\%}}   &\footnotesize{27.02\%} &\footnotesize{{\bf 22.48\%}} \\
\footnotesize{ResNet34} 		&\footnotesize{5.93\%} & \footnotesize{{\bf 4.26\%}}  &\footnotesize{6.32\%}  &\footnotesize{8.31\%} &\footnotesize{{\bf 6.11\%}}   &\footnotesize{26.47\%} &\footnotesize{{\bf 20.27\%}} \\
\footnotesize{ResNet50} 		&\footnotesize{6.24\%} & \footnotesize{{\bf 4.17\%}}  &\footnotesize{6.63\%}  &\footnotesize{9.64\%} &\footnotesize{{\bf 6.49\%}}   &\footnotesize{29.69\%} &\footnotesize{{\bf 20.19\%}} \\
\footnotesize{PreActResNet18}   &\footnotesize{6.21\%} & \footnotesize{{\bf 4.74\%}}  &\footnotesize{6.38\%}  &\footnotesize{8.20\%} &\footnotesize{{\bf 6.61\%}}   &\footnotesize{27.36\%} &\footnotesize{{\bf 21.88\%}} \\
\footnotesize{PreActResNet34} 	&\footnotesize{6.08\%} & \footnotesize{{\bf 4.40\%}}  &\footnotesize{5.88\%}  &\footnotesize{8.52\%} &\footnotesize{{\bf 6.34\%}}   &\footnotesize{23.56\%} &\footnotesize{{\bf 19.02\%}} \\
\footnotesize{PreActResNet50} 	&\footnotesize{6.05\%} & \footnotesize{{\bf 4.27\%}}  &\footnotesize{5.91\%}  &\footnotesize{9.18\%} &\footnotesize{{\bf 6.05\%}}   &\footnotesize{25.05\%} &\footnotesize{{\bf 18.61\%}} \\
\bottomrule
\end{tabular}
\end{table*}

Tables \ref{Cifar10} and \ref{Cifar100} list the error rates of 15 different vanilla networks and WNLL activated networks on CIFAR10 and CIFAR100 datasets. On CIFAR10, WNLL activated DNNs outperforms the vanilla ones with around 1.5$\%$ to 2.0$\%$ absolute, or 20$\%$ to 30$\%$ relative error rate reduction. The improvements on CIFAR100 are more significant. We independently ran the vanilla DNNs on both datasets, and our results are consistent with the original reports and other researchers' reproductions \cite{DRN:2016,IdentityMap:2016,Huang:2017CVPR}. We provide experimental results of DNNs' performance on SVHN data in the appendix. Interestingly, the improvement are more significant on harder tasks, suggesting potential for our methods to succeed on other tasks/datasets. For example, reducing the sizes of DNNs is an important direction to make the DNNs applicable for generalize purposes, e.g., auto-drive, mobile intelligence, etc. So far the most successful attempt is DNNs weights quantization\cite{BinaryConnect:2015}. Our approach is a new direction for reducing the size of the model: to achieve the same level of accuracy, compared to the vanilla networks, our model's size can be much smaller.

\begin{table}[!h]
\renewcommand{\arraystretch}{1.3}
\centering
\caption{Error rates of the vanilla DNNs v.s. the WNLL activated DNNs over the whole CIFAR100 dataset. (Median of 5 independent trials)}
\label{Cifar100}
\centering
\resizebox{\textwidth}{!}{\begin{tabular}{cccccc}
\hline
\textbf{Network} & \textbf{Vanilla DNNs} & \textbf{WNLL DNNs} & \textbf{Network} & \textbf{Vanilla DNNs} & \textbf{WNLL DNNs}\\
\hline
VGG11           & 32.68\% & {\bf 28.80\%}  & ResNet110       & 28.86\% & {\bf 23.74\%}\\
VGG13           & 29.03\% & {\bf 25.21\%}  & ResNet18 	     & 27.57\% & {\bf 22.89\%}\\
VGG16           & 28.59\% & {\bf 25.72\%}  & ResNet34 	     & 25.55\% & {\bf 20.78\%}\\
VGG19           & 28.55\% & {\bf 25.07\%}  & ResNet50        & 25.09\% & {\bf 20.45\%}\\
ResNet20 		& 35.79\% & {\bf 31.53\%}  & PreActResNet18  & 28.62\% & {\bf 23.45\%}\\
ResNet32 		& 32.01\% & {\bf 28.04\%}  & PreActResNet34  & 26.84\% & {\bf 21.97\%}\\
ResNet44        & 31.07\% & {\bf 26.32\%}  & PreActResNet50  & 25.95\% & {\bf 21.51\%}\\
ResNet56        & 30.03\% & {\bf 25.36\%}  &                 &         &        \\
\hline
\end{tabular} }
\end{table}

\section{Concluding Remarks}
We are motivated by ideas from manifold interpolation and the connection between ResNets and control problems of transport equations. We propose to replace the classical output activation function, i.e., softmax, by a harmonic extension type of interpolating function. This simple surgery enables the deep neural nets (DNNs) to make sufficient use of the manifold information of data. An end-to-end greedy style, multi-stage training algorithm is proposed to train this novel output layer. On one hand, our new framework resolves the degradation problem caused by insufficient data; on the other hand, it boosts the generalization accuracy significantly compared to the baseline. This improvement is consistent across networks of different types and different number of layers. The increase in generalization accuracy could also be used to train smaller models with the same accuracy, which has great potential for the mobile device applications.

\subsection{Limitation and Future Work}
There are several limitations of our framework to improve which we wish to remove. Currently, the manifold interpolation step is still a computational bottleneck in both speed and memory. During the interpolation, in order to make the interpolation valid, the batch size needs to be quasilinear with respect to the number of classes. This pose memory challenges for the ImageNet dataset \cite{imagenet_cvpr09}. Another important issue is the approximation of the gradient of the WNLL activation function. Linear function is one option but it is far from optimal. We believe a better harmonic function approximation can further lift the model's performance.

Due to the robustness and generalization capabilities shown by our experiments, we conjecture that by using the interpolation function as output activation, neural nets can become more stable to perturbations and adversarial attacks \cite{Papernot:2015}. The reason for this stability conjecture is because our framework combines both learned decision boundary and nearest neighbor information for classification.

\clearpage
\section*{Acknowledgments}
This material is based on research sponsored by the Air Force Research Laboratory and DARPA under agreement number FA8750-18-2-0066. And by the U.S. Department of Energy,
Office of Science and by National Science Foundation, under Grant Numbers DOE-SC0013838
and DMS-1554564, (STROBE). And by the NSF DMS-1737770, NSFC 11371220 and 11671005, and the Simons foundation. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.

redpajama_set_name: RedPajamaArXiv


Passengers have described their terror after British Airways Flight BA2276 was engulfed in flames moments before take-off as it gathered speed on the runway of Las Vegas airport.
Passengers fled the aircraft after the pilot had brought it to a standstill after it had caught fire at McCarran International Airport in Nevada. Flames and black smoke consumed the undercarriage of the aircraft before firefighters brought the blaze under control.
Some of those on board said the preparations for take-off went normally and the plane was speeding along the runway, when a loud thud or explosion was heard.
"It all happened so quickly. I was jolted awake up by a huge thud. It took me a while to register what was going on. How long had I been asleep? Were we in the air? Were we still on the ground?" writes Guardian reporter Joshua Steinberg, who was aboard the flight.
He writes that passengers were initially told to remain seated, before they smelt burning.
"Our first move was towards the back of the plane. A flight attendant opened the door and had a look of utter panic on her face when smoke started to come into the plane. There were shouts to turn around and move towards the front. Some people looked chastened. There was a lot of smoke behind us and even the flight attendant was screaming at us to run."
Passengers then exited the plane from a door at the front, he writes.
"Once we reached the bottom, we ran. We didn't know where to go. We just ran. When we looked back and saw the other side of the plane the extent of the damage was clear." Some passengers described seeing windows melted by the intensity of the blaze.
Alex Crimmens, 29, was returning from holiday on the flight, which was scheduled to land at London's Gatwick Airport.
He told the Mirror that as the plane began to take off he heard a massive bang.
"People were understandably nervous, and then all of sudden I could hardly see out of my window because of all this smoke. We were travelling at the speed you'd expect a plane to be taking off at, so it was alarming."
As passengers evacuated the plane, he said there was increasing fear.
"Obviously people were worried, and in everyone's mind there's the fear that we've got a plane with smoke and flames and it's full of fuel -- we thought it could explode."
All 159 passengers on the plane and 13 crew members were able to get off the Boeing 777 safely. 14 people were taken to the Sunrise Hospital with minor injuries.
Aviation authorities delayed flights to the airport for up to two hours to stem the flow of air traffic, as the blackened aircraft blocked a runway.