text:
Puppy Refuses To Take A Compliment From His Dad
We've all seen the viral posts showing people before and after receiving a compliment. These people look so happy and genuinely happy after receiving a nice compliment. Apparently it's not just a human thing to have a positive reaction when being told they're funny, smart, or pretty. Our dogs can also have similar reactions. And there was one adorable pup who had the best response ever! It's impossible not to feel your heart melt while watching this video. I have to admit, I've watched it nearly five times because it's just so wholesome to see.
In the video, it starts off with the most precious little white French Bulldog puppy who happens to be wearing a red bandana. Adorable, right? Well, the puppy is seated on the couch just behind his owner, looking a little shy. When the owner asks the pup how he likes his red bandana, the little pup begins to nuzzle and lick his owner's neck and ear. But when his owner calls the puppy "handsome," it seems to boost this little guy's confidence a bit more.
The puppy then becomes more playful and extroverted. He starts to do what puppies do best – he barks a little, nibbles at his owner, and even plays peek-a-boo with the camera. It's just so cute! We assume that he's just learning how to take a compliment and needs a little bit more encouragement. Of course, his owner is more than happy to continue making this pup feel special because he continues to reassure his little pup that he's a handsome boy.
What do you think of the little puppy's reaction? Do you share similar moments with your dog? Do you think there is some benefits to chatting with your pup in the same way? Let us know!
PrevPreviousAdorable Sheepdogs In Training Meet Sheep For The First Time
NextTherapy Dog Picks Out A Different Toy Every NightNext
Sports Broadcaster Has Nothing To Commentate On Except His Dogs Eating Their Food
Sports fans everywhere are reeling from the recent pandemic's shut down of all matches everywhere.
Dog Drags A Sprinkler Inside To Beat The Heat
Dogs are known to be quite clever individuals as we've seen with plenty of online
Cafe In Australia Lets You To Play With Adoptable Puppies While You Drink Coffee
There are more and more establishments that are dog-friendly. But those are great for people


One of Imagine Prep's newest clubs is all about filling the halls with music. IPS Glee Club is a student-run club that introduces students to the joy of singing and performing.
The definition of the word glee is "exultant high-spirited joy" and that is exactly what the students and teachers in charge of the club are hoping to achieve.
Glee Club is open to both high school and middle schoolers, and is run by high schoolers Melody Steele, Kaih Whitfield and Annabelle Daigle. Jacob Culver, a math teacher at the Prep, and Alexandra Zito, an English teacher, oversee and coach the club members.
Zito has been helping out since the beginning of the year.
"I saw a great opportunity to work with some great young musicians," Zito said.
Her overall goal for the club is for everyone to step out of their comfort zone.
Having Zito around has definitely improved how the Glee Club functions. Her students are very excited for her to return next year.
The club has been running since the second semester of last year and was started by juniors Steele and Whitfield.
Whitfield said their goal for the club was to have our group come together and perform against other glee clubs.
Eighth Grader Izabel Boroski is an original member of Glee and has enjoyed being a part of the group. She has encouraged many of her friends to join the club and plans to return as a freshman next year.
Boroski said Glee Club has helped her to become more confident in front of people. It has also taught her how to have fun while learning how to improve her vocal skills. Since she has joined Glee Club, she has auditioned for solos and performed in duets and trios for the club.
Another original member of Glee is Eighth Grader Kaylin Hayes. She has also enjoyed coming to glee to form new friendships and get more comfortable singing in front of other people.
Her favorite part of the glee club is how interactive it is.
"When I first began singing I wasn't very comfortable, but by the end of the year I was more confident and singing came easier to me," Hayes said.
Unfortunately, Hayes plans to attend Valley Vista for high school and will not be returning to the Glee club next year.
Though some members come and go, the bond that students who joined glee club have built are unbreakable. Glee has become a place where people can feel comfortable about being themselves and learning about one of life's greatest treasures: music.


"The Wiz Live's" Shanice Williams followed the yellow brick road home to New Jersey on Tuesday.
The Rahway, NJ native paid a visit to Harmony House, "a transitional living facility that helps homeless families find permanent housing," in Newark to hand out Christmas gifts to local kids. Shanice had some help from "Follow the Rules" star Ja Rule.
The unlikely duo made some kids' Christmas wishes come true when they each rolled out new bikes for them.
The holiday giveaway was made possible by the Godfather of Soul's daughter Deanna Brown. Continuing her father's tradition of toy giveaways through the James Brown Family Foundation whose aim is to help those who are less fortunate.
Judging by the smiling faces and hugs they all met the mark.


text:
Contacts Maps My Bangor Site Search Cymraeg
A Welcome from the Head of School
Resident Ensembles
Bangor & the Area
Why Bangor?
Get ready for University
Applying to Bangor
Careers & Employability
UCAS Visit Days
Postgraduate Community and Research Environment
Media & New Media
Creative Bangor
Music Summer School 2020
Musicology Research Group
Welsh Music Research Group
Composition Research Group
Knowledge Partnerships
Journals and Editorial Boards
Choirs and Orchestras
Our full list of Media-related courses can be found here.
Interested in Postgraduate study? Take a look at the courses we offer here.
An Open Day is a great way to get to know Bangor University and find out what we have to offer. You'll have the chance to visi tthe department, talk to staff and students and view our facilities.
If you have any questions about the subjects we offer, our courses or the applications process, please contact our Admissions Team.
Tel. +44 (0)1248 382085 or E. media@bangor.ac.uk
Why study Film?
Want to turn that film that's in your head into a reality? Our school will provide you with all the tools you need.
We offer an exciting range of degrees and modules that will give you a theoretical and practical tools needed to put your vision into practice. The school is home to a dynamic and vibrant community of professionals and students who want to make and study films.
The department is fully kitted out with the latest facilities, including cameras, full lighting and sound equipment, digital video edit suites running Final Cut Pro and Final Cut Express on Apple Macs, and a cinema space with professional theatre lighting facilities for rehearsal and performance work.
At our Pontio Arts and Innovation Centre, you will benefit from a world-class facility to watch and even screen your own films. Every year, our filmmaking students showcase their productions to a paying audience. Working with the cinema coordinator, you will also have the opportunity to suggest and curate your own film evenings. Every year the university film society and students from the School of Music and Media curate their own film festival open to the public and industry invitees.
The School of Media and Music of which our Film Studies Programme is a part, is a member of MeCCSA, NAHEMI (National Association of Higher Education in Moving Image) and has formal partnerships with BAFTA.
The 2015/16 academic year saw the opening of a brand new arts and innovation centre, Pontio, which includes a 200-seater cinema.
The broad range of other subjects studied at the school, including screenwriting, performance, music and sound, and journalism, means that you'll never be short of actors, scriptwriters, musicians and critics to aid you, or publicise your magnum opus!
A career in Film
You will have exceptional employment prospects. Every year approximately 90% of film studies graduates find employment or undertake further study within six months of graduating (Destination of Leavers from Higher Education). Bangor film graduates work in the development and production of feature films and television, with credits like HBO (Watchmen Series), Doctor Dolittle, BBC, SKY, ITV, S4C, Channel 4 and international independent productions.
Our graduates' work has been recognised by awards from BAFTA and the Royal Television Society and has screened at world-class festivals, secured development from the BFI and BAFTA, and been published in titles like DETAILS, The Guardian. Student placements have included BBC, S4C, International Feature Films, Wales International Documentary Festival, Sheffield Documentary Festival, and Llangollen International Festival.
You will gain practical involvement and experience with organising events, including screenings., film festivals and special events. These bring together academic scholars and industry professionals with students. Recent examples include 50 Years of the New Wave 2001: Beyond 50, the only event of its kind in the UK to celebrate the fiftieth anniversary of this groundbreaking film and which played host to a range of people who worked on the film. Alien at 40 celebrated another remarkable movie, including those who designed its unforgettable special effects.
Jan Harlan, Stanley Kubrick's brother-in-law and producer regularly visits to Bangor to lecture and teach. We work very closely with The Kubrick Estate and the Kubrick Archive. We have collaborated with Festival No. 6. In conjunction with BAFTA, we will host Keeping Faith & Film Composer Samuel Sim.
Studying Film
Your Bangor Film Studies degree programme will be flexible. Your course will uniquely balance subject breadth with opportunities to specialise in film history, theory, screenwriting, film music or film production.
The school of Music and Media is also home to expertise in related subject areas, including composition for film and television, music, theatre and performance, journalism and video game design, meaning that you can combine multiple passions and choose your own pathway.
In your first year, you will be introduced to the essential concepts and ideas needed to study film at degree level. In Film History, you will learn about the ways in which the medium has developed technologically.
In Film Language, you will learn the skills necessary to analyse the film image in detail. In Film Criticism, you will learn about the fierce debates that have occurred about what's constitutes the medium's great works.
You will have the opportunity to learn about screenwriting and filmmaking, as well as to produce your own screenplays and short films.
In your second and third years you will be able to choose from a range of optional modules accordingly to your own interests:
Learn about what goes into the marketing and distribution of film and the way the industry has developed
Study closely films of renowned director Stanley Kubrick with one of the world's leading scholars in the subject
Think about the ways American history has been imagined on screen, or debate the ways in which cinema has engaged in debates about race, gender and politics.
There are opportunities to study films from all over the world, from a variety of different perspectives: aesthetics, politics, history, industry.
You can also explore, in more depth, your creative abilities in screenwriting, film production and performance, submitting a piece of creative work as your final assessment.
You will benefit from high-quality teaching, guided by the world-acclaimed expertise of staff who are not only leading academics in their fields, but are also practicing professionals, many of whom have come from the industry.
Film Staff
Professor Nathan Abrams lectures, write and broadcast widely (in English and Welsh) on UK and American popular culture, history film and intellectual culture. He co-founded Jewish Film and New Media: An International Journal and is a world-leading expert on the films of Stanley Kubrick. He teaches modules on Stanley Kubrick: Auteur, America on Film, and Film History.
Jo Wright is a filmmaker and creative producer, working in Wales and internationally.
She works with documentary film, photography, sound and archive collections, often working in collaboration with communities to produce projects that re-examine established narratives. Her work has been screened internationally, including at the Institute for Contemporary Art, The British Film Institute, British Council, Channel 4 and BBC. She teaches filmmaking.
A Location to Film
Nestled in a breath-taking location between the Snowdonia National Park and the Irish Sea, Bangor is one of the UK's most inspirational locations to study film. Bangor is 'the best university setting in the UK' according to The Independent's A-Z of Universities.
Bangor's location is so inspiring that it has brought HBO, Netflix, the BBC, S4C and a host of a major film companies to shoot here. Recent productions include Watchmen at Penrhyn Castle or The Crown in Caernarfon Castle.
As a student, you can take advantage of these major productions to learn valuable skills, or even be a paid extra, as well as use these locations to shoot your own productions.
The Film Studies programme is situated at the heart of the University campus, right next door to the BBC Wales Studios in Bangor. It offers a close-knit and friendly community atmosphere for students, alongside the bustle of cafes, shops and city life, as well as the beautiful scenery.
You are also less than two hours from Liverpool and Manchester, as well as their airports, and enjoy excellent direct train services to London, Birmingham and Cardiff, with the fast ferry to Dublin just half an hour away.
You will study in a creative atmosphere, working closely with world leading scholars and practising industry professionals and consultants, in a dynamic, friendly and professional learning environment.
We have an award-winning radio station, Storm FM, as well as extremely successful clubs and societies in areas such as film making, games and drama, as well as two student newspapers.
Film students play an active role in these societies, with the activities mainly located within the school.
The compact and well-appointed nature of the university campus allows a great deal of cross-disciplinary co-operation with other schools, and this has proved to be extremely beneficial to our Film students, as well as students from the other schools.
Studying for an undergraduate degree in the School is a rich and stimulating activity. The emphasis here is on creative exploration and critical understanding.
If you choose practical modules, we have specialist resources to support media production work.
As well as our extensive loans service for equipment such as digital recorders, video cameras, stills cameras and more, we have a range of industry-standard facilities:
Digital video edit suites running Final Cut Pro and Final Cut Express on Apple Macs Professionally equipped Radio studio
Audio editing suites running Adobe Audition
Cinema Space with Dolby Surround Sound
John Phillips Hall with a large performance space with professional theatre lighting facilities for rehearsal and performance work
Professional equipment and computing facilities
Video and Radio kits for use on location work in the field of radio and TV
Multi-media computer room
All these facilities are available to you in addition to the University resources which you have access to as one of our students.
Our new multi-million pound Pontio centre is a major social hub for students as well as an international centre for learning, innovation and the performing arts.
Bangor University, College Road, Bangor, Gwynedd LL57 2DG, Wales, UK
Email: marketing@bangor.ac.uk

redpajama_set_name: RedPajamaCommonCrawl


text:
Martyn Waghorn finding his feet at Derby County after tough start to Rams career
The summer signing from Ipswich Town scored his first goals for the club last week
Martyn Waghorn looks to be finding his feet at Derby County after a tough start to his Rams career.
Waghorn joined Derby from Ipswich Town last month in a deal believed to be worth £5m rising to £7.5m with add-ons.
The 28-year-old striker arrived at Pride Park on the back of a highly impressive season with Ipswich, where he scored 16 goals and provided 11 assists.
He made his Derby debut in the first home game of the 2018-19 campaign - but it proved to be a chastening experience as Frank Lampard's men were soundly beaten 4-1 by Leeds United.
Waghorn came on as a late substitute in the next match, the Carabao Cup first-round win at Oldham Athletic, and almost opened his account, only to be denied by the post in the dying moments.
Derby County match reports
Derby 0-0 Blackburn
Rotherham 1-0 Derby
Hull 1-2 Derby
He returned to the starting line-up for the following fixture, away to Millwall in the league, but again ended up on the losing side as Derby went down 2-1.
The visitors were made to pay for a poor start, even though they improved after the break and pulled a goal back through a fine strike from David Nugent, who replaced Waghorn midway through the second half.
The Rams responded by winning their next two matches but Waghorn had to watch the back-to-back home league victories over former club Ipswich and Preston North End from the bench.
He got his chance to impress in the Carabao Cup second-round tie against Hull City as Lampard made wholesale changes to his side.
Waghorn - operating in a wide attacking role - certainly made the most of his opportunity as he opened the scoring with his first goal for the club in fine style, deftly bringing down a pass from Bradley Johnson before lifting the ball ove the keeper.
Derby County live - Lampard talks Mount's England ambition, Elsnik on Mansfield move, plus more Rams transfer news
The former Sunderland and Rangers man then set up the second goal, winning back possession and feeding Florian Jozefzoon, who doubled the lead.
Derby cruised to a 4-0 victory and Waghorn had done enough to retain his starting spot when the Rams returned to Humberside for a league fixture a few days later.
Once again, he got the ball rolling, setting the Rams on their way to a 2-1 win from the penalty spot midway through the first half.
Waghorn was "delighted" to have scored for the second game running - and is looking to kick on now.
"As a striker, you want to come in and hit the ground running," he told Derbyshire Live.
Martyn Waghorn
Waghorn relishing spot of pressure
Waghorn reaction to Hull win
Waghorn opens up about Rams move
Waghorn almost joined Derby in 2011
"The first couple of games, for myself, didn't really go to plan.
"So to come in and score on Tuesday and again here is really pleasing.
"It's always going to take time to settle when you come to a new club and into a big squad.
"There's a lot of competition - great competition - we've got strength in depth all over the pitch.
"It takes a bit of time but I'm getting to know the lads a bit more, how they play and how the gaffer wants us to play.
poll loading
Who has been Derby County's best summer signing so far?
2000+ VOTES SO FAR
Harry Wilson
Florian Jozefzoon
Jack Marriott
George Evans
Scott Malone
Duane Holmes
"Hopefully, I can build on the last couple of games and go from there.
"I'm the kind of player that will work hard and give my all.
"Sometimes, you have those off days but I like to give my all for the team.
"And it's always rewarding to get the three points for the fans."
Waghorn has been in the process of moving his family to the Midlands from East Anglia.
Martyn Waghorn scores from the penalty spot against Hull (Image: Andy Clarke)
"It's been a difficult few weeks away from them but we're getting that sorted now," he added.
"The schools are starting up again and my little boy will be in there.
"And my wife and family are moving up, which is nice."


Another great year – thank you Denbies, and thank you to all our exhibitors and visitors for 2019.
Returning to Dorking, you can once again be in the Garden Room at Denbies Wine Estate on Sunday 7th April showcasing a sensational selection of chocolatiers and chocolate makers who are exhibiting with us. Denbies has a wide range of options for the day, including a conservatory restaurant, a farm shop, vineyard tours, an art gallery, and a micro brewery too.
This year the Chocolate Fair is proud to be sponsored by Pure Grenada – Spice of the Caribbean, who we hope will bring along a little sunshine (plus chocolate & rum).
This year we are proud to announce our sponsors 'Pure Grenada' will be hosting a Taste With Colour® workshop.
Hazel Lee, creator of Taste With Colour®: The Chocolate Tasting Flavour Map, is also a Chocolate Consultant, member of the Academy of Chocolate, judge for the Academy of Chocolate Awards, Northwest Chocolate Awards and Premio Bean to Bar Brasil, along with being a home bean-to-bar chocolate maker and 'Cocoa Adventurer'. Pure Grenada will also be highlighting just a few of the reasons for adding Grenada to your holiday destination wish-list. Tickets for this workshop are £5.00 per person and include everything you will need: paint, paper, brushes, chocolate samples, rum – and a glass of wine. Duration 55 minutes.
An age-limit is not imposed but please note that under 18's will not be able to sample any alcohol during the session but will still need a ticket to attend.
Back by popular demand, we have a 'Meet the Maker' talk where you will be in a private session with award-winning chocolatiers Fifth Dimension Chocolates.
Discover their story, and uncover their journey in to chocolate. With a selection of samples you will discover how to taste chocolate, what to look for in a bar, and why it is important to know where your chocolate is coming from in order to maintain a future supply of quality produce. Water will be supplied throughout, and a glass of Denbies wine to finish. Tickets for this talk are £5.00 per person and include everything you will need: the chocolatier, chocolate samples and a glass of wine. Duration 45 minutes.


Fun fun fun without the cavities!
I'm done. Don't bother with this app. Does nothing but lock up for days. If I could give less than 1 star I would !!!! Freezes when the bonus wheel it lands on the jackpot Freezes and the only way to get in frozen is to delete and reinstall which I ask why bother to synchronize with Facebook, it has never reinstalled with the level completed, prizes you've won, or lives your friends you. The latest freeze occurred yesterday and it remained even after rebooting iPad.
This is beyond an addiction, can't put it down!
Overall it's fun and addictive game to play. Especially fun to play when you have others who play as well to give you necessary lives! Lol like I said addicting.
I love the game and am playing it nonstop!!


text:
Great Expectations-part 2 این آلبوم بهادار می‌باشد
The readers
: زبان های خارجی
From the opening passage itself of Great Expectations by Charles Dickens, the reader is drawn into the world of the hero, Pip, who is at that time, seven years old. The author creates an unforgettable atmosphere: the gloom of the graveyard, the melancholy of the orphan boy, the mists rising over the marshes and the terrifying appearance of an escaped convict in chains. Told in first person (one of the only two books that Dickens used this form for, the other being David Copperfield) Great Expectations is a classic coming of age novel, in which we trace the growth and evolution of Pip or Philip Pirrip to give his full name. Pip has lost his parents very early in life and is being brought up by his much older sister and brother-in-law Joe Gargery. His sister is a dominating and shrewish woman, while Joe is an affectionate man. Joe's uncle Mr Pumblechook (another of Dickens' delightfully evocative names) asks Joe to send Pip to the stately mansion Statis House ostensibly to play with the owner Miss Havisham's adopted daughter Estella. This marks the beginning of a life-changing experience for Pip. Miss Havisham is a wealthy eccentric whose twisted agenda is that of taking revenge on the man who jilted her at the altar. Pip is unwittingly drawn into her malevolent schemes. When Pip grows older, Miss Havisham's lawyer Mr Jaggers informs Pip that he has come into a large fortune from an unknown benefactor and that he must move to London and learn to become a "gentleman." What follows is a strange and labyrinthine tale of Pip's journey into adulthood.
بعدازظهری در پارک (انگلیسی)
A Princess of Mars - Part 1
Dead Men's Money - Part1
Dead Men's Money - Part 3


text:
(function(){

	angular.module('archivoApp', [])

	.controller('tablaCtrl', function($scope, $http)
	{
		$scope.curPage = 0;
	 	$scope.pageSize = 5;
	 	$scope.difuntos = [];

		$http.get("http://localhost/panteones/archivo/getrows")
		.then(function (response) {
			$scope.difuntos = response.data;
		});

	 	$scope.numberOfPages = function() {
			return Math.ceil($scope.difuntos.length / $scope.pageSize);
		};

	})

	.filter('pagination', function()
	{
	 return function(input, start)
	 {
	  start = +start;
	  return input.slice(start);
	 };
	});

})();

redpajama_set_name: RedPajamaGithub


text:
Holly City Hall
Built as a WPA project in 1938, the Holly City Hall housed the fire and police departments, jail, and library for many decades and is now a museum operated by the Holly Historical Society.
Located at the corner of East Cheyenne and South Third Streets, the Holly City Hall was built in 1938 as a Works Progress Administration (WPA) project designed to consolidate the town of Holly's administrative offices and departments. It never accomplished that goal, but the town fire and police departments, jail, and library occupied the one-story building for about six decades. In 2003 the building was listed on the National Register of Historic Places, and the Holly Historical Society now uses it as a local history museum.
A City Hall for the Town of Holly
Before the 1930s, the town of Holly's administrative offices moved around a lot. Town government offices originally operated out of a building on Main Street before moving to a nearby residence and then, in 1929, to a rented room. In 1932 the town secured a grant to buy land on East Cheyenne Street and build a town office. In January 1938 the town of Holly submitted a proposal for a WPA project to construct a new building that would consolidate all the town's administrative offices and departments. The proposal was approved that summer and work began immediately on a site across East Cheyenne Street from the existing town office.
The Holly City Hall cost nearly $18,000 and opened on September 18, 1938. The rectangular, one-story building had a spare style with almost no ornamentation. Like the Holly Gymnasium—the town's other WPA project—it was made using local Niobrara limestone, which could be cut easily but hardened when exposed to air. Lintels and sills were made of local sandstone. The main entrance faced south onto East Cheyenne Street. Inside, the building had a T-shaped plan, with two large front rooms on the south end separated by a central hall. At the north end of the building, smaller rooms and offices opened off a hall that ran from east to west. The building's northeast corner had a garage that opened onto South Third Street.
The building had "City Hall" painted in block letters over the entry and carved into the cornerstone, but the name was a misnomer in two ways. First, Holly was a town, not a city; second, the town board and administrative offices never moved into the building. The town board used the southeast room for a short time in 1943 but then returned to the older town office across the street. Nevertheless, the City Hall building became a civic and social hub. In 1938 the police and fire departments moved into the rear of the building, which included two jail cells and an office. Two years later, the public library moved into the building's large southwest room. Across the hall, the meeting room intended for the town board became a community room used for club meetings, elections, parties, and other events.
In 1972 the fire department moved to a new building, and in 1998 the police department also moved. In 2000 the library moved to the newly renovated Holly Santa Fe Depot, which also houses the town hall and other municipal offices. The empty City Hall building started to be used for storage.
In 2003 the Holly City Hall was listed on the National Register of Historic Places. In 2008 the Holly Historical Society received a State Historical Fund grant to restore the building's exterior and soon started to use the building as a local history museum that occasionally opened for special events. In 2016 the historical society renovated the building's interior to make it suitable for year-round use, with the goal of opening the museum on a regular basis in May 2017.
Patt Campbell, "Holly City Hall," National Register of Historic Places Registration Form (June 4, 2003).
Abigail D. Christman, The Legacy of the New Deal on Colorado's Eastern Plains (Denver: Colorado Preservation, 2008).
Ava Betz, A Prowers County History (Lamar, CO: Prowers County Historical Society, 1986).
Colorado Preservation Inc., Drought, Depression, and Dust: The New Deal in Eastern Colorado, Phase I Survey Report (2005).
James F. Wickens, Colorado in the Great Depression (New York: Garland, 1979).
Works Progress Administration
Niobrara limestone
American Legion Hall (Eads)
Collegiate Peaks Stampede Rodeo Grounds
Englewood Post Office
Florence Post Office
Hartman Gymnasium
Holly Gymnasium
Holly Santa Fe Depot (Town Hall)
Indian Reorganization Act (Indian New Deal)
Julesburg Public Library
La Junta City Park
Lamar Post Office
Lands End Observatory
Morgan County Courthouse and Jail
New Deal in Colorado
Notch Mountain Shelter
Prowers County Welfare Housing
Red Rocks Park and Amphitheatre
Tigiwon Community House
Wiley Rock Schoolhouse
Willow Creek Park


text:
Rockets star Russell Westbrook speaks out on facing an 'unbelievable talent' like Ja Morant
By Reid Goldsmith · 1 min read
Memphis Grizzlies rookie point guard Ja Morant already has a huge fan in former NBA MVP point guard Russell Westbrook of the Houston Rockets.
Ahead of Houston's Wednesday bout with Memphis, Westbrook called the leading Rookie of the Year candidate an "unbelievable talent" when asked about his thoughts on facing Morant.
"I never worry about my opponent or what they're doing. I know when I'm at my best I'm able to do whatever I need to do. Ja is going to be a great player. He's an unbelievable talent," Westbrook said, via Jonathan Feigen in the Houston Chronicle.
"He's been having a great year. But it doesn't matter who I go against, what day or what time, I don't really worry about or spend time on what they're doing. I worry about myself, how i can affect my team. That's it."
Ja Morant's legend grows with another incredible franchise record vs. Blazers
Rexwell Villas · 6 hours ago
Ja Morant is averaging 17.6 points, 3.4 rebounds and 6.8 assists per game in 51 appearances in his debut season in the NBA out of Murray State. The second overall pick of the 2019 NBA Draft has miraculously led the young Grizzlies to the last spot in the Western Conference playoff field at the moment, as Memphis clings to a 2.5 game advantage for the eight seed.
Russell Westbrook was once the young, hotshot floor general with springy hops out of UCLA as a point guard drafted in the lottery for the then-Seattle SuperSonics in their relocation to Oklahoma City. Westbrook was traded in a blockbuster deal to the Rockets this past offseason for Chris Paul and draft picks, reuniting the two-time All-Star Game MVP with his former Thunder backcourt teammate in James Harden.
Tags: Houston Rockets, Ja Morant, Russell Westbrook
R.P. Salao · 1 hour ago
Jesseyriche Cortez · 2 hours ago


text:
\section{Introduction}

With stationary training resources and various advanced neural network structures, deep learning models have exceeded human performance in many areas. However, a well-known limitation of deep learning models is the so-called ``catastrophic forgetting.'' That is, while acquiring new knowledge, the models may forget the knowledge learned in the past. In other words, after tuning a pre-trained model with new data, the model can have a poor performance on patterns learned from old data.

In this paper, we investigate the possibility of a model to generalize from old patterns and new patterns. Towards formalizing the setting of our study, we assume that we have a model with a generic loss function, and it has been well-trained on a big dataset. Our aim is that, upon presenting a new dataset,
the model can be adjusted so that it can handle the new dataset well -- while simultaneously  not forgetting  the patterns learned from the old dataset. In many real-world systems, new and old data are commonly stored in a buffered manner, where the model always has access to a roughly fixed size of training data. This fact is at the core of what differentiates our model retraining from traditional continual learning with bounded computational complexity. In our study, the old and the new datasets share the same prediction space. For instance, the number of classes in the two datasets are the same, but the background and object angles may differ.

The issue of catastrophic forgetting in neural networks had been addressed in the literature (cf.~\cite{french1999catastrophic, kirkpatrick2017overcoming}). Once again, the source of the problem is that when neural networks are trained continually on datasets with different feature distributions and/or trained for different objectives, the knowledge learned in an old training session can be lost in a new training session ~\cite{yosinski2014transferable}.

We note that our study aligns closely to continual/incremental learning and multi-task/sequential learning. Continual learning focuses on tuning a model with continual new data without forgetting old knowledge~\cite{albesano2006adaptation, silver2013lifelong, ruvolo2013ella, li2017learning, DBLP:journals/corr/abs-1802-07569}.
Compared to continual learning, multi-task learning focuses on more dissimilar tasks~\cite{Ruder2017, zhang2017, henderson2017benchmark, rusu2016progressive}, e.g., incremental class learning~\cite{roy2018tree, rebuffi2017icarl, kemker2017fearnet, Ganegedara2017, li2018}, where different tasks have different classes. Moreover, different tasks usually only have access to new data~\cite{aljundi2018selfless} or just a small portion of old data~\cite{lopez2017gradient}. From the training objective aspect, multi-task learning solves task-specific problems: parameter space shifting using priors~\cite{lee2017overcoming, nguyen2018variational, swaroop2019improving}, path traversal among different tasks~\cite{lopez2017gradient, Fernando2017}, parameter sharing in different tasks~\cite{Elliot2018}, and hard attentions~\cite{serra2018overcoming} or masks~\cite{mallya2018packnet} for switching tasks. Their contributions address challenges in different experimental settings~\cite{Maltoni2018}, such as how data in different tasks are distributed, how tasks are ordered~\cite{Meyerson2018, standley2019tasks}, and how performance should be evaluated among a sequence of tasks.

Our study may be perceived as a special case of continual learning -- however, we note that our primary focus and data setting differ from continual learning and, especially, sequential learning. Our retraining models aim to have good performance on both new and old data. In continual learning and sequential learning, a model can only access new data. However, when tuning a pre-trained model in real-world systems, the model should access some old data to guarantee adequate retraining. It may not be feasible in the long run to train on the union of all old data and new data. In our study, old data and new data are stored in a buffered manner. Given limited computation resources that can process a fixed capacity of data, we feed the same full capacity of training data during training or retaining. In every future retraining process, a trained model is tuned on a fixed-size data set, which is composed of the new dataset and a subset of the old dataset. We boost the performance of a trained model by gradient boosting and by learning a meta-network. The two methods are generic and can be applied to any neural network models and loss functions.

Our major contributions are that (1) we propose an additive ensemble model that can tune a pre-trained model based on its performance gap, and (2) we further propose a meta-network model that can balance new patterns and old patterns and eventually keep the number of parameters unchanged.

\section{Related Work}
\label{Related Work}
We now detail the major directions of continual learning: regularization, ensemble methods, and memory consolidation.

\subsection{Regularization}
Regularization is the most common way of solving catastrophic forgetting via approaches that measure the importance of weights of a trained model. The elastic weight consolidation (EWC) algorithm proposed in~\cite{kirkpatrick2017overcoming} uses the Fisher matrix to represent weight importance. A regularization term is added to the original loss function to constrain important weights to stay close to their old values when retraining on a new session/task. Despite the usefulness of the Fisher matrix, the diagonal Fisher matrix assumption may not hold; therefore, approximations of the Fisher Matrix~\cite{tu2016ranking, liu2018rotate, Schwarz2018, zeno2018task} and additional KL-divergence constraints~\cite{chaudhry2018riemannian} were applied in later studies. Besides the Fisher matrix, gradient magnitude~\cite{aljundi2018memory}, unsupervised self-organizing maps~\cite{kohonen1990self}, and the Hessian matrix~\cite{zenke2017continual} are common techniques for measuring weight importance. In particular, the memory aware synapses (MAS) model~\cite{aljundi2018memory} has become a new widely-used benchmark after EWC. MAS uses the gradient magnitude of network outputs with regard to weights to measure weight importance. In contrast, the Fisher matrix is based on loss and weights in EWC. When outputs are multi-dimensional, as in classification tasks, MAS calculates the squared l2 norm of outputs. An improvement over EWC and MAS by adding an additional regularization term on neurons was presented in~\cite{aljundi2018selfless}, proposing that sparsity at the neuron level was also important. We show in our experiments that our ensemble models perform better than this regularization strategy.

\subsection{Ensemble Methods}
Intuitively, ensemble models try to solve catastrophic forgetting by training multiple sub-networks~\cite{polikar2001learn++, Dai2017, hinton2015distilling, chen2017learning, wang2017model, YuSlimmable, Luong2016}. Ren et al.~\cite{Ren2017} and Juefei-Xu et al.~\cite{juefei-xu2018pnn} made breakthroughs by proposing dynamic ways of constructing an ensemble model by tuning a sub-network of a pre-trained model or expanding a pre-trained model. The general limitation is that the memory usage scales up with the number of training sessions. Related meta-learning works generally focus on learning network architectures in order to adjust a pre-trained model to be robust on different datasets and tasks~\cite{yoon2018lifelong, lin2013network, Rusu2016, xu2018reinforced, javed2019meta, juefei-xu2018pnn}. For instance, knowledge distillation~\cite{hinton2015distilling} tries to distill knowledge learned from a large (teacher) network to a small (student) network. The goal is to make the teacher network and the student network behave as similarly as possible. However, our goal is to learn a more generalized network continually, being able to fix errors in the old trained model by acquiring new data.

\subsection{Memory Consolidation}
Memory consolidation solves catastrophic forgetting from the data aspect. Memory consolidation models~\cite{kemker2017fearnet, lopez2017gradient, Nitin2017, chaudhry2018efficient, chaudhry2019continual, Mehta2018, Sodhani2019} learn to store the patterns/memories of different training sessions by strategically sampling a subset of the old data. Generative Adversarial Networks (GANs) are suitable in nature for learning networks~\cite{shin2017continual, Seff2017, Rios2018, Thanh-Tung2018, liang2019generative, Aljundi2019, Subakan2019} in an incremental way. A generator trained on old data can be used in a new training session for replaying old learning experiences. Our study solves a more generic issue from the modeling aspect, and memory consolidation can be applied to our models for providing a reasonable data buffer.

\begin{figure*}[pt]
\begin{subfigure}[b]{0.5\textwidth}
  \includegraphics[width = \textwidth]{BoostNet}
  \caption{BoostNet}
  \label{fig:BoostNet}
\end{subfigure}\hspace{0.1\textwidth}%
\begin{subfigure}[b]{0.5\textwidth}
  \includegraphics[width =0.86\textwidth]{FrostNet}
  \caption{FrostNet}
  \label{fig:FrostNet}
\end{subfigure}
\end{figure*}

\section{Proposed Models}
\label{Proposed Models}

In this section, we present two ways of ``frosting'' weights for better retraining. One is based on classical gradient boosting, the other is based on neural network meta-learning.

\subsection{Frosting Network Weights by One-step Gradient Boosting}
\label{Frosting Network Weights by One-step Gradient Boosting}

We train our proposed network BoostNet, which can be simpler or the same as the trained network. We firstly apply a retraining dataset, composed of new training data and a subset of the old training data, to the trained model and calculate the model outputs and ``residuals''~\cite{friedman2001greedy}. Let $L$ denote the softmax cross-entropy loss function, $o_i$ denote the output of the $i^{th}$ neuron before the softmax layer, $p_i$ denote the softmax probability of the $i^{th}$ neuron, and let $y_i$ denote the one-hot label. The residuals are calculated as $\frac{\partial L}{\partial o_i} = y_i - p_i$, measuring how much the trained model needs to improve in order to perform well on the retraining dataset. It is very likely that the residual values are close to 0 for the old training data, since the loss was originally minimized, and the residual values for the new training data are relatively larger. The weights of the trained model stay unchanged. Figure~\ref{fig:BoostNet} illustrates this training procedure. The BoostNet takes the residuals as the training targets (``labels''). The loss function for the simpler model is the mean squared error (MSE) loss. In the end, we sum up the outputs of the old trained model and the outputs of the BoostNet model for future inference. This method does not change the trained model but adds new weights, which come from the simpler model. The limitation is that the number of total weights increases every time a new dataset comes in. While in knowledge distillation (where a student model is trained to mimic the behavior of the teacher model), a base network was also trained and kept frozen~\cite{hinton2015distilling}, BoostNet aims to have different training targets  -- i.e., errors from former learners.

\subsection{Frosting Network Weights by Meta-learning}
\label{Frosting Network Weights by Meta-learning}

We now explain our second ensemble model that can keep the number of parameters unchanged for inference. We add ``frosting'' layers for weights in a pre-trained model by taking weights as ``neurons.'' More specifically, we train a meta-network for weights in addition to the trained network. Both weights in the meta-network and weights in the base network are optimized during re-training. We have attempted freezing weights in the base network, but optimizing both networks yielded faster convergence. Each frosting layer is of the same shape as its corresponding weights. The weights in the frosting layers and the weights in the pre-trained model are both updated during retraining. In the end, the frosting layers are merged with the original weights so that the total number of weights is still unchanged. The merging operation is done by replacing the weights in the base network with the product of those weights and the weights in the corresponding frosting layer -- after which the frosting layers are discarded. The base network with the updated weights is used for inference -- thus, there is no performance loss after the merging operation. We call this ensemble model FrostNet, and it is illustrated in Figure~\ref{fig:FrostNet}. We have attempted common activation functions such as $tanh$ and $ReLU$; $tanh$ had a better performance compared to $ReLU$ in our experiments. Although $ReLU$ has many good metrics when applied to neurons, we do not find that it contributes to our frosting layer activation. Regardless, we have found that simply applying the frosting layers to trained weights without any activation functions already outperformed the benchmark models in our experiments.

\section{Experiments}

We now discuss the experiments used in evaluating the benefits of the proposed approaches.

\subsection{Datasets}
We use the MNIST and the CIFAR-10 datasets for comparing our BoostNet and FrostNet models with benchmark models. Similar to the data variation experiments in continual learning, we create a retraining dataset from the training set of the MNIST or CIFAR-10 by taking a random half of the original training set and augmenting the other half. The random half is taken by sampling each mini-batch during retraining instead of keeping them fixed before retraining. We obtain the augmented validation set using the same method. To this end, the sizes of the training and the validation sets stay unchanged in the initial training and retraining. We augment the entire test set as the new test set. The augmentations were conducted on the original training, validation, and test data separately to avoid data leakage. The augmentation factors are randomly rotating images by 45 degrees (clockwise and counterclockwise), shifting images by 20 percent (left and right), and zooming in by 80-90 percent. We have found that a well-trained model on the original MNIST or CIFAR-10 performs poorly on this augmentation setting.

Similarly to incremental class experiments in sequential learning, we split both the original MNIST and CIFAR-10 training datasets into two parts: the first part has classes 0-4; the second part has classes 5-9. We use the first part for training a LeNet model. We then take a random half of the first and the second parts during retraining. The original validation and test datasets are split into two parts under the same setting. Finally, we obtain a retraining dataset that is of the same size as the initial training dataset.

\begin{table}[t]
\centering
\begin{tabular}{|l|r|r|r|}
\hline
                    & Original & Augmented & Total   \\ \hline
Base                & 0.9921  & 0.2829  & 0.6352  \\ \hline
Train-from-scratch  & 0.9870  & 0.7547  & 0.8699  \\ \hline
Fine-tuning         & 0.9921  & 0.2662  & 0.6267  \\ \hline
EWC                 & 0.9890  & 0.8117  & 0.8995  \\ \hline
MAS                 & 0.9889  & 0.6907  & 0.8385  \\ \hline
Selfless            & 0.9860  & 0.3036  & 0.6430  \\ \hline
BoostNet            & 0.9891  & 0.8830  & 0.9352  \\ \hline
FrostNet            & \textbf{0.9920}   & \textbf{0.9012}  & \textbf{0.9818} \\ \hline
\end{tabular}
\caption{Variational MNIST}
\label{MNIST-result}
\end{table}

\begin{table}[t]
\centering
\begin{tabular}{|l|r|r|r|}
\hline
                    & Original & Augmented & Total   \\ \hline
Base                & 0.6599  & 0.3013   & 0.4810  \\ \hline
Train-from-scratch  & 0.6077  & 0.4309   & 0.5155  \\ \hline
Fine-tuning         & 0.6207  & 0.4463   & 0.5200  \\ \hline
EWC                 & 0.6191  & 0.4640   & 0.5415  \\ \hline
MAS                 & 0.6094  & 0.4363   & 0.5230  \\ \hline
Selfless            & 0.6198  & 0.3397   & 0.4795   \\ \hline
BoostNet            & 0.6363  & 0.4580   & 0.5426   \\ \hline
FrostNet            & \textbf{0.6456}  & \textbf{0.4646}   & \textbf{0.5552} \\ \hline
\end{tabular}
\caption{Variational CIFAR-10}
\label{cifar-10-result}
\end{table}

\begin{table}[t]
\centering
\begin{tabular}{|l|r|r|r|}
\hline
                    & Classes 0-4 & Classes 5-9 & Total   \\ \hline
Base                & 0.9984  & 0.0000  & 0.4992  \\ \hline
Train-from-scratch  & 0.9957  & 0.9562  & 0.9760 \\ \hline
Fine-tuning         & 0.9953  & 0.9626  & 0.9790 \\ \hline
EWC                 & 0.9949  & 0.9562  & 0.9756  \\ \hline
MAS                 & 0.9916  & 0.9008  & 0.9462  \\ \hline
Selfless            & 0.9940  & 0.9074  & 0.9507  \\ \hline
BoostNet            & \textbf{0.9973}  & 0.9628  & 0.9800  \\ \hline
FrostNet            & 0.9963   & \textbf{0.9665}  & \textbf{0.9814} \\ \hline
\end{tabular}
\caption{Split MNIST}
\label{MNIST-Split-result}
\end{table}

\begin{table}[t]
\centering
\begin{tabular}{|l|r|r|r|}
\hline
                    & Classes 0-4 & Classes 5-9 & Total   \\ \hline
Base                & 0.7590  & 0.0000   & 0.3795  \\ \hline
Train-from-scratch  & 0.4359  & \textbf{0.6518}   & 0.5439  \\ \hline
Fine-tuning         & 0.6010  & 0.5639   & 0.5824 \\ \hline
EWC                 & 0.5921  & 0.5520   & 0.5721  \\ \hline
MAS                 & 0.6402  & 0.5184   & 0.5230  \\ \hline
Selfless            & \textbf{0.6535}  & 0.4344  & 0.5440  \\ \hline
BoostNet            & 0.6252   & 0.5791  & 0.6022  \\ \hline
FrostNet            & 0.6498   & 0.5693  & \textbf{0.6096} \\ \hline
\end{tabular}
\caption{Split CIFAR-10}
\label{cifar-10-Split-result}
\end{table}

\subsection{Experimental Settings}
\label{Experimental Settings}
We compare our models to three baseline models: the initial LeNet model (base), training a new model (train-from-scratch), fine-tuning from pre-trained weights (fine-tuning), and three state-of-the-art models: EWC, MAS, and Selfless~\cite{aljundi2018selfless}. We use LeNet with batch normalization as the network framework. The pre-trained weights from the base model are used for measuring weight and neuron importance for EWC, MAS, and Selfless. All benchmark models are tuned based on the optimized weights from the base LeNet model. We use the truncated normal distribution for weight initialization for all network models. All models are trained at most 50 epochs with an early stopping of 10 consecutive epochs. The weights that have the highest accuracy and, secondarily, the lowest loss value (if the accuracy does not improve) on the validation dataset are used for inference. We report the accuracy of the corresponding test datasets in tables \ref{MNIST-result}--\ref{cifar-10-Split-result}. Boldface indicates the highest value in each column except the base model. We publish the code for our experiments on GitHub\footnote{https://github.com/XiaofengZhu/frosting\_weights}.

\section{Results}
\label{Results}

We average the accuracy of each model using three runs. Tables \ref{MNIST-result}-\ref{cifar-10-result} summarize the performance of the data variation experiments using the baseline models, the benchmark models, and our ensemble models. The base model has the highest accuracy on the original test datasets and the lowest accuracy on the augmented test datasets. Due to early-stopping and data augmentation, fine-tuning and benchmark models do not necessarily outperform train-from-scratch models. All the models except the base one suffer from catastrophic forgetting to some degree as the accuracy on an original test dataset is lower than the accuracy using the base model. All benchmark models have higher accuracy on the augmented dataset compared to the baseline models. EWC, MAS, and Selfless can prevent a trained model forgetting old knowledge, but their common limitation is that they have trouble learning from new knowledge. Our FrostNet model performs best among all benchmark models, and our BoostNet comes second. The FrostNet model yields the highest accuracy on the original datasets and the augmented datasets as the FrostNet model uses a network to learn how to adjust weights. The BoostNet model learns from performance gaps and pushes its weights to minimize the gaps.

Tables \ref{MNIST-Split-result}-\ref{cifar-10-Split-result} summarize the incremental class experiments. The base model has the highest accuracy on the datasets of classes (0-4) and the lowest accuracy on the ones of classes (5-9). Since the initial training dataset only has classes (0-4), the trained base model cannot predict any data from classes (5-9). Training-from-scratch on the retraining dataset has noticeable catastrophic forgetting on the initial training dataset. Although EWC, MAS, and especially Selfless models have an advantage in reducing catastrophic forgetting, they have inferior results on the new classes (5-9). Our BoostNet and FrostNet models can balance memorizing old knowledge and learning new knowledge, with a good overall performance.

\section{Conclusion}
\label{Conclusion}
In this paper, we propose two generic ensemble methods for boosting the performance of retraining. Regarding parameter capacity, BoostNet slowly adds a small number of parameters to a pre-trained model, similar to adding new trees in random forest modeling. Although FrostNet adds frosting layers for trained weights, the merging operation brings the number of total parameters down to the original size. Our work can be extended to other fields, such as general continual learning and network compression. Our work can be improved by taking the time series characteristic of training into account to further boost the performance. An interesting future direction would be strategically selecting training data. Another interesting direction would be selecting synapse paths from a sophisticated network model trained for a large domain to specialize for small domains.

\bibliographystyle{abbrv}

\section{Introduction}

With stationary training resources and various advanced neural network structures, deep learning models have exceeded human performance in many areas. However, a well-known limitation of deep learning models is the so-called ``catastrophic forgetting.'' That is, while acquiring new knowledge, the models may forget the knowledge learned in the past. In other words, after tuning a pre-trained model with new data, the model can have a poor performance on patterns learned from old data.

In this paper, we investigate the possibility of a model to generalize from old patterns and new patterns. Towards formalizing the setting of our study, we assume that we have a model with a generic loss function, and it has been well-trained on a big dataset. Our aim is that, upon presenting a new dataset,
the model can be adjusted so that it can handle the new dataset well -- while simultaneously  not forgetting  the patterns learned from the old dataset. In many real-world systems, new and old data are commonly stored in a buffered manner, where the model always has access to a roughly fixed size of training data. This fact is at the core of what differentiates our model retraining from traditional continual learning with bounded computational complexity. In our study, the old and the new datasets share the same prediction space. For instance, the number of classes in the two datasets are the same, but the background and object angles may differ.

The issue of catastrophic forgetting in neural networks had been addressed in the literature (cf.~\cite{french1999catastrophic, kirkpatrick2017overcoming}). Once again, the source of the problem is that when neural networks are trained continually on datasets with different feature distributions and/or trained for different objectives, the knowledge learned in an old training session can be lost in a new training session ~\cite{yosinski2014transferable}.

We note that our study aligns closely to continual/incremental learning and multi-task/sequential learning. Continual learning focuses on tuning a model with continual new data without forgetting old knowledge~\cite{albesano2006adaptation, silver2013lifelong, ruvolo2013ella, li2017learning, DBLP:journals/corr/abs-1802-07569}.
Compared to continual learning, multi-task learning focuses on more dissimilar tasks~\cite{Ruder2017, zhang2017, henderson2017benchmark, rusu2016progressive}, e.g., incremental class learning~\cite{roy2018tree, rebuffi2017icarl, kemker2017fearnet, Ganegedara2017, li2018}, where different tasks have different classes. Moreover, different tasks usually only have access to new data~\cite{aljundi2018selfless} or just a small portion of old data~\cite{lopez2017gradient}. From the training objective aspect, multi-task learning solves task-specific problems: parameter space shifting using priors~\cite{lee2017overcoming, nguyen2018variational, swaroop2019improving}, path traversal among different tasks~\cite{lopez2017gradient, Fernando2017}, parameter sharing in different tasks~\cite{Elliot2018}, and hard attentions~\cite{serra2018overcoming} or masks~\cite{mallya2018packnet} for switching tasks. Their contributions address challenges in different experimental settings~\cite{Maltoni2018}, such as how data in different tasks are distributed, how tasks are ordered~\cite{Meyerson2018, standley2019tasks}, and how performance should be evaluated among a sequence of tasks.

Our study may be perceived as a special case of continual learning -- however, we note that our primary focus and data setting differ from continual learning and, especially, sequential learning. Our retraining models aim to have good performance on both new and old data. In continual learning and sequential learning, a model can only access new data. However, when tuning a pre-trained model in real-world systems, the model should access some old data to guarantee adequate retraining. It may not be feasible in the long run to train on the union of all old data and new data. In our study, old data and new data are stored in a buffered manner. Given limited computation resources that can process a fixed capacity of data, we feed the same full capacity of training data during training or retaining. In every future retraining process, a trained model is tuned on a fixed-size data set, which is composed of the new dataset and a subset of the old dataset. We boost the performance of a trained model by gradient boosting and by learning a meta-network. The two methods are generic and can be applied to any neural network models and loss functions.

Our major contributions are that (1) we propose an additive ensemble model that can tune a pre-trained model based on its performance gap, and (2) we further propose a meta-network model that can balance new patterns and old patterns and eventually keep the number of parameters unchanged.

\section{Related Work}
\label{Related Work}
We now detail the major directions of continual learning: regularization, ensemble methods, and memory consolidation.

\subsection{Regularization}
Regularization is the most common way of solving catastrophic forgetting via approaches that measure the importance of weights of a trained model. The elastic weight consolidation (EWC) algorithm proposed in~\cite{kirkpatrick2017overcoming} uses the Fisher matrix to represent weight importance. A regularization term is added to the original loss function to constrain important weights to stay close to their old values when retraining on a new session/task. Despite the usefulness of the Fisher matrix, the diagonal Fisher matrix assumption may not hold; therefore, approximations of the Fisher Matrix~\cite{tu2016ranking, liu2018rotate, Schwarz2018, zeno2018task} and additional KL-divergence constraints~\cite{chaudhry2018riemannian} were applied in later studies. Besides the Fisher matrix, gradient magnitude~\cite{aljundi2018memory}, unsupervised self-organizing maps~\cite{kohonen1990self}, and the Hessian matrix~\cite{zenke2017continual} are common techniques for measuring weight importance. In particular, the memory aware synapses (MAS) model~\cite{aljundi2018memory} has become a new widely-used benchmark after EWC. MAS uses the gradient magnitude of network outputs with regard to weights to measure weight importance. In contrast, the Fisher matrix is based on loss and weights in EWC. When outputs are multi-dimensional, as in classification tasks, MAS calculates the squared l2 norm of outputs. An improvement over EWC and MAS by adding an additional regularization term on neurons was presented in~\cite{aljundi2018selfless}, proposing that sparsity at the neuron level was also important. We show in our experiments that our ensemble models perform better than this regularization strategy.

\subsection{Ensemble Methods}
Intuitively, ensemble models try to solve catastrophic forgetting by training multiple sub-networks~\cite{polikar2001learn++, Dai2017, hinton2015distilling, chen2017learning, wang2017model, YuSlimmable, Luong2016}. Ren et al.~\cite{Ren2017} and Juefei-Xu et al.~\cite{juefei-xu2018pnn} made breakthroughs by proposing dynamic ways of constructing an ensemble model by tuning a sub-network of a pre-trained model or expanding a pre-trained model. The general limitation is that the memory usage scales up with the number of training sessions. Related meta-learning works generally focus on learning network architectures in order to adjust a pre-trained model to be robust on different datasets and tasks~\cite{yoon2018lifelong, lin2013network, Rusu2016, xu2018reinforced, javed2019meta, juefei-xu2018pnn}. For instance, knowledge distillation~\cite{hinton2015distilling} tries to distill knowledge learned from a large (teacher) network to a small (student) network. The goal is to make the teacher network and the student network behave as similarly as possible. However, our goal is to learn a more generalized network continually, being able to fix errors in the old trained model by acquiring new data.

\subsection{Memory Consolidation}
Memory consolidation solves catastrophic forgetting from the data aspect. Memory consolidation models~\cite{kemker2017fearnet, lopez2017gradient, Nitin2017, chaudhry2018efficient, chaudhry2019continual, Mehta2018, Sodhani2019} learn to store the patterns/memories of different training sessions by strategically sampling a subset of the old data. Generative Adversarial Networks (GANs) are suitable in nature for learning networks~\cite{shin2017continual, Seff2017, Rios2018, Thanh-Tung2018, liang2019generative, Aljundi2019, Subakan2019} in an incremental way. A generator trained on old data can be used in a new training session for replaying old learning experiences. Our study solves a more generic issue from the modeling aspect, and memory consolidation can be applied to our models for providing a reasonable data buffer.

\begin{figure*}[pt]
\begin{subfigure}[b]{0.5\textwidth}
  \includegraphics[width = \textwidth]{BoostNet}
  \caption{BoostNet}
  \label{fig:BoostNet}
\end{subfigure}\hspace{0.1\textwidth}%
\begin{subfigure}[b]{0.5\textwidth}
  \includegraphics[width =0.86\textwidth]{FrostNet}
  \caption{FrostNet}
  \label{fig:FrostNet}
\end{subfigure}
\end{figure*}

\section{Proposed Models}
\label{Proposed Models}

In this section, we present two ways of ``frosting'' weights for better retraining. One is based on classical gradient boosting, the other is based on neural network meta-learning.

\subsection{Frosting Network Weights by One-step Gradient Boosting}
\label{Frosting Network Weights by One-step Gradient Boosting}

We train our proposed network BoostNet, which can be simpler or the same as the trained network. We firstly apply a retraining dataset, composed of new training data and a subset of the old training data, to the trained model and calculate the model outputs and ``residuals''~\cite{friedman2001greedy}. Let $L$ denote the softmax cross-entropy loss function, $o_i$ denote the output of the $i^{th}$ neuron before the softmax layer, $p_i$ denote the softmax probability of the $i^{th}$ neuron, and let $y_i$ denote the one-hot label. The residuals are calculated as $\frac{\partial L}{\partial o_i} = y_i - p_i$, measuring how much the trained model needs to improve in order to perform well on the retraining dataset. It is very likely that the residual values are close to 0 for the old training data, since the loss was originally minimized, and the residual values for the new training data are relatively larger. The weights of the trained model stay unchanged. Figure~\ref{fig:BoostNet} illustrates this training procedure. The BoostNet takes the residuals as the training targets (``labels''). The loss function for the simpler model is the mean squared error (MSE) loss. In the end, we sum up the outputs of the old trained model and the outputs of the BoostNet model for future inference. This method does not change the trained model but adds new weights, which come from the simpler model. The limitation is that the number of total weights increases every time a new dataset comes in. While in knowledge distillation (where a student model is trained to mimic the behavior of the teacher model), a base network was also trained and kept frozen~\cite{hinton2015distilling}, BoostNet aims to have different training targets  -- i.e., errors from former learners.

\subsection{Frosting Network Weights by Meta-learning}
\label{Frosting Network Weights by Meta-learning}

We now explain our second ensemble model that can keep the number of parameters unchanged for inference. We add ``frosting'' layers for weights in a pre-trained model by taking weights as ``neurons.'' More specifically, we train a meta-network for weights in addition to the trained network. Both weights in the meta-network and weights in the base network are optimized during re-training. We have attempted freezing weights in the base network, but optimizing both networks yielded faster convergence. Each frosting layer is of the same shape as its corresponding weights. The weights in the frosting layers and the weights in the pre-trained model are both updated during retraining. In the end, the frosting layers are merged with the original weights so that the total number of weights is still unchanged. The merging operation is done by replacing the weights in the base network with the product of those weights and the weights in the corresponding frosting layer -- after which the frosting layers are discarded. The base network with the updated weights is used for inference -- thus, there is no performance loss after the merging operation. We call this ensemble model FrostNet, and it is illustrated in Figure~\ref{fig:FrostNet}. We have attempted common activation functions such as $tanh$ and $ReLU$; $tanh$ had a better performance compared to $ReLU$ in our experiments. Although $ReLU$ has many good metrics when applied to neurons, we do not find that it contributes to our frosting layer activation. Regardless, we have found that simply applying the frosting layers to trained weights without any activation functions already outperformed the benchmark models in our experiments.

\section{Experiments}

We now discuss the experiments used in evaluating the benefits of the proposed approaches.

\subsection{Datasets}
We use the MNIST and the CIFAR-10 datasets for comparing our BoostNet and FrostNet models with benchmark models. Similar to the data variation experiments in continual learning, we create a retraining dataset from the training set of the MNIST or CIFAR-10 by taking a random half of the original training set and augmenting the other half. The random half is taken by sampling each mini-batch during retraining instead of keeping them fixed before retraining. We obtain the augmented validation set using the same method. To this end, the sizes of the training and the validation sets stay unchanged in the initial training and retraining. We augment the entire test set as the new test set. The augmentations were conducted on the original training, validation, and test data separately to avoid data leakage. The augmentation factors are randomly rotating images by 45 degrees (clockwise and counterclockwise), shifting images by 20 percent (left and right), and zooming in by 80-90 percent. We have found that a well-trained model on the original MNIST or CIFAR-10 performs poorly on this augmentation setting.

Similarly to incremental class experiments in sequential learning, we split both the original MNIST and CIFAR-10 training datasets into two parts: the first part has classes 0-4; the second part has classes 5-9. We use the first part for training a LeNet model. We then take a random half of the first and the second parts during retraining. The original validation and test datasets are split into two parts under the same setting. Finally, we obtain a retraining dataset that is of the same size as the initial training dataset.

\begin{table}[t]
\centering
\begin{tabular}{|l|r|r|r|}
\hline
                    & Original & Augmented & Total   \\ \hline
Base                & 0.9921  & 0.2829  & 0.6352  \\ \hline
Train-from-scratch  & 0.9870  & 0.7547  & 0.8699  \\ \hline
Fine-tuning         & 0.9921  & 0.2662  & 0.6267  \\ \hline
EWC                 & 0.9890  & 0.8117  & 0.8995  \\ \hline
MAS                 & 0.9889  & 0.6907  & 0.8385  \\ \hline
Selfless            & 0.9860  & 0.3036  & 0.6430  \\ \hline
BoostNet            & 0.9891  & 0.8830  & 0.9352  \\ \hline
FrostNet            & \textbf{0.9920}   & \textbf{0.9012}  & \textbf{0.9818} \\ \hline
\end{tabular}
\caption{Variational MNIST}
\label{MNIST-result}
\end{table}

\begin{table}[t]
\centering
\begin{tabular}{|l|r|r|r|}
\hline
                    & Original & Augmented & Total   \\ \hline
Base                & 0.6599  & 0.3013   & 0.4810  \\ \hline
Train-from-scratch  & 0.6077  & 0.4309   & 0.5155  \\ \hline
Fine-tuning         & 0.6207  & 0.4463   & 0.5200  \\ \hline
EWC                 & 0.6191  & 0.4640   & 0.5415  \\ \hline
MAS                 & 0.6094  & 0.4363   & 0.5230  \\ \hline
Selfless            & 0.6198  & 0.3397   & 0.4795   \\ \hline
BoostNet            & 0.6363  & 0.4580   & 0.5426   \\ \hline
FrostNet            & \textbf{0.6456}  & \textbf{0.4646}   & \textbf{0.5552} \\ \hline
\end{tabular}
\caption{Variational CIFAR-10}
\label{cifar-10-result}
\end{table}

\begin{table}[t]
\centering
\begin{tabular}{|l|r|r|r|}
\hline
                    & Classes 0-4 & Classes 5-9 & Total   \\ \hline
Base                & 0.9984  & 0.0000  & 0.4992  \\ \hline
Train-from-scratch  & 0.9957  & 0.9562  & 0.9760 \\ \hline
Fine-tuning         & 0.9953  & 0.9626  & 0.9790 \\ \hline
EWC                 & 0.9949  & 0.9562  & 0.9756  \\ \hline
MAS                 & 0.9916  & 0.9008  & 0.9462  \\ \hline
Selfless            & 0.9940  & 0.9074  & 0.9507  \\ \hline
BoostNet            & \textbf{0.9973}  & 0.9628  & 0.9800  \\ \hline
FrostNet            & 0.9963   & \textbf{0.9665}  & \textbf{0.9814} \\ \hline
\end{tabular}
\caption{Split MNIST}
\label{MNIST-Split-result}
\end{table}

\begin{table}[t]
\centering
\begin{tabular}{|l|r|r|r|}
\hline
                    & Classes 0-4 & Classes 5-9 & Total   \\ \hline
Base                & 0.7590  & 0.0000   & 0.3795  \\ \hline
Train-from-scratch  & 0.4359  & \textbf{0.6518}   & 0.5439  \\ \hline
Fine-tuning         & 0.6010  & 0.5639   & 0.5824 \\ \hline
EWC                 & 0.5921  & 0.5520   & 0.5721  \\ \hline
MAS                 & 0.6402  & 0.5184   & 0.5230  \\ \hline
Selfless            & \textbf{0.6535}  & 0.4344  & 0.5440  \\ \hline
BoostNet            & 0.6252   & 0.5791  & 0.6022  \\ \hline
FrostNet            & 0.6498   & 0.5693  & \textbf{0.6096} \\ \hline
\end{tabular}
\caption{Split CIFAR-10}
\label{cifar-10-Split-result}
\end{table}

\subsection{Experimental Settings}
\label{Experimental Settings}
We compare our models to three baseline models: the initial LeNet model (base), training a new model (train-from-scratch), fine-tuning from pre-trained weights (fine-tuning), and three state-of-the-art models: EWC, MAS, and Selfless~\cite{aljundi2018selfless}. We use LeNet with batch normalization as the network framework. The pre-trained weights from the base model are used for measuring weight and neuron importance for EWC, MAS, and Selfless. All benchmark models are tuned based on the optimized weights from the base LeNet model. We use the truncated normal distribution for weight initialization for all network models. All models are trained at most 50 epochs with an early stopping of 10 consecutive epochs. The weights that have the highest accuracy and, secondarily, the lowest loss value (if the accuracy does not improve) on the validation dataset are used for inference. We report the accuracy of the corresponding test datasets in tables \ref{MNIST-result}--\ref{cifar-10-Split-result}. Boldface indicates the highest value in each column except the base model. We publish the code for our experiments on GitHub\footnote{https://github.com/XiaofengZhu/frosting\_weights}.

\section{Results}
\label{Results}

We average the accuracy of each model using three runs. Tables \ref{MNIST-result}-\ref{cifar-10-result} summarize the performance of the data variation experiments using the baseline models, the benchmark models, and our ensemble models. The base model has the highest accuracy on the original test datasets and the lowest accuracy on the augmented test datasets. Due to early-stopping and data augmentation, fine-tuning and benchmark models do not necessarily outperform train-from-scratch models. All the models except the base one suffer from catastrophic forgetting to some degree as the accuracy on an original test dataset is lower than the accuracy using the base model. All benchmark models have higher accuracy on the augmented dataset compared to the baseline models. EWC, MAS, and Selfless can prevent a trained model forgetting old knowledge, but their common limitation is that they have trouble learning from new knowledge. Our FrostNet model performs best among all benchmark models, and our BoostNet comes second. The FrostNet model yields the highest accuracy on the original datasets and the augmented datasets as the FrostNet model uses a network to learn how to adjust weights. The BoostNet model learns from performance gaps and pushes its weights to minimize the gaps.

Tables \ref{MNIST-Split-result}-\ref{cifar-10-Split-result} summarize the incremental class experiments. The base model has the highest accuracy on the datasets of classes (0-4) and the lowest accuracy on the ones of classes (5-9). Since the initial training dataset only has classes (0-4), the trained base model cannot predict any data from classes (5-9). Training-from-scratch on the retraining dataset has noticeable catastrophic forgetting on the initial training dataset. Although EWC, MAS, and especially Selfless models have an advantage in reducing catastrophic forgetting, they have inferior results on the new classes (5-9). Our BoostNet and FrostNet models can balance memorizing old knowledge and learning new knowledge, with a good overall performance.

\section{Conclusion}
\label{Conclusion}
In this paper, we propose two generic ensemble methods for boosting the performance of retraining. Regarding parameter capacity, BoostNet slowly adds a small number of parameters to a pre-trained model, similar to adding new trees in random forest modeling. Although FrostNet adds frosting layers for trained weights, the merging operation brings the number of total parameters down to the original size. Our work can be extended to other fields, such as general continual learning and network compression. Our work can be improved by taking the time series characteristic of training into account to further boost the performance. An interesting future direction would be strategically selecting training data. Another interesting direction would be selecting synapse paths from a sophisticated network model trained for a large domain to specialize for small domains.

\bibliographystyle{abbrv}

redpajama_set_name: RedPajamaArXiv


That morning on the 1st of April, 2015 I caught a train from Brussels to Liège to see Barbara (and the city) but it was definitely not a joke. The first thing she said to me when I hopped off the train was "this is so crazy" and I agree. Less than a year ago we met and became classmates; 3 months ago we became friends but she was about to leave Hong Kong, and I believe we both thought we wouldn't see each other again any time soon. Turns out we made it to meet in Europe and yes we had a good time ! Of course I can't say I went "all the way" from Hong Kong to Liège, as I am in the UK for my exchange and in Belgium for my holidays and it literally just took me an hour from Brussels, but it indeed required a lot of coincidences for this rendezvous to happen. I feel so lucky and so blessed.
Liège is exciting in its own terms. It is obviously not a popular tourist destination but it is worth visiting if one is already in Belgium. Ask any of my friends and you will know food is the most important thing in my travels. In Liège I had my first and the best gaufre (waffles) and moules frites (fries). (Told by Barbara and also later confirmed with my Belgian friends, waffles of Liège are slightly different from that of Brussels, so it was worth the visit !) The gaufre we had was made fresh to order, and slightly sweeter than that of Brussels. So when the shop lady handed me the gaufre it was still hot. It was definitely one of the best feelings in life when you have warm food in your hands under a cool, windy and rainy weather.
Also, I could never forget sharing the basket of moules frites with Barbara in a 2-floor food place because 1. the fries was superb; 2. the food place was very interesting; and 3. Barbara is such an amazing companion. Moules frites is special (and different from all other fries in the world) because it was fried twice. In the food place we saw how the moules frites were prepared and it was absolutely fascinating. The daily demand is so high the raw potato fries were stored literally in plastic bins and we were at once shocked because we thought the bin was used for trash ! Everything happened in a really high speed and the shop smelled so good.
After we got our moules frites we moved upstairs to enjoy the food, the warmth and had a nice chat. The food place was very cozy with the wooden furniture. We had a 4-people table so very soon 2 other customers sat next to us. It was funny because while they eavesdropped our conversation Barbara also eavesdropped theirs. We were talking in English and they were in French. According to Barbara one of them said "I could never speak English as fluent as them" and the other one mentioned about a piece of food we left on the tray. It was absolutely funny obviously because they didn't realize that Barbara is very good at French !


DETROIT (AP) — A Detroit man is awaiting sentencing after pleading guilty to making threats on Facebook against police officers.
The office of Michigan Attorney General Bill Schuette says 42-year-old Nheru Littleton entered the plea Tuesday in a Detroit courtroom to making a threat of terrorism. Sentencing is scheduled for April 10.
Authorities say he used social media in July 2016 to encourage the killing of white officers after the fatal shooting of five officers in Dallas.
Littleton's attorney had earlier argued that Littleton's posts were protected by the First Amendment. The defense had challenged on appeal whether the charge could proceed, but higher courts including the Michigan Court of Appeals let it move forward.


Our design and production process take a lot of time and effort. To please esports players and their fans is no small task!
We at Maika'i Copenhagen choose our manufactures carefully, and only work with known companies that live up to the strict standards of our quality and the legislation in EU.
Therefore, our Head of buying & Logistics Nis Johanssen and Head of Design Sebastian Cann-Svärd flew to China to meet our collaborators and go through all our products being developed and produced.
Here they got to see and feel many of the products they have been working on for a long time for the first time! Both were very excited, but also a bit nervous, since orders had already been filled by sales.
Luckily, they saw that everything lived up to standards and expectations. Our producers had lived up to our standards and expectations. At the powerbank factory, were our signature LED powerbank is produced, they suddenly realized how big this project is! Rows and rows of powerbanks with our logo, the Astralis logo and from the teams competing in BLAST Pro Series Copenhagen were lined up and be controlled before shipping.
The biggest focus on the trip was going to several tech trade fairs, were new products are shown to companies from all over the world. This is were many ideas for new products, and patents are made. Since Sebastian is always thinking about new products to design, this was like sending a kid into a candy store!
The fairs are HUGE and takes several days to get through all the vendors there. Safe to say that Sebastian found several new products that he fell in love with and decided to implement in our product line! Already on the ride home he was scribbling in his notebook with rough designs, ideas and finished products.
For anyone that knows Sebastian, this is when he enters his rain man mode! No one can get through to him, and he simply turns off the world and starts designing. Annoying when you need him, amazing when you see what he can do with silence, a pencil, paper and a computer!
Stay tuned to see what came out of the trip….


Parts listed here are compatible with Model A Indoor Rowers. Model As were manufactured from 1981–1986.
Model A There are 35 products.
For use on the wooden handle originally sold on the Model A, B and C Indoor Rower. These are the black foam grips that fit on either end of the handle. Two per handle. Sold individually. Order two to have a pair. See more info.
For use with the Model A and B Indoor Rower (Ergometer) only. This screw is part of the heel rest assembly.
For use with the Model A and B Indoor Rower (Ergometer) only. The is the S.C.A.M. that secures the static end of the shock cord in the return mechanism. See more info.
For use with the handle on all models of indoor rowers. Connects the handle to the chain. The corresponding nuts are part number 240 for a Model A, B or C and part number 1235 for Model D, E and Dynamic.
This is the 1/4-20 heavy Nylock nut that is used with the wooden handle that came standard on the Model A, B and C Indoor Rowers. The corresponding U-bolt is part number 244. Order two per handle.
For use with all Model B Indoor Rowers, as well as Model A Indoor Rowers that have been retrofitted with the skewer assembly. Holds the skewer assembly in place in the monorail. Order four per machine.


Traveler's Tales has given out new details on its highly anticipated LEGO Star Wars III, a title that promises to significantly improve the already impressive array of action adventure game title which is set in the universes of LEGO and Star Wars.
A new gameplay level will also be there based on the movies, for example the Battle of Geonosis in Episode II.
Traveler's Tales has also revealed that gamers are going to dive into battle with up to 300 characters on screen, which undoubtedly will give the game a truly spectacular level.


text:
'Scrubs' co-EP Eric Weinberg sued for double sexual assault; Producer behind bars for 18 counts of rape
Leave a Comment / Entertainment / By Adam
Ancient Scrubs Co-executive producer Eric Weinberg, currently incarcerated in downtown Los Angeles on 18 counts of rape, was sued in civil court by two different women for sexual assault.
"Plaintiff CD suffered physical, emotional, and economic harm as a result of Weinberg's assault, and brings this action for sexual assault, sexual assault, sexual harassment, gender-based violence, assault, battery, intentional infliction of emotional distress and negligence," says one of the filings (read it here) filed Monday night in Los Angeles Superior Court. Detailing an alleged 2019 rape at the producer's Los Angeles home, the jury complaint seeking a trial seeks sweeping but unspecified damages.
Initially arrested by Los Angeles police in July for multiple scatter rapes from 2012 to 2019, Weinberg was initially released on $3.2 million bail. Later increased to $5 million, that bail was revoked and Weinberg was taken into custody on October 25 after pleading not guilty at his arraignment. "The defendant engaged in a pattern of violence for at least six years," Judge Victoria B. Wilson said despite objections from Philip Cohen and other members of Weinberg's defense team who were present. . The LASC judge went on to label Weinberg a potential "serial rapist" and called his decades of despicable behavior "brazen and predatory."
Held in downtown Twin Towers Correctional Facility, Weinberg is next due to have a hearing in his criminal case on November 15. Looking at a period of up to 100 years in state prison if convicted, the ex-Californication The executive producer faces six counts of forcible sexual penetration, four counts of oral copulation, three counts of forcible rape, two counts of coerced sexual assault and one count each of assault with likely force. to cause serious bodily harm, attempted sexual penetration by the use of force and forcible confinement.
"Eric Weinberg is a calculated serial predator who targeted these women and caused them immense trauma, mental anguish and physical harm," said Micha Star Liberty, lead civil litigation attorney for the two women. "Weinberg must be held accountable to these women for his horrific actions," the Liberty Law managing partner added. By partnering with Gibbs Law Group, Liberty says she and her colleagues are looking into more potential civil lawsuits from other potential victims.
Since Weinberg's arrest this summer, the LAPD and the LA County District Attorney's Office have received more than 70 tips and accounts from women who say they, too, were lured to the producer's home and raped on the promise of a photoshoot to advance a career in entertainment. industry.
Weinberg targeted women in their 20s and 30s "in grocery stores, cafes and other public places," according to police. "He is a danger to society," Assistant District Attorney May Martinez said of Weinberg during the Oct. 25 hearing. "He is a danger to all women."
#Scrubs #coEP #Eric #Weinberg #sued #double #sexual #assault #Producer #bars #counts #rape


text:
Marine Light Attack Helicopter Squadron 367 (HMLA-367) is a United States Marine Corps helicopter squadron consisting of AH-1Z Viper attack helicopters and UH-1Y Venom utility helicopters. Originally commissioned during World War II, the squadron participated in combat operations on Peleliu and Okinawa. Reactivated during the Vietnam War, the squadron has served during numerous conflicts since. The squadron is based at Marine Corps Air Station Camp Pendleton, California and falls under the command of Marine Aircraft Group 39 (MAG-39) and the 3rd Marine Aircraft Wing (3rd MAW).

Mission
The mission of HMLA-367 is to support the Marine Air-Ground Task Force (MAGTF) Commander by providing offensive air support, utility support, armed escort and airborne supporting arms coordination, day or night, under all weather conditions during expeditionary, joint, or combined operations.

History

World War II
Artillery Spotting Division, Marine Observation Squadron 351 (ASD(VMO-351)) was commissioned on December 1, 1943 at Marine Corps Base Quantico, Virginia. On January 15, 1944, the squadron was re-designated as Marine Observation Squadron 3 (VMO-3) while preparing to deploy overseas. During February 1944, VMO-3 departed San Diego and arrived at Espiritu Santo on March 4, 1944. The squadron spent the next few months in the Russell Islands continuing to train.  On September 17, 1944, the forward echelon of VMO-3 landed on Peleliu Airfield, two days after the initial invasion. The squadron supported operations on Peleliu and on September 28 it also supported the assault on Ngesebus. On October 22, 1944 VMO-3 departed Peleliu for the Russell Islands to begin training with III Marine Amphibious Corps artillery.

On March 10, 1945, VMO-3 sailed for Ulithi where it transferred to escort carriers for movement to Okinawa. On April 3, 1945, two OY-1 Grasshoppers from VMO-3 landed on Yontan Airfield. These were the first American planes to land on the island. The squadron commenced operations from Yontan the following day immediately supporting four artillery battalions. By April 11, VMO-3 was supporting eleven artillery batteries ashore.

Following the surrender of Japan, VMO-3 departed for Northern China in September 1945. On October 10, the  arrived at the Taku Forts, and disembarked VMO-3, which was tasked to support the 3rd Marines. The squadron participated in the occupation of North China from October 1945 through June 1947. VMO-3 relocated to Guam in July 1947 and moved again in May 1949 to Marine Corps Air Station Cherry Point, North Carolina being assigned to the 2nd Marine Aircraft Wing. The squadron was decommissioned on August 20, 1949.

Vietnam War
As the United States increased its presence in the South Vietnam, the Marine Corps reactivated VMO-3 at Marine Corps Air Station Camp Pendleton on 1 August 1966 and assigned it to Marine Wing Support Group 37. The squadron arrived in Vietnam in December 1966 and were now flying the UH-1E. VMO-3 became fully operational at Huế/Phu Bai on 16 January 1967.

In 1966 and for the first half of 1967, the squadron's radio call sign was "Oakgate," which sometimes got confused for the term OK.  Major Bill Murphy is credited with first using the call "Scarface" while on TAD, and lobbied to have the substantially more macho Scarface designation adopted as VMO-3's signature when all Marine squadrons in-country changed call signs during the summer of 1967. On November 14, 1967 Major General Bruno Hochmuth, then commanding general of the 3rd Marine Division was killed when a UH-1E Huey from VMO-3 exploded and crashed 5 miles northwest of Huế. Four others also died in this crash.

The squadron assumed a new identity in March 1968 when VMO-3 was re-designated HML-367 and immediately began operations in direct support of ground combat units. Based at Hue and Phu Bai, the squadron flew the Bell UH-1E Huey throughout I Corps in support of the 1st and 3rd Marine Divisions, the United States Army and the Army of the Republic of Vietnam. In addition to the countless other missions flown they most notably supported Operations Medina, Fortress Ridge, Prairie Fire, Project Delta, Sparrow Hawk, Dewey Canyon, Apache Snow, Swift Saber and Lam Son 719.

In addition to participating in the above operations, the Scarface crews also flew near daily missions in support of MACV-SOG on what were then top secret reconnaissance missions deep into Laos.

Squadron assets quickly grew from eighteen aircraft (UH-1E) to twenty five. In 1969, HML-367 exceeded 2,000 flight hours for the month of June. October 1969 saw the departure of Marine Air Group 36 from the Republic of Vietnam. Due to the continued necessity for armed helicopter support, HML-367 remained in Vietnam and was assigned to Marine Aircraft Group 16 (Forward) at Hue/Phu Bai.

The squadron supported almost every type of helicopter mission flown by Marines in Vietnam.  Most missions were flown in support of Marine units, but many were flown for the U.S. Army and for the Korean Marines.  With the increased Vietnamization of the War, numerous sorties were flown supporting the Army of the Republic of Vietnam.

On April 24, 1970, the Squadron had all of their twenty-five aircraft in an "Up" status. This was the first time a Helicopter Squadron had attained 100% aircraft availability in Vietnam. To celebrate the event, a 25-plane "fly-over" of Marble Mountain Air Facility was conducted with Plane Captains called on to fill the front seats of all 25 Cobras. Total flight time for Fiscal Year 1970 was 22,378 hours.

HML-367 completed its combat tour in South Vietnam in June 1971, and relocated to MCAS Futenma, Okinawa, rejoining MAG-36. Once established, the Squadron began peacetime operations using the UH-1E.

Post Vietnam 1970s

In April 1975, HML-367(REIN), with nine UH-1E's and twelve CH-46's deployed aboard the  and the  in support of Operations Eagle Pull and Talon Vise/Frequent Wind, participating in the evacuation of Phnom Penh and Saigon. In June 1975, HML-367 returned to Okinawa and resumed normal operations.  In 1976, the squadron took receipt of the new UH-1N and in 1977, AH-1J's Cobra gunships.

After more than eight years on the island of Okinawa, HML-367 was reassigned to Marine Aircraft Group 39 (MAG-39), 3rd Marine Aircraft Wing, at MCAS Camp Pendleton. The majority of squadron personnel stayed behind, having been transferred to other units of the 1st Marine Air Wing.  New personnel were selected from the parent command, and by August 1981, unit strength was up to 165 men and 16 aircraft.

1980s & 1990s

During this timeframe the HMLA-367 spent approximately six of each eighteen months split into rotations to Marine Corps Air Station Futenma, Okinawa, Japan as well as attached as Det "A" in Subic Bay, Philippines, and as part of an Aviation Combat Element (ACE) aboard an amphibious assault ship, typically the  while in rotation with two other squadrons also based at MCAS Camp Pendleton.

While not on their rotation overseas, HMLA-367 supported major exercises deploying for training frequently to the Marine Corps Air Ground Combat Center Twentynine Palms, California, Marine Corps Air Station Yuma, Arizona, Nellis Air Force Base, Nevada and Naval Air Station Fallon, Nevada.

HMLA-367 also supported combined US Navy & USMC security operations, "Kernel Usher & Kernel Blitz" aboard the USS Tarawa and other naval ships for exercises around the Pacific Coast of California.

Desert Storm/Desert Shield

Shortly after the invasion of Kuwait on 2 August 1990 HMLA-367 was quickly chosen as the Light Attack component for Marine Aircraft Group 70 (MAG-70) in support of Operation Desert Shield/Storm and immediately deployed.

Scarface arrived in Al Jabail, Saudi Arabia on 29 August 1990, as the first operational U.S. Marine Light Attack Helicopter Squadron in theater.   HMLA-367 recorded the first Iraqi tank kill by an AH-1W "Super Cobra" Helicopter during the battle of "Al Elbow" in Southwestern Kuwait on 30 January 1991.

HMLA-367 also had confirmed kills of military targets during both the Battle of Khafji on 30–31 January 1991 and  the ground war which commenced on 24 February 1991.

HMLA-367 accrued confirmed kills of 48 tanks, 24 armored personnel carriers and 23 other troop transport vehicles, trucks, etc., as well as many other military targets, more than any other operational combat squadron during Desert Storm/Desert Storm.

On 12 March 1991, Scarface concluded operations and began departing Saudi Arabia, returning to Camp Pendleton, California.

HMLA-367 Scarface Marines were all awarded the National Defense Service Medal, Southwest Asia Service Medal with 2 bronze stars, the Kuwait Liberation Medal (Saudi Arabia), & the Kuwait Liberation Medal ( Kuwait) for their service during Operation Desert Storm & Desert Shield combat operations.

Operations in Iraq & Afghanistan

The squadron was deployed for 13 months to Okinawa from 2002–03 and this was followed less than a year later by a deployment to Al Taqaddum, Iraq in support of Operation Iraqi Freedom.

While there, the squadron flew in support of numerous ground units including the 1st Marine Division during the Battle of Fallujah. In the next three years, Scarface would return to Iraq two more times in support of OIF 05-07.2 and OIF 08.1-08.2.

After three deployments to Iraq in only four years, Scarface returned to MCAS Camp Pendleton in November 2008 to transition its Huey pilots to the new UH-1Y, and to prepare for upcoming operations in Afghanistan.

After yet another turnaround of less than a year, HMLA-367 deployed to Afghanistan in October 2009 to support international security and assistance forces during Operation Enduring Freedom. The seven-month deployment was the first combat deployment of the UH-1Y.  The squadron's skills were again put to the test as Scarface simultaneously provided support for seven infantry battalions and participated in the battles for Marjah and Now Zad in the Helmand Province of Afghanistan.

In April 2011 "Scarface" HMLA-367 was the first squadron to have both the newest upgraded four-blade rotor UH-1Y "Yankee" Huey utility helicopter and the AH-1Z "Zulu" Cobra gunships along with a state of the art flight simulator for each helicopter.

In 2012, Scarface moved to its current location of Marine Corps Air Station Kaneohe Bay on island of Oahu in Hawaii.  The unit reverted to the venerable AH-1W Super Cobra prior to the relocation to MCAS Kaneohe Bay.  They have also conducted numerous deployments as part of the 31st MEU and to Okinawa as part of the Unit Deployment Program (UDP).

In March 2018, Scarface and the Marine Corps bid their final farewell to the AH-1W Super Cobra helicopters on Marine Corps Base Hawaii.

HMLA-367 was decommissioned in April 2022 as a part of the Commandant of the Marine Corps Force Design 2030 initiative and reactivated on December 16, 2022 aboard Marine Corps Air Station Camp Pendleton, California.

Unit awards

A unit citation or commendation is an award bestowed upon an organization for the action cited. Members of the unit who participated in said actions are allowed to wear on their uniforms the awarded unit citation. HMLA-367 has been presented with the following awards:

See also
 List of United States Marine Corps aircraft squadrons
 United States Marine Corps Aviation

Notes

References
Bibliography

Web

LA367
Inactive units of the United States Marine Corps

redpajama_set_name: RedPajamaWikipedia