text:
Home » Urban » Introducing Our Editors: Jered B. Carr
Introducing Our Editors: Jered B. Carr
Carr received his Ph.D degree in public administration from the Askew School at Florida State University and an M.A. in economics and B.A. in finance from Florida Atlantic University.
His dissertation, "The Political Economy of Local Government Boundary Change: State Laws, Local Actors, and Collection Action," received the 2001 Leonard White Award from the American Political Science Association.
Carr is co-editor of City-County Consolidation and Its Alternatives: Reshaping the Local Government Landscape (ME Sharpe, 2004). His research has been published in a wide range of journals in public administration and urban affairs, including the American Review of Public Administration, Journal of Urban Affairs, Local Government Studies, Political Research Quarterly, Public Administration Review, Publius, State and Local Government Review, Urban Affairs Review and Urban Studies.
Previous to joining UIC, Carr was a faculty member at the University of Missouri-Kansas City, Wayne State University, the College of Charleston and worked as a policy analyst for the Florida Legislature. He is an active member in the American Political Science Association, the Urban Affairs Association and the Section on Intergovernmental Administration and Management (SIAM) of the American Society for Public Administration.
Jered B. Carr is Department Head and Professor in the Department of Public Administration at the University of Illinois at Chicago. Carr's teaching and research interests are in intergovernmental management, metropolitan governance, and local government administration. He teaches courses in intergovernmental management, public administration, urban policy and local government administration. He has ongoing research programs focused on effective metropolitan governance, public services contracting, risk perceptions in intergovernmental service collaborations, the effects of city size and density on civic and political engagement.
Many of Carr's published articles can be downloaded from the research network website ResearchGate (http://www.researchgate.net/) and a full list of his published research is included in his CV available on the main faculty page at http://cuppa-pa.uic.edu/the-department/faculty/dr-carr/


Wetlands improve water quality through denitrification, but these ecosystems are also an important source of the greenhouse gas, methane. The objective of this research was to determine the effect of two common macrophyte species (Juncus effusus and Salix nigra) on denitrification and on the methane cycle. The research was conducted in a newly constructed wetland on the Columbus campus of The Ohio State University during two growing seasons. In the wetland, some plots were left unplanted, while others were planted with Salix or Juncus species (i.e., 3 treatments; n = 15 per treatment). For each treatment, we quantified concentrations of methane at two depths (15 and 25 cm) in the sediment, emissions of methane from the sediment and through the plants, and denitrification rates. During most of the second growing season, both species had a limited effect on denitrification and methanogenesis. The effects of the plants became evident by the end of the second growing season and during the third growing season. During the third growing season, Salix species enhanced the release of the greenhouse gas methane to the atmosphere, while Juncus limited the emission of methane. In comparison to the unplanted plots, the long-term removal of nitrate by denitrification was favored in the plots planted with Juncus and was not affected by Salix. Our study provides evidence that certain plants (such as Juncus) can be planted in constructed wetlands to favor denitrification, while buffering methane emission.


You are visiting the placeholder page for John Ball. This page is here because someone used our placeholder utility to look for John Ball. We created this page automatically in hopes John Ball would find it. If you are not John Ball, but are an alumni of Everett High School Everett, MA, register on this site for free now.


text:
Crocs appoints Michelle Poole as new President
World leader in casual footwear, Crocs, has today announced that Michelle Poole has been appointed as President, effective immediately.
As President, Poole will assume responsibility for America, Asia and EMEA regional commercial teams as well as continue to lead Product Design and Management, Merchandising, and Marketing teams.
She will continue to report to Andrew Rees, Chief Executive Officer.
Poole has served as Crocs' Executive Vice President, Chief Product and Merchandising Officer since April 2020. Prior to this, she served as Senior Vice President and Chief Product and Merchandising Officer.
With nearly three decades of experience, Poole has proven success in the development of several other product and lifestyle brands including Sperry Top-Sider, Timberland, Kangol, Converse, MTV Europe and Pepe Jeans.
Andrew Rees, Chief Executive Officer, said: "Since joining the Crocs team, Michelle has had a profound impact on the trajectory and success of the Crocs business. I am proud to appoint Michelle to the role of president"
"As a brand, we have proven resilient in the face of adversity and are emerging from this crisis with renewed confidence and optimism. This change will provide an even more effective and cohesive go-to-market strategy to achieve our long-term brand and commercial goals. We are confident that Michelle's track record and leadership will continue to fuel sustainable growth in 2021 and beyond."
"It is a privilege to lead such a talented team and to act as steward of this incredible global brand," added Poole.
In April, the footwear brand donated over 200,000 pairs of shoes to hospitals across multiple territories, including the UK, US, France, Germany and the Netherlands.
Retail prices slide lower amid...
John Lewis extends its Anyday value...
Reiss reports strong Christmas...

redpajama_set_name: RedPajamaCommonCrawl


Manimation is Manchester's annual day-long gathering for the animation sector, bringing everyone together to gain business understanding and creative inspiration.
Watch full videos of Manimation 2016 sessions here.
Manimation Keynote from Cat Lewis, CEO, Nine Lives Media.
Brave New Worlds Will VR 'eat' traditional media? With Aardman's David Sproxton member of the BAFTA VR Advisory Board. Aardman recently moved into VR working with Google Spotlight on the 360° short 'Special Delivery' and collaborating with the BBC on the VR/CGI short 'We Wait'. Moderated by Simon Smith (Studio Liddell) and featuring Ben Mills (Desk Dragons) and Nick Pittom (Fire Panda).
Creative Snapshot with the BBC Commissioning executives included Sarah Legg (CBeebies), Jo Allen (CBBC) and Tim Searle (CBBC). Moderated by Westley Wood (CITV).
Fund and Flow Speakers included Erica Darby (Spider Eye), Tim Patterson (Larkshead Media) and Chris Halliday, (BFI).
One-to-one meetings The time to meet many of the speakers in one-to-one meetings was bookable on the day. This was your opportunity to meet legal and finance experts and get free and valuable advice, as well as meet commissioners.
Realtime VR Storytelling – Heading for the Holodeck with Nick Pittom (Fire Panda).
North by North West Showcase presented by Jonathan Ashworth (Creative Director, Mighty Giant).
Training and Development: New Initiatives, New Entrants Discover how your company can access training funding and how the animation levy is being refined for 2017. Speakers included Abigail Fairhurst (Brown Bag Films); Adrian Healy (Director of Business Development, The Manchester Growth Company); Steve Swanborough (Freelance); Praveen Zacharias (Brown Bag Films).
Animation UK Update with Oli Hyatt (Blue-Zoo Animation).
PLUS Presentations from BAFTA Award nominee Kath Shackleton (Fettle Animation) and the latest research on kids' viewing habits from Fiona Scott (University of Sheffield).
Manimation 2016 was organised by CMC and funded by Creative Skillset's Skills Investment Fund.


text:
Car & Tracks
The Tracks
RaceGrid VR SimulatorsVirtual Racing Centre
AZURE COAST
05/04/2020 938 0 0
ShareTrx_addons_twitterTrx_addons_facebookTrx_addons_tumblrE-mail
Azure Circuit Monaco is a street racing circuit in south of France
AZURE CIRCUIT
FERRARI IMOLA
Oulton Park Circuit is a motor racingtrack close to the village of Little Budworth, Cheshire, England
Aston Martin Vantage GTE 2015
he Aston Martin Vantage GTE (also known as the Aston Martin Vantage AMR…
Aston Martin Vantage GT3 2013
In 2012, Aston Martin unveiled a racing version of the V12 Vantage ….
ASTON MARTIN DBR1/300 1959
The Aston Martin DBR1 was a sports racing car built by Aston Martin starting in 195….
The Aston Martin DB11 is a grand tourer produced by British luxury car…
Aston Martin Vantage GT12
The Aston Martin Vantage GT12 is a lightweight and extrem….
Aston Martin Vantage GT4
The Aston Martin Vantage GT4 is an updated version of the Aston Martin V8…
Race Grid VR Stoke
Next to Vue Cinema
ST51PT
Mon 5:00pm-9:00pm
Tue 5:00pm-9:00pm
Wed 2:00pm-9:00pm
Thur 2:00pm-9:00pm
Fri 2:00pm-10:00pm
Sat 12:00am-10:00pm
Sun 12:00am-9:00pm
Phone from 9:00am Every Day
Email: info@racegrid.co.uk
RaceGrid © VR Simulators Staffordshire. Terms and Conditions | Privacy Policy


Pafos Zoo, the biggest licensed zoo on the island, is set in a beautiful environment of lakes, ponds and gardens of more than 100.000 sq.m. It is home to an extensive variety of birds and animals from every corner of the planet. The collection of parrots and venomous snakes is one of the largest in Europe!
Zoo: the perfect place for children, for everlasting memories!
Pafos Zoo is situated just a few minutes drive from Aphrodite Sands Resort. Wildlife includes giraffe, monkeys, meerkats, crocodile, kangaroos, albino wallabies, mouflon, white lions, tigers and many more. In the amphitheater, performers like the brightly coloured macaws and the smart owls, will amaze everybody with their tricks.


text:
layout:     post
title:      HA issue in cinder.
subtitle:   "[ OPENSTACK ISSUE ]"
categories: [general, issues, openstack]
tags:       [openstack, issues]
date:       2016-05-15 15:10:00 (UTC +5:30)
contributor_name:     Abhishek Shrivastava
contributor_image:    abhishek.jpg
contributor_linkedin: abhishek-shrivastava-03921390
contributor_emailid:  abhishek@cloudbyte.com
---

## Observation :

- Check for host property in /etc/cinder/cinder.conf

  ![alt text](https://raw.githubusercontent.com/CloudByteStorages/blog/gh-pages/post_images/cinder_ha1.jpg)

- Check the host column in volumes table in cinder database

  ![alt text](https://raw.githubusercontent.com/CloudByteStorages/blog/gh-pages/post_images/cinder_ha2.jpg)

## Solution :

- Above screenshot typically imply, host property was set to FQDN of the cinder node. This has got issues in the current HA implementation in cinder.
- Uncomment the **host=cinder** in **/etc/cinder/cinder.conf**.
- Update the volumes records host column to cinder irrespective of the host in which the volume was created.

redpajama_set_name: RedPajamaGithub


text:
AT WALTON ARTS CENTER: Musical version of "A Christmas Story" feels familiar in a chaotic world
By Kevin Kinder · Wednesday, Dec 11, 2019
We start this review much as the Walton Arts Center did to opening their newest Broadway musical, "A Christmas Story," on Tuesday (Dec. 10) night.
Less than two hours before the doors opened for the show, the Walton Arts Center's main entrance plaza was the site of a candlelight vigil held in honor of the late Stephen Carr, a Fayetteville Police Department officer who was murdered in his patrol car on Saturday night. He was 27. And because his beat included the downtown entertainment district, he no doubt frequented the Walton Arts Center during his patrols.
As Peter Lane, CEO of the Walton Arts Center, told the assembled crowd before the curtains went up, "officer Carr was like family to us." A moment of silence was held in honor of Carr and those mourning his departure.
Lane then offered the audience a challenge to use the arts as a means of escapism.
What: "A Christmas Story"
When: Through Dec. 15
Where: Walton Arts Center, Fayetteville
Cost: Starting at $40 plus fees
Tickets: 479-443-5600 or waltonartscenter.org
Which brings us back to "A Christmas Story," which was adapted from the beloved 1983 movie about family, traditions and a boy's pursuit of an elusive BB gun for Christmas. And just how beloved? A recent nationwide poll asked respondents to name their favorite Christmas movie. Residents from 24 states picked "A Christmas Story" as their favorite, including those from Arkansas. In my home state of Kansas, "A Christmas Story" wasn't the top pick – "Die Hard" was – but nevertheless the annual 24-hour repeated showing of the film on Christmas day was a constant backdrop at my grandmother's house growing up.
With that kind of popularity and familiarity with the source material, the work must tread lightly. Because audience members will know much of the dialogue, there's only so far that the plot can stray away. Many of the vignettes that you would know from the film come to life in the stage show. When nine-year-old protagonist Ralphie (played alternately by Tommy Druhan and Ian Shaw) swears while helping his dad change a tire, he is forced to bite soap. When there's a triple dog dare issued for a classmate to touch a frozen pole with his tongue, we watch the action unfold. When Ralphie misses his chance to tell Santa the gift he wants and then scales a slide to reach Santa again, we see that, too. And, importantly, when family patriarch Mr. Parker (always referred to as The Old Man) wins a contest, he is given the major award of a leg lamp, just as in the movie. A few of the movie vignettes are missing, notably Ralphie's pursuit of clues during the radio show "Little Orphan Annie."
The premise of a radio show is how the plot of the stage version moves along. Original story author Jean Shepherd (played in Fayetteville by Chris Carsten) opens the evening on stage behind a microphone, reminiscing his past for his own radio program. His elaborate stories come to life behind him, then in front of him as he wanders into the action.
Because the stage version of the house, a department store and a Chinese restaurant near the Parker family home all look similar to their film counterparts, what's left to discover are the song and dance elements. One of the two is better than the other.
On the dance side of things, we get to watch the much-discussed leg lamp come to life, including a chorus of them that dance around The Old Man (played by Christopher Swan). The cancan sequence with leg lamps swinging into the air was well executed and fun. So was a fantasy dance sequence involving Ralphie's teacher Miss Shields (played by Lauren Kent) and his classmates. The number featured an extended tap segment involving about a dozen of the children that tour with this musical. It's the kind of gratuitous scene included to give the kids a showcase piece – but this one was nearly too cute to ignore.
As for the songs, I have a program that tells me what they were, but I don't recall any of them without looking back to that guide. The first song of the musical is "It All Comes Down to Christmas" and we're told to panic because we're only 21 days from the main event. Many of the other songs felt forced, too. The Old Man's "The Genius on Cleveland Street" is the kind of major key, propulsive number that's a staple of musical theater – and much too bright and cheery for a character I don't recall as having many of those qualities. I can't say any of the songs were poorly done. I thought the acting, singing and dancing were solid throughout the show, but there's only so much an actor can do with less-than-inspiring song material.
At the heart of "A Christmas Story," we're left with a nostalgic look at holidays of the past. The Jean Shepherd character of the show more than once hearkens back to days gone by in his monologues. We all once had a gift we coveted, and our Christmases did revolve around the pursuit of those toys – I was a Transformers kid myself. I should note that Ralphie's fetishizing of a gun and his dream sequences of vanquishing bad guys with it struck me in the wake of the very fresh incident of gun violence in Fayetteville and the memorial service just outside the front doors of the show's host venue moments before the show began. It may be something for you to consider before you attend the show.
We are ultimately encouraged to consider our own Christmases, and our families. I consider "A Christmas Story" a holiday classic, like many in Arkansas, apparently. But I think I'd rather watch the movie version at my grandmother's house over a slice of her chocolate meringue pie, if I still had that option. Or, perhaps another way to say it: The holidays are chaos. Be grateful for yours, however it works for you.


If you search the tag "my boyfriend does my makeup" on YouTube...forget about your productivity for the rest of the day, but you can guarantee some laughs.
We decided to do one of our own. Enter the willing participants: Amanda and Jared. Check it out below.


text:
by Lupe Fiasco
Album: Tetsuo and Youth (2015)
This song finds Fiasco lamenting on the simple problems of inner-city life such as getting a deep-dish pizza delivered to your crib. It was inspired by his upbringing in Chicago's Westside. He explained to Billboard magazine: "A song like 'Deliver' is the autobiography of a myth in the hood. You don't really recognize it until you are there. The hood doesn't really have the basic amenities that things that aren't the hood has. The pizza man might have two or three thoughts before he comes to your neighborhood. The pizza man might pull up, see your building and then keep driving. It's almost like a myth. Does that really happen? Does the pizza man really not come to the hood? In some cases it's true."
"In some places it's probably more of a story," Fiasco continued. One of the interesting things when we put the record out was that people were posting news stories about pizza men getting killed. There are some very serious reasons as to why. But it also speaks to the nature of those places like the places I grew up in, West Side of Chicago, South Side of Chicago. It's things like that, that are those odd aspects of the hood that don't really get a lot of attention like all drug dealers. But they have effects too."
Fiasco is assisted on the song by frequent collaborator Ty Dolla $ign who croons the hook and background vocals. Their other work together include Fiasco's cuts "Snitches" and "Next To It."
The song's music video was directed by Alex Nazari, who's also worked with 2 Chainz, YG and Tyga. Fiasco and Nazari previously teamed up on the Chicago rapper's cuts "Around My Way (Freedom Ain't Free)" and "#1234."
Here are a few more songs that reference pizza:
"Chic 'N' Stu " by System of a Down (Deals with compulsions and addictions using pizza as a metaphor.)
"This Town" by Rehab (One of the places mentioned is Mazzio's, which is a pizza place and a hang out spot in Rehab's home town of Winder, Georgia.)
"Shell Shocked" by Juicy J, Wiz Khalifa, Ty Dolla $ign (The three rappers spit rhymes referencing the Teenage Mutant Ninja Turtles' love of pizza.)
"Pizza And Pinball" by Rumer (The singer's imaginings about being a child in the 1970s include eating pizza and playing pinball.)
This was released as the first single from Tetsuo and Youth. The album cover was painted by Fiasco. He told Billboard: "I paint a lot - probably too much. I paint more than I write raps. It's the same creative thing for me. I started painting two years ago, and I gave myself 10 years to really get good. I'll sit and paint for 11 hours and get lost in it, the technique of it, trying to execute it clean, colors and palettes, etc."
"Van Gogh said he wasn't happy unless he was painting, and I'm starting to realize that's becoming true for me," Fiasco added. "If I'm not in a creative mode and I'm dealing with the outside world, I'm not really happy."
More songs from Lupe Fiasco
More songs about people with financial problems
Lyrics to Deliver
Lupe Fiasco Artistfacts
Todd RundgrenSongwriter Interviews
Todd Rundgren explains why he avoids "Hello It's Me," and what it was like producing Meat Loaf's Bat Out of Hell album.
Timothy B. Schmit of the EaglesSongwriter Interviews
Did this Eagle come up with the term "Parrothead"? And what is it like playing "Hotel California" for the gazillionth time?
Rick SpringfieldSongwriter Interviews
Rick has a surprising dark side, a strong feminine side and, in a certain TV show, a naked backside. But he still hasn't found Jessie's Girl.
Dave MasonSongwriter Interviews
Dave reveals the inspiration for "Feelin' Alright" and explains how the first song he ever wrote became the biggest hit for his band Traffic.
Producer Ron NevisonSong Writing
Ron Nevison explains in very clear terms the Quadrophenia concept and how Heart staged their resurgence after being dropped by their record company.
Francis Rossi of Status QuoSongwriter Interviews
Doubt led to drive for Francis, who still isn't sure why one of Status Quo's biggest hits is so beloved.


The opposite of anger is peace. Hebrews 12:14-15 commands us to, Make every effort to live in peace with everyone and to be holy; without holiness no one will see the Lord. See to it that no one falls short of the grace of God and that no bitter root grows up to cause trouble and defile many..
In processing the realization that my child died at my own hands, red hot anger against myself fired through my soul. As my abortion recovery class progressed, that anger was replaced with God's amazing peace. The calm was unexpected. I never realized how much energy was taken up by this hostile emotion!
A few years later I was standing on the steps of the Capitol, waiting to address the 750,000 people gathered for a pro-life event. As I waited, I listened to many angry tirades from those who spoke before me. They were angry that abortion was still legal. I then heard one speaker offering judgmental words against those of us who had made this choice.
As I realized that I was following these angry speakers – who held a great deal of respect from the crowd due to the thunderous applause – I prayed for God's help. He put Romans 12:1 on my heart, Therefore, I urge you, brothers and sisters, in view of God's mercy, to offer your bodies as a living sacrifice, holy and pleasing to God—this is your true and proper worship. I knew then what God wanted me to say.
"I am living proof that He (God) can use a sinner. You know, there's no sin that the Lord can't forgive. My sin in this choice is always before me. None of you can judge me any more than I have condemned myself over the years since my son's death. I know that my child rests in the arms of Jesus and I'll see him someday. God has chosen to heal my heart and given me a task in this work.
Many of you share my sin. You are here to lend your voice to stop this bloodshed. I am here to simply ask you to consider those of us who are walking in shame as you proceed. We are everywhere and need to know that God loves us too.
Afterwards I headed down into the crowd alone. Only one person stopped me to say, "Thanks." Most didn't notice me. I was a still small voice that day for God's own purposes.
You never know when God will amplify His wonderful message of hope through your own sacrifice. It is your reasonable service to be obedient in His leading!


John R. Bowlin is the Robert L. Stuart Professor of Philosophy and Christian Ethics at Princeton Theological Seminary, where he chairs the Theology Department. His primary interests are in theology and ethical theory, the history of moral theology, and social ethics and criticism. He is the author of Contingency and Fortune in Aquinas's Ethics (Cambridge 1999) and Tolerance among the Virtues (Princeton 2016).
7:00 PM | The Human Soul and Neuroscience: Is Belief in the Soul Obsolete?


With how busy our lives have become, mobile banking is as popular as ever. In fact, 60% of smartphone users say they use mobile banking in some way versus 36% who said so five years ago.* That's why at Great Western Bank we are working to provide you with the services and features you need at your fingertips - through the Great Western ebanking mobile app!
Remove payees no longer needed.
Our hope is that these enhancements help make everyday life easier with more flexibility and convenience! For more information on other services and features available to Great Western Bank customers, explore www.greatwesternbank.com or visit your local branch.


The archipelago off the west coast of Africa has just over 500,000 inhabitants, making it 100 times smaller than South Africa who they beat 2-1 in Praia on Friday and even more surprisingly by the same scoreline in Durban on Tuesday.
Cape Verde Islands are in contention for a shock World Cup spot with successive wins over South Africa giving them a fairytale chance to become the smallest nation to play at the finals.
The two victories took bottom side Cape Verde level with leaders Burkina Faso in Group D, offering the tantalising prospect of World Cup qualification for the first time as they eye a place at the finals in Russia next year.
"It is the dream of every footballer to go to the World Cup. It's the biggest tournament in the world. I dream about that and for me, nothing is impossible," said Dutch-born Garry Rodrigues, whose long-range wonder strikes secured Tuesday's win.
Rodrigues is one of many in the squad drawn from emigrant communities in France, the Netherlands, Portugal and the United States who have vastly strengthened the team in recent years.
Key for the Cape Verdeans is their next match at home to Senegal in Praia on Oct. 7 before they finish their qualifying campaign away to Burkina Faso in November.
It is not the first time the island nation, a former Portuguese colony, has punched above its weight.
Their 3-2 aggregate elimination of Cameroon over two legs in the 2013 African Nations Cup qualifiers saw them go to the continental championship for the first time where they continued the fairytale run by reaching the quarter-finals.
They were unbeaten at the 2015 finals but eliminated after three successive draws in their first round group and climbed as high as 39 in the FIFA rankings.
They were also keenly placed to reach the African playoffs for the last World Cup in Brazil before being docked points for using a suspended player in one of their qualifiers.
Coach Lucio Antunes, who burst into song at a news conference when the islanders reached the Nations Cup quarter-finals, was again full of emotion after Tuesday's triumph.
Trinidad & Tobago, with a population of 1.3 million, are the smallest country to have played at the World Cup when they qualified in 2006.


Did You Know Metal Roofing offers Quality Energy Efficiency For Commercial Properties?
As a business owner, you do everything you can to ensure your business and building run efficiently without issues. One thing that business owners worry about with their properties is energy efficiency. If your property does not consume energy at a normal rate, this will reflect in rising utility bills. What can you do to lower this? Updating your commercial roof may provide an answer. Metal roofing, a popular material, has been shown to offer businesses benefits that increase the energy efficiency, lowering electric and utility bills for the structures with it.
While well known for its durable protection and longevity, metal roofing can help with energy efficiency. This material is a reflective one, which means it can act as a cool roof system. Sunlight and UV rays can heat up a property and wear at materials. However, if your roof has a metal roof, it reflects these rays away. This prevents UV rays from affecting your building, keeping it from heating up. Your air conditioner will be able to cool the property without having to run all the time, meaning while you and your customers stay cool, your utilities can lower. During winter, it is equally as efficient, keeping cool air out and warm air in.
Need to get started on your new metal roofing system today? WeatherShield Roofing Group is always available to assist you with quality installations services in Longwood and Orlando, FL. Call us today at (407) 298-0100 for improved energy efficiency for your commercial property with metal roofing.
Posted in Uncategorized on July 20, 2018 at 8:11 am by WeatherShield Roofing.


Micah was the Nimir-Raj, and he had to make a decisions for the good of the pard.
Edward finally comes after Anita. With a roadside motel as their battleground, who will be the last man standing?
Anita hasn't ever realized just how much she relies on Edward.
If Edward won't stop talking, Anita just may shoot him. Richard approves.


We hope everyone enjoyed the short work week. Today we have some great selections. Enjoy.
Fall is in the air and Cristina from Remodelando la Casa shows us how to make these great DIY fall signs.
Jennifer from Classic Style Home reveals the new neutral office which used to be their formal living room.
Tania from Run To Radiance gives us a step by step on how to paint a rug on your concrete patio.
It is apple season, and Tanya from Twelve O Eight blog has a recipe for Baked Apple Pie Granola Bars.- Yummy!
We love a good hutch, and the collection that Alison put together over at The Polohouse has something for everyone's taste.
Please pin from the original posts. Now, go out and enjoy the rest of your week-end!
Okay the apple pie granola bars are on my list to make this week!!
It is a great project so we are glad we could share with others.


If we confess our sins, he is faithful and just and will forgive us our sins and purify us from all unrighteousness. … Yet to all who did receive him, to those who believed in his name, he gave the right to become children of God.
This entry was posted in 1 John, 2 Corinthians, Acts, Isaiah, John, Romans. Bookmark the permalink.


text:
The Thoresby Society
The Historical Society for Leeds and District
Contribs. Index
Volume Indexes
Leeds History
Leeds People
Leeds in Pictures
Leeds Timeline
Leeds Townships
Leeds Regalia
Leeds Bibliography
Thoresby's Diary
Library Treasures
Leeds Elections
Adel Parish Church Registers, 1606-1812, and Monumental Inscriptions
ed. by G. D. Lumb
ERRATA AND ADDENDA vi.
PREFACE vii.
FIRST BOOK BAPTISMS, 1606-1655 1
MARRIAGES ,, 19
BURIALS ,, 26
MISCELLANEOUS NOTES, 1648-1661 40
SECOND BOOK BAPTISMS, 1655-1679 43
MARRIAGES " 53
BURIALS " 57
THIRD BOOK BAPTISMS, 1680-1782 67
MARRIAGES, 1680-1753 110
BURIALS, 1680-1782 122
FOURTH BOOK MARRIAGES, 1754-1812 155
FIFTH BOOK BAPTISMS. 1783-1812 172
BURIALS ,, 190
APPENDIX MR. KIRKE'S PREFACE 202
MEMORANDA 203
MONUMENTAL INSCRIPTIONS 205
INDEX LOCORUM 214
INDEX NOMINUM 216
Plates.
ADEL CHURCH (EXTERIOR) frontispiece
DOORWAY to face page 1
PREFACE.
THE present Volume gives the entries contained in the earliest Five Books of the REGISTERS of ADEL CHURCH, and covers a period extending from the year 1606 to the year 1812.
The First and Second Books, and the transcript of them by Mr. Thomas Kirke, of Cookridge (a friend of Thoresby), are bound in one volume. The First Book contains 31 parchment leaves, measuring 11¼ by 5½ inches, and it will be noticed that some leaves are missing. The Second Book contains 15 parchment leaves, 12¼ by 7 inches, and one-third of another one, and chasms have been filled by Mr. Kirke from Oglethorp's pocket-book (see p. 50). The entries in the Third Book are written on 66 parchment leaves (folio size). The Fourth Book (folio) contains 48 parchment and three paper leaves, and also a few paper leaves with the marriage forms not filled in. And the Fifth Book (folio) has 34 parchment leaves on which entries have been made, and also a number of spare leaves at the end, on some of which various memoranda connected with the parish have been written.
Superfluous words, such as " baptized," have been omitted in the first and second Books, and the entries in the fourth Book have been abstracted, but the entries in the third and fifth Books are printed as they appear in the original.
The Baptisms, Marriages, and Burials of each Book are printed separately, but in the original the baptisms are generally entered on the page to the left, and the burials and marriages on the opposite page; but often in the earlier Books they are mixed up indiscriminately.
In the first and second Books the writing is occasionally ex-ceedingly difficult to decipher, and many entries in the third Book have been written by illiterate parish clerks. Nearly the whole of the fourth and fifth Books is in the handwriting of the Rev. Henry Nicholson, curate and rector.
The following explanation of the abbreviations which occur in the Registers may be found useful:—
b for "her," as Robt for Robert.
p for "per," as Cowp for Cowper.
9 for " er," as Pe? for Peter.
— for "m" or "n" omitted, as Ada for Adam, so for son.
for some letters omitted. Jo. for John. X| ofer for Christopher.
Various memoranda, and copies of the tombstones contemporary with the Registers, appear in the Appendix.
Descriptions of the church and parish of Adel have been written at different times by Thoresby, Whitaker, and Lewthwaite, and in the first volume of The Churches of Yorkshire, 1845; some of which works contain excellent views of the church.
The thanks of the Society are due to the rector, the Rev. Charles H. Owen, M.A., for permission to publish the Registers, and the Editor desires to acknowledge the uniform courtesy and kindness he has received from Mr. Owen, and the facilities given for transcription.
The Editor desires also to express his appreciation of the interest which has been taken in the work by the Rev. Charles Best Norcliffe, M.A., of Langton Hall, who has not only added many valuable notes to the Registers from his large genealogical collections, but has made several useful suggestions respecting the transcript.
The thanks of the Society are likewise due to Mr. John Kirk, of Headingley, for the plate of the etching giving a view of the church as it existed from 1843 to 1879, and also to Mr. Herbert Denison, of Leeds, for preparing a photogravure of the doorway; and to Mr. James Midgley, of Burley Grove, for defraying the cost of printing the views.
The Editor has carefully compared the printed copy with the Registers, and has endeavoured to make the transcript a correct one, and the index reliable.
G. D. L. LEEDS, Jan. 17th, 1895.
Full scan on archive.org includes the Calverley Charters volumes
The Thoresby Society, The Leeds Library, 18 Commercial Street, Leeds, LS1 6AL.
Company Limited by Guarantee No 6649783 Registered Charity No 1126086
Copyright ©2000-2019 The Thoresby Society


text:
Q: Backstepping control of second order nonlinear system $\dot{x_{1}}=x_{2}^2-3\sin(x_{1})x_{2}$
$\dot{x_{2}}=x_{1}^3-3x_{2}\cos(x_{1})+u^{1/2}$
Question: Using the backstepping method and Lyapunov function, design the controller $u$ that will make the following system asymptotically stable.
For backstepping control, $x_{1}$ must be in this form;
$\dot{x_{1}}=f(x_{1})+g(x_{1})x_{2}$
then,
$e_{1}=x_{1}-x_{1}^d,\space \dot{e_{1}}=\dot{x_{1}}-\dot{x_{1}}^d$, where $e_{1}: error, \space x_{1}^d: desired\space x_{1}$
and Lyapunov function is $L_{1}=\frac{1}{2}e_{1}^2, \space \dot{L_{1}}=e_{1}\dot{e_{1}}$.
For asymptotically stability, $\dot{L_{1}}=-k_{1}e_{1}^2, k_{1}>0$.
Now we have a $x_{2}$ that extracted from $\dot{L_{1}}$ and this $x_{2}=x_{2}^d$ for $e_{2}=x_{2}-x_{2}^d$. Where, $\dot{L_{2}}=e_{2}\dot{e_{2}}^2$ and $\dot{L_{2}}=-k_{2}e_{2}^2, k_{2}>0$. Finally we can extract $u$ from $\dot{L_{2}}$.
But in this question $\dot{x_{1}}$ is not in form $\dot{x_{1}}=f(x_{1})+g(x_{1})x_{2}$. So how can I solve this question with backstepping control?

redpajama_set_name: RedPajamaStackExchange


text:
Blockchain Analytics Firm Nansen Raises $75 Million in Funding Led by Accel
The blockchain analytics platform Nansen raised $75 million via a Series B funding round led by the venture capital firm Accel. The investment will aim to expand the former's global presence and accelerate the development of innovative product offerings for customers.
Nansen's Newest Funding Round
The Singapore-based blockchain platform announced the news on its official website. Apart from Accel, the funding round was supported by several other venture capital firms, including Andreessen Horowitz (a16z), GIC, Tiger Global, SCB 10X, and more.
Nansen informed that the investment will fuel the company's next phase of growth and development. It also intends to add innovative platform features, multichain integrations and expand globally.
The last funding round which Nansen raised was in June this year. The company noted that since then, it had introduced several advancements. Some of them include broader blockchain support from Ethereum and Polygon to Binance Smart Chain (BSC), Fantom, Avalanche, Celo, and Ronin.
Nansen also joined the non-fungible token space and took part in over 18,000 NFT collections. Additionally, the company hired 55 new team members across 28 countries.
"2021 has been an inspirational year for us. We're grateful for how much our customers love Nansen, and with this latest round, we can serve them even better. In addition, we can now onboard new customer segments that will benefit from market-leading on-chain analytics", said Alex Svanevik – Co-Founder and CEO at Nansen.
Following the most recent initiative, Nansen is ready to provide global investors with a "world-class product to explore the latest crypto trends and make more informed decisions," Svanevik assured.
Andrei Brasoveanu – Partner at Accel – predicted that the funding round could provide "huge" opportunities for Nansen. It even has the potential to become "investors' go-to platform for crypto insights and data."
Nansen's Previous Endeavors
At the beginning of October, the data-analytics provider added support for the decentralized smart contracts platform – Fantom Network.
Consequently, Nansen began showing Fantom's data on its dashboards. Speaking on the matter, Michael Kong – Fantom's CEO – said:
"Nansen analytics will bring visibility to innovative projects on the network and amplify their successes. We're excited about the increased usage Nansen will bring to Fantom and the incredible wealth of information that the analytics platform will make available."
Interestingly, shortly after announcing the news, the native token of Fantom Network – FTM – soared by 30%.
PrevPreviousthis is why it's rising
NextBTC/USD Trades Marginally as Bitcoin Pauses Above $45.5kNext


So I was at the pool today doing laps and because I'd forgotten my goggles I was wearing my glasses, which meant I could do a lot of people watching. A couple caught my attention – she was very pregnant and also getting a toddler ready for a swim with her husband. The lesson began and it all went well, he had a good time, the toddler was happy, the mum just sat there enjoying a bit of a rest.
Then the husband got out, handed the toddler over and stood in front of her. At this point it became obvious that he was a classic Silver Fox: a bit older, maybe mid forties, nice face and a very fit body (I'm only human). He had on some kind of white rash vest and little lime green shorts, and he stood in front of her and peeled off the shirt slowly, then stood for a while in front of her in just his shorts, hoping, I suspect for a bit of richly deserved admiration.
She was doing that awkward dressing toddler with baby bump thing where a shoe dropped on the floor necessitates much grunting and puffing to retrieve it and you get completely out of breath because it feels like your lungs are squashed up into your neck and it's all just a bit difficult and she didn't so much as glance up at her husband. He stood there for a few more seconds, towelling his chest, flexing a leg. Not a flicker of interest. A bit of a stretch, a light towelling off the hair. Nothing.
Eventually he just wrapped a towel around himself, sat down and started fiddling with his phone. At that point she finally turned to him and said something, and he reached into one of their fourteen bags and pulled out a packet of Tiny Teddies.
And in a round about way that is a bit how I am feeling about East London and this blog. It's a fabulous place, there's so much there to see and enjoy, but I'm just a bit distracted with child rearing and the most gorgeous scenery or fabulous restaurant isn't quite as absorbing and all-encompassing as the sheer will of a toddler and the mewlings of a newborn.
I'm also not living in East London and more, and haven't been for quite some time. So I'm going to leave it for now, until I can summon up the energy to move back again….


text:
Photos and video: X-Teams race for the right to rove on the moon
10. W3af - Web application attack and audit framework – ComputerWeekly.com
Attacking Angry Droids : Mobile Malware – ComputerWeekly.com
HAL - 2001: A space odyssey – ComputerWeekly.com
Synergy Moon - X-Teams race for the right to rove on the moon
Source: Interorbital Systems
Synergy Moon also participated in the Ansari X Prize and is the only other team entering for the Lunar X Prize that is building its own rocket.
The rest plan to find their lunar rover a ride on a commercial rocket taking satellites into orbit. Synergy Moon is aiming to complete the mission by launching a rocket built by Interorbital Systems directly to the moon's surface.
"I think we're the only ones who have a chance," says Randa Milliron, CEO of Interorbital Systems. "If you haven't reserved a rocket by this time, there's no way a team would be able to get a moon flight going."
More than a dozen teams are racing to claim the $30 million Lunar X Prize for landing a robot rover on the moon.
This article first appeared on New Scientist.


text:
\section{Introduction}
Over the past decade, deep neural networks (DNNs) achieved profound success in many fields including, computer vision and natural language processing. However, training such state-of-the-art DNNs may necessitate thousands of petaflops and many Gigabytes of storage to contain trillions of parameters. Thus,  researchers from academia and the industry are actively seeking new techniques to reduce the power and increase the throughput of DNNs models for both training and inference.  The main techniques in this area include quantization, knowledge distillation, and pruning.

Pruning DNNs is one of the most effective and widely studied methods to improve DNN resource efficiency. Since DNNs are over-parametrized most researchers focused on weights pruning. Yet, recently researchers suggested that sparsity of activation \cite{Jaszczur2021SparseIE,pmlr-v119-kurtz20a} and gradients \cite{Chmiel2020NeuralGA} could be exploited as well. All these types of \textit{unstructured} pruning only reduce the memory footprint \cite{lottery,evci2020rigging}. It is possible to also reduce the compute footprint by enforcing some structure such as block sparsity \cite{blockSparsity}, filter sparsity \cite{l1Filter}, or N:M fine-grained sparsity \cite{Nvidia,Hubara2021AcceleratedSN,Mishra2021AcceleratingSD}.

N:M fine-grained sparsity can be used in one of  Nvidia's sparse tensor cores  \cite{Nvidia,Mishra2021AcceleratingSD} in order to accelerate matrix multiplication. N:M fine-grained sparsity requires that, at least for one of the two matrices involved in the matrix multiplication, N out of every M contiguous elements would be pruned. For example, Nvidia demonstrated \cite{Nvidia} a 2:4 format can accelerate inference by x2. They suggested using a three-step scheme: (a) train a dense model, (b) prune weights to obtain a fine-grained 2:4 fixed mask,
and (c) use the original training regime to retrain with the masked weights.

Following work suggested methods to accelerate different parts of this scheme. First, \citet{n:mStructured} was able to omit steps (a) and (b) by training with an N:M mask from scratch using a straight-through estimator (STE) and additional regularization. Specifically, they keep a dense copy of the weights and set different weight decays rates to the masked and unmasked weights

Next, \citet{Hubara2021AcceleratedSN} focused on accelerating the remaining step (c), i.e., sparse training. Generally, there are three General Matrix Multiplications (GEMMs) involved in a training step of a single DNN layer: (1) forward pass with a GEMM between the weights and activations, (2) backward pass with a GEMM between the gradients and transposed weights, and (3) update phase with a GEMM between the gradients and activations. While Nvidia suggested accelerating only the inference phase (i.e., the forward pass), the backward and update passes were kept dense. Noting that the backward phase uses the transposed (sparse) weight matrix, \citet{Hubara2021AcceleratedSN} used a transposable mask, i.e., a mask that can be transposed and still match the N:M fine-grained structure. This enabled the acceleration of the backward phase. Although \citet{Hubara2021AcceleratedSN} suggested different methods to find the optimal transposable mask efficiently, they did not offer ideas on how to accelerate the update phase matrix multiplication.

In this work we explore different methods to accelerate the update phase as well. To that end, we examine gradients with fine-grained pruning and establish a tensor-level optimallity criteria. Our experiments suggest that while the traditional method crashed, our method with 1:2 sparsity is sufficient  for  training,  and  2:4  sparsity  is  usually enough when this is not the case. Moreover, we suggest to combine several such methods together (fine-grained sparse gradient and sparse transposable fine-grained weights) in order to speed up training even more and for the first time accelerate all training phases with N:M fine-grained sparsity. Moreover, we studied the effect of applying N:M
structured sparsity on the activations and showed results of N:M fine grained sparsity for inference that are equivalent to 2-bit quantization achieving  on par results. In \cref{fine-grained_methods} we summarize all the N:M fine-grained structured sparsity methods, which part of the network they accelerate and the relevant optimallity criteria we use. Additionally, we showed the configurations used to fullly accelerate training or inference.

This paper makes the following contributions:
\begin{itemize}
\item We developed unbiased minimum variance optimality criteria for pruning neural gradients with N:M structured sparsity.
\item We propose 1:2 and 2:4 unbiased minimum variance methods to prune the neural gradients and demonstrate that they achieve small or no degradation where previous methods failed.
\item We are able to speed up training by 2x for the first time with our new method combined with previous methods of N:M structured sparsity.
\item We show combining N:M sparsity for activations and weights, together with and 4-bit quantization achieves on par results with quantization-aware-training results for 2-bit inference.
\end{itemize}

\begin{table}[h!]
\centering
\caption{Exploring fine-grained sparsity on different tensors  (activations,weights,gradients) and with different sampling methods (MVUE, MSE). For inference scheme we used  sparse weights  and activations and for our training scheme we used transposable weights and sparse gradients. }
\label{fine-grained_methods}
\resizebox{\linewidth}{!}{
\begin{tabular}{lccc}
\toprule
\diagbox[width=9.5em]{Tensor}{Phase}& \multicolumn{1}{|l|}{Forward} & \multicolumn{1}{l|}{Backward} & \multicolumn{1}{c}{Update} \\ \midrule
\multicolumn{1}{l|}{\multirow{2}{*}{\vspace{10pt} Weights}} &  \multicolumn{1}{l|}{\cmark (MSE)} & \multicolumn{1}{l|}{ \xmark} & \xmark\\
\multicolumn{1}{l|}{\cite{Nvidia}} & \multicolumn{1}{l|}{} & \multicolumn{1}{l|}{}\\\hline
\multicolumn{1}{l|}{\multirow{2}{*}{ \vspace{10pt} T-Weights}}  & \multicolumn{1}{l|}{\cmark (MSE)} & \multicolumn{1}{l|}{\cmark (MSE)} & \xmark\\
\multicolumn{1}{l|}{\cite{Hubara2021AcceleratedSN}} & \multicolumn{1}{l|}{} & \multicolumn{1}{l|}{}\\\hline
\multicolumn{1}{l|}{Activations (\textbf{ours})} & \multicolumn{1}{l|}{\cmark (MSE)} & \multicolumn{1}{l|}{\xmark} & \multicolumn{1}{c}{\xmark}  \\ \hline
\multicolumn{1}{l|}{Gradients (\textbf{ours})} & \multicolumn{1}{l|}{\xmark} & \multicolumn{1}{l|}{\xmark} & \multicolumn{1}{c}{\cmark (MVUE)}   \\
\midrule \midrule
\multicolumn{1}{l|} {\multirow{2}{*}{\vspace{40pt} \underline{Inference Config.}}} & \multicolumn{1}{l|}{\cmark (MSE)} & \multicolumn{1}{l|}{\xmark} & \multicolumn{1}{c}{\xmark}  \\
\multicolumn{1}{l|}{  Weights+Activations } & \multicolumn{1}{l|}{} & \multicolumn{1}{l|}{}& \multicolumn{1}{l}{}\\
\hline
\multicolumn{1}{l|} {\multirow{2}{*}{\vspace{40pt} \underline{Training Config.}}} & \multicolumn{1}{l|}{\cmark (MSE)} & \multicolumn{1}{l|}{\cmark (MSE)} & \multicolumn{1}{c}{ \cmark (MVUE)}   \\
\multicolumn{1}{l|}{  T-Weights+Gradients } & \multicolumn{1}{l|}{} & \multicolumn{1}{l|}{}& \multicolumn{1}{l}{}\\
\bottomrule
\end{tabular}
}
\end{table}

\section{Related works}
\label{sec:related}
Pruning has been extensively investigated in the last few years. Most of the pruning methods focus on pruning of the weights \cite{evci2020rigging, lottery,Janowsky1989,liu2018rethinking}. Unstructured pruning methods achieved impressive sparsity ratio with minimal or no accuracy degradation, e.g. \citet{Renda2020ComparingRA} achieved over 80\% sparsity in ResNet50- ImageNet dataset without sacrificing accuracy. Despite this impressive achievement, the ability of unstructured pruning methods to actually reduce computational resources of modern hardware is limited \cite{Nvidia,Mishra2021AcceleratingSD}.

Structured pruning methods vary between coarse-grained and fine-grained methods. Coarse-grained methods  such as filter-wise or layer-wise pruning \cite{l1Filter,thinet,blockSparsity}  are naturally supported by hardware and software but none of these methods were able to maintain the test accuracy for sparsity ratio higher than 50\%. Recently, Nvidia introduced the Ampere GPU architecture \cite{Nvidia,Mishra2021AcceleratingSD} hardware with software support for N:M fine-grained structured sparsity. Specifically, they showed that 2:4 fine-grained structured sparsity, where two of every four contiguous elements are zero, achieves a 2x improvement in the GEMM operation. They suggested a three-step scheme to accelerate inference. Later, \citet{n:mStructured} accelerated their method by avoiding the first two steps. Next, \citet{Hubara2021AcceleratedSN} accelerated the remaining training step by suggesting transposable mask, which accelerates both the forward and backward phases ($\frac{2}{3}$ of the training). \citet{Stosic2021SearchSF} further demonstrated the transposable mask can accelerate training with minimal accuracy degradation on 20 different models for various tasks and datasets. \citet{ChannelPermute} suggested permuting the weight matrices to improve accuracy of sparse models for inference. \citet{sun2021dominosearch} suggested a mixed layer-wise N:M sparsity schemes to improve the uniform sparsity scheme with similar complexity constraints. \citet{Holmes2021NxMTransformerSS} suggests a new learning framework to improve the performance of N:M sparse NLP models on downstream tasks.

Beyond pruning the weights, recent work also focuses on unstructured sparsity of the activations or neural gradients. \citet{pmlr-v119-kurtz20a} suggested a parametrized activations function called Forced-Activation-Threshold Rectified Linear Unit (FATReLU) which increases the naturally sparse of ReLU with any accuracy loss. \citet{Jaszczur2021SparseIE} studied the sparsification of the activations in Transfomer-based models. "MeProp" \citep{meProp2017} prunes the K smallest absolute-valued entries of the neural gradients on the fly, using the top-k algorithm. \citet{SWAT2020} used top-k pruning on the copies of weights and activations used in the backpropagation. \citet{acceleratedSpars2019}, suggested "stochastic pruning", reaching higher sparsity levels on the neural gradient.  \citet{Chmiel2020NeuralGA} improved their results with a lognormal distribution approximation for the neural gradient achieving more than 80\% sparsity on the neural gradients without accuracy degradation.

In parallel to our work, \citet{McDanel2022AcceleratingDT}  suggested a method to use N:M structured data pruning for the neural gradients to accelerate the backward phase, which was also accelerated in \citet{Hubara2021AcceleratedSN}. In \cref{sec:AppCompareSdgp} we show the degradation of applying \citet{McDanel2022AcceleratingDT} method also in the update phase.  As far as we know, no previous work suggested using N:M fine-grained sparsity to accelerate the update phase, by pruning the neural gradients. Moreover, we are the first work that suggest to use N:M fine grained sparsity on the activations to accelerate inference.

\section{Which optimality criteria to use?}
When pruning weights during training, we require local (tensor level) criteria to select which weights to prune. Recently, \citet{LUQ} investigated which optimality criteria to use, but in the context of quantization. They found, that when quantizing the activations, we should minimize the mean-square-error (MSE) of the quantization error. In contrast, for the neural gradients, they found that it is much more critical to use unbiased quantization (stochastic rounding). Based on their findings, we suggest to apply N:M fine-grained sparsity on the activations and neural gradients with the same optimality criteria:

(1) For the activations we focus on reducing the MSE. This is done with the greedy method of keeping the $M-N$ largest elements in each block of size $M$. Exploiting the inherent ReLU unstructured sparsity we experiment with forcing this form structured sparsity to vision models with ReLU activation functions, as discussed in section \cref{sec:sparseAct}.

(2) For the neural gradients we focus on finding an unbiased estimator. From all the possible unbiased estimators, we will prefer the one that reduce the MSE. The MSE can be represented as follows: for a random variable $x$ and a N:M sparsity operator $\theta$ we can write $\text{MSE}[\theta(x)] = E[\theta(x) - x]^2 = \operatorname{Var}[\mathrm{\theta}(x)] +  \operatorname{Bias}^2[\mathrm{\theta}(x)]\,$. Since we focus on an unbiased estimator (i.e., $\operatorname{Bias}[\mathrm{\theta}(x)]=0$), we arrived to the optimality criteria of the minimum variance unbiased estimator (MVUE) for the neural gradients.

\section {N:M Minimum variance unbiased estimator}
In this section, we propose two unbiased estimators with minimum variance. The first corresponds to the 1:2 case and the second to the 2:4 case.

\subsection{Minimum variance unbiased estimator for 1:2}
\label{sec:1_2_mvue}
For a block $[a,b]$, one entry needs to be pruned:
\begin{equation}
\theta\left([a,b]\right)= \begin{cases}{[v_1, 0]} & ,  \text{w.p. }   p \\
{\left[0, v_2\right]} & ,  \text{w.p. }  1-p\end{cases}
\label{unbiasedEq}
\end{equation}
We wish to design an unbiased estimator for this pruning method where $E[\theta\left([a,b]\right)]=[a,b]$:
\begin{equation}
    E[\theta\left([a,b]\right)] = p\cdot [v_1, 0] + (1-p) \cdot [0, v_2] = [a,b]
\end{equation}
To find an unbiased estimator which minimizes the total block variance, we now calculate the variance for each element in the block $\theta\left([a,b]\right)=[x,y]$ as follows:
\begin{equation}
\begin{aligned}
&\operatorname{Var}[x]=E\left[x^{2}\right]-E^{2}(x)=v_{1}^{2} \cdot p -a^2\\
&\operatorname{Var}[y]=E\left[y^{2}\right]-E^{2}[y]=v_{2}^{2} \cdot(1-p) -b^2
\end{aligned}
\end{equation}
Then, the total variance of a block is the sum of its element variances:
\begin{equation}
 \mathrm{Var}_B\left[\theta\right] = \mathrm{Var}[x] + \mathrm{Var}[y] = v_{1}^{2} p-a^{2} +  v_{2}^{2}-v_{2}^{2} p-b^{2}
 \label{TotalVariance_main}
\end{equation}
Using equations \ref{unbiasedEq} and \ref{TotalVariance_main} together we obtain the following expression for the total variance in the block specified by $v_1$ alone (for more information, see \cref{sec:App-1_2_mvue}):
\begin{equation}
\begin{aligned}
\operatorname{Var}_B[\theta] = v_{1}\cdot a - a^{2}+\frac{b^{2} \cdot v_{1}}{v_{1}-a}-b^{2}
\end{aligned}
\label{Var_thetha_main}
\end{equation}
By by finding the derivative of equation \ref{Var_thetha_main} with respect to $v_1$ and setting it to zero we get the  following equation:
\begin{equation}
\frac{\partial Var_B[\theta]}{\partial v_{1}}=\frac{b^{2}}{v_{1}-a}-\frac{b^{2} v_{1}}{\left(v_{1}-a\right)^{2}}+a=0
\label{equation_main}
\end{equation}
The solution to Equation \ref{equation_main} gives the following unbiased estimator, which has the lowest variance of all unbiased estimators (full details in \cref{sec:App-1_2_mvue}).
\begin{equation}
\theta\left([a,b]\right)= \begin{cases}{[\text{sign}(a)\cdot (|a|+|b|), 0]} & ,  \text{w.p. }   \frac{|a|}{|a|+|b|} \\ {\left[0, \text{sign}(b)\cdot(|a|+|b|)\right]} & ,  \text{w.p. }  \frac{|b|}{|a|+|b|}\end{cases}
\label{our_method_main}
\end{equation}

Let us calculate the mean square error of this unbiased method. By substituting into Equation \ref{Var_thetha_main} the optimal solution for $v_1$, the optimal method outlined in \ref{our_method_main} has a variance of $2ab$. Therefore, since the method is unbiased it results with a mean-square-error of
\begin{equation}
    \mathrm{MSE} = \text{Bias}^2 + \text{Var} =  0+ 2ab = 2ab
\end{equation}

In \cref{tab:1_2_ablation} we compare different  1:2 structured sparsity on the neural gradients on ResNet18 Cifar10 dataset. We show that although the proposed MVUE method doesn't minimize the MSE, it gets the best results.

\begin{table}[h!]
\centering
\caption{1:2 sparsity on the neural gradients of ResNet18 cifar10 dataset. 'Greedy' refers to the traditional method of choosing the smallest element for each block. 'Biased' refers to the case $[v_1,v_2] = [a,b]$ in \cref{unbiasedEq}. 'Uniform' refers to uniform sample, i.e. $p=0.5$, $[v_1,v_2] = [a,b]$. 'Unbiased' refers to unbiased uniform sample, i.e $p=0.5$, $[v_1,v_2] = [2a,2b]$. 'MVUE' refers to the minimimum variance unbiased estimator of \cref{our_method_main}.}
\label{tab:1_2_ablation}
\begin{tabular}{lc}
\toprule
\multicolumn{1}{l|}{Method} & \multicolumn{1}{c}{Accuracy (\%)}   \\ \midrule
\multicolumn{1}{l|}{Baseline} & \multicolumn{1}{c}{90.02}  \\ \hline
\multicolumn{1}{l|}{Greedy} & \multicolumn{1}{c}{85.5}  \\ \hline
\multicolumn{1}{l|}{Biased} & \multicolumn{1}{c}{71.8}  \\ \hline
\multicolumn{1}{l|}{Uniform} & \multicolumn{1}{c}{85.8}  \\ \hline
\multicolumn{1}{l|}{Unbiased} & \multicolumn{1}{c}{87.2}  \\ \hline
\multicolumn{1}{l|}{MVUE (Ours)} & \multicolumn{1}{c}{\textbf{89.98}}  \\
\bottomrule
\end{tabular}
\end{table}

\begin{figure*}[h!]
\begin{subfigure}{0.33\textwidth}
  \centering

  \includegraphics[width=\linewidth,trim=15mm 0 15mm 0]{imgs/deter_illus.pdf}
  \caption{}
 \end{subfigure}
\begin{subfigure}{0.33\textwidth}
  \centering

  \includegraphics[width=\linewidth,trim=15mm 0 15mm 0]{imgs/unbiased_illus.pdf}
  \caption{}
\end{subfigure}
\begin{subfigure}{0.33\textwidth}
  \centering

  \includegraphics[width=\linewidth,trim=15mm 0 15mm 0]{imgs/mvue_illus.pdf}
  \caption{}
\end{subfigure}
\caption{1:2 Sparsity for blocks located on the first quarter of the unit circle. The blocks [a,b] (represented by red dots) are sampled 100 times each, and then averaged (green dots) using one of three methods: (a) \textit{greedy} is the traditional method that generates the block [0,b] if $a\leq b$, or [a,0] otherwise. In this method, all 100 samples are the same for each block, resulting in a biased average. (b) unbiased -  each block [a,b] is equally likely to be pruned to [2a, 0] or [0,2b]. Although the average of the 100 samples is unbiased, it does not have minimum variance. (c) Our unbiased method with minimum variance (outlined in Equation \ref{our_method}), exhibits a smaller spread here than in (b).}
\label{fig:Illustration}
\end{figure*}

\subsection {Optimality criteria for 2:4}

We now extend the results from the previous section to 2:4. With a block $[a_1,a_2,a_3,a_4]$, we wish to construct an unbiased 2:4 block with minimum variance. It can be seen that by assigning for each index $ i $ an element $ \frac {a_i}{p_i} $ with a probability $ p_i $, the expected value of the block is $ [a_1, a_2, a_3, a_4]$ since
\begin{equation}
E[\theta(a_i)]=\frac{a_i}{p_i}\cdot p_i + 0\cdot (1-p_i) = a_i
\end{equation}
for each $i\in \{1,2,3,4\}$

The variance of each element in the pruned block is given as follows:

\begin{equation}
\label{eq:varElement}
\begin{aligned}
\operatorname{Var}\left[\theta\left(a_{i}\right)\right] &=E\left[\theta\left(a_{i}\right)^{2}\right]-E\left[\theta\left(a_{i}\right)\right]^{2}=\\
&=\underbrace{\left(\frac{a_{i}}{p_{i}}\right)^{2} \cdot p_{i}+0^{2} \cdot\left(1-p_{i}\right)}_{E\left[\theta\left(a_{i}\right)^{2}\right]} -a^{2}= \\
&=\frac{a_{i}^{2}}{p_{i}}-a_{i}^{2}
\end{aligned}
\end{equation}

Therefore, total variance $\operatorname{Var[\theta]} = \operatorname{Var}\left[\theta\left(a_{1},a_2,a_3,a_4\right)\right]$ would be
\begin{equation}
    \operatorname{Var}\left[\theta\right] =  \sum_{i}\left(\frac{a_{i}^{2}}{p_{i}}-a_{i}^{2}\right)
    \label{variance}
\end{equation}
For each $i\in\{1,2,3,4\}$ we have the equality constraint
\begin{equation}
    \sum_{ i} p_{i}-2=0
\end{equation}
and inequality constraint:
\begin{equation}
    p_i -1 \leq 0
\end{equation}
From KKT conditions:
\begin{equation}
L = \sum_{j}\left(\frac{a_{j}^{2}}{p_{j}}-a_{j}^{2}\right)+ \sum _{j}\lambda_{j}(p_{j}-1)+\mu \sum_{j} (p_{j}-2)
\end{equation}
then, taking the derivative with respect to $p_i$
\begin{equation}
\frac{\partial L}{\partial p_i}=-\frac{a_{i}^{2}}{p_{i}^{2}}+\lambda_{i}+\mu=0
\label{kkt}
\end{equation}

With each $i$, the constant $\lambda_i$ could be zero or not. Using Equation \ref{kkt} for $\lambda_i=0$ we get that
\begin{equation}
p_{i}=\frac{a_{i}}{\sqrt{\mu+\lambda_i}} = \frac{a_{i}}{\sqrt{\mu}}
\label{lambda_zero}
\end{equation}
This coupled with the normalization constraint $\frac{\partial L}{\partial \mu}=0$ implies that
\begin{equation}
\sum_{j} p_{j}=\frac{\sum_{j} a_{j}}{\sqrt{\mu}}=2 \Rightarrow  \sqrt{\mu}=\frac{\sum_{i} a_{i}}{2}
\end{equation}
Finally, together with Equation \ref{lambda_zero} we have that for each $i\in\{1,2,3,4\}$
\begin{equation}
\label{eq:optimality_24}
    p^*_{i}=\frac{2 a_{i}}{\sum_{i} a_{i}}
\end{equation}
Turning to the case where $\lambda_i>0$, we have $p_i-1=0$ (i.e., $p^*_i=1$) because of the complementary slackness condition. The normalization constraint guarantees that $\sum_{k \neq i} p_{k}+1=2$ and therefore $\sum_{k \neq i} p_{k}=1$ i.e., all other probabilities are in the range $(0,1)$ and so $\lambda_k=0$ for every $k,k \neq i$. Therefore, from equation \ref{kkt} we have that
\begin{equation}
-\frac{a_{k}^{2}}{p_{k}^{2}}+\mu=0 \Rightarrow p_{k}=\frac{a_{k}}{\sqrt{\mu}} \quad k \neq i
\label{p_k}
\end{equation}
Since $\sum_{k \neq i} p_{k}=1$  we have that  $\sum_{k \neq i}\frac{a_{k}}{\sqrt{\mu}}=1$, and therefore $\sqrt{\mu}=\sum_{k \neq i} a_{i}$. Thus, we can substitute $\sqrt\mu$ in Equation \ref{p_k}
\begin{equation}
    p^*_{k}=\frac{a_{k}}{\sum_{k \neq i} a_{i}} \quad k \neq i
    \label{opt_p_1}
\end{equation}

We provide an algorithm in \cref{sec:min_varAlgo} for identifying pairs of elements for 2:4 policy that meets the optimization criteria stated in Equations \ref{opt_p_1} and \ref{eq:optimality_24}.

\subsection {A comparison of optimal 1:2 and optimal 2:4}
Given a block $B = [a_1,a_2,a_3,a_4]$, we can either apply optimal 2:4 method directly on that block $\theta_{2:4}(B)$ or we can break it into two sub-blocks $b_1=[a_1,a_2]$ and $b_2=[a_3,a_4]$, and apply optimal 1:2 method twice i.e.,  $\theta_{1:2}(b_1)$ and $\theta_{1:2}(b_2)$. The following proves that the former alternative is preferable and introduces less variance, i.e.,
\begin{equation}
    \operatorname{Var}[\theta_{2:4} (B)] \leq \operatorname{Var}[\theta_{1:2}(b_1)] + \operatorname{Var}[\theta_{1:2}(b_2)]
\label {claim}
\end{equation}
To prove this claim we first find $\operatorname{Var}[\theta_{2:4} (B)]$  by assigning a probability $p^*_i=\frac{a_i}{a_1+a_2+a_3+a_4}$ to each element $i$ in Equation \ref{variance}:
\begin{equation}
\begin{aligned}
\operatorname{var}\left[\theta^{*}\right]&= \frac{a_{1}}{2}\left(a_{1}+a_{2}+a_{3}+a_{4}\right)-a_{1}^{2}\\
&+\frac{a_{2}}{2}\left(a_{1}+a_{2}+a_{3}+a_{4}\right) -a_{2}^{2}\\
& +\frac{a_{3}}{2}\left(a_{1}+a_{2}+a_{3}+a_{4}\right) - a_{3}^{2}\\
& + \frac{a_{4}}{2}\left(a_{1}+a_{2}+a_{3}+a_{4}\right)-a_{4}^{2} \\
& = a_{1}\cdot a_{2}+ a_{1} \cdot a_{3}+ a_{1} \cdot a_{4}\\
&+ a_{2} \cdot a_{3}+ a_{2} \cdot a_{4}+ a_{3} \cdot a_{4}\\
&-\frac{a_1^2}{2}-\frac{a_2^2}{2}-\frac{a_3^2}{2}-\frac{a_4^2}{2}
\label{variance2:4}
\end{aligned}
\end{equation}

the left hand side of Equation \ref{claim} is given by Equation \ref{variance2:4} and its right-hand side equals to $2a_1a_2 + 2a_3a_4$. Let $D$ be the difference between the left-handside of Equation \ref{claim} and its right handside. To prove our claim we need to show that $D$ is negative:
\begin{equation}
\begin{aligned}
& D = a_{1} a_{2}+ a_{1}  a_{3}+ a_{1} a_{4} + a_{2} a_{3}+ a_{2} a_{4}+ a_{3}  a_{4}\\
&-\frac{a_1^2}{2}-\frac{a_2^2}{2}-\frac{a_3^2}{2}-\frac{a_4^2}{2} - (2a_1a_2 + 2a_3a_4)\\
=&\underbrace{\left(-\frac{a_{1}^{2}}{2}+a_1 a_{4}-\frac{a_{4}^{2}}{2}\right)}_{-0.5\left(a_{1}-a_{4}\right)^{2} }+\underbrace{\left(-\frac{a_{2}^{2}}{2}+a_{2} a_{3}-\frac{a_{3}^{2}}{2}\right)}_{-0.5\left(a_{2}-a_{3}\right)^{2}}+ \\
&+\underbrace{a_{4} a_{2}+a_{1} a_{3}-a_{1} a_{2}-a_{3} a_{4}}_{-\left(a_{1}-a_{4}\right) \cdot\left(a_{2}-a_{3}\right)}
\end{aligned}
\end{equation}

Let $A \triangleq (a_1-a_4)$ and $B \triangleq (a_2-a_3)$ then we have that
\begin{equation}
    D = -\frac{A^2}{2} -A\cdot B -\frac{B^2}{2}= -\frac{1}{2}(A+B)^2<0
\end{equation}
Our claim is thus proven.

\subsection{Approximated optimal 2:4}
\label{sec:approx2_4}
As shown in \cref{tab:overhead}, in terms of time complexity, the optimal method may not be feasible. Using insights gained from the optimal solution, we now present a simple approximation method called approx-MVUE. The idea is simple. We first choose an element $a_i$ with a probability $p_i = \frac{a_i}{a_1+a_2+a_3+a_4}$ and remove $a_i$ from the block. In order to select a second element, we repeat the same procedure for the three remaining elements with probability $p_j =\frac{{a}_{j}}{{a_{1}}+{a_{2}}+{a_{3}}+a_4-a_i}$. Thus, each element is chosen with probability:
\begin{equation}
\begin{split}
    p_i &= \frac{|a_i|}{\sum_j |a_j|} + \sum_{k \neq i}\frac{|a_k|}{\sum_j|a_j|}\frac{|a_i|}{\sum_{j \neq k} |a_j|} \\

\end{split}
\end{equation}
In order to check the effect of this approximated method to the variance of the estimator, we present in \cref{fig:optVsOpt} the ratio of the variance between the two methods: $\frac{\mathrm{Var}(\theta^{\mathrm{approx}}_{2:4})}{\mathrm{Var}(\theta^{\mathrm{opt}}_{2:4})}$, where both variances are calculated analytically using \cref{eq:varElement}). Without loss of generality for a block $[a_1,a_2,a_3,a_4]$ where $a_1 \leq a_2 \leq a_3 \leq a_4$, we set $a_4 = 1$ and scan with small steps all combinations of $a_1,a_2,a_3$. The scan suggests the variance ratio is bounded below two\footnote{The largest values are near the left edge of the scan, which represents the limit where $a_4 \gg a_1,a_2,a_3$. Near this edge, we additionally checked with very small (logarithmically spaced) step sizes that the variance ratio is bounded below two.}, and therefore the approximate method is a 2-approximation of the old method. As can be seen in \cref{tab:overhead} the approximated method reduces the complexity time of MVUE 2:4 by $\sim 70\%$, in our non-optimized implementation.

\begin{figure}[h]
  \centering

  \includegraphics[width=\linewidth]{imgs/optVsApproxVar.pdf}
\caption{Ratio between the variance (\cref{eq:varElement}) of the approx-MVUE 2:4 and MVUE 2:4 in a scan of all possible values of the block $[a_1,a_2,a_3,1]$, where $0 \leq a_1,a_2,a_3 \leq 1$ with a step size 0.005. Notice that ratio is bounded below 2. The maximum is achieved near the left edge, when $a_4 \ll \max(a_1,a_2,a_3)$.  }
\label{fig:optVsOpt}
\end{figure}

\section{Hardware Constraints}
So far fine-grained sparsity papers focused on pruning only the weights matrix \cite{Nvidia,n:mStructured,Hubara2021AcceleratedSN}. Here we investigate pruning gradients and activations as well. Therefore, we need to understand how much acceleration can we expect, when both matrices involve in the matrix multiplication are fine-grained pruned. Assuming we have two N:M fine-grained blocks $b_W$ $b_x$ with masks $M_{b_W}$ and $M_{b_X}$, the number of Multiply and Accumulate operations (MACs) required for multiplying and accumulating the blocks may vary from zero to N. For example for 2:4 fine-grained sparsity, there are $\binom{4}{2}=6$ possible mask configuration. Thus, the expected number of MACs in a block, assuming uniform distribution on the non-zeros in the blocks, would be:
\begin{equation}
\begin{split}
    \mathbb{E}[\mathrm{\#MACs(b_x,b_W)}]&= \mathbb{E}\left [\mathbb{E}[\mathrm{\#MACs(b_x,b_W)}|\mathrm{M_{b_X}}]\right]\\
    &=\frac{1}{6}\cdot 0 +\frac{1}{6}\cdot 2 +\frac{4}{6}\cdot 1 =1
    \end{split}
\end{equation}
Thus on average, each block contains $N/2$ MACs. While some architectures (such as CPU) can avoid all unnecessary multiplication, architectures with a systolic array at the core of their matrix multiplication engine must always assume the worst case. Therefore, for these types of architectures, we cannot achieve an additional compute reduction by pruning both matrices involved in the matrix multiplication. Yet, the bandwidth reduction for both matrices is the same. This property helps support sparse and dense matrix multiplication without creating dedicated hardware which has twice the bandwidth to one of the matrices. It is specifically important when targeting higher sparsity. For instance, if only the weights obey 1:4 fine-grained structure then the activations bandwidth is 4x higher than the weights bandwidth as for every single block we bring one weight and four activations to the engine.

Additionally, we noticed that doing N:M fine-grained sparsity on the activations is not useful for accelerating the update phase, since: (1) we must sparsify the activations for each sample separately (for inference), and (2) our experiments indicated that structured sparsity on the spatial dimension does not work well. In contrast, neural gradients can be sparsified in the batch dimension, and so can accelerate the update phase.

\section{Experiments}
In this section, we demonstrate the effectiveness of our proposed methods over several vision models. First we show the effect of the proposed method for the fine-grained N:M structured sparsity on the neural gradients and demonstrate results when combined with the fine-grained N:M transposable weight \citep{Hubara2021AcceleratedSN} allowing for the first time the acceleration with N:M structured sparsity in all training GEMM. Then we show the results of applying greedy fine-grained N:M structured sparsity on the activations and finally combined the greedy fine-grained N:M structured sparsity on the weights and activations to achieved near state-of-the-art results in the deployment of a model that is similar to 2-bit quantization in terms of bit-operations(BOPS) \cite{wang2020differentiable}.

\subsection{N:M structured sparsity on the neural gradients}
In \cref{fig:r18_mvue,fig:r50_mvue} we showed the results of applying the suggested N:M structured sparsity for ResNet18 and ResNet50 ImageNet dataset respectively. The 1:2 results refer to the MVUE method (\cref{sec:1_2_mvue}) while the 2:4 results refer to the approximate method (\cref{sec:approx2_4}). In \cref{fig:r18_mvue} we compare the proposed method also with the traditional greedy method of keeping the largest elements in each block. While the greedy method had a very significant degradation, the proposed method achieved baseline accuracy with the simple MVUE 1:2. Additional experiments appear in the \cref{sec:AppExp}.

We used the standard pre-processing of ImageNet ILSVRC2012 dataset. We train for 90 epochs, use an initial learning rate of 0.1 with a $\frac{1}{10}$ decay at epochs 30,60,80. We use standard SGD with momentum of 0.9 and weight decay of 1e-4.  The batch size used is 256. Following the DNNs quantization conventions \citep{Banner2018ScalableMF,Nahshan2019LossAP,PACT} we kept the first and last layer (FC) at higher precision.
\begin{figure}[h]
  \centering

  \includegraphics[width=\linewidth]{imgs/R18_gradients.pdf}
\caption{Top-1 validation accuracy in ResNet18 ImageNet dataset with the suggested fine-grained structured sparsity MVUE 1:2. We compare it with the traditional greedy 2:4 method of keeping the two larger elements in each block of size four. While the MVUE 1:2 has no degradation, the greedy 2:4 get a big degradation. }
\label{fig:r18_mvue}
\end{figure}

\begin{figure}[h]
  \centering

  \includegraphics[width=\linewidth]{imgs/R50_gradients.pdf}
\caption{Top-1 validation accuracy or ResNet50 trained over ImageNet dataset with the suggested fine-grained structured sparsity using MVUE 1:2 and approx-MVUE 2:4 criteria for the neural gradients. The MVUE 1:2 gets degradation of 0.8\% while the approx-MVUE 2:4 achieved baseline accuracy.}
\label{fig:r50_mvue}
\end{figure}

\subsection{Accelerating all training phases}
In \cref{tab:accelerateAll} we showed the results of the combination between the proposed N:M MVUE for the neural gradients and the N:M transposable weights presented in \cite{Hubara2021AcceleratedSN}. The combination between both methods allowed for the first time to accelerate with N:M structured sparsity for all training GEMM operations. This demonstrates that one can achieve 2x acceleration in all training with minimal or no accuracy degradation. In these experiments, we used the exact same experiment setting as \citet{Hubara2021AcceleratedSN}.

\begin{table}[h!]
\centering
\caption{Top-1 validation accuracy on ImageNet dataset of the combination of the suggested MVUE 1:2 and approx-MVUE 2:4 with the transposable weights of \citet{Hubara2021AcceleratedSN}. While the transposable weights enable accelerating the forward and backward phases, the suggested methods are able to accelerate the update phase. With this combination, we are able to accelerate for the first time all training phases with N:M fine-grained structured sparsity.}
\label{tab:accelerateAll}
\begin{tabular}{l|c|c|c}
\toprule
\multicolumn{1}{l|}{Model} & \multicolumn{1}{c|}{G} & \multicolumn{1}{c|}{$W^T$} & \multicolumn{1}{c}{Acc (\%)}   \\ \midrule
\multirow{4}{*}{ResNet18} & \multicolumn{1}{c|}{FP32} & \multicolumn{1}{c|}{FP32} & \multicolumn{1}{c}{70.6} \\ \cline{2-4}
& \multicolumn{1}{c|}{FP32} & \multicolumn{1}{c|}{2:4}  & \multicolumn{1}{c}{70.5} \\ \cline{2-4}
& \multicolumn{1}{c|}{1:2} & \multicolumn{1}{c|}{2:4}  & \multicolumn{1}{c}{70.4} \\ \cline{2-4}
& \multicolumn{1}{c|}{2:4} & \multicolumn{1}{c|}{2:4}  & \multicolumn{1}{c}{70.6} \\
\hline
\multirow{3}{*}{ResNet50} & \multicolumn{1}{c|}{FP32} & \multicolumn{1}{c|}{FP32} & \multicolumn{1}{c}{77.2} \\ \cline{2-4}
& \multicolumn{1}{c|}{FP32} & \multicolumn{1}{c|}{2:4}  & \multicolumn{1}{c}{77.1} \\ \cline{2-4}
 & \multicolumn{1}{c|}{1:2} & \multicolumn{1}{c|}{2:4}  & \multicolumn{1}{c}{75.6} \\
\bottomrule
\end{tabular}
\end{table}

\subsection{N:M structured sparsity on the activations}
\label{sec:sparseAct}
We experimented with greedy N:M fine-grained sparse activations on ResNet18 and ResNet50 over ImageNet, wherein for each block of size M we keep the M-N bigger elements. Note that in CNNs the activations memory footprint is much larger than the weights footprint (especially for the first set of layers). Throughout our experiments, we did not change the training regime and applied the sparsity from scratch. Since the training regimes and model initialization are tailored to the ReLU function our pruning criterion was the activation value and not its absolute value (i.e., magnitude). As can be seen in \cref{fig:activation_cifar} applying only fine-grained sparsity results in notable accuracy degradation for both training and validation.  However, a simple fix that requires only to apply ReLU before the fine-grained sparsity results in on par accuracy over ResNet18 and ResNet50 (\cref{tab:activations_R18_R50}).

\begin{figure}[h]
  \centering

    \includegraphics[width=\linewidth]{imgs/R18_cifar_sparseAct.pdf}
    \label{fig:sub-first}
\caption{Top-1 accuracy on ResNet18 Cifar10 dataset with 4:8 fine-grained sparsity. ReLu + greedy refers to applying ReLU and then the structured sparsity, while greedy does not include the ReLU function. As can be seen, applying only N:M structured sparsity as the activation function leads to accuracy degradation.}
\label{fig:activation_cifar}
\end{figure}

\begin{table}[h!]
\centering
\caption{ResNet18 (R18) and ResNet50 (R50) top-1 accuracy on ImageNet dataset with greedy N:M fine-grained sparsity on the activations}
\label{tab:activations_R18_R50}
\begin{tabular}{lcc}
\toprule
\multicolumn{1}{l|}{Method} & \multicolumn{1}{l|}{R18 top-1 (\%)} & \multicolumn{1}{c}{R50 top-1 (\%)}  \\ \midrule
\multicolumn{1}{l|}{Baseline} & \multicolumn{1}{l|}{70.6}& \multicolumn{1}{c}{76.6}  \\ \hline
\multicolumn{1}{l|}{4:8 activations}  & \multicolumn{1}{l|}{70.6}  & \multicolumn{1}{c}{76.45}\\
\bottomrule
\end{tabular}
\end{table}

\subsection{Accelerating inference}
Accelerating inference requires compressing only the weights and activations. Thus, we experimented in \cref{tab:inferece_R18} with greedy N:M fine-grained sparsity of weight and activations. To compete with the latest inference acceleration results based on quantization-aware techniques, we further quantize the weights and activation to 4-bit using the SAWB method \cite{SAWB}. Since on average we have $N/2$ MAC operations for each block, the BOPS \cite{wang2020differentiable} reduction is equivalent to 2-bit quantization. We experimented with N:M fine-grained sparse activations and weights using ResNet18 over ImageNet using two training schemes:
\begin{itemize}[noitemsep]
    \item \textbf{Scheme A} is similar to \citet{n:mStructured} regime in which  sparse weights are used from scratch only here we additionally applied 4-bit asymmetric quantization for both weights and activations.
    \item \textbf{Scheme B} is similar to \citet{Nvidia} regime and has three phases: (a) train a  sparse activation full precision model; (b) set a mask for the weights; and (c) train a sparse weights and activation model while using 4bit quantization-aware training.
\end{itemize}
In all our experiments we kept last fully connected layer dense and in full precision. We used 2:4 structure for the weights and 4:8 structure for the activations.
\begin{table}[h!]
\centering
\caption{Inference acceleration of ResNet18 on ImageNet dataset. We quantize the weights and activations to 4-bit and apply in both greedy N:M fine-grained sparsity getting an equivalent of 2-bit inference. We compare our results with different 2-bit quantization-aware training methods.}
\label{tab:inferece_R18}
\begin{tabular}{lc}
\toprule
\multicolumn{1}{l|}{Method} & \multicolumn{1}{c}{Accuracy (\%)}   \\ \midrule
\multicolumn{1}{l|}{Baseline} & \multicolumn{1}{c}{70.6}  \\ \hline
\multicolumn{1}{l|}{4:8 activations}  & \multicolumn{1}{c}{70.6}  \\ \hline
\multicolumn{1}{l|}{Scheme A (ours)} & \multicolumn{1}{c}{66.5}  \\ \hline
\multicolumn{1}{l|}{Scheme B (ours)} & \multicolumn{1}{c}{67.33}  \\ \hline
\bottomrule
\multicolumn{1}{l|}{PACT 2-bit \cite{PACT}} & \multicolumn{1}{c}{64.4 }  \\ \hline
\multicolumn{1}{l|}{QIL 2-bit \cite{jung2019learning}} & \multicolumn{1}{c}{65.7}  \\ \hline
\multicolumn{1}{l|}{LSQ 2-bit \cite{esser2019learned}} & \multicolumn{1}{c}{66.9}  \\ \hline
\bottomrule
\end{tabular}
\end{table}

\section{Discussion}
In this work, we studied the effect of N:M structured sparsity on the neural gradients and activations:
\paragraph{Sparse Gradients} Basing on a previous work \cite{Chmiel2020NeuralGA} which show the importance of unbiasedness of the neural gradients, we suggest an unbiased minimum variance method to prune the neural gradient with 1:2 and 2:4 structured sparsity. Moreover, since the optimal 2:4 method may not be feasible in term of complexity, we suggest an approximate method which increases the variance only by a factor of 2 (making it a 2-approximation). We showed that our method achieved small or no degradation while the traditional greedy method completely failed. Moreover, we combine our method with a previous method for transposable weights \cite{Hubara2021AcceleratedSN}. This enabled  N:M fine grained sparsity, for the first time, to accelerate by 2x with all training process.

\paragraph{Sparse Activations.} ReLU is a very common activation function that usually combined with some normalization function (e.g., batch norm). Thus commonly applying ReLU result in 50\% sparse output. Here for the first time we investigate fine-grained N:M sparsity for the activations. While simply applying N:M sparsity on the pre-ReLU outputs results in severe degradation, using ReLU and then enforcing the structure result in an on-par accuracy. Notably, N:M structured sparsity in the activations can lead to acceleration of the proceeding layer's backward phase (in our case batch-norm) as the gradient's share the same mask. We experimented with different training regimes and initialization schemes for fine-grained sparse activation but did not observe accuracy boost.  Finally, we suggested for inference to use weights and activation that are fine-grained sparse and quantized.

redpajama_set_name: RedPajamaArXiv


MINISTERS were last night forced to U-turn on claims they'd found new cash for a flagship £100m fund to stop rough sleeping.
Housing Secretary James Brokenshire admitted half the cash to stamp out homelessness in 10 years had already been committed to rough sleeping and the other half was "reprioritised" from existing budgets.
It followed claims from his senior aide Chris Philp that the opposite was the case.
Mr Brokenshire said: "Yes, some of this is reprioritised from within existing budgets where we have underspends. There are significant sums of money being focused and targeted.
Mr Philp, parliamentary private secretary to Mr Brokenshire, had told the BBC's Westminster Hour programme that the money was all new investment.
The Government's integration czar Dame Louise Casey has said additional funds needed to be committed if the pledge to end rough sleeping by 2027 was to be met.


Social Physics is really interesting so far. The only thing really bothering me about this book is the author – he really likes to pat himself on the back. It seems like there isn't a single page where he isn't talking about some amazing work he's done. But the information so far is exactly what I wanted to learn about when I picked this book up.
The Fallen Kingdom series is getting pretty crazy. I've just started Crystal Storm, and it's just… ugh. All the things are happening and I can't.
Recently Finished: Once again, I didn't post last week so this is two weeks worth of books read!
Since You've Been Gone was pretty cute. I can't imagine having a friend like Sloan do something to me like what she did to Emily, but I do have to say the whole list/journey was really fun and really good for Emily.
Fifth Grave Past the Light literally had me grinning and giggling through the whole thing. I love this series.
To the Nines and Ten Big Ones were both funny as well.
Arrowheart was cute. Here is my review. And here is my review for The Secret Lives of Royals.
The Dark Oracle was great. I can't imagine I'll ever say anything bad about a Rick Riordan book.
Tyler Johnson was Here hurt my heart. I think it is a really great way for people who are not black to get some serious insight into what is happening these days from their point of view.
I am rereading the Red Queen series so I can finally read War Storm. I think I like Red Queen a lot better the second time through.
Like I said above, the Fallen Kingdom series is getting pretty crazy. Frozen Tides had a lot of characters switching sides and recruiting new characters to their sides and creating new sides and there is so much happening.
I'm getting to the last Rick Riordan series I haven't read yet (:() with The Son of Sobek. Blind the Eyes is my next review copy to read. And I'm continuing my reread of the Red Queen series with Glass Sword.
Since You've Been Gone and Arrowheart look pretty good! I still need to read a Rick Riordan book. Have a great reading week!


Lively debate marked the Seattle kickoff event of Reed on the Road 2013 on Wednesday, February 20. More than 60 alumni, parents, and friends gathered at the Women's University Club to toast the success of the recently completed Centennial Campaign and discuss the future of the college.
At a time when other colleges are considering layoffs and shutting down programs, Reed has never been in better financial shape, Reed College president John Kroger said in his opening remarks.


The Rev. Jesse Jackson today called for an end to the indifferent killing of innocent people and transparency in the investigation into the death of a 21-year-old Hueytown man shot by Hoover police at the Riverchase Galleria on Thanksgiving night.
Jackson's comments came as he gave the eulogy at the funeral of Emantic "E.J." Bradford Jr. at Boutwell Auditorium in Birmingham.
At least 1,000 people attended the 2 1/2-hour ceremony at Boutwell, which was part funeral, part church service and partly a rally calling for justice for Bradford.
A Hoover police officer shot Bradford as he fled the scene of another shooting at the mall just seconds earlier on the night of Nov. 22. Police said Bradford had a gun in his hand as he was fleeing.
Initially, police labeled Bradford as the gunman in the first shooting, but the next day, police said their initial statements were incorrect and that, while Bradford may have been involved in the altercation that led to the first shooting, he likely was not the one who shot 18-year-old Brian Wilson of Birmingham in the torso.
U.S. marshals and Georgia and Alabama law enforcement agents on Thursday arrested a 20-year-old Bessemer man in Georgia after he was charged with attempted murder in the first shooting.
In Hoover, protesters have claimed the police killing of Bradford was unjustified and demanded that either Hoover or the Alabama Law Enforcement Agency release police body camera videos and any other videos of the incident. So far, no videos have been released.
Today, Jackson joined the chorus of voices calling for the public release of any videos of the shootings.
"The officer who took Emantic's life and snuffed out his dreams must face justice," Jackson said. "No one is above the law. Even the president and certainly no police officer is above the law.
He said some communities have grown accustomed to weeping in the face of expanding police violence.
There's something magic about Birmingham, he said. The city has a history of resiliency, of people not allowing dogs, fire hoses, bombs, churches or bullets to stop them.
Jackson noted that Birmingham is the place where singer Nat King Cole was attacked by the Ku Klux Klan in 1956 and where Martin Luther King Jr. spent time in jail for protesting for racial equity.
He said Bradford joins a long list of martyrs whose innocent blood has transformative power, mentioning Trayvon Martin, Walter Scott, Rodney King and Michael Brown as being among them.
Jackson encouraged the family to not give up their fight for answers and justice.
In an interview after the funeral, when asked what prompted him to come to Birmingham to speak at Bradford's funeral, Jackson said Bradford's family asked him to come.
During the funeral, Jackson told the story of how Jesus was in a boat with his disciples when a storm came upon them suddenly. He noted how Bradford's family had an unexpected storm arise but encouraged them to stay strong and allow God's power to see them through.
The Rev. Mike McClure, senior pastor of Rock City Church in Birmingham, who served as master of ceremonies for the funeral, told the crowd he saw the tears in their eyes and understands that they are hurting as they try to make sense of Bradford's killing.
Jefferson County Commissioner Lashunda Scales, another speaker at the funeral, said "it's not just a black thing. It's not just a white thing. It's a humane thing." She encouraged the family to stay focused and unified.
Carlos Chaverst Jr., one of the leaders of the protests in Hoover since the shooting, quoted Martin Luther King Jr. in saying that justice delayed is justice denied.
Bradford's father, Emantic Bradford Sr., holding back tears, talked about the good memories he had with his son, from the time he took his first breath, to trying to feed him beets as a baby and watching Batman marathons with him all day as he grew up.
Then, when Bradford Sr. got sick with cancer, he noticed his son begin to show more responsibility. "He started taking care of me and calling me and coming to check on me, making sure I didn't want for nothing," the father said.
Jackson's daughter, Santita Jackson, also sang several songs just before he spoke. And at the end of the funeral, McClure gave an altar call, with a special plea to young people to turn their lives over to Jesus and receive Him as their Saviour. Numerous people came forward and stood around Bradford's casket as McClure led in a prayer of surrender.
A large contingent of family members accompanied Bradford's parents into Boutwell Auditorium, and Bradford's mother fainted as she visited his casket and had to be taken out in a wheelchair briefly.
The funeral drew numerous pastors and even former Gov. Don Siegelman, who said he just wanted to come to show support to the family and help bring them some comfort.
This story was updated at 4:46 p.m. with additional details from the funeral.
See all stories on the Riverchase Galleria police shooting here.
E.J. Bradford Carlos Chaverst Jr. Hoover Police Department Riverchase Galleria police shooting Riverchase Galleria shooting Hoover police Emantic Bradford Jr. fatal shooting homicide Lashunda Scales Jesse Jackson Emantic Fitzgerald Bradford Jr. funeral Boutwell Auditorium Mike McClure Jr.


Ex-ministers and former civil servants who lobby on behalf of big companies and other organisations will have to publish their client lists for the first time, under a new crackdown on lobbying.
Mark Harper, Minister for Political and Constitutional Reform, said former ministers and civil servants who lobbied for other companies would have to declare their client lists.
In 2010 three former Labour ministers were stripped of parliamentary passes after they breached lobbying rules and the House of Lords was rocked by a "cash for amendments" scandal.
The sleaze clean-up was promised in the Coalition agreement to deal with what Prime Minister David Cameron warned was "the next big scandal waiting to happen".
Under the plans, unions and charities would be forced to sign up to a new statutory register of parliamentary lobbyists. Lobbyists who deliberately failed to comply with the anti-sleaze rules would face up to two years in prison.
A 12-week consultation on how the register should work has now begun and legislation is expected to be introduced in the next session of Parliament.
Mr Harper admitted that one weakness was that lobbyists with large client rosters would not have to declare what they were discussing with ministers. But he said: "We already publish an unprecedented amount of information about who ministers and senior officials meet.
The plans were criticised by a major rail union. Bob Crow, the leader of the RMT union, said: "The idea that trade unions, representing millions of workers up and down the country, should be bracketed in with the chancers and schmoozers from the shadowy world of political lobbying is a gross insult to men and women fighting for a fair deal in the workplace.