Hello fellow Walking Dead fanatics! We're marching closer and closer to the ultimate conflict between Rick's crew and the Woodbury crew. This episode seemed to be the fallout from Phillip's welcoming party at the prison last week. If there would be anything that was comparable to last season in terms of pacing, it would be this one. But, of course, I rather enjoyed the change of pace and the chance to catch up with our characters. Definitely some interesting chess pieces moved in the episode and in a place that I didn't quite expect. We'll see where it all goes, but let's dive into this episode in the meantime!
The Governor continues to play his mind games with Andrea. Meanwhile, she wanted to try and create a merger of camps between her friends and the Woodbury crew. But, Woodbury is gearing up for war. She was able to "sneak" out with Milton's assistance (secretly approved by the Governor). She even created herself a Michonne-like Walker pet. But, it was all ulimately part of The Governor's plan.
But, it was nice to have a reunion of characters with Andrea coming to the prison. Rick did not make her feel very welcome though. She was frisked and questioned as if she was an outsider. She got information about Shane and Lori and was shocked at the things that happened to them all. I love how she needed to be reminded about T-Dog. Good times!
Loved that Carol has come so far since her crazy husband in season 1 that she suggested to Andrea that she needs to give ol' Phillip a night he will never forget and then kill him in his sleep. That's pretty dark stuff! I never got the impression that Andrea was considering this when she left the prison though.
Rick is still dealing with his issues, and got reamed out by Hershel in this episode. Hershel was furious and said that he has trusted Rick with his family and he better sort his issues out and LEAD. Hershel also thinks it's time to leave the prison. Rick does not agree. Things have gotten so bad that even Carl is suggesting for his dad to step down from the leadership position.
Michonne and Andrea had their little reunion scene as well. Michonne was furious with Andrea for falling under the Governor's spell. So, she returned to expose him and intentionally hurt Andrea in the process. Ouch!
There was also the whole "should we or should we not trust Merle" storyline throughout the episode. By the end of the episode, he had seemed to earn his place. He and Hershel traded war stories about their hand and foot. Merle explained what kind of person in the Governor they're dealing with. How he'll kill Merle and Michonne first. Then he'll kill the women and children and save Rick for last. I still am not 100% sure we can trust Merle, but I'm also sure that's by design. But, considering he did lie about killing Michonne, it's tough to think that Phillip would've given him a 2nd chance to prove himself (off camera). But Merle seems like the kind of guy who will switch allegiances in a heartbeat if it benefits him. So, we'll see where he stands when the conflict comes!
Oh right, and Tyreese and crew made their appearance again this episode! They ran into Andrea and Milton in the woods. And that stand-up guy Tyreese was still offering a hand, wanting to help out. But, they sent them to Woodbury. It was interesting seeing the Governor welcome Tyreese and crew similar to the way he welcomed Michonne and Andrea. Of course, this time, we've peeled back the layers of the Governor a little bit more.
Things took a turn once Phillip and Team Tyreese realize they're talking about the same people that they've been in contact with. Crazy Rick and the Prison crew. One of Tyreese's guys said that they are up for a war with Rick if it would come to it. All Tyreese wants is a safe haven from the walkers, so he seemed game as well. The Governor asked if they could recall the layour of the prison, and Tyreese seemed to think he could. Looks like we might be getting a secret invasion to the prison! Now, while in the long run, I'm sure we'll see Tyreese aligned with Rick and co this certainly was a twist I was not expecting! Rick's bad decisions have given them additional enemies to deal with. Oh boy!
I'm sure there's a lot I missed, but that's what the comments sections are for! So, just a heads up, I will be out of town this coming weekend so I probably won't have time to get a recap up next week. But, I'll still post a separate discussion post for that episode and I'll comment after I've seen the episode! Hope you enjoyed my ramblings and I'll see you in one form or another next week!
I was a little disappointed that Tyrese didn't explain more about how the rest of the group treated them when the arrived at the prison -- I mean Carl actually saved them and provided a safe haven -- that it was only Rick who insisted that they leave.
I got the impression that Carol has thought of that whole "murder him in his sleep" scenario many times before this. Remember, her husband was a wife-beater. I think that's why she was able to mention it so calmly.
Don't know why so many are up in arms over Andrea not killing the Gov - hello! then we would not get a Rick/Gov showdown ! Strangely off-putting that Carol is the one who told Andrea to give the gov a great night and then end him. Awesomely cool when Andrea made her own 'pet' to get her to the prison. Don't know why so many Andrea haters - she wants to live in a nice place with decent surroundings - after the last few years why wouldn't she !?! But glad Mishone finally spoke to how she felt betrayed by Andrea staying in Woodbury. But oh - was not happy to see Tyresse and his crew land in Woodbury. I thought it was good that they went thru all the peeps they had lost - sometimes its easy to forget what Andrea doesn't know now. And Carl stating to Rick that he should step aside I think is a real turning point for their relationship.
Tyreese did seem a little uncomfortable with the conversation that was taking place with the governor, but yeah surprising he didn't talk to how nice everyone else was. I really think Tyreese will be able to tell the difference between Rick and the Governor when the conflict begins! And his true allegiance will show.
Good point with Carol thinking about possibly murdering her own husband in his sleep. Makes sense for her character to go there. But, the old her may have never suggested it out loud!
You may not be far off with a mutiny on the Governor. No, it's not a happy ending type show but the audience does want something to root for. And our group, after last season and this season deserve some kind of a victory! Sure, it will lead to further conflicts and troubles to tackle, but that's all television shows.
You guys covered most of my thoughts already. Kinda funny that they had Merle separated. I wouldn't trust him either, but he's more likely to stay loyal to Daryl than anyone, but I think Merle looks out for Merle.
Agree about Andrea making her own "pet" and Michonne having her say finally. Loved when she said something about not knowing the Messiah complex was contagious. Didn't expect Tyreese to show up in Woodbury.
Funny when Carol told Andrea she couldn't leave until meeting "little a** kicker." Agree that Carol probably had those fantasies about putting an end to her husband, but no way the Governor would go out that way. I kept expecting him to suddenly grab her hand.
Well it cant have been 9 mos betw 2 and 3. She knew she was expecting at end of 2, women usually a month or more when they realize they are preggers. Still, did not realize Andrea s pent more time with Mishone than Carol and group. Seems wrong somehow.
Yeah, I estimated with the 9 months. But Andrea and Michonne were together for the majority of Lori's pregnancy which is longer than the group had spent together. You have to think of season 1 and 2 as not a lot of time passing by...then between season 2 and 3 we had a huge time jump. Now..I don't know how long Carol and Andrea were together before Rick joined the group. I forget how long he was in a coma. But I still think the break between seasons 2 and 3 still wins out.
Thought we knew that already, but I guess they just made it official.


text:
China reports no new coronavirus cases for first time since pandemic began
FILE PHOTO: Volunteers measure the body temperature of a man in Jilin
SHANGHAI (Reuters) - China recorded no new confirmed COVID-19 cases on the mainland for May 22, the first time it had seen no daily rise in the number of cases since the pandemic began in the central city of Wuhan late last year.
The National Health Commission (NHC) said in a statement on Saturday that this compared to four new cases on the previous day. It said, however, there were two new suspected cases: an imported one in Shanghai and locally transmitted case in the northeastern province of Jilin.
New asymptomatic cases of the coronavirus fell to 28 from 35 a day earlier, the NHC said.
China has seen a sharp fall in locally transmitted cases since March as major restrictions on people movement helped it to take control of the epidemic in many parts of the country.
However, it has continued to see an influx of imported cases, mainly involving Chinese nationals returning from abroad, while new clusters of infections in the northeastern border provinces of Jilin and Heilongjiang have emerged in recent weeks.
Wuhan also reported this month its first cluster of infections since a lockdown on the city ended on April 8, prompting authorities to warn that counter-epidemic measures could not be relaxed and to launch a campaign to test all of Wuhan's 11 million residents for COVID-19.
The number of confirmed cases in the mainland stood at 82,971 and the death toll remained unchanged at 4,634.
(Reporting by Brenda Goh; Editing by Lincoln Feast and Muralikumar Anantharaman)
Countries in Central America have ramped up their border controls. As thousands of migrants from Honduras and Guatemala are heading to the U.S. for a new life. The region is reeling from a growing hunger crisis after hurricanes hit, as well as violence and lockdown measures that disrupted the job markets. On Friday (January 15), the first migrant caravan of the year departed Honduras. Small skirmishes broke out between Honduran security forces and migrants at the border with Guatemala as night fell. But groups of Hondurans continued to trickle past the border, even as Guatemalan military detained hundreds of migrants, including many families with young children. Guatemalan officials estimate around 6,500 people to be heading north to the U.S. Dora Sanchez is one of them: "We just want a chance to work. We are defeated, we have nothing, we are left on the streets, we lost everything. Here most of us all (in the caravan) have nothing, not even a blanket to tuck into." Guatemala declared emergency powers in seven border provinces on Thursday, citing pandemic concerns. Mexico also deployed soldiers and riot police to its border with Guatemala on Friday. The exodus comes less than a week before the inauguration of U.S. President-elect Joe Biden, who has promised a more humane approach to migration.
Israeli authorities on Sunday advanced plans to build nearly 800 homes in West Bank settlements, in a last-minute surge of approvals before the friendly Trump administration leaves office later this week. COGAT, the Israeli defense body that authorizes settlement construction, confirmed the approvals, which drew swift condemnations from the Palestinians. The anti-settlement monitoring group Peace Now said that over 90% of the homes lay deep inside the West Bank, which the Palestinians seek as the heartland of a future independent state, and over 200 homes were located in unauthorized outposts that the government had decided to legalize.
Snow backs down, but leaves landscape coated
Biden plans to roll back controversial Trump policies on his first day in office, including reversing the Muslim travel ban and rejoining the Paris climate accord
Impeaching Trump after he leaves office would be 'plainly unconstitutional': Alan Dershowitz
New Jersey gym owners claim the state has seized legal defense funds

redpajama_set_name: RedPajamaCommonCrawl


Adrian Glass-Moore Sep 19th 2014 - 9pm.
FARGO – After 26 years, the private owners of Fargo-Moorhead's only toll bridge must hand it over to the two cities, a judge ruled Wednesday.
The decision was a victory for the city of Moorhead, which sued Bridge Co. and the city of Fargo in May, arguing that, according to a contract signed in 1986, it was time for the bridge to transfer into public hands and that the company was exploiting the bridge for private profit.
Bridge Co. entered into a 25-year agreement in 1986 with both cities to construct, maintain and operate the bridge between Fargo's 12th Avenue North and Moorhead's 15th Avenue North and earn revenue from tolls.
Judge Frank Racek wrote that "the Bridge Company is required to transfer the bridge to the Cities of Fargo and Moorhead" without specifying a timeline, in an order filed in Cass County District Court.
Moorhead City Manager Michael Redlinger said the city would present a timeline on future plans for the bridge next week. Whether the 75-cent toll will remain "hasn't been determined yet," he said.
Moorhead Mayor Del Rae Williams said she thought the toll would be removed after the bridge is turned over to the two cities.
Fargo Mayor Dennis Walaker said Friday he is concerned about the financial impact of taking over the bridge, which he says needs hundreds of thousands of dollars in repairs.
"I don't have any problem with … taking away the toll," Walaker said. "It's just I don't feel that it's our responsibility to take on any more debt for the repair of the structure."
The 1986 contract gave Bridge Co. control over the bridge until Feb. 5 of this year, after accounting for 249 days when the bridge closed because of flooding. The city of Fargo offered to extend the contract for another five years, but the city of Moorhead balked.
The company, which is owned by Clifford Moore and James Dixon, argued that it needed more time before handing the bridge over because it had more than $480,000 in debt stemming from construction, maintenance, and repairs – arguments the judge dismissed.
Racek also found that the company's profits in 2012 totaled $215,982. The toll generated $399,623 that year.
"What the city of Fargo and my clients wanted was simply the right to operate the bridge for another several years," Randolph Stefanson said.
Stefanson said he will meet with his clients Monday to discuss the possibility of an appeal.
Thousands of dollars in unpaid taxes to North Dakota and Minnesota need to be paid before Bridge Co. transfers the bridge to the cities, the judge wrote.


a day in the life: Done!
Offset wraplan, Garter stripe baby socks, Aviator. I still have to attach buttons and sew in a few ends. I really want to make those baby socks in an adult size; I love the pattern. :) Next I have three elephants to make; one for my boss (who does circus research) and two for babies for my mom's co-workers. I also have to get back to Josh's sweater which I haven't worked on in much too long. I just like these quick little projects so much better! Plus I have the wedding yarn to use and I'm getting more for graduation from Mom and Dad and I just want to use it now, lol.


Having passed my degree in Information Systems, I've decided to have a slight career change and to go down the accounting route. I do qualify to start straight on the fundamentals level of ACCA but would like an Accounting qualification much sooner. Hence me opting for the AAT, (Does any find this silly?). So I was just wondering if it was possible to start straight at the Technician stage, and has anyone done this?
I think you only get the ACCA exemptions if you have passed both Intermediate and Technician. I think the only exceptions to this are if you hold a relevant degree (eg Accounting) - and if you do hold such a degree I think you get the ACCA exemptions too.
To gain the AAT qualification, you must complete both intermediate and technician levels.
This will then give you exemptions from papers F1, F2 and F3 from ACCA Fundamental levels.
If you visit the ACCA website, you will be able to see what exemptions AAT and your degree will give you. Or alternitavely, give their student services a call.
I think you only have to pass technician.
When I got my exemptions I only had to provide my technician statement of achievement.
You can check your degree and AAT for exemptions at the link above - however, it only shows exemptions if you have completed all AAT levels to NVQ4 (Tech level), it doesn't say what happens if you only have NVQ 4/Tech level.
However, I don't see the point in just sitting Tech level - enrdbrow said they want a qualification sooner, but they won't have AAT - you need to complete both intermediate and technician level to be AAT qualified, so you might aswell just get on and do ACCA and sit the 3 papers that AAT would give you exemption from?
you might aswell just get on and do ACCA and sit the 3 papers that AAT would give you exemption from?
Couldn't agree more especially as you have to pay for the exam exemptions anyway!
You must have completed both the Intermediate (NVQ/SVQ level 3) and Technician (NVQ/SVQ level 4) levels of the AAT Accounting Qualification. If you hold a UK degree in Accounting or are part-qualified with one of the UK's chartered or certified bodies you need only to have completed the Technician level.
I knew I had read it somewhere. Obviously I wouldn't expect that a degree in Information Systems would give you that exemption so as phunkyphantom says, you would need to do the two levels.
Thanks for all the replies. I've found that you do only require the Technician certificate to gain the exemptions from ACCA F3 - F1. But I would like to gain the AAT qualification so I'm going to complete both the Intermediate and Technician Stages. Thanks again for your help.


text:
Q: XSLT: Using xsl:key in nested XML structures EXAMPLE
Here is an example from MSDN. It show how to transform a XML file into HTML using xsl:key.
Example XML (input)
<books>
  <book title="XML Today" author="David Perry" release="2016"/>
  <book title="XML and Microsoft" author="David Perry" release="2015"/>
  <book title="XML Productivity" author="Jim Kim" release="2015"/>
</books>

Example XSL (input)
<xsl:key name="title-search" match="book" use="@author"/>

<xsl:template match="/">

        <xsl:for-each select="key('title-search', 'David Perry')">

         <xsl:value-of select="@title"/>

      </xsl:for-each>

</xsl:template>

Example HTML (output)

XML Today

XML and Microsoft

MY PROBLEM
I would like to produce the same HTML output but using a different XML input. How should the corresponding XSL file look like?
My XML (input)
<books>
  <book>
    <s>David Perry</s>
    <s>XML Today</s>
    2016
  </book>
  <book>
    <s>David Perry</s>
    <s>XML and Microsoft</s>
    2015
  </book>
  <book>
    <s>Jim Kim</s>
    <s>XML Productivity</s>
    2015
  </book>
</books>

My XSL (input)
???

My HTML (output)

XML Today

XML and Microsoft

A: In the first example, your key matched book elements by their author attribute, but in the new XML, you want to match them by the a element where the n attribute is "author", so the key looks like this.
<xsl:key name="title-search" match="book" use="a[@n='author']/s"/>

Then, to get the title for the a matched book you would do this...
<xsl:value-of select="a[@n='title']/s"/>

Therefore your XSLT would look like this:
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output method="html" />

    <xsl:key name="title-search" match="book" use="a[@n='author']/s"/>

    <xsl:template match="/">

            <xsl:for-each select="key('title-search', 'David Perry')">

             <xsl:value-of select="a[@n='title']/s"/>

          </xsl:for-each>

    </xsl:template>
</xsl:stylesheet>

You could actually drop the /s in the expressions here, if the s element was only ever going to be the only element under each a element.
This would work too:
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output method="html" />

    <xsl:key name="title-search" match="book" use="a[@n='author']"/>

    <xsl:template match="/">

            <xsl:for-each select="key('title-search', 'David Perry')">

             <xsl:value-of select="a[@n='title']"/>

          </xsl:for-each>

    </xsl:template>
</xsl:stylesheet>

A: This is basic XSLT transformation, you can do something like this:
<xsl:template match="book">

  Author: <xsl:value-of select="a[@n='author']/s" />
  title: <xsl:value-of select="a[@n='title']/s" >
  release: <xsl:value-of select="a[@n='release']/s" >

</xsl:template>

redpajama_set_name: RedPajamaStackExchange


This hand painted and framed white Hello Autumn sign features a custom wood frame and is the perfect accent to your rustic home decor. It also makes a great gift. It would look wonderful on the mantle of your living room to welcome friends and family. At Oconee Sign Shack, I specialize in pieces that reflect what home means to you.
This sign is 100% handmade and hand painted with extra care. My paints are inspired by the rich archives of stylish fabric designs and provide a fresh, yet classic vision for your home decor. The patina of the hand distressed Ivory background adds a cozy charm, and the striking Mineral lettering contrast perfectly. The light Walnut frame perfects the rustic look.
Back to the Shop: www.OconeeSignShack.


1998 Cam - Vintage Camera is a Photography app developed by The FFFF Studio. We only share original APK files. You can download and install the latest version of 1998 Cam - Vintage Camera APK from our direct download link. On this page we provide 1998 Cam - Vintage Camera 1.4.3 apk file for 4.1 and up .This is a free app listed in Photography category of apps. Use the download button on the top to download this apk. App Installs: 1000000 +.


How to Stop Feefo Emails Being Sent?
You can immediately stop emails being sent by your Feefo eCommerce plugin by logging into the Feefo Hub and accessing your campaigns page. From there you will be able to select 'Stop sending feedback requests' from the 'Actions' menu.
When Do Feedback Request Emails Get Sent?


K&G Lettings Limited specialise in buying houses directly from the owner. If you have sufficient equity and need to sell your property quickly we can help you.
Telephone: 01482 961961 (Low cost call) or Text: 07860021976 to start to relieve the pressure you feel.
Are you currently going through the stress of divorce or separation? Selling via the traditional route can take many months, if not years. Do you want to wait this long before moving on and starting a new and happier life? Can you cope with the additional stress on your life for the months or years while the property sells? Selling to K&G Lettings Limited can give you a quick sale, giving you a quick, clean, hassle free way into your new life. If it is a quick sale you are looking for then you need look no further, contact us today on E-mail: Enquiries@kandglettings.co.uk Telephone: 01482 961961 (Low cost call) or Text: 07860021976 to start to relieve the pressure you feel to begin your new life.
Many people are facing the anxiety and stress of repossession. More and more people are falling behind on mortgage repayments. Are you one of them? If you are, don't delay, don't wait for the repossession notice or eviction order – contact us today 01482 961961. A fast house sale may be just the solution to stop your repossession, we have experience of this and can offer you an excellent service allowing you to sell your house and stop your repossession, contact us today on E-mail: Enquiries@kandglettings.co.uk Telephone: 01482 961961 (Low cost call) or Text: 07860021976.
Have you found your dream home but are struggling to sell yours? Many people have found their dream home but are unable to buy it because their current home won't sell quickly enough. Others have managed to agree a sale but, then at the crucial moment, the chain breaks and they lose their dream home. At K&G Lettings Limited, we can help you buy that dream home by buying your home fast – allowing you to move when you choose, not when your chain dictates. Do you want to lose that dream house? Contact K&G Lettings Limited today to ensure you get the house you have always wanted and deserve – E-mail: Enquiries@kandglettings.co.uk Telephone: 01482 961961 (Low cost call) or Text: 07860021976.
Health problems can radically change your life. They often do not just affect your quality of life but, also your financial situation; forcing you to stop work or reduce your hours. This all has the effect of lowering your income. This can lead to debt problems which can lead to repossession. Redundancy is similar, it effects your finances and does so extremely quickly, with almost immediate effect. K&G Lettings Limited can help to remove the financial stresses at this sensitive time by buying your home quickly, removing the burden of unaffordable mortgage payments from you. Contact us today if you want to learn more – E-mail: Enquiries@kandglettings.co.uk Telephone: 01482 961961 (Low cost call) or Text: 07860021976.
Are you thinking of moving abroad or relocating because of work or family reasons? Have you only got a matter of weeks or days to sell your property? The traditional route for selling your property may not allow you to relocate in the time you have; a quick sale could be the answer. Selling to K&G Lettings Limited will allow you to buy your new house quickly and allow you to start planning your new life, rather than your property sale dragging on, delaying your move. A quick sale may be just the thing you need – contact us today to find out more – E-mail: Enquiries@kandglettings.co.uk Telephone: 01482 961961 (Low cost call) or Text: 07860021976.
Are you a landlord who has a tenanted property and want to sell your house? Are you an accidental landlord (perhaps through inheritance) who has problem tenants that refuse to move out of the property. Are you simply fed –up of being a landlord? K&G Lettings Limited can help to solve you problem by buying your house quickly and efficiently – freeing you from the hassles of dealing with unreliable tenants. Contact us today to find out how we can help you – E-mail: Enquiries@kandglettings.co.uk Telephone: 01482 961961 (Low cost call) or Text: 07860021976.
How to sell to a property investor?
Subscribe with us today and stay tuned for the updates!


text:
Jon M. Bramnick
John C. Rodriguez
Gary Grabas
Jonathan P. Arnold
Patrick J. Mangan
Kelly A. Lynch
Robert C. Rowbotham II
Brian J. Trembley
Michael Noriega
José Vilariño
Richard Brockway
Brent Bramnick
Carl Salisbury
Russell M. Woods
Jonathan Holtz
Stephen A. Satkin
Wade Suthard
Gavin Handwerker
Cristina Carreno
We Accept Cases Throughout NJ
Glouceter County
CapeMay County
Want to find out if you have a case? Click Here.
Cherry Hill Law Office
Clifton, NJ Law Office
East Brunswick Law Office
Elizabeth Law Office
Westfield Law Office
Newark Law Office
Scotch Plains Law Office
Ways to Prepare for an Upcoming Trial
BRAMNICK LAW
Free Consultation for Personal Injury
and Criminal Cases only
If you have been charged with a crime and the case is headed to trial in superior court or municipal court, or if you were injured in an accident and the case is being presented to a jury in civil court, you are going to need an experienced trial attorney on your side. In fact, when it comes to preparing for an upcoming trial, the best preparation is hiring an attorney who understands the nuances and complexities of the NJ legal system.
How Attorneys Prepare for Trial
Your attorney is going to need as much time as possible to prepare your case, which is why you need to contact a lawyer as soon as possible. If you have been charged with a crime, or if you have suffered an injury due to another person's negligence, do not delay in speaking with an attorney. The success of your case could hang in the balance.
When you decide which attorney should represent you, make sure that you hire an attorney who prepares every case as if it is going to trial. Keep in mind, however, that most criminal and civil cases are settled long before reaching the trial stage. This is usually a good thing because it means that you will avoid the costs and stress that come with a trial.
Here are a few of the ways that attorneys prepare for trial:
Gather evidence: You want an attorney who utilizes private investigators to gather evidence and conduct research in preparation for trial. This is crucial because the judge or jury in your case will likely reach a conclusion based on the evidence presented at trial.
Negotiate with the other side: Your attorney should speak with the other side, whether it's the prosecutor in a criminal case or the person who caused your injuries in a civil case, and try to work out a deal that is favorable to you. It is also not uncommon for these kinds of deals to be struck during the trial, especially if one side appears to be "winning" the case in front of the judge or jury.
Attend pretrial hearings and file motions: There are usually several pretrial hearings that occur before an actual trial in New Jersey. Your attorney will need to be present at these hearings because important motions could be filed, including motions to dismiss your case. Depending on the circumstances, your lawyer may need to be ready to answer motions filed by the other side.
Subpoena witnesses: Witnesses whose testimony may prove crucial to your case will need to be located and subpoenaed for trial. Sometimes, a private investigator might need to be used to track down hard-to-find witnesses in advance of the trial date.
Prep for testimony: Anyone who is expected to take the stand in your defense or on your behalf will need to be prepared to do so. There are strict rules about the ways information is elicited at trial, so witnesses should be prepped for testimony.
Develop a theme: Despite the complicated procedures involved in a trial, at the end of the day it is a judge or jury who will be rendering a final verdict. And judges and jury members are ultimately just people. That's why your lawyer will probably want to develop a theme for the trial and tell a story that explains your case to the judge or jury.
What to Expect in the Courtroom
The reality of a trial is often far different from the fictional representation of trials in movies and TV shows. In movies and on television, trials tend to be condensed and exhilarating. In real life, a trial can be long and tedious.
Some of the basic components of your trial may include:
Opening arguments
Witnesses testifying for the plaintiff (or for the State in a criminal case)
Summary judgment motions
Witnesses testifying for the defendant
Preparing the record for appeal
Throughout the trial, attorneys for both sides may have to make objections to certain testimony or evidence. This means that your attorney will need to be knowledgeable and familiar with your case so that they can anticipate exactly what kind of claims will be made by the other side. Moreover, your lawyer will need to have a solid understanding of courtroom procedure so that testimony or evidence that hurts your case is not presented during the trial.
If you were arrested and charged with a criminal offense in NJ, you need a skilled criminal attorney on your side throughout the legal process. The experienced criminal defense lawyers at Bramnick, Rodriguez, Grabas, Arnold & Mangan, LLC can help you fight your charges. Contact us anytime to schedule a free consultation about your case.
LET OUR
Our law firm is committed to protecting your rights against large insurance companies and governmental abuse. Our lawyers will fight to get you the results you deserve.
Serving NJ Since
Since 1984 Bramnick, Rodriguez, Grabas, Arnold & Mangan have built a solid reputation for protecting the rights of New Jersey residents and achieving outstanding results.
Whether you're in the hospital, at the police station or need to speak with someone after hours, we ca help. We are available and ready to start working for you immediately.
Recent Personal Injury Case Results
$5,500,000 / MOTORCYCLE ACCIDENT
Since 1984, the law office of Bramnick, Rodriguez, Grabas, Arnold & Mangan, LLC has worked tirelessly to deliver proper compensation to those involved in personal injury accidents throughout New Jersey. If the best settlement for your claim is not offered, our experienced trial lawyers are prepared to take your case before a New Jersey Superior Court jury.
Speak With An Attorney Now / 877.423.4878
OR Request A Call Back
A CALL BACK
1827 East Second Street
45 Bleeker Street
1135 Broad Street, Suite 211
7 Broad Street, 2nd Floor
940 South Ave W. 2C
2G Auer Court
20 Brace Road, Suite 115
Call or Visit One of Our
Seven New Jersey Locations
What to Do After a Slip and Fall Accident During [...]
What is a Wrongful Death Lawsuit? A wrongful death lawsuit [...]
Guilty Verdict for NJ Resident On November 22, 2019, a [...]
Changes to DUI Law in NJ The holiday season is [...]
2020 © Bramnick Law. Disclaimer Sitemap


15% discount on repairs. No overtime charges for after hours or weekend work. Includes 2 tune-ups every 6 months & 12 air filters for the year! Same day service.
Air Conditioning Installation. Air Conditioning Maintenance. Air Conditioning Repair. Call The Pros!
Need AC repair in Port St. Lucie, FL? Air Temp Air Conditioning helps our customers have clean, reliable air year round. We have uniformed technicians and can help you 24 hours a day. Whether you need an AC repair or a HVAC repair in Port St.Lucie, we're here to help.
Our local AC and HVAC company office has provided services to this area for 10 years. We've built a reputation for reliability. We stand by our mission to offer professional services by professional people. As a licensed and insured AC company, our customer referrals prove we are working hard to keep our customers happy, and the best AC service provider near you.


Ah! A rest day in Boise. I love the community and could easily move here. I decided to have a "me" day. I had no agenda. Too often I get into a fog as I create an agenda and strive to do all that I wish for that day. Today was different. I woke up and was invited by Kert to breakfast at Goodys. We had a great time and met a couple that we chatted with for quite some time...mostly about Idaho and the Fuller Center. Then we headed to mass at the First Presbyterian Church. After church, I spoke with a number of people about the FCBA and my experiences. Then, my adventure began. I pulled on my tennis shoes and began walking in nonspecific direction. I ended it at the Julia Harris Park where I went to the zoo, watched some great tennis, hit a few balls, stopped to sit on Abraham Lincolns lap and stopped to talk to so many people. Today my goal was to be spontaneous and just take advantage of every opportunity that comes before me. I stopped to smell the roses, looked at the architecture of the city, talked with people and just stopped to take a photo once in a while. To often,no am concerned with how far I go or what I accomplish. Today, I was focusing on the journey itself. The word agenda was not in my vocabulary and this gave me such a greater appreciation for all that is around me.
As the day is coming to a close, I think about our journey on the FCBA.  We have all taken the risk to walk away from their daily agenda and into a world far different than any other experience we have ever had.  Yes, we are focused on reaching Vancouver (it's Grand Coulee for me), but we are more focused on the journey itself.  We make every effort to experience each community, to interact with its people, to appreciate the topography  of the miles and miles we have cycled through, to smell the fragrant air and to feel the wind  against our face.  We have become so much more sensitive of our surrounding.  We have also become more sensitive to the needs of others.  I can only speak for myself, but I am gaining a love for life as it is, not as I wish it to be.  Sometimes I just need to sit back and not get so engulfed in my goal....now I am really beginning to focus on the journey.  Thanks FCBA.  You not only help to build homes, but also you build an appreciation for the needs of others.  Thank you Dan Zassick for your care package of bike essentials.


American Swan Winter Carnival is featuring 30,000+ Styles in Winterwear and more. Sale is Offering Flat 55% off on listed price in sale.
Shop for the latest winter styles with heavy discounts. Grab cool styles for this Winter.
Just Apply the Coupon Code WARM55 to avail discount. you will get Flat 55% Discount on your Cart. You must shop for Rs. 1499 or above to avail this coupon.


text:
« Too Expensive To Waste | Main | Commentary »
Tufts Judicial Activism
by Douglas Kingman | The TCUJ does its part to ruin democracy.
Indignation is a common reaction of Tufts students to the smallest appearance, real or perceived, of election impropriety. Yet, when elections are held on campus, scandalous interpretation of bylaws and precedents are tolerated to support the agendas of overzealous ideologues. This was the case last spring when Tufts Elections Board (ELBO), responsible for conducting campus elections, held a referendum vote on the issue of subsidizing wind power through a mandatory $20 student fee sponsored by the student organization Environmental Consciousness Outreach (ECO). ELBO's poor judgment helped advance ECO's position. This prompted THE PRIMARY SOURCE to challenge ELBO's practices with a Tufts Community Union Judiciary (TCUJ) complaint that was ultimately decided in ELBO's favor. The TCUJ struck with impressive judicial activism when it trivialized ELBO's bylaw transgressions and went on to legislate destructive new bylaw protocol.
ELBO's bylaws charge it to "ensure fair elections." Yet, when ELBO notified the Tufts community of the upcoming ECO referendum, it grossly violated its charter by providing a link titled "why vote yes for wind power." The contention that this constitutes election bias is supported by TCUJ precedent. In 2001, a misleading e-mail authorized by a Tufts administrator was sent out in favor of approving a new constitution for the Tufts Community Union. Even though the e-mail did not originate from ELBO or advocate an ELBO position by association with the email (both the case this time), the TCUJ invalidated that election because, "a fair election would technically not be possible" due to the impression that ELBO had endorsed one of the referendum positions. The current situation is even more straightforward. ELBO supplied information to students on its own behalf that directly advocated one side of the referendum question. ELBO minimally showed poor judgment and at worst exerted a concerted effort to impact the outcome. This is hardly what could be considered "fair."
Yet, when ELBO Chair Denise Weisman was asked to defend her organizations' actions, she asserted to the Tufts Daily that "'Fair' to ELBO is giving all students an opportunity to vote." This logic is severely flawed. Consider a fair election by this standard in pre-liberation Iraq. Her "fair elections" would legitimize the elections that Saddam Hussein held in his country that made a mockery of democracy. Never mind the fact that the constituency was coaxed into voting a particular way by force, propaganda, and other tactics. Look how fair it was—everyone voted.
The TCUJ, however, did not agree with ELBO's definition of fair—or at least as much as one group looking to protect their allies is able.
The TCUJ reprimanded ELBO with a warning against breaking fundamental bylaw guidelines yet gave them flagrant underhanded approval by declaring that the results of the referendum would stand regardless of the established wrongdoing. The TCUJ found that, "while ELBO was incorrect in posting the ECO information, the margin of victory of this referendum was such as to negate any possible effects of the website posting." This absurd ruling shows a major fallacy. If a necessary condition to conduct an election is ignored, the outcome under unlawful circumstances is worthless regardless of a decisive outcome. Because it is not possible to determine how the irregularity of the vote affected the margin of victory, the results are meaningless and the vote must be recast. Furthermore, because the referendum election was intended only to show a vote of confidence to the trustees for the proposed measure, each percentage of votes cast is meaningful. Having perused the information provided illegally by ELBO that advocated the position, a student might have felt compelled to vote for the measure. It would be impossible to tally all of the possible effects the partisan information had on voters in the referendum. This election was skewed by unlawful variables. This is precisely the reason that the vote must be recast if it is to hold.
To add insult to injury, the TCUJ made recommendations to the senate to improve referendum votes in the future. The TCUJ ordered that a senate "fact finding mission be responsible for making every effort to research information and thoroughly understand both sides of the issues called into question" in future referendums. While the TCUJ ordered no more ELBO election endorsements, it recommended that the senate fact-finding mission write a "pro" and "con" argument for each future referendum. This ridiculous assertion of power is not only unnecessary, since groups should educate students about the questions themselves, but dangerous because of the power the senate would hold to frame all future debates on campus. By the discretion of the senate president (elected with a smaller voter turnout than the referendum in question) the ruling gives the senators the final say over what voting "yes" or "no" for a particular measure means. This amount of power is not only profound, but eliminates the ability of controversial minority opinion to triumph over the ruling powers.
The TCUJ's lack of jurisprudence and ELBO's incompetency won't stand unanswered. THE PRIMARY SOURCE will appeal this case to the Committee on Student Life in the upcoming weeks. Hopefully, they will recognize that as overwhelming as an election outcome is, results obtained in a flawed manner are not valid.
Mr. Kingman is a junior majoring in Chemical Engineering.
in Campus Activism | Permalink


text:
\section{Introduction}\label{sec:intro}
Sparse general matrix multiplication (SpGEMM) is a fundamental building block in many real-world applications such as algebraic multigrid solvers~\cite{AMG, AMG2}, multi-source breadth first search~\cite{BFS}, Markov clustering~\cite{markov}, finite element simulations based on domain decomposition~\cite{finite-element}, and molecular dynamics simulations~\cite{molecular}.

As a memory-bounded application, the memory access efficiency of both the main memory and multi-level cache memory is critical for SpGEMM's performance~\cite{PB-SpGEMM, roofline}. Two of the most critical memory access patterns in performing SpGEMM are 1) the vast and irregular memory accesses to the $B$ matrix for the row-wise SpGEMM~\cite{yusuke}, or to the $\hat{C}$ matrix for the outer-product SpGEMM~\cite{PB-SpGEMM}, and 2) the memory accesses in the accumulation method, which accumulates multiple intermediate lists with varying lengths.

The existing Heap-SpGEMM~\cite{yusuke} and Hash-SpGEMM~\cite{yusuke} only consider the computation complexity while putting no consideration to the memory access efficiency mentioned above. Specifically, the Heap-SpGEMM may have high translation lookaside buffer (TLB) cache miss rate when accessing the $B$ matrix, whereas the Hash-SpGEMM may waste memory bandwidth utilization when accumulating the intermediate lists due to the random access pattern caused by hashing operations. As a result, the two libraries suffers suboptimal performance on the CPU architectures (Section~\ref{sec:result}).

Although the PB-SpGEMM~\cite{PB-SpGEMM} considers the memory access efficiency, they fail to fully assess the gains and overheads in their method. They focus on solving the vast and irregular memory accesses to the $B$ matrix in the row-wise dataflow by adopting the outer-product dataflow~\cite{yusuke, PB-SpGEMM}. However, their memory accesses to the $\hat{C}$ matrix are still irregular with an even larger amount~\cite{PB-SpGEMM}. Therefore, the PB-SpGEMM may suffer worse TLB cache hit rate than the Heap-SpGEMM. As a result, PB-SpGEMM suffers exceptionally lower performance than its precedent Heap-SpGEMM and Hash-SpGEMM (Section~\ref{sec:result}).

To avoid or minimize the inefficient memory access issues in the existing SpGEMM libraries, we propose a novel and more efficient accumulation method, named BRMerge, for SpGEMM on the CPU architectures. To better understand the proposed work and its architectural benefits, we first describe several backgrounds and the BRMerge accumulation method. And then, we discuss the architectural benefits of the BRMerge accumulation method and the inefficient memory access issues of the existing methods in Section~\ref{sec:memory}. We denote the proposed accumulation method as BRMerge since it is based on the binary-row-merging algorithm described in Section~\ref{sec:brmerge}.

The BRMerge accumulation method adopts the row-wise dataflow to perform SpGEMM, which multiplies each row of $A$ with the whole $B$ matrix for the corresponding row of $C$~\cite{gustavson, yusuke, nsparse, speck}. It consists of a multiplying phase and an accumulating phase to compute each output row. In the first multiplying phase, the multiple required rows of $B$ are accessed, multiplied with the nonzeros in the corresponding row of $A$, and the generated intermediate lists are stored in a consecutive memory space, which is implemented with a ping-ping buffer. In the second accumulating phase, the multiple intermediate lists generated in the previous phase are merged two by two in a tree-like hierarchy between two ping-pong buffers.
The architectural benefits of BRMerge are the streaming access patterns, minimized TLB misses, and reasonably high L1/L2 cache hit rates, which result in both low access latency and high bandwidth utilization when performing SpGEMM on the multi-core CPU architectures (Section~\ref{sec:memory}).

Based on the BRMerge accumulation method, we further propose two SpGEMM libraries named BRMerge-Upper and BRMerge-Precise. The BRMerge-Upper and BRMerge-Precise use the upper-bound~\cite{bhsparse} and precise allocation methods~\cite{nsparse}, respectively. Comprehensive evaluations with 26 commonly used benchmarks on two Intel Xeon CPUs show that the proposed SpGEMM libraries outperform the state-of-the-art SpGEMM libraries significantly. Specifically, BRMerge-Precise achieves on average $1.42\times$ and $1.39\times$ performance speedups compared to the state-of-the-art best-performing SpGEMM library (i.e., Hash-SpGEMM~\cite{yusuke}) on the Intel Xeon Platinum 8163 CPU and the Intel Xeon Gold 6254 CPU, respectively.

The key technical contributions of this work are as follows:
\begin{itemize}
\item We present a novel accumulation method, named BRMerge, for the SpGEMM algorithm on multi-core CPU architectures.
\item We analyze the architectural benefits of the proposed BRMerge accumulation method on multi-core CPU architectures, emphasizing the memory subsystem.
\item We propose two SpGEMM libraries named BRMerge-Upper and BRMerge-Precise based on the BRMerge accumulation method and different allocation methods.
\item Evaluations with 26 commonly used benchmarks on two Intel Xeon CPUs show that the proposed SpGEMM libraries outperform the state-of-the-art SpGEMM libraries significantly.
\end{itemize}

The rest of the paper is organized as follows. Section~\ref{sec:bg}  provides backgrounds and related work on the SpGEMM algorithm. Section~\ref{sec:method} describes the binary-row-merging algorithms, the proposed BRMerge accumulation method, the performance analysis of the BRMerge accumulation method, and two proposed SpGEMM libraries based on the BRMerge accumulation method and other methods. Section~\ref{sec:evaluate} first describes the performance evaluation environments, then provides the results and discussions. Section~\ref{sec:conclude} concludes this paper.

\section{Backgrounds and Related Work}\label{sec:bg}
In this section, we provide the backgrounds and related work of SpGEMM algorithms mainly on CPUs.
\subsection{SpGEMM Preliminaries}
\subsubsection{Notations}\label{sec:note}
We define several notations for the rest of this paper. Capital letters $A$, $B$, and $C$ denote matrices. The two input matrices are $A$ and $B$ with sizes of $M \times K$ and $K \times N$, respectively. The result matrix is denoted as $C$ with a size of $M \times N$.

$A_{ij}$ represents an element located at the $i_{th}$ row and $j_{th}$ column of $A$. $A_{i*}$ represents all the nonzero elements in the $i_{th}$ row of $A$. Similarly, $B_{*j}$ represents all the nonzero elements in the $j_{th}$ column of $B$.

\subsubsection{Dataflows to Perform SpGEMM}\label{sec:dataflow}
Given two input matrices $A$ and $B$, the text-book definition of matrix multiplication is computed as:
\begin{equation}C_{ij} = \sum_{k= 1}^{K}A_{ik}\cdot{B}_{kj}.\label{eq:inner}\end{equation}
Equation~(\ref{eq:inner}) describes the inner-product dataflow to perform SpGEMM.

One variation of Equation~(\ref{eq:inner}) is:
\begin{equation}C_{i*} = \sum_{k}A_{ik}\cdot{B}_{k*},\label{eq:row-wise}\end{equation}
where $k$ belongs to the set of column indices of the nonzero elements in each row of $A$.
Equation~(\ref{eq:row-wise}) describes the row-wise dataflow to perform SpGEMM. A similar variation of the row-wise dataflow is the column-wise dataflow described in Equation~(\ref{eq:column-wise}):
\begin{equation}C_{*j} = \sum_{k}A_{*k}\cdot{B}_{kj},\label{eq:column-wise}\end{equation}
where $k$ is the set of row indices in each column of $B$.

The SpGEMM can also be computed as:
\begin{equation}C = \sum_{k = 1}^{K}A_{*k}\cdot{B}_{k*},\label{eq:outer}\end{equation}
where the $\sum$ operation computes the sum of multiple sparse partial matrices. Equation~(\ref{eq:outer}) describes the outer-product dataflow to perform SpGEMM.

All the four dataflows mentioned above can generate the correct result matrix $C$ but with different performance issues on modern hardware platforms. Generally, the row-wise dataflow is reported to have the best performance than other dataflows by recent SpGEMM work on CPUs~\cite{yusuke}, GPUs~\cite{rmerge, rmerge2, nsparse, speck} and accelerators~\cite{matraptor, gamma}. Note that, based on Equation~(\ref{eq:row-wise}) and Equation~(\ref{eq:column-wise}), the implementation of the row-wise and column-wise dataflows are dual of each other. Therefore, we only discuss the row-wise dataflow for simplicity.

There are three key benefits of the row-wise dataflow compared to the inner-product dataflow: 1) zero elements are entirely skipped in both the computation and memory access, 2) the computation of each output row is independent of each other; therefore, the row-wise SpGEMM can be easily parallelized, and 3) accumulating the intermediate lists of each output row has a good temporal locality. As a result, we adopt the row-wise dataflow in our proposed SpGEMM algorithm.

\subsubsection{CSR Storage Format}\label{sec:CSR}
The compressed sparse row (CSR) storage format is one of the most commonly used sparse storage formats for SpGEMM, which is commonly used in the row-wise SpGEMM~\cite{yusuke, PB-SpGEMM, bhsparse, rmerge, rmerge2, nsparse, speck}.

Fig.~\ref{fig:CSR} illustrates the CSR storage format. The CSR consists of three arrays to record the nonzero elements and their corresponding indices. The $val$ and $col$ arrays record the values of the nonzero elements and their corresponding column indices in a sorted row-major and column-major order. The lengths of $val$ and $col$ arrays are both the number of nonzero elements ($n_{nz}$) of the sparse matrix. The $rpt$ array records the start and end offsets for each row's values and column indices in the $val$ and $col$ arrays. Since the $i_{th}$ row's end offsets can be encoded the same as the $(i+1)_{th}$ row's start offsets in the $rpt$ array, the $rpt$ array is further compressed to $M + 1$ entries, where $M$ is the number of rows. One of the key performance benefits of using CSR is that it is easy to access the elements of an entire row. In this paper, we also use CSR as the storage format for the input and output matrices in the proposed SpGEMM libraries.

\begin{figure}[h]
\centering
\includegraphics[width=0.45\textwidth]{fig/CSR.pdf}
\caption{Illustration of the CSR storage format. Left: matrix $A$ in the dense storage format. Right: matrix $A$ in the CSR storage format}
\label{fig:CSR}
\end{figure}

\subsubsection{Compression Ratio (CR)}\label{sec:CR}
In performing SpGEMM, usually, multiple intermediate products are combined to generate one result element due to the duplicate indices. The compression ratio is defined by dividing the total number of intermediate products ($n_{prod}$) in performing SpGEMM by the result matrix's total number of nonzero elements ($n_{nz}$) (Equation~(\ref{eq:CR})). In other words, the compression ratio shows the average number of intermediate products that generate one nonzero element in the result matrix.
\begin{equation}Compression\ ratio = \frac{Total\ n_{prod}\ to\ compute\ C}{Total\ n_{nz}\ of\ C}.\label{eq:CR}\end{equation}

\subsection{Related SpGEMM Designs}
In this section we describe the existing methods to address two important design issues: the accumulation method and the allocation method. The accumulation method generates multiple intermediate lists and merges these lists to one result row. Since the structure of the result matrix cannot be not known in advance, where to store the result row in the accumulation method remains a challenging task. The allocation method tackles the unknown memory allocation challenge mentioned above.
\subsubsection{Accumulation Methods}\label{sec:accumulation}

The heap-based accumulation method~\cite{yusuke} merges all the intermediate lists of one result row simultaneously by using a heap data structure. Since the intermediate lists are already in sorted ascending order, the heap keeps all the intermediate lists' front elements to sort the result row. In each iteration, the heap pops the smallest element to the result row. The popped element will be added to the current position in the result row or be stored in a new position in the result row, based on the indices of the popped element and the current element in the result row. After this operation, the heap reads the next element from the one of the multiple intermediate lists, which provides the smallest element last time. The read element should be pushed to the heap for the next iteration.

The most two time-consuming operations in the heap-based accumulation method are the pop and push operations, which are both with $O(log(k))$ complexity, where $k$ is the heap size. In this paper, we use the binary-row-merging-based accumulation method, which can be seen as a heap-based method when the heap size is two. As a result, our accumulation method's pop and push operations are simply one comparison operation and one pointer addition operation for the two input intermediate lists.

The hash-based accumulation method~\cite{yusuke} merges the intermediate lists by inserting all the intermediate elements into a hash table. After the insertion, the valid nonzero elements are extracted and sorted to generate one result row~\cite{yusuke, nsparse}. The hashvec-based accumulation method~\cite{yusuke} follows the same computation flow as the hash-based accumulation method. The difference is that the hashvec-based accumulation method uses the Intel's SIMD instructions~\cite{wiki-AVX} to implement the hashing operation. The hashvec-based accumulation method is reported to have better performance than the hash-based counterpart when the hash collision rate is high~\cite{yusuke}.

The ESC-based accumulation method is first proposed in the GPU work~\cite{cusp} and is recently used by the CPU work~\cite{PB-SpGEMM} with the outer-product dataflow. The ESC represents \textbf{E}xpand, \textbf{S}ort, and \textbf{C}ompress. It first expands the intermediate lists by storing them consecutively without order. Then it sorts the unordered intermediate lists with duplicate column indices. At last, it compresses the sorted list by merging the duplicate elements which are consecutive to each other.

The row-merge-based accumulation method is proposed on the GPU architectures~\cite{rmerge, rmerge2}. It merges a pre-defined power-of-two number (e.g., 2, 4, 8, 16 et al.) of intermediate lists simultaneously using subwarp~\cite{cuda-doc}. It is similar to the heap-based accumulation method in that it merges multiple intermediate lists simultaneously. Since it uses one thread to maintain one or more intermediate lists and accesses each of these intermediate lists consecutively within the thread, the warp-level accesses to these intermediate lists are completely non-coalesced, which is detrimental to its performance on GPU architectures.

\subsubsection{Allocation Methods}\label{sec:allocation}
Since the structure of the result matrix (i.e., the number of nonzero elements of each row of the result matrix) cannot be determined before performing SpGEMM, allocating the memory space to store the result matrix is a challenging task. We describe two commonly used allocation methods to tackle the above challenge. The two allocation methods are also implemented in the two proposed SpGEMM libraries.

The upper-bound allocation method~\cite{bhsparse, yusuke} first computes the number of intermediate products ($n_{prod}$) of each output row as the upper-bound memory space (temporary memory space) to store the result matrix. The computation of the $n_{prod}$ is a very low-cost operation compared to SpGEMM. And then the result matrix is computed and stored into the temporary memory space. Since the result matrix should be stored in a standard storage format (e.g., CSR format), an additional memory copy operation is required to move the result matrix from the temporary memory space to the memory space conforming to the standard storage format.

The precise allocation method~\cite{yusuke, nsparse, speck} first computes the number of nonzero elements ($n_{nz}$) of each result row. This process only computes the indices information of the input matrices without floating-point multiplication, which is commonly named as the symbolic computation~\cite{nsparse}. Then the precise memory space can be allocated based on a low-cost prefix-sum operation on the computed $n_{nz}$ information. At last, the result matrix is computed and directly stored in the standard memory space without the need for the additional copy operation. However, one additional cost when using the precise allocation method is the nontrivial symbolic computation, which has a similar complexity to the SpGEMM itself.

\section{Method}\label{sec:method}
In this section, we first describe the binary-row-merging algorithm and the BRMerge accumulation method. And then we analyze the architectural benefits of BRMerge and the inefficient memory access issues of the existing methods. At last, we describe two proposed SpGEMM libraries, named BRMerge-Upper and BRMerge-Precise, based on the BRMerge accumulation method.
\subsection{Binary-row-merging Algorithm}\label{sec:brmerge}
The binary-row-merging algorithm merges multiple sorted lists into one list. Fig.~\ref{fig:brmerge} illustrates how the binary-row-merging algorithm merges six sorted lists into one list.

\begin{figure}[h]
\centering
\includegraphics[width=0.45\textwidth]{fig/brmerge.pdf}
\caption{Illustration of the binary-row-merging algorithm. There are six sorted lists. the elements of the lists are $col$ and $val$ pairs, where $col$ represents the column indices and $val$ represent the values. Two lists are merged each time by comparing their $col$s of two front elements, which is similar to the mergesort algorithm~\cite{merge-sort}, except that the $val$s with duplicate $col$s are combined into one $val$. All the lists are merged two by two in a tree-like hierarchy. After each round of merging, the number of intermediate lists is reduced by half. Therefore, there are totally $\lceil log(6)\rceil = 3$ rounds of merging until one list is left as the result list.}
\label{fig:brmerge}
\end{figure}

\subsection{BRmerge Accumulation Method}\label{sec:brmerge-accumu}
Algorithm~\ref{alg:brmerge} illustrates the proposed BRMerge accumulation method in a single thread version. BRMerge mainly consists of two phases: 1) generating the intermediate lists, named multiplying phase, and 2) merging the generated lists into one result row, named accumulating phase.

\begin{algorithm}
\caption{BRMerge accumulation method}
\label{alg:brmerge}
\begin{algorithmic}[1]
	\Require A, B in CSR format

	\State {Compute max\_row\_nprod\_C, max\_row\_nnz\_A}

	\State {ping\_buffer = \textbf{new double} [max\_row\_nprod\_C]}
	\State {pong\_buffer = \textbf{new double} [max\_row\_nprod\_C]}
	\State {ping\_list\_offset = \textbf{new int} [max\_row\_nnz\_A + 1]}
	\State {pong\_list\_offset = \textbf{new int} [max\_row\_nnz\_A + 1]}
	\For {$i \gets 1..M$ }

	\State{dst\_buffer = ping\_buffer}
	\State{dst\_list\_offset = ping\_list\_offset}
	\State{buffer\_incr = 0, list\_incr = 0, dst\_list\_offset[0] = 0}
	\For {$A_{ik}$ in $A_{i*}$} \Comment{Multiplying phase}
	\For{$B_{kj}$ in $B_{k*}$}
	\State {dst\_buffer[buffer\_incr++] = $A_{ik}$ * $B_{kj}$}
	\EndFor
	\State {dst\_list\_offset[++list\_incr] = buffer\_incr}
	\EndFor

	\State {num\_list = list\_incr}
	\State {src\_buffer = dst\_buffer}
	\State {src\_list\_offset = dst\_list\_offset}
	\State {dst\_buffer = pong\_buffer}
	\State {dst\_list\_offset = pong\_list\_offset}
	\While {num\_list $>$ 1} \Comment{Accumulating phase}
	\State {inner\_num\_list = num\_list, num\_list = 0}
	\While {inner\_num\_list}
	\If {inner\_num\_list $\geq$ 2}
	\State {consume two lists and store to dst\_buffer}
	\State {inner\_num\_list -= 2}
	\ElsIf {inner\_num\_list == 1}
	\State {copy the last list to dst\_buffer}
	\State {inner\_num\_list -= 1}
	\EndIf
	\State {num\_list++}
	\EndWhile
	\State {swap(src\_buffer, dst\_buffer)}
	\State {swap(src\_list\_offset, dst\_list\_offset)}
	\EndWhile
	\State {Store the $i_{th}$ result row in the src\_buffer into the appropriate memory space}

	\State {Release the allocated ping-pong buffers}
	\EndFor
\end{algorithmic}
\end{algorithm}

Since the accumulating phase in BRMerge consists of multiple rounds of merging, we use ping-pong buffers to store the input and output lists in each round of merging. For simplicity in Algorithm~\ref{alg:brmerge}, the \emph{ping\_buffer} and the \emph{pong\_buffer} represent both the $col$'s and $val$'s pair of the ping-pong buffers. We use an additional pair of ping-pong buffers named \emph{ping\_list\_offset} and \emph{pong\_list\_offset} to record the list offsets of the intermediate lists, which are stored consecutively in the \emph{ping\_buffer} and \emph{pong\_buffer}.

Line 1 in Algorithm~\ref{alg:brmerge} computes the sizes of the two pairs of ping-pong buffers, which are the maximum $n_{prod}$ of the output rows and the maximum $n_{nz}$ of the rows in $A$. Line 2--5 allocates the ping-pong buffers. The multiplying phase (Line 10--15) follows the row-wise dataflow (Equation~(\ref{eq:row-wise})). It multiplies the nonzero elements in each row of $A$ with the corresponding rows in $B$ to generate the intermediate lists. The multiple generated intermediate lists are stored consecutively in one of the ping-pong buffers (pointed by the \emph{dst\_buffer}). The start offset of each list is recorded in one of the \emph{list\_offset} ping-pong buffers  (pointed by the \emph{dst\_list\_offset}). Note that each generated list is naturally sorted since each row of $B$ is sorted due to the CSR format.

\begin{figure}[h]
\centering
\includegraphics[width=0.55\textwidth]{fig/brmerge-imp.pdf}
\caption{Illustration of Line 21--35 in Algorithm~\ref{alg:brmerge}, which is the implementation of the binary-row-merging algorithm. The illustrated implementation uses the same input data as used in Fig.~\ref{fig:brmerge}.}
\label{fig:brmerge-imp}
\end{figure}

The accumulating phase (Line 21--35) implements the binary-row-merging algorithm, which is further illustrated in Fig.~\ref{fig:brmerge-imp}. The accumulating phase is implemented with two while-loops. The outer while-loop determines how many rounds of merging are required. It monitors the \emph{num\_list} information, which is reduced by half after each round of merging. The exit condition of the outer while-loop is when \emph{num\_list} equals $1$ (Line 21). The inner while-loop consumes the \emph{num\_list} intermediate lists from the \emph{src\_buffer} two by two and stores the merged result list to the \emph{dst\_buffer} (Line 25). The consume operation of two input lists is similar to the merge sort algorithm~\cite{merge-sort}, except that the elements with duplicate indices are added together. When only one intermediate list is left in the inner while-loop, it is directly copied to the \emph{dst\_buffer} (Line 28). After the completion of each round of merging (i.e., after the completion of one inner while-loop), the pointers of the ping-pong buffers are swapped (Line 33--34) without data movement. As a result, in each inner while-loop, we always read input lists from the \emph{src\_buffer} and write the result lists to the \emph{dst\_buffer}. After the two while-loops, the result row is stored in the \emph{src\_buffer} due to the last swap operation. Line 36 writes the result row in the \emph{src\_buffer} to the appropriate memory space determined by the allocation method.

\subsection{Performance Analysis of the BRMerge Accumulation Method and Other Methods}\label{sec:memory}
Two of the most critical memory access patterns in performing SpGEMM are 1) the vast and irregular memory accesses to the $B$ matrix for the row-wise SpGEMM~\cite{yusuke}, or to the $\hat{C}$ matrix for the outer-product SpGEMM~\cite{PB-SpGEMM}, and 2) the memory accesses in the accumulation method, which accumulates multiple intermediate lists with varying lengths.
In this section, we analyze the architectural benefits of the BRMerge accumulation method for the two critical memory access patterns on modern multi-core CPU architectures. We also analyze the inefficient memory access issues of the existing methods compared to BRMerge.

In the following, we present the analysis mentioned above when we show the three architectural benefits of the proposed BRMerge, which are the streaming access patterns, minimized TLB cache misses, and reasonably high L1/L2 cache hit rate.

\subsubsection{Streaming Access Patterns}
The streaming access patterns is mainly in the accumulating phase in BRMerge. The accumulating phase uses one CPU thread to merge all the intermediate lists two by two. Therefore, at any given time, it only merges two lists into one result list. The memory read and write patterns are always consecutive when accessing the input and output lists. Furthermore, the duration of the read and write operation is short since it only performs one simple comparison with one potential addition operation before the next data read and write operations. Therefore, the memory access pattern of the input and output lists is a typical streaming access pattern.

There are mainly two benefits of the streaming access patterns in modern CPU architectures. The first benefit is to maximize the memory bandwidth utilization between any two high-level memory systems (e.g., between L1 and L2 caches, or between L2 and L3 caches). In modern CPUs, the memory access granularity between the CPU and the L1 cache can be as small as 4 or 8 bytes. However, the minimum memory access granularity between high-level memory systems is usually at least a cache line (i.e., 64 bytes).
When a program randomly accesses a 4-bytes memory space, which loads a cache line from the L2 to the L1, the other 60 bytes of the loaded cache line may not be used after a long duration. And after the long duration, the above-loaded cache line may have been evicted from the L1 cache, which means the bandwidth utilization between the L1 and L2 caches is only $4/64 = 6.25\%$. In contrast, when a program has a streaming access pattern and accesses a 4-bytes memory space, which loads a cache line from the L2 to the L1, the other 60 bytes of the loaded cache line will be used immediately. Therefore, a program with the streaming access patterns can achieve up to $64/64 = 100\%$ bandwidth utilization between the high-level memory systems (e.g., between L1 and L2 caches).

The second benefit of the streaming access pattern is to take advantage of the hardware pre-fetching mechanism, which can increase the L1 cache hit rate. Modern CPUs usually have a hardware pre-fetching mechanism, which automatically pre-fetches the following cache line when a few consecutive memory accesses are detected~\cite{intel-opt}. Consider a typical case where a CPU program consecutively accesses and processes data in a memory space with a size between the L1 and L2 cache sizes. Due to the pre-fetching mechanism, most to-be-accessed data may have been pre-fetched from L2 to L1, improving the L1 cache hit rate. This benefit is provided that the CPU also takes time to process the accessed data; therefore, part or all of the pre-fetching latency from the L2 cache is hidden, guaranteeing an improved L1 cache hit rate.

Since the accumulating phase of the BRMerge has a typically streaming access pattern when merging the intermediate lists, BRMerge achieves the maximized memory bandwidth utilization and takes advantage of the hardware pre-fetching mechanism. In contrast, the hash-based and hashvec-based accumulation methods do not have the streaming access pattern since the hashing operation can direct the insertion to potentially any place in a hash table. Therefore, when the random memory accesses miss the L1 cache and load a cache line from the L2 cache, most memory bandwidth between L1 and L2 may be wasted. Moreover, the hash-based and hashvec-based accumulation methods cannot take advantage of the pre-fetching mechanism.

\subsubsection{Minimized Translation Lookaside Buffer (TLB) Cache Misses}
We discuss the TLB hit rate for two reasons: 1) both the row-wise and outer-product SpGEMM algorithms have to randomly access a large matrix (either the $B$ matrix for the row-wise SpGEMM or the $\hat{C}$ matrix for the outer-product SpGEMM~\cite{PB-SpGEMM}), which may cause high pressure for the TLB cache, and 2) the TLB cache hit rate is also critical for the performance of CPU programs. Some may argue that the outer-product SpGEMM can merge the partial matrices on the fly while generating them. However, this will require too many merging operations and data movement operations, which quickly cause more overheads than the gains, especially for the medium-to-large-size matrices which cannot fit in the cache~\cite{PB-SpGEMM}. Therefore, we do not consider that situation.

We first explain the importance of the hit rate of L1-d TLB (i.e., the first level data TLB). Each time the CPU accesses a memory space, it generates a virtual address. The virtual address should be translated to a physical address (with page number) by consulting the L1-d TLB before finishing the querying to the L1 cache. If accessing the L1-d TLB misses, the querying to the L1 cache is hanged, and the CPU may consult the shared L2 TLB, which has a similar cost to the access to L2 cache~\cite{intel-opt, DLL-TLB}. Therefore, the L1-d TLB miss also causes a similar cost to L2 cache access. Also, note that if accessing the L2 TLB misses again, the CPU should start a page table walk process, which involves much more expensive main memory accesses~\cite{DLL-TLB}.

In most cases, one TLB miss penalty can be alleviated by multiple following accesses to the same 4KiB page, which always hits the L1-d TLB. However, if each TLB miss is not followed with many accesses to the same 4KiB page, the penalty will of course not be alleviated, which is the case for the Heap-SpGEMM. We then shall discuss the possible TLB misses for the interested SpGEMM algorithm, how the proposed BRMerge accumulation method minimizes the TLB misses, and the inefficient TLB access issue in the Heap-SpGEMM.

We provide an upper-bound estimation of the possible TLB misses for the row-wise SpGEMM algorithm on Intel's recent Skylake server-class CPU. The Skylake CPU has a 64-entry private L1-d TLB per physical core backed by a 1536-entry shared L2 TLB per CPU~\cite{intel-opt}. The hyper-threading technology is often enabled to improve the performance of the multi-threaded programs~\cite{intel-opt}. When the hyper-threading technology is enabled, there will be two logical cores in each physical core, which share the L1-d TLB resources~\cite{intel-opt}. As a result, each logical core only has 32 entries of the L1-d TLB.

The upper-bound estimation assumes that the memory distance of different accessed rows in $B$ to compute one result row is more than the 4KiB page size. In other words, accessing each different row in $B$ occupies two TLB entries, where the "two" entries are occupied by the $col$ and $val$ arrays of a row. Therefore, when a logical core computes a result row that needs to access $32/2 = 16$ different rows in $B$, there may be capacity-caused L1-d TLB misses. Note that the number of the accessed rows of $B$ is the same as the $n_{nz}$ of the corresponding row in $A$. In the real situation, the $n_{nz}$ of rows in A can be much larger than 16, which can cause many TLB misses in the SpGEMM algorithm.

Yet, the capacity-caused TLB misses are hard to be avoided in performing SpGEMM. However, minimizing the TLB misses can be easily achieved by ensuring the the TLB is only updated once when loading each row from the $B$ matrix. To achieve this, in many cases, we only need to make sure each row of the $B$ matrix is loaded consecutively in a short duration. The proposed BRMerge accumulation method meets this requirement since it accesses all the required rows in $B$ consecutively and stores the multiplied intermediate lists in a local ping-pong buffer in the multiplying phase.

In contrast, the Heap-SpGEMM does not meet the above requirement. It maintains the accesses to all the required rows of $B$ during the its accumulating phase. Therefore, the corresponding TLB entries for accessing each row of $B$ may be inserted into and evicted out of the TLB multiple times, suffering not-alleviated TLB misses penalties when there are capacity-caused TLB misses.

\subsubsection{Reasonably High L1/L2 Cache Hit Rates}
When accumulating the multiple intermediate lists in the accumulation method, the data may be processed multiple times in the L1/L2 cache. Therefore, the L1/L2 cache hit rates are important for the performance of the accumulation method.

Since BRMerge is a row-wise accumulation method, it only processes the intermediate lists for one result row in each iteration, which is usually small and comparable to the L1/L2 cache size. Moreover, after each round of merging, the required memory size of the intermediate lists is reduced due to the merging of duplicate indices. Therefore, the expected cache hit rate of the L1/L2 caches increases after each round of merging. Furthermore, since BRMerge has a typical streaming access pattern during the accumulating phase, the hardware pre-fetching mechanism further increases the L1/L2 cache hit rate. As a result, the L1/L2 cache hit rate of BRMerge during the accumulating phase is reasonably high.

In summary, the architectural benefits of the proposed BRMerge accumulation method are the streaming access patterns, minimized TLB misses, and reasonably high L1/L2 cache hit rates, which result in both low access latency and high bandwidth utilization when performing SpGEMM on the multi-core CPU architectures.

\subsection{BRMerge-Upper and BRMerge-Precise}\label{sec:spgemm}
To implement a high-performance SpGEMM algorithm, the allocation and the load balance methods are also important~\cite{yusuke, nsparse}. Since the upper-bound and precise allocation methods are commonly used and yields high performance~\cite{bhsparse, yusuke}, we use the two allocation methods and implements two SpGEGMM libraries named BRMerge-Upper and BRMerge-Precise. The two SpGEMM libraries are based on the BRMerge accumulation method.

The load balance of the two SpGEMM algorithms is the same as the previous work~\cite{yusuke}, which is reported to have high performance. First, the $n_{prod}$ of computing each output row is counted. Then the work is statically divided into $p$ groups in a per-row granularity, where $p$ is the number of CPU threads. The division policy is to keep the same total $n_{prod}$ in each group of work. Each group of work is computed by one CPU thread.

\subsubsection{BRMerge-Upper}\label{sec:upper}
Fig.~\ref{fig:spgemm}~(a) shows the computation steps of the BRMerge-Upper SpGEMM algorithm. Step~1 computes the \emph{row\_nprod} of each output row and performs prefix sum on the \emph{row\_nprod} for two purposes: 1) load balance (step~2) and 2) allocating the temporary memory space for the $C$ matrix (step~3). The temporary memory space for the $C$ matrix is denoted as $C\_bar$. Step~4 allocates the ping-pong buffers and computes the \emph{row\_size}, $col$, and $val$ arrays of the $C$ matrix by the BRMerge accumulation method. The \emph{row\_size} represents each row's number of nonzero elements. Step~5 performs prefix sum on the \emph{row\_size} to obtain the $rpt$ array and total $n_{nz}$ of the $C$ matrix at the same time. The $col$ and $val$ arrays of $C$ are allocated according to the total $n_{nz}$ of $C$. Step~6 copies the $C\_bar$ matrix to the standard $C$ matrix which conforms to the CSR storage format.

\begin{figure}[h]
\centering
\includegraphics[width=0.55\textwidth]{fig/spgemm.pdf}
\caption{Computation steps of BRMerge-Upper (Left) and BRMerge-Precise (Right)}
\label{fig:spgemm}
\end{figure}

Step~3 allocates the $C\_bar$ matrix parallelly, which means each CPU thread only allocates the part of $C\_bar$ for its private use. The load balance of step~4 and step~6 is based on the $n_{prod}$ information, which means each CPU thread computes approximately the same number of intermediate products. Whereas, the load balance of step~1 and step~5 is based on the number of rows of $C$, which means each CPU thread computes approximately the same number of rows. Step~3 is computed in a single CPU thread since its computation complexity is very low (i.e., $O(p)$, where $p$ is the number of CPU threads).

\subsubsection{BRMerge-Precise}\label{sec:precise}
Fig.~\ref{fig:spgemm}~(b) shows the computation steps of the BRMerge-Precise SpGEMM algorithm. Step~1 computes the \emph{row\_nprod} of each output row and performs prefix sum on the \emph{row\_nprod} for the load balance in step~2. Step~3 computes the \emph{row\_size} of the $C$ matrix by the hashing method described in the previous work~\cite{yusuke}. Step~4 performs the prefix sum on the computed \emph{row\_size} to obtain the $rpt$ array and the total $n_{nz}$ of $C$. Then the $col$ and $val$ arrays of $C$ are allocated according to the total $n_{nz}$ of $C$. Step~5 allocates the ping-pong buffers and computes the $col$ and $val$ arrays by the BRMerge accumulation method. Each computed row is directly written into the $col$ and $val$ arrays, which conform to the CSR storage format.

The load balance of step~3 and step~5 is based on the $n_{prod}$ information. Whereas the load balance of step~1 and step~4 is based on the number of rows of the $C$ matrix.

\section{Performance Evaluation}\label{sec:evaluate}
\subsection{Evaluation Setup}\label{sec:env}

We compare the proposed SpGEMM libraries BRMerge-Upper and BRMerge-Precise to Heap-SpGEMM~\cite{yusuke}, Hash-SpGEMM~\cite{yusuke}, Hashvec-SpGEMM~\cite{yusuke}, PB-SpGEMM~\cite{PB-SpGEMM}, and MKL-SpGEMM~\cite{mkl-doc}. The source code of our SpGEMM libraries and the other evaluated SpGEMM libraries are provided in the Supplementary Materials of this paper. The evaluation is based on the FLOPS performance of the matrix square benchmarks~\cite{bhsparse, nsparse, speck}, which is twice the number of the intermediate products divided by the execution time. The execution time is obtained by averaging the execution time over ten-times of program running after one warm-up. In all benchmarks, we measure the execution time of the approaches in double precision.

We evaluate all the approaches on two CPU servers with different CPUs. All the evaluated approaches are compiled and executed with the same hardware and software environment on each CPU server. The detailed information about the two CPU servers and the software environment is summarized in Table~\ref{tab:env}.

\begin{table}[h]
\caption{Detailed Information about the two CPU servers and the software environment.}
\fontsize{7.5}{12}\selectfont
\label{tab:env}
\centering
\begin{tabular}{|l|l|l|}
	\hline
	\textbf{Server name} & Platinum & Gold \\
	\hline
	CPU name & Intel Xeon Platinum 8163 & Intel Xeon Gold 6254 \\
	\hline

	\#CPUs & 2 & 4 \\
	\hline
	\#Cores/CPU & 24 & 18 \\
	\hline
	\#Threads/core & 2 & 1 \\
	\hline
	\#Threads & 96 & 72\\
	\hline
	\#Threads (\textbf{used}) &  \textbf{80} &  \textbf{58}\\
	\hline
	Clock frequency & 2.7GHz & 3.1GHz \\
	\hline
	L1-d cache & 32KiB/core & 32KiB/core \\
	\hline
	L1-i cache & 32KiB/core & 32KiB/core \\
	\hline
	L2 cache & 1024KiB/core & 1024KiB/core \\
	\hline
	L3 cache & 33MiB/CPU & 24.75MiB/CPU \\
	\hline
	Main memory type & DDR4  & DDR4\\
	\hline
	Main memory capacity & 755.1GiB & 4031.2GiB \\
	\hline
	Linux kernel & \multicolumn{2}{l|}{3.10.0} \\
	\hline
	Compiler & \multicolumn{2}{l|}{Intel icc (ICC) 2021.6.0 20220226} \\
	\hline
	Compiler option & \multicolumn{2}{l|}{-O2 -fopenmp -std=c++17} \\
	\hline
\end{tabular}

\end{table}

We select 26 square matrices from the SuiteSparse Matrix Collection~\cite{suitesparse} for the evaluation. The selected matrices are commonly used for the performance evaluation of SpGEMM on CPUs~\cite{yusuke, PB-SpGEMM} and GPUs~\cite{bhsparse, nsparse, speck}. The detailed information of these matrices is summarized in Table~\ref{tab:matrix}. Note that the 26 matrices in Table~\ref{tab:matrix} are sorted in an ascending order by their compression ratio.

\begin{table*}[ht]
\centering
\fontsize{7.5}{12}
\selectfont
\caption{Detailed information about the 26 sparse matrices.}
\label{tab:matrix}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
	\hline
	\textbf{Id} & \textbf{Name} & \textbf{Rows} & \textbf{Nnz} & \textbf{Nnz/row} & \textbf{Max nnz/row} & \textbf{Nprod of $A^2$} & \textbf{Nnz of $A^2$} & \textbf{Compression ratio of $A^2$} \\
	\hline
	1 & m133-b3 & \num{200200} & \num{800800} & 4.0 & \num{4} & \num{3203200} & \num{3182751} & 1.01 \\
	\hline
	2 & mac\_econ\_fwd500 & \num{206500} & \num{1273389} & 6.2 & \num{44} & \num{7556897} & \num{6704899} & 1.13 \\
	\hline
	3 & patents\_main & \num{240547} & \num{560943} & 2.3 & \num{206} & \num{2604790} & \num{2281308} & 1.14 \\
	\hline
	4 & webbase-1M & \num{1000005} & \num{3105536} & 3.1 & \num{4700} & \num{69524195} & \num{51111996} & 1.36 \\
	\hline
	5 & mc2depi & \num{525825} & \num{2100225} & 4.0 & \num{4} & \num{8391680} & \num{5245952} & 1.60 \\
	\hline
	6 & scircuit & \num{170998} & \num{958936} & 5.6 & \num{353} & \num{8676313} & \num{5222525} & 1.66 \\
	\hline
	7 & delaunay\_n24 & \num{16777216} & \num{100663202} & 6.0 & \num{26} & \num{633914372} & \num{347322258} & 1.83 \\
	\hline
	8 & mario002 & \num{389874} & \num{2101242} & 5.4 & \num{7} & \num{12829364} & \num{6449598} & 1.99 \\
	\hline
	9 & cage15 & \num{5154859} & \num{99199551} & 19.2 & \num{47} & \num{2078631615} & \num{929023247} & 2.24 \\
	\hline
	10 & cage12 & \num{130228} & \num{2032536} & 15.6 & \num{33} & \num{34610826} & \num{15231874} & 2.27 \\
	\hline
	11 & majorbasis & \num{160000} & \num{1750416} & 10.9 & \num{11} & \num{19178064} & \num{8243392} & 2.33 \\
	\hline
	12 & wb-edu & \num{9845725} & \num{57156537} & 5.8 & \num{3841} & \num{1559579990} & \num{630077764} & 2.48 \\
	\hline
	13 & offshore & \num{259789} & \num{4242673} & 16.3 & \num{31} & \num{71342515} & \num{23356245} & 3.05 \\
	\hline
	14 & 2cubes\_sphere & \num{101492} & \num{1647264} & 16.2 & \num{31} & \num{27450606} & \num{8974526} & 3.06 \\
	\hline
	15 & poisson3Da & \num{13514} & \num{352762} & 26.1 & \num{110} & \num{11768678} & \num{2957530} & 3.98 \\
	\hline
	16 & filter3D & \num{106437} & \num{2707179} & 25.4 & \num{112} & \num{85957185} & \num{20161619} & 4.26 \\
	\hline
	17 & cop20k\_A & \num{121192} & \num{2624331} & 21.7 & \num{81} & \num{79883385} & \num{18705069} & 4.27 \\
	\hline
	18 & mono\_500Hz & \num{169410} & \num{5036288} & 29.7 & \num{719} & \num{204030968} & \num{41377964} & 4.93 \\
	\hline
	19 & conf5\_4-8x8-05 & \num{49152} & \num{1916928} & 39.0 & \num{39} & \num{74760192} & \num{10911744} & 6.85 \\
	\hline
	20 & cant & \num{62451} & \num{4007383} & 64.2 & \num{78} & \num{269486473} & \num{17440029} & 15.45 \\
	\hline
	21 & hood & \num{220542} & \num{10768436} & 48.8 & \num{77} & \num{562028138} & \num{34242180} & 16.41 \\
	\hline
	22 & consph & \num{83334} & \num{6010480} & 72.1 & \num{81} & \num{463845030} & \num{26539736} & 17.48 \\
	\hline
	23 & shipsec1 & \num{140874} & \num{7813404} & 55.5 & \num{102} & \num{450639288} & \num{24086412} & 18.71 \\
	\hline
	24 & pwtk & \num{217918} & \num{11634424} & 53.4 & \num{180} & \num{626054402} & \num{32772236} & 19.10 \\
	\hline
	25 & rma10 & \num{46835} & \num{2374001} & 50.7 & \num{145} & \num{156480259} & \num{7900917} & 19.81 \\
	\hline
	26 & pdb1HYS & \num{36417} & \num{4344765} & 119.3 & \num{204} & \num{555322659} & \num{19594581} & 28.34 \\
	\hline

\end{tabular}
\end{table*}

The MKL-SpGEMM is implemented based on the C version of Intel's oneAPI Math Kernel Library (oneMKL)~\cite{mkl-doc}. The key routine in the MKL-SpGEMM is the \emph{mkl\_sparse\_spmm} routine. The parallelism in all the approaches including MKL-SpGEMM is based on the OpenMP framework~\cite{openmp}. Therefore, we use \emph{omp\_set\_num\_threads} to set the used number of threads ($\#threads$). All the evaluations are set with the same $\#threads$ on each CPU server~(Table~\ref{tab:env}).

During the performance evaluation, we observe that when the used $\#threads$ is set as the maximum $\#threads$ on a CPU server, which is 96 on the Platinum server and 72 on the Gold server, even the average execution time can vary widely in different runs. The possible reasons for this might be that the occasional operating system services may seize several CPU threads for execution and evict the executing SpGEMM tasks. Based on our experiments, finally, we set the $\#threads$ as 80 on the Platinum server and 58 on the Gold server for all the evaluations and observed a relatively stable average execution time among different runs.

Another performance note is that the \emph{scalable\_malloc} and \emph{scalable\_free} in Intel's oneAPI Thread Building Block (oneTBB)~\cite{tbb-doc} are used in the state-of-the-art SpGEMM libraries including Heap-SpGEMM, Hash-SpGEMM, Hashvec-SpGEMM, and PB-SpGEMM. And the \emph{scalable\_malloc} and \emph{scalable\_free} yield better performance for the parallel memory allocation and deallocation~\cite{yusuke}. Therefore, we also use the \emph{scalable\_malloc} and \emph{scalable\_free} in our SpGEMM implementations.

\subsection{Results and Discussions}\label{sec:result}

\subsubsection{Performance of SpGEMMM on the Platinum Server}

\begin{figure*}[ht]
\centering
\includegraphics[width=0.95\textwidth]{fig/perf-cs08.pdf}
\caption{Performance of SpGEMM on the Platinum server}
\label{fig:perf-cs08}
\includegraphics[width=0.95\textwidth]{fig/perf-cs11.pdf}
\caption{Performance of SpGEMM on the Gold server}
\label{fig:perf-cs11}
\end{figure*}

Fig.~\ref{fig:perf-cs08} shows the overall performance of the evaluated SpGEMM libraries on the Platinum server. The results show that the BRMerge-Precise outperforms other libraries on most benchmarks. The average performance speedups of BRMerge-Precise over BRMerge-Upper, Heap-SpGEMM, Hash-SpGEMM, Hashvec-SpGEMM, MKL-SpGEMM, and PB-SpGEMM are $1.05\times$, $2.29\times$, $1.42\times$, $1.56\times$, $1.68\times$, $8.46\times$, respectively. The best-performing and the second best-performing SpGEMM algorithms are the proposed BRMerge-Precise and BRMerge-Upper. Moreover, the BRMerge-Precise achieves an average $1.42\times$ performance speedup over the state-of-the-art best-performing SpGEMM library, which is the Hash-SpGEMM.

The PB-SpGEMM suffers exceptionally lower performance on \textbf{all} the tested benchmarks, including benchmarks with low compression ratios. We attribute the exceptionally low performance of PB-SpGEMM to the outer-product dataflow, which suffers worse memory access efficiency to the $\hat{C}$ matrix compared to the memory access efficiency to the $B$ matrix for the row-wise dataflow. The previous work also reported that the outer-product dataflow has significant drawbacks compared to the row-wise dataflow~\cite{matraptor, gamma}.

We compare the performance of the BRMerge-Upper with the Heap-SpGEMM since the two SpGEMM libraries implement the same allocation and load balance methods. For the benchmarks with low compression ratios and low $n_{nz}$ per row of the $A$ matrix (i.e., the matrix with ids ranges from 1 to 8, except for the webbase-1M matrix), the performance of the Heap-SpGEMM is comparable to the proposed BRMerge-Upper SpGEMM library. The reason is that the two main drawbacks of the Heap-SpGEMM are avoided in these benchmarks, which are 1) the high computation complexity on benchmarks with high compression ratios~\cite{yusuke}, and 2) the high TLB cache misses due to accessing the rows of the $B$ matrix in a longer duration (Section~\ref{sec:memory}). For other evaluated benchmarks, BRMerge-Upper significantly outperforms Heap-SpGEMM despite the two SpGEMM libraries implementing the same allocation and load balance methods. For the evaluated benchmarks with the matrix ids above 20, BRMerge-Upper even achieves more than $3\times$ better peroformance than the Heap-SpGEMM. We attribute this large performance gap to the architectural benefits of the proposed accumulation method and the high TLB cache misses of the Heap-SpGEMM (Section~\ref{sec:memory}).

We compare the performance of the BRMerge-Precise with the Hash-SpGEMM since the two SpGEMM libraries implement the same allocation and load balance methods. Moreover, the symbolic phase of the BRMerge-Precise also uses the hash-based method to count the $n_{nz}$ per row~\cite{yusuke} which is the same as the Hash-SpGEMM. The Hash-SpGEMM is reported to have better performance than other SpGEMM libraries on benchmarks with high compression ratios~\cite{yusuke}. The reason is that, when combined with the precise allocation method, the $n_{nz}$ per output row is known before the numeric phase; therefore, the accessed hash table size can be as small as the $n_{nz}$ per output row.

Nevertheless, for benchmarks with relatively high compression ratios (i.e., the matrices with ids ranging from 15 to 26), the BRMerge-Precise outperforms the Hash-SpGEMM. We attribute this to three reasons. First, the BRMerge accumulation method has streaming memory access patterns, which can achieve maximized bandwidth utilization and increase the L1 cache hit rate by taking advantage of the hardware pre-fetching mechanism. Whereas, the hash-based accumulation method does not have the architectural benefits mentioned above. Second, the BRMerge accumulation method has reasonably high L1/L2 cache hit rate as a row-wise accumulation method~\ref{sec:memory}. Third, the result list in the BRMerge accumulation method is natively sorted. However, the hash-based accumulation method has to perform the additional extracting and sorting operations after the hashing operation.

As for the benchmarks with low compression ratios, the BRMerge-Precise significantly outperforms the Hash-SpGEMM. Note that in our evaluation, the Hashvec-SpGEMM seldom outperforms the Hash-SpGEMM in the evaluated benchmarks.

\subsubsection{Performance of SpGEMMM on the Gold Server}
Fig.~\ref{fig:perf-cs11} shows the overall performance of the evaluated SpGEMM libraries on the Gold server. The BRMerge-Precise achieves on average $1.29\times$, $2.62\times$, $1.39\times$, $1.50\times$, $1.73\times$, and $9.37\times$ performance speedup over the BRMerge-Upper, Heap-SpGEMM, Hash-SpGEMM, Hashvec-SpGEMM, MKL-SpGEMM, and PB-SpGEMM, respectively. The best-performing and second-best-performing SpGEMM libraries are still the proposed BRMerge-Precise and BRMerge-Upper. Moreover, the state-of-the-art best-performing SpGEMM library is the Hash-SpGEMM.

The performance trends of all the evaluated SpGEMM libraries except for the MKL-SpGEMM on the Gold server are similar to the performance trends on the Platinum Server. The MKL-SpGEMM outperforms the proposed BRMerge-Precise on benchmarks with high compression ratios (i.e., the matrices with ids ranging from 18 to 26). In contrast, the proposed BRMerge-Precise outperforms the MKL-SpGEMM on benchmarks with compression ratios lower than 4. Note that on the highly irregular webbase-1M benchmark, the BRMerge-Precise significantly outperforms the MKL-SpGEMM on both the Platinum server and the Gold server. On average, the BRMerge-Precise outperforms the MKL-SpGEMM by $1.73\times$.

\section{Conclusion}\label{sec:conclude}
In this paper, we present a novel binary-row-merging-based accumulation method, named BRMerge, to optimize the performance of SpGEMM on multi-core CPU architectures. We also analyze the architectural benefits of the proposed accumulation method, which are the streaming access patterns, minimized TLB cache misses, and reasonably high L1/L2 cache hit rates. As a result, the proposed accumulation method can achieve both low access latency and high bandwidth utilization when performing SpGEMM on multi-core CPU architectures. Based on the proposed accumulation method, we further propose two parallel SpGEMM libraries named BRMerge-Precise and BRMerge-Upper based on different allocation methods. The proposed SpGEMM libraries significantly outperform the state-of-the-art SpGEMM libraries on 26 commonly used matrices with two CPU servers.  Specifically, our best-performing SpGEMM library (i.e., BRMerge-Precise) achieves on average $1.42\times$ and $1.39\times$ performance speedups compared to the state-of-the-art best-performing SpGEMM library (i.e., Hash-SpGEMM~\cite{yusuke}) on the Intel Xeon Platinum 8163 CPU and the Intel Xeon Gold 6254 CPU, respectively.

\bibliographystyle{unsrt}

redpajama_set_name: RedPajamaArXiv


text:
<!DOCTYPE html>
<html>
<head lang="en">
    <meta charset="UTF-8">
    <title></title>
</head>
<body>
<script src="MostFrequentNumber.js"></script>
</body>
</html>

redpajama_set_name: RedPajamaGithub


Gorgeous rock urns are perfect memorials for your beloved pet's ashes. Made from a mixture of granite rock powder and premium resin, these premium rock urns feel and look life actual stone. Rock urns are perfect for outdoor or indoor use.