Mediheal is known for their effective sheet masks, which are developed by a team of expert dermatologists. They are at the forefront of the home-use concept, creating products that give beautiful skin without the need for surgical treatment. With over 10 million pieces of mask sheets sold, Mediheal continues to lead the charge in formulating innovative skincare products that are simple yet delivers amazing results.


text:
Spiker, Drexel agree to multiyear extension
Zach Spiker gets new deal after 4-win improvement
Drexel has agreed to a multiyear contract extension with coach Zach Spiker, the school announced Thursday.
Terms of the new deal were not disclosed.
Spiker said in a statement that he was "humbled and grateful" by the commitment from the school.
He is 22-43 in two seasons as the Dragons' coach, but Drexel improved to 13-20 last season, posting four more victories than it did in Spiker's first season and its most since 2013-14.
"Zach has been tireless in his efforts leading our program over the last two seasons," Drexel deputy athletic director Nick Gannon said in a statement. "The hard work and energy that Zach has put in over the last two years has established the foundation for long-term success. His championship vision for Drexel is contagious and we are excited to see the fruits of his labor in the years to come."
Prior to taking the Drexel job, Spiker spent seven seasons as the coach at Army, with a 102-112 record.

redpajama_set_name: RedPajamaCommonCrawl


A full list of my races on DVD is available here.
What's the quality like for your 1982 CART races?
If theuy are the same as mine, phoenix is the poor one of the 3 the other 2 are pretty good.
Id be interested in these if i have something of interest to you Mark?
I'm still looking for a new tool to play DVDs (AVS DVD gives region code errors which I never had before). From what I have checked is that they are acceptable for a race considering its age, but then again I have only loaded the dvd to write down the running time, so i'm not completely sure.
Yeah sure, but I have to finish off some other trades first, and i'm kinda busy with school and other stuff at the moment. I guess I have time in 2 or 3 weeks or so.
I have decided to return to an excel sheet, in a more detailed way.
I have made a large update to my List.
I have reworked my website (and my lists) to make them better readable.


888 Media is always seeking quality people with the drive and passion for digital media. Now, more than ever, we need talented people to join the team. 888 Media is looking for the best and brightest engineering talent to innovate, craft and refine our cutting edge online advertising platform, we also have positions available in Sales, Finance and Marketing. Vacancies are available in a number of countries namely; Australia, Canada, Czech Republic, India and the United Kingdom.
If you are interested in joining our team please contact us today.


text:
'Continental Divide': A Q&A with author Alex Myers
Alex Myers (Courtesy Better Wyoming)
March 10, 2020 by Katie Klingsporn 1 Comment
One summer day in the '90s, a newly out transgender Harvard student named Ron Bancroft hops off a Greyhound bus in Cody. Ron's experience in the West is limited to the tales of fur trappers and explorers he read obsessively when he was growing up as a girl on the East Coast, but it seems a good place to find what he's looking for:
"A new start, a rugged life, a sense of the physical and real rather than the theoretical."
So begins "Continental Divide," a new young-adult novel by author Alex Myers. The book, a page-turning coming-of-age tale packed with bruising life lessons, romance, self discovery and uniquely Wyoming scenes, is informed by Myers' own experience.
In the summer of 1995, Myers, who was the first openly transgender student at Harvard, came to Wyoming, where he found a job on a dude ranch. This was well before the transgender rights movement had much of a profile in America. It was a heady time in Myers' life, and he was seeking to escape some of the issues that came with being transgender in an era when it was still an unfamiliar concept.
"This word, as much as it freed me, it also confined me," Myers said. "I had this real impulse to go somewhere where being transgender was not a thing. Wyoming was one of those places."
The cover of Alex Myers' new novel, "Continental Divide." (Courtesy Better Wyoming)
His transformative experience became the inspiration for "Continental Divide." In the book, protagonist Bancroft tries on his new identity as a man while working on a dude ranch and later for the U.S. Forest Service. He explores Wyoming's rugged mountains, rides horses, meets a girl and tries to blend in at a ranch staffed by hypermasculine wranglers — all while hiding the fact that he grew up as a girl named Veronica.
That major omission threads the story with tension. Myers' apt descriptions of Wyoming landscapes and people, meanwhile, imbue it with the genuine knowledge only accrued through experience. And as Ron stumbles, corrects himself and learns how to inhabit the identity he has long yearned for, it's hard not to become invested in his journey. Figuring yourself out, after all, is a universal experience.
Myers kicks off a Wyoming book tour today in Laramie. The tour, which is being organized by Better Wyoming and Wyoming Equality, will also entail meetings with Gay-Straight Alliance chapters at schools along the way.
Myers sat down with WyoFile to talk about Wyoming masculinity, the novel and the symbolism of the Continental Divide.
WyoFile: Tell me about your own experience traveling to Wyoming in the summer of 1995.
Alex Myers: I think there's a lot of contact points with the novel. I had just finished my freshman year in Harvard and I very much wanted an experience that was not the East Coast, where I was born and raised. I got a job as a cook at a dude ranch and headed out. I was just excited to try a different part of the country and to see more or less what I would make of it and what it would make of me.
I worked for a couple weeks at that dude ranch. Then a friend of mine sent me a postcard there, and the back of the postcard asked, "How's your summer going? I hope it's not a problem being transgender in Wyoming." My boss read that and fired me on the spot.
So that was sort of a wake-up call. I had that moment, saying, do I want to just head back to the East Coast, or do I want to stay? But within two weeks of being there I was totally enchanted. I had gone for a couple hikes, I just loved the terrain — it was like nothing I had ever seen before. So I called around and looked around for a job in Cody and I got a job with the Forest Service with a wonderful ranger named Mick.
By a total fluke, the station that I worked at, it was all guys, there were no women, so it was this super saturated masculine environment. But I had just an amazing time.
WF: Why was Wyoming a compelling place to explore the concept of masculinity?
AM: Unlike Ron, I actually did some advance work. I applied to ranches in Utah, Arizona, Montana, Wyoming, and truly I was looking to get far off of the East Coast. But I have to say, when I got to Cody, it hit sort of what I had imagined the American West to be, and a somewhat dated version of it. Ranches and cowboys and that sort of thing. But I found a lot of people who live a lot of their lives outdoors and had a sort of ruggedness to them that I associate with American masculinity of the West. And it still felt largely like a frontier compared to Boston. I grew up in a rural area, an apple orchard in Maine, so I know rural life. But that is different than the sort of wide-open spaces and the distance between people that I found out West.
WF: At what point did you recognize that this experience was the inspiration for a novel?
AM: I started thinking about writing about it pretty soon after I came back. Because Matthew Shepard was murdered not long after I had returned. That was sort of the wake-up call to me that I had done something that was exciting and that meant a lot to me, but that was also risky. I'm not a person who takes a ton of uncalculated risks. And I wondered at that moment why I had been willing to take such a risk, and why it was worth it.
So I wanted to answer that question, and to me that's where writing comes from, especially fiction writing: having a question that you want to answer.
WF: What was the answer to that question?
AM: I think in part being young, but I really think — to me, and to cisgendered people as well — gender is a really fundamental core part of who we are and how we organize ourselves in the world and how the world puts us in a place it views as appropriate for our gender. I had a really serious case of self-doubt and desire or need even to prove that I was a man on my own terms. I think what going really far from home to a place that didn't know transgender, and that didn't know me as anything except how I presented myself, gave me the confidence to say, 'Yep I can do this as who I am.' I think that need to prove myself was so compelling that it kind of overrode any concerns for what could be the worst possible outcome.
WF: To me, there are two symbols that stand out. One is the Continental Divide, the other is the jackalope. Can you talk about what interested you about those symbols and what they stand for in the novel?
AM: Ron is obsessed with maps and he's obsessed with this question of where is he and where does he belong, existentially as well as physically. So I wanted to look at maps as a way that he's trying to orientate himself until he realizes maps have limitations. As a feature, the Continental Divide is mysterious. It's definitely real, but you can't necessarily see it. But maps will still put it on there. To me, that's kind of how gender kind of works as well. We see gender, we experience gender, we put words to gender. But can you tell me what that really means?
Jackalopes seem to be this other mystical thing. They don't really exist, in flesh, but they do really exist in this bone form. And they are kind of this thing that's real and unreal, again like the Continental Divide.
WF: Were you writing for a particular readership, and what do you hope readers take from the book?
AM: I do a lot of talking at schools and schools are always asking me, "What are good books? What should we have on our library shelves?" And I can rattle off a bunch of really good nonfiction books about trans and nonbinary characters. I cannot name that many, especially YA, fiction books. It doesn't exist, and what does exist is not that good. And it's also not written by trans authors. The audience I was writing for is people who want that. That is going to be queer people, but it's going to be teachers, it's going to be parents, it's going to be allies. It's people who want to understand not just "what does being transgender mean," which is a lot of what the non-fiction addresses, sort of the practicalities of coming out and getting support. To me the more interesting question of: How does a transgender person live their life in America? It doesn't stop when you come out. That is where the story begins.
Support arts and culture coverage with a donation today.
This interview was edited for length and clarity.
Week 44: The pandemic in Wyoming from Jan. 9-15
Filed Under: Art, Featured, People
About Katie Klingsporn
Katie Klingsporn is WyoFile's managing editor. She is a journalist and word geek who has been writing about life in the West for 15 years. Her pieces have appeared in Adventure Journal, National Geographic Adventure and Patagonia's Cleanest Line blog.
Don Thomas says
Hi Dan Neal Thanks for publishing this article. I have recently moved out of my hometown of Casper. . Anxious to read the story.
City: Yucca Valley
State of Residence: California


text:
View Lab Photos
Barts Cancer Institute
10.0/10 Overall Rating 100% Positive
2 Orders Completed
About Barts Cancer Institute
Divison of: Queen Mary University of London
BCI was created in 2003, and brought together some of the most eminent cancer research teams in London. We are one of the six Institutes within the Barts and The London School of Medicine and Dentistry, Queen Mary University of London.
Under the leadership of the Director, Professor Nick Lemoine, and supported by an Executive Board of senior investigators, we have created an academic environment fitting for an internationally recognised, comprehensive cancer centre.
Barts Cancer Institute has a long tradition of pioneering research and innovation. Many surgical procedures were first discovered here as well as the use of radiotherapy to treat cancer.
By investing in and undertaking world-class research, we aim to prevent cancer and develop better diagnostic techniques and treatments. This will ultimately increase patient survival and enhance the quality of life for those with long-term disease.
A wide range of research is currently underway and includes:
Therapeutic and diagnostic target identification and validation in both haematological and solid malignancies
Clinical trials exploring new therapies
The development of novel molecular approaches for diagnosis
Classification and treatment of human cancers
Investigations into the regulation of tumour spread and host anti-tumour responses
We are recognised as one of the leading Cancer Research Institutions in the UK and are a Cancer Research UK Centre of Excellence - one of only 17 in the country.
Our Services (12)
Targeted Metabolomics
Request a quote for more information about this service.
Non-Targeted Metabolomics
The MS facility can provide access to advanced MS, proteomics and metabolomics technologies.
A series of analytical techniques that enable:
Protein identification
Determination of protein post-translational modifications such as: phosphorylation, acetylation, and more
Relative quantification of proteins and their modifications
Absolute protein quantification in targeted mode
Molecular weight determination of intact proteins (top-down proteomics)
The mass spectrometry (MS) laboratory is a Core Service at Barts Cancer Institute and is currently managed by the Centre for Haemato-Oncology. MS has a variety of applications in cell and molecular biology and cancer research, including the analysis of:
Pharmacokinetics and pharmacodynamics of drugs
Samples' protein composition
Protein structures
Post-translational modifications of proteins
Enzymatic activity
MS is also the preferred analytical method in metabolomics and proteomics.
The MS Laboratory supports BCI researchers through collaborations in a variety of high quality m
Identify barriers preventing data being used to their maximal potential and develop bioinformatics methods to address these.
Work closely with basic and clinical scientists to provide the bioinformatics backbone to support cutting edge research and translate data into diagnostics and therapeutics.
Biostatistics & Bioinformatics
Computational Modeling
Get info on this provider's capabilities without requesting a quote.
David Britton
Pedro Cutillas
Positive review for Mass Spectrometry:
"We had a very good experience with Dr. Pedro Cutillas and team at Barts Cancer Institute, they followed our tight timelines and provided very robust data, we recommend this group for phosphoproteomics studies."
Barts Cancer Institute has not received any endorsements.


text:
\section{Background}
\label{sec:2_background}

We explain the pertinent details of HE with an emphasis on CKKS.
We adopt notations and terminologies from \cite{2021-arxiv-does, rsa-2020-better}.
Parameters and notations are summarized in Table \ref{table:notation}.

\subsection{Homomorphic encryption (HE)}
\label{sec:2_1_he}
HE is an encryption method that allows computation on encrypted data.
It is based on a computational problem called ring learning with errors (RLWE), which is known to be difficult to solve even for quantum computers~\cite{iacr-2019-standard}.
There are a variety of HE schemes that support different data types, such as integers~\cite{toct-2014-bgv, siam-2014-bfv1, iacr-2012-bfv2}, boolean~\cite{jc-2020-tfhe, 2015-eurocrypt-fhew}, and fixed-point complex (real) numbers~\cite{asia-2017-ckks}.
In particular, CKKS~\cite{asia-2017-ckks}, our primary target HE scheme, supports encryption of fixed-point complex  vectors, making it suitable for numerous real-world applications such as machine learning.
HE schemes use a fixed set of computations called \emph{HE ops} to process encrypted data without decryption.

In CKKS, a \emph{message} is an unencrypted vector of length $n$.
Each entry of a message vector is called a \emph{slot}, which can contain a complex number.
Encryption of a message consists of two steps.
First, we transform a message $\textbf{m}$ into a polynomial $\mathtt{P}_{\textbf{m}}$ in a cyclotomic polynomial ring $\mathrm{R}_Q\!=\!\mathbb{Z}_Q[X] / (X^N\! +\! 1)$, which is a degree-$(N\!-\!1)$ polynomial with coefficients in $ \mathbb{Z}_Q\! =\! \{0, 1, \cdots, Q\! -\! 1\} $, the set of integers modulo $Q$; we treat a polynomial as a vector in $ \mathbb{Z}_Q^N$ where each entry is a coefficient.
The maximum number of slots included in a single message $\textbf{m}$ is determined by $N$ as $n \leq \sfrac{N}{2}$.
We perform this transformation by computing the Inverse Discrete Fourier Transform (IDFT) of $\textbf{m}$, multiplying a scale factor $\Delta$, (typically ranging from $2^{40}$ to $2^{60}$)
and rounding the result:
\begin{equation} \label{eq:encode}
        \mathtt{P}_\mathbf{m} \simeq \Delta \cdot \text{IDFT}(\mathbf{m})
\end{equation}

$\mathtt{P}_{\textbf{m}}$ is encrypted into a \emph{ciphertext} $[\![ \mathbf{m} ]\!]= ( \mathtt{B}_{\textbf{m}}, \mathtt{A}_{\textbf{m}} )$ based on Eq.~\ref{eq:encryption}. $\mathtt{A}_{\textbf{m}}$ is a randomly sampled polynomial, $\mathtt{S}$ is a secret key polynomial, and $ \mathtt{E} $ is a small random error polynomial.
\begin{equation} \label{eq:encryption}
    [\![ \mathbf{m} ]\!]= ( \mathtt{B}_{\textbf{m}}, \mathtt{A}_{\textbf{m}} ),\ \ \ \ \mathtt{B}_\mathbf{m} =  \mathtt{A}_\mathbf{m} * \mathtt{S} + \mathtt{P}_\mathbf{m} + \mathtt{E}
\end{equation}

Decryption is the inverse of the encryption process.
We recover the polynomial $\mathtt{P}_\mathbf{m}$ using the secret key $\mathtt{S}$ and apply DFT as shown in the following equation.
\begin{equation} \label{eq:decode}
    \mathbf{m} \simeq  {\Delta}^{-1} \cdot \text{DFT}(\mathtt{P}_\mathbf{m}+\mathtt{E})={\Delta}^{-1} \cdot \text{DFT}( \mathtt{B}_{\textbf{m}}-\mathtt{A}_{\textbf{m}}*\mathtt{S} )
\end{equation}
Due to the random error polynomial $\mathtt{E}$ included in the encryption process, the decrypted result is not identical to $\textbf{m}$.
However, we can consider the decrypted result as $\textbf{m}$ because $\mathtt{E}$ is a polynomial with small coefficients.

\begin{table}[tb!]
\renewcommand{\arraystretch}{1.2}
  \centering
  \caption{CKKS parameters and notations. \label{table:notation}}
  \begin{tabularx}{\columnwidth}{lX}
    \toprule
    \textbf{Params} & \textbf{Description}\\
    \midrule
    $N$ & Degree. Length of a plaintext polynomial. \\
    $n$ & Length of a message. $ n \leq N / 2 $.\\
    $Q$  & Polynomial modulus.\\
    $P$ & Special modulus.\\
    $\mathcal{B}$ & $\{ p_0, p_1, \cdots, p_{\alpha-1}\} $, a set of prime limbs of $P$. $ P = \prod_{i=0}^{\alpha-1} p_i $.\\
    $\mathcal{C}$ & $\{ q_0, q_1, \cdots, q_L\}$, a set of prime limbs of $Q$. $ Q = \prod_{i=0}^{L} q_i $.\\
    $\mathcal{C}_i$ & Partial limb group of $\mathcal{C}$. $ \mathcal{C}_i = \{q_{\alpha i}, q_{\alpha i + 1}, \cdots, q_{\alpha (i + 1) - 1}\}$, $i = 0, 1, \cdots, \mathtt{dnum}\! -\! 1 $.\\
    $\mathcal{D}$ & $\mathcal{B} \cup \mathcal{C}$, a set of prime limbs of $PQ$.\\
    $L$ & Maximum multiplicative level.
    \\
    $\ell$ & (Current) multiplicative level. A level-$\ell$ polynomial has $ \ell\! +\! 1 $ limbs. $ 0\! \leq\! \ell\! \leq\! L $.\\
    $\mathtt{dnum}$ & Decomposition number.\\
    $\alpha$ & $ (L + 1) / \mathtt{dnum} $. \# of prime limbs in $\mathcal{C}_i$ and $\mathcal{B}$. \\
    $ \Delta $ & Scale multiplied\\
    $ \mathbf{m} $ & Message, a vector of $n$ slots.\\
    $\mathtt{P}, \mathtt{P}_\mathbf{m}$ & Polynomial. Subscript $ \mathbf{m} $ emphasizes that it corresponds to a message $ \mathbf{m} $.

    \\
    $[\![ \mathbf{m} ]\!]$ &
     Ciphertext encrypting a message $ \mathbf{m} $.\\
    $[\mathtt{P}]_{q_i}$ & $q_i$-limb of $\mathtt{P}$.\\
    $[\mathtt{P}]_{\mathcal{C}}$ &
    $\{ [\mathtt{P}]_{q_i} : q_i \in {\mathcal{C}} \}$,
    the set of $q_i$-limbs of $\mathtt{P}$ for $q_i \in {\mathcal{C}}$.\\
    $\textbf{evk}$
 & Evaluation key. An $\textbf{evk}$ for HMult ($\textbf{evk}_{\text{mult}}$) and $\textbf{evk}$s for HRots with different rotation amount $r$'s ($\textbf{evk}_{\text{rot}}^{(r)}$'s) exist.\\
    \bottomrule
   \end{tabularx}
  \vspace{-0.10in}
\end{table}

\begin{table*}[tb!]
\renewcommand{\arraystretch}{1.3}
  \centering
  \caption{Primitive HE ops of CKKS.\label{table:operations}}
  \begin{tabularx}{\textwidth}{l>{\raggedright\arraybackslash}Xp{3.2in}}
    \toprule
    \textbf{Operation} & \textbf{Result} & \textbf{Description}\\
    \midrule
    $\text{CAdd}([\![\mathbf{m}]\!],c)$ & $[\![\mathbf{m}+\mathbf{c}]\!]=(\mathtt{B}_{\textbf{m}}+\textbf{c},\mathtt{A}_{\textbf{m}})$ & Add a scalar $c$ to a ciphertext. $\textbf{c}$ is a length-$N$ vector with every entry $c$.\\
    $\text{CMult}([\![\mathbf{m}]\!],c)$ & $[\![\mathbf{m} \cdot \mathbf{c}]\!]=(\mathtt{B}_{\textbf{m}} \cdot \textbf{c},\mathtt{A}_{\textbf{m}} \cdot \textbf{c})$ & Multiply a scalar to a ciphertext.\\
    $\text{PAdd}([\![\mathbf{m}]\!], \mathtt{P}_{\textbf{m}^\prime})$ & $[\![\mathbf{m}+\mathbf{m}^\prime]\!] = (\mathtt{B}_{\textbf{m}} + \mathtt{P}_{\textbf{m}^\prime},\mathtt{A}_{\textbf{m}})$ & Add an unencrypted polynomial to a ciphertext.\\
    $\text{PMult}([\![\mathbf{m}]\!], \mathtt{P}_{\textbf{m}^\prime})$ & $[\![\mathbf{m} \cdot \mathbf{m}^\prime]\!] = (\mathtt{B}_{\mathbf{m}} * \mathtt{P}_{\mathbf{m}^\prime},\mathtt{A}_{\mathbf{m}} * \mathtt{P}_{\mathbf{m}^\prime})$ & Multiply an unencrypted polynomial to a ciphertext.\\
    $\text{HAdd}([\![\mathbf{m}]\!],[\![\mathbf{m}^\prime]\!])$ & $[\![\mathbf{m}+\mathbf{m}^\prime]\!] = (\mathtt{B}_{\mathbf{m}} + \mathtt{B}_{\mathbf{m}^\prime},\mathtt{A}_{\mathbf{m}} + \mathtt{A}_{\mathbf{m}^\prime})$ & Add two ciphertexts.\\
    $\text{HMult}([\![\mathbf{m}]\!], [\![\mathbf{m}^\prime]\!], \mathbf{evk}_{\text{mult}})$ & $[\![\mathbf{m} \cdot \mathbf{m}^\prime]\!] = \text{KeySwitch}(\mathtt{A}_{\mathbf{m}} * \mathtt{A}_{\mathbf{m}^\prime}, \mathbf{evk}_{\text{mult}}) + (\mathtt{B}_{\mathbf{m}} * \mathtt{B}_{\mathbf{m}^\prime}, \mathtt{A}_{\mathbf{m}} * \mathtt{B}_{\mathbf{m}^\prime} + \mathtt{A}_{\mathbf{m}^\prime} * \mathtt{B}_{\mathbf{m}})$ & Multiply two ciphertexts.\\
    $\text{HRot}([\![\mathbf{m}]\!],r, \mathbf{evk}_{\text{rot}}^{(r)})$ & $[\![\mathbf{m} \ll r]\!] = \text{KeySwitch}(\psi_r(\mathtt{A}_{\mathbf{m}}), \mathbf{evk}_{\text{rot}}^{(r)}) + (\psi_r(\mathtt{B}_{\mathbf{m}}), \mathbf{0})$ & Perform a circular left shift by $r$ slots. $\psi_r$ is an automorphism performed on polynomials.\\
    $\text{HRescale}([\![\mathbf{m}]\!])$ & $[\![\Delta^{-1} \cdot \mathbf{m}]\!] = (\Delta^{-1}\mathtt{B}_{\mathbf{m}}, \Delta^{-1}\mathtt{A}_{\mathbf{m}})$ & Restore the scale of a ciphertext having scale $\Delta^2$ back to $\Delta$ \\
    \bottomrule
  \end{tabularx}
  \vspace{-0.1in}
\end{table*}

\subsection{Computational optimizations for HE ops}
\label{sec:2_2_ntt_and_rns}

HE ops are expensive.
A polynomial in $\mathrm{R}_Q$ is a long ($N=2^{16}$) vector of large integers ($Q > 2^{1000}$).
In general, two techniques are used to mitigate the cost of HE ops.

\noindent\textbf{Number-Theoretic Transform (NTT):} NTT is used to lower the complexity of a polynomial multiplication (mult).
Multiplying two polynomials in $\mathrm{R}_Q$ is
a (negacyclic) convolution of two length-$N$ vectors and has an $\mathcal{O}(N^2)$ complexity.
NTT is a variant of DFT, which converts the convolution into an element-wise mult, reducing the complexity to $\mathcal{O}(N)$:

\begin{equation*} \label{eq:ntt}
    \text{NTT}(\mathtt{P}_1 * \mathtt{P}_2) = \text{NTT}(\mathtt{P}_1) \cdot \text{NTT}(\mathtt{P}_2)
\end{equation*}
Fast Fourier Transform (FFT) algorithms~\cite{cooley-tukey} can be applied to compute NTT, which has $\mathcal{O}(N \log N)$ complexity.
Still, NTT is an expensive function, so we keep polynomials in their NTT-applied versions without applying the Inverse NTT (INTT).
We refer to this NTT-applied version of a polynomial as the \emph{evaluation representation} of the polynomial, and we assume every polynomial is in its evaluation representation unless otherwise stated.
However, as there are functions that cannot be performed when a polynomial is in its evaluation representation, we often perform INTT to bring back the polynomial to the original representation, which we refer to as the \emph{coefficient representation}.

\noindent\textbf{Residue Number System (RNS) and limbs:} $\mathtt{P}_{\mathbf{m}}$ has coefficients in $ \mathbb{Z}_Q $ having up to 1,000s of bits.
RNS is widely used to reduce the high complexity of performing multi-precision arithmetic between large integer coefficients~\cite{sac-2018-frns-ckks, sac-2016-behz}.
We first choose $(L+1)$ word-sized prime numbers to form an ordered set $\mathcal{C}=\{q_0, \cdots , q_L\}$ such that $ Q=\prod_{i=0}^{L}q_i $; we refer to each $q_i$ as a \emph{prime limb} of $Q$.
Based on the Chinese Remainder Theorem (CRT), any coefficient $c \in \mathbb{Z}_Q $ can be uniquely represented with a vector $(c\! \text{ mod }\! q_0, \cdots, c\! \text{ mod }\! q_L)$, allowing a single polynomial $\mathtt{P}_{\mathbf{m}} \in \mathrm{R}_Q$ to be decomposed into $(L+1)$ polynomials in $ \{ \mathrm{R}_{q_i}\}_{0 \leq i \leq L } $.
Under such representation, we treat $\mathtt{P}_{\mathbf{m}}$ as a matrix of size $(L+1)\times N$ words and denote it as $[\mathtt{P}_{\mathbf{m}}]_\mathcal{C}$.
We refer to each row of the matrix as the polynomial's \emph{limb}.
The $i$-th row corresponding to the modulus $q_i$ is specifically called the polynomial's $q_i$\emph{-limb} and is denoted as $[\mathtt{P}]_{q_i}$.
Using RNS and NTT, a polynomial op becomes element-wise ops between $(L+1)\times N$ matrices that only involve word-sized arithmetic.

Base conversion (BConv)~\cite{sac-2016-behz} is used to change the prime limb set of a polynomial.
For example, $[\mathtt{P}_{\textbf{m}}]_{\mathcal{B}}$ for a prime limb set $\mathcal{B} =\{p_0, p_1, \cdots, p_{\alpha-1}\}$ can be converted into $[\mathtt{P}_{\textbf{m}}]_{\mathcal{C}}$ by the following equation, where
$\hat{p}_j$ denotes $\prod_{i \neq j} p_i, p_i \in \mathcal{B}$.

\begin{equation} \label{eq:bconv}
\begin{split}
    [\mathtt{P}_{\textbf{m}}]_{\mathcal{C}} &= \underset{\mathcal{B} \rightarrow \mathcal{C}}{\text{BConv}}([\mathtt{P}_{\mathbf{m}}]_{\mathcal{B}}) \\
    &=
    \bigg\{ \sum_{j=0}^{\alpha\!-\!1}\! \big( [\mathtt{P}_\mathbf{m}]_{p_j} \! \cdot \hat{p}_j^{-1}\! \text{ mod }\!p_j\big) \cdot \hat{p}_j\! \text{ mod }\! {q_i}\bigg\}_{0 \leq i \leq L}
\end{split}
\end{equation}
The polynomial must be in its coefficient representation to perform BConv.
Thus, it is common in CKKS to perform a series of INTT $\rightarrow$ BConv $\rightarrow$ NTT, which we refer to as a \emph{BConvRoutine} (Alg.~\ref{alg:bconvroutine}).

\begin{figure}[tb!]
\begin{algorithm}[H]
\caption{BConvRoutine}\label{alg:bconvroutine}
\algnewcommand\algorithmicnotation{\textbf{Notation:}}
\algnewcommand\Notation{\item[\algorithmicnotation]}
\begin{algorithmic}[1]
\Notation{$\mathtt{P}_\mathtt{coeff}$ = coefficient representation of $\mathtt{P}$}
\Procedure{BConvRoutine}{$[\mathtt{P}]_\mathcal{B}, \mathcal{C}$}
    \State $[\mathtt{P}_\mathtt{coeff}]_\mathcal{B} \leftarrow \text{INTT}([\mathtt{P}]_\mathcal{B})$
    \State $[\mathtt{P}_\mathtt{coeff}]_\mathcal{C} \leftarrow \underset{\mathcal{B} \to \mathcal{C}}{\textrm{BConv}}([\mathtt{P}_\mathtt{coeff}]_\mathcal{B})$
    \State $[\mathtt{P}]_\mathcal{C} \leftarrow \text{NTT}([\mathtt{P}_\mathtt{coeff}]_\mathcal{C})$
    \State \textbf{return} $[\mathtt{P}]_\mathcal{C}$
\EndProcedure
\end{algorithmic}
\end{algorithm}
\vspace{-0.32in}
\end{figure}

\subsection{Primitive HE ops in CKKS}
\label{sec:2_3_he_ops}

In Table~\ref{table:operations}, we summarize the \emph{primitive} HE ops of CKKS.
These HE ops become the building blocks of complex HE ops such as homomorphic linear transform.
We describe some of the important details of the primitive HE ops below.

\noindent\textbf{HRescale:} After multiplying (CMult, PMult, or HMult) a ciphertext with another operand having a scale of $\Delta$, the scale of the result becomes ${\Delta}^2$.
HRescale is a primitive HE op that divides the ciphertext by $\Delta$ to recover its scale back to $\Delta$.

As it is difficult to perform a division when polynomials are represented in limbs using the RNS, CKKS utilizes an alternative way to obtain the divided result.
In CKKS, the value of each $q_i$ is set close to $\Delta$ and HRescale is performed by eliminating the last limb of the polynomial ($q_L$) and multiplying $q_L^{-1} \text{ mod } q_i (i\!=\!0,1, \cdots, L-1)$ to the remaining limbs~\cite{sac-2018-frns-ckks}.

We can successively apply HRescale after each mult until the polynomial only has the $q_0$-limb left. At this moment, it is impossible to perform additional mults.
Thus, the maximum number of possible mults is $L$, which we refer to as the \emph{maximum multiplicative level}.
We call the remaining number of possible mults the \emph{(current) multiplicative level} and denote it as $\ell$.
A ciphertext at a multiplicative level $\ell$ is represented as a pair of $(\ell\!+\!1)\times N$ matrices.

\noindent\textbf{Automorphism:} To circularly shift a message by $r$ slots through HRot, we first compute the automorphism of polynomials.
Automorphism moves the $i$-th coefficient of a polynomial to the position of the $\psi_r(i)$-th coefficient by applying the mapping in Eq. \ref{eq:automorphism} to each limb.
We reuse the notation to denote automorphism on a polynomial $\mathtt{P}_{\textbf{m}} $ as $\psi _r(\mathtt{P}_{\textbf{m}})$ and refer to $r$ as the \emph{rotation amount}.
\begin{equation} \label{eq:automorphism}
    \psi_r : i\mapsto i \cdot 5^r \text{ mod } N
\end{equation}

\noindent\textbf{Key-switching:} HE ops such as HMult and HRot produce polynomials that are only decryptable with keys such as $\mathtt{S}^2$ or $\psi_r(\mathtt{S})$.
To make such polynomials decryptable with the secret key $\mathtt{S}$,
we perform \emph{key-switching} during HMult or HRot, which multiplies a polynomial with a special public key called the  \emph{evaluation key} ($\textbf{evk}$):
\begin{equation}
    \text{KeySwitch}(\mathtt{P}_\mathbf{m}, \mathbf{evk}) = P^{-1} \cdot \mathtt{P}_\mathbf{m} \boldsymbol{\cdot} \mathbf{evk}
\end{equation}
In particular, the $\textbf{evk}$ encrypting $\mathtt{S}^2$ is used for HMult and is denoted as $\textbf{evk}_{\text{mult}}$.
The $\textbf{evk}$ encrypting $\psi_r(\mathtt{S})$ is used for HRot and is denoted as $\textbf{evk}_{\text{rot}}^{(r)}$.
A separate $\textbf{evk}_{\text{rot}}^{(r)}$ is required for each rotation amount $r$.

In general, key-switching is an expensive HE op as it involves multiple NTTs and BConvs. Moreover, because key-switching is performed frequently, it dominates the execution time of an HE application~\cite{tches-2021-100x}.
Multiplying $ \mathtt{P}_{\mathbf{m}} $ and $ \mathbf{evk} $ requires additional steps as they have different modulus.
While $ \mathtt{P}_{\mathbf{m}} \in \mathrm{R}_Q$ has modulus $Q$, $ \mathbf{evk} \in  \mathrm{R}_{PQ}$ has modulus $PQ$ where $P$ is the \emph{special modulus} that can be decomposed into a set of $ \alpha $ prime limbs $\mathcal{B}=\{ p_0, p_1, \cdots, p_{\alpha-1}\}$.
BConvs are performed to bring $\mathtt{P}_{\mathbf{m}}$ into $\mathrm{R}_{PQ}$ by generating the $[\mathtt{P}_{\mathbf{m}}]_\mathcal{B}$ part. After multiplying $\mathtt{P}_{\mathbf{m}}$ with $ \mathbf{evk}$, BConvs bring the multiplied results back to $\mathrm{R}_Q$, and the results are divided by $P$ to reduce error in a similar way to HRescale.

\noindent\textbf{Security of HE and generalized key-switching:}
We use the generalized key-switching method in \cite{rsa-2020-better}.
It introduces a new parameter, $ \mathtt{dnum} $, and sets the number of special prime limbs as
$ \alpha\! =\! (L\! +\! 1) / \mathtt{dnum} $.
A single $ \mathbf{evk} $ consists of $ \mathtt{dnum} $ pairs of polynomials: $\mathbf{evk}\! =\!  ( \mathbf{evk}_0,  \mathbf{evk}_1, \cdots, \mathbf{evk}_{\mathtt{dnum} - 1} )$, $\mathbf{evk}_i \in \mathrm{R}_{PQ}^2$.
Key-switching is performed by Alg.~\ref{alg:key-switching}.
We first decompose $\mathcal{C}$
into  $\mathtt{dnum}$ \emph{partial limb groups}, $ \mathcal{C}_0, \cdots, \mathcal{C}_{\mathtt{dnum} - 1} $ where $ \mathcal{C}_i\! =\! \{ q_{\alpha i}, q_{\alpha i + 1}, \cdots,  q_{\alpha (i + 1) - 1}\} $.
We equally decompose $(L\!+\!1)$ limbs of $ \mathtt{P}_{\mathbf{m}} $ into $ \mathtt{dnum} $ pieces: $\{ [\mathtt{P}_{\mathbf{m}}]_{\mathcal{C}_i} \}_{0 \leq i < \mathtt{dnum}}$.
We perform BConvRoutine on each piece to extend the limbs to $\mathrm{R}_{PQ}$ and multiply each piece with the corresponding $\mathbf{evk}_i$. We accumulate the $\mathtt{dnum}$ pieces of multiplied results, then bring the result back to $\mathrm{R}_{Q}$ through BConvRoutine.

The computational complexity of key-switching and the size of an $ \mathbf{evk} $ increase as $ \mathtt{dnum} $ gets larger, degrading overall performance.
Despite such limitations, generalized key-switching is used to acquire more multiplicative levels.
The \emph{security level} of HE is roughly decided by $N$ and $( \alpha\! +\! L\! +\! 1) $, which increases as $N$ gets larger or $( \alpha\! +\! L\! +\! 1) $ gets smaller~\cite{wahc-2019-curtis}.
Recent HE studies~\cite{eurocrypt-2021-efficient,iacr-2020-varmin,eurocrypt-2021-invsine} and libraries~\cite{github-lattigo, online-palisade} typically target a security level of 128 bits.
Under such security level constraints, $N$ is chosen as low as possible and $( \alpha\! +\! L\! +\! 1) $ as high as possible.
Table~\ref{tab:params} shows exemplar parameter selection satisfying 128-bit security.
If $\mathtt{dnum}$ is large, the value of $ \alpha\! =\! (L\! +\! 1) / \mathtt{dnum} $ reduces, enabling a higher $L$ under the same  $( \alpha\! +\! L\! +\! 1) $.
In particular, when using a small $N$, it is inevitable to use a $\mathtt{dnum}$ value close to its maximum ($=\!L\!+\!1$) to secure as many multiplicative levels as possible.

\begin{figure}[tb!]
\vspace{-11pt}
\begin{algorithm}[H]
\caption{Generalized key-switching}\label{alg:key-switching}
\algnewcommand\algorithmicnotation{\textbf{Notation:}}
\algnewcommand\Notation{\item[\algorithmicnotation]}
\begin{algorithmic}[1]
\Notation{$\mathbf{evk} = \left( \mathbf{evk}_0, \mathbf{evk}_1, \cdots, \mathbf{evk}_{\mathtt{dnum} - 1} \right) $}
\Procedure{KeySwitch}{$ \mathtt{P}$, $ \mathbf{evk} $}
  \For{i $\leftarrow$ 0, 1,$\cdots$, $\mathtt{dnum}$-1} \label{alg:key-switching:line:for}
    \State $ [\mathtt{P}_{i}]_\mathcal{D} \leftarrow [\mathtt{P}]_{\mathcal{C}_i} \cup \text{BConvRoutine}([\mathtt{P}]_{\mathcal{C}_i}, \mathcal{D}\setminus \mathcal{C}_i) $ \label{alg:key-switching:line:bconvroutine}
  \EndFor
  \State $(\mathtt{B}, \mathtt{A}) \leftarrow \sum_{i=0}^{\mathtt{dnum}-1} \text{PMult}(\mathbf{evk}_i, [\mathtt{P}_{i}]_D) $ \label{alg:key-switching:line:accum}
  \State $ \mathtt{B}^{\prime} \leftarrow [\mathtt{B}]_\mathcal{C} - \text{BConvRoutine}([\mathtt{B}]_\mathcal{B}, \mathcal{C})$
  \State $ \mathtt{A}^{\prime} \leftarrow [\mathtt{A}]_\mathcal{C} - \text{BConvRoutine}([\mathtt{A}]_\mathcal{B}, \mathcal{C})$
  \State \textbf{return} $(P^{-1} \cdot \mathtt{B}^{\prime}, P^{-1} \cdot \mathtt{A}^{\prime})$
\EndProcedure
\end{algorithmic}
\end{algorithm}
\vspace{-0.32in}
\end{figure}

\subsection{CKKS bootstrapping}
When the ciphertext $[\![ \mathbf{m} ]\!] $'s multiplicative level $\ell$ reaches 0 after multiple HRescales, it becomes impossible to perform more mult ops.
As such, one must recover the multiplicative level through a special op called \emph{bootstrapping} to enable further ops on $[\![ \mathbf{m} ]\!] $.
Because bootstrapping allows an unlimited number of ops, it becomes an essential op for complex HE applications.
The HE schemes that support bootstrapping are generally referred to as \emph{Fully HE} (FHE).

Bootstrapping is a complex sequence of HE ops which consists of four major steps: LevelRecover, Homomorphic IDFT (H-IDFT), EvalMod, and Homomorphic DFT (H-DFT).
During LevelRecover, we raise $[\![ \mathbf{m} ]\!] $'s multiplicative level to its maximum, $L$, by changing the modulus from $q_0$ to $Q$.
However, a multiple of $q_0$, which was previously removed through modular reduction, appears in the resulting $[\![ \mathbf{m'} ]\!]$ (i.e., $\mathtt{P}_{\mathbf{m'}}\! =\! \mathtt{P}_{\mathbf{m}}\! +\! q_0 \cdot \mathtt{I} $).
To recover $[\![\mathbf{m} ]\!]$ from $[\![ \mathbf{m'} ]\!]$, we have to remove the $q_0 \cdot \mathtt{I}$ term from $\mathtt{P}_{\mathbf{m'}}$.
To enable homomorphic ops on $\mathtt{P}_{\mathbf{m'}}$, we perform H-IDFT on $[\![ \mathbf{m'} ]\!]$ to get $[\![\text{IDFT}(\mathbf{m'})]\!]\!$, which we denote as $\![\![ \mathtt{P}_{\mathbf{m'}} ]\!]$ to emphasize that the ciphertext encrypts the coefficients of $\mathtt{P}_{\mathbf{m'}}$ considering Eq.~\ref{eq:encode}.
EvalMod homomorphically performs the modulo operation on $ [\![ \mathtt{P}_{\mathbf{m'}} ]\!] $. The modulo operation is not a linear function, so it is approximated by a high-degree polynomial function.
Homomorphic evaluation of the function consists of multiple HMults and CMults, consuming many multiplicative levels~\cite{eurocrypt-2018-heaanboot}.
This step recovers $ [\![ \mathtt{P}_{\mathbf{m}} ]\!] $ by calculating $ [\![ \mathtt{P}_{\mathbf{m'}}\text{ mod }q_0 ]\!] $ and H-DFT recovers $ [\![ \mathbf{m}]\!]$ from $ [\![ \mathtt{P}_{\mathbf{m}} ]\!] $.

The resulting ciphertext has $ (L - L_{boot}) $ multiplicative levels as multiple HRescales consume $ L_{boot} $ levels during bootstrapping.
Thus, $L$ must be high enough to support CKKS bootstrapping. However, the security limit requires that $L$ can only be increased proportionally to $N$, so $N$ should be sufficiently high ($\geq 2^{15}$) to allow bootstrapping~\cite{iacr-2020-varmin}.
Prior works on FHE~\cite{github-lattigo,tches-2021-100x,iacr-2020-varmin, eurocrypt-2021-efficient} mostly use $N\! =\! 2^{15}$ to $2^{17}$ to preserve sufficient  multiplicative levels for other ops.
Thus, we use the parameter set summarized in Table \ref{tab:params} in this work, which is a slightly modified version of Lattigo~\cite{github-lattigo}'s parameter set and guarantees 128-bit security.

\setlength{\tabcolsep}{3.4pt}
\begin{table}[tb!]
\centering
\begin{threeparttable}
\caption{Representative parameters used in HE acceleration works and corresponding data sizes.}
\begin{tabular}{l|ccccc|ccc}
\hline
\multirow{2}{*}{Work} & \multicolumn{5}{c|}{Parameters} & \multicolumn{3}{c}{Data size (MB)} \\
& $N$ & $L$  & $L_{\text{boot}}$ & $\mathtt{dnum}$ & $\alpha$ & $\mathtt{P}_{\mathbf{m}}$ & $[\![\mathbf{m}]\!]$ & $\mathbf{evk}$ \\
\hline
\hline
Lattigo~\cite{github-lattigo} & $2^{16}$ & 24 & 15 & 5 & 5 & 12.5 & 25 & 150 \\
100x~\cite{tches-2021-100x}\textsuperscript{\dag} & $2^{17}$ & 29 & 19 & 3 & 10 & 30 & 60 & 240 \\
F1~\cite{micro-2021-f1} & $2^{14}$ & 15 & - & 16 & 1 & 1 & 2 & 34  \\
ARK\xspace & $2^{16}$ & 23 & 15 & 4 & 6 & 12 & 24 & 120 \\
\hline
\end{tabular}
\begin{tablenotes}\footnotesize
  \item[\dag] \cite{tches-2021-100x} used a 173b secure parameter set, while others 128b.
\end{tablenotes}
\label{tab:params}
\vspace{-0.19in}
\end{threeparttable}
\end{table}
\setlength{\tabcolsep}{5.4pt}

\section{conclusion}
\label{sec:8_conclusion}
In this paper, we present a set of fabrication-technology-aware algorithmic optimizations and hardware design for fully homomorphic encryption (FHE).
To mitigate the off-chip memory bandwidth bottleneck of bootstrapping,
we present the following key algorithmic innovations: \emph{multi-hop HRot} and \emph{on-the-fly limb extension}.
These two optimizations significantly reduce  off-chip memory accesses,
fully reaping the high computational capabilities of  domain-specific architectures.
We propose ARK\xspace, an accelerator addressing new computation and data movement constraints of FHE through an efficient functional unit (FU) microarchitecture that minimizes on-chip data movement.
ARK\xspace features a data-access-pattern-aware data distribution policy and a memory organization simplifying dataflow, enabling streamlined execution across the FUs.
Overall, ARK\xspace provides two to four orders of magnitude higher performance
over prior works. This enables ARK\xspace to provide
real-time encrypted CNN inference, showing 0.294 seconds of inference time for the ResNet-20 model.

\section{FHE Memory Bottlenecks}
\label{sec:3_bottleneck}

\subsection{Limitations of prior HE accelerators}
\label{sec:3_1_prior_accelerators}

All HE ops can be broken down into \emph{primary functions} that can be executed in parallel: (I)NTT, BConv, automorphism, and other \emph{element-wise functions}.
By using well-known FFT algorithms~\cite{cooley-tukey}, (I)NTT can be effectively parallelized by dividing it into $\log N$ stages, where each stage consists of parallel $\sfrac{N}{2}$ butterfly ops.
Since $N$ is as large as $2^{16}$, the degree of parallelism is high.
Given such, domain-specific architectures that employ a massive number of functional units (FUs) to reap such opportunities become an appealing option for accelerating HE.
To this end, designing a memory system that is capable of feeding these abundant FUs on-chip becomes a primary design objective.

Prior work has identified that HE ops have very low arithmetic intensity and are bounded by  off-chip memory bandwidth~\cite{2021-arxiv-does, tches-2021-100x, hpca-2019-roy}.
Even a simple HE op, such as HMult, has a working set including several ciphertexts and an $\mathbf{evk}$\xspace, which amounts to several hundreds of MBs.
Unfortunately, conventional CPU/GPU/FPGA solutions typically do not come with an on-chip memory large enough to hold the entire working set of HE ops (e.g., even the latest NVIDIA H100 GPU ``only'' has 50MB of on-chip cache~\cite{authorea-2022-h100}), causing frequent off-chip memory accesses and  high memory bandwidth pressure.

F1~\cite{micro-2021-f1} is the first ASIC HE accelerator to evaluate the performance of a single-slot CKKS bootstrapping.
F1 uses small parameters, $(N, L) = (2^{14}, 15)$, and
deploys a massive amount of computation logic to utilize the parallelism in HE ops.
In particular, F1 utilizes a dedicated
\emph{NTT unit} (NTTU) that
implements a pipelined 2D-FFT using  $\frac{1}{2}\sqrt{N} \log N\! =\! 896$ modular multipliers.
F1 consists of 16 \emph{clusters}, and each cluster is a vector core with $\sqrt{N}\! =\! 128$ lanes that feed data to an NTTU and other FUs.
Overall, F1 has a massive amount of 18,432 modular multipliers in a chip: 14,336 for NTTUs and 4,096 for element-wise multipliers.
Every HE op is essentially a sequence of modular mults and modular adds, and a modular mult requires a costly modular reduction op~\cite{eurocrypt-1986-barrett, 1985-montgomery}. Thus, the computational capability of an HE accelerator can be quantified by the number of modular multipliers.

Although F1 provides high speedup vs. conventional systems, its applicability is limited to small problem sizes and simple workloads; when it comes to practical FHE parameters and complex workloads that require frequent bootstrapping, the performance of F1 can be  degraded significantly.\footnote{Because of the small parameters used in F1, F1 cannot perform bootstrapping of ciphertexts encrypting a message vector, only supporting a message with a single slot ($n=1$).
Thus, F1's bootstrapping throughput is severely limited.
Single-slot bootstrapping does not need H-(I)DFT steps, so it is less costly and consumes fewer levels.
F1 also uses 32-bit machine word and prime sizes, but practical bootstrapping requires the use of larger primes for error mitigation.}
We first explain the details of bootstrapping for analysis.

\subsection{State-of-the-art bootstrapping algorithm}
\label{sec:3_2_sota_bootstrapping}

H-(I)DFT is a memory-intensive op that takes up most of the execution time in bootstrapping.
H-(I)DFT can be performed in a similar pattern to FFT, as shown in Alg.~\ref{alg:fft-like-hdft} where $2^k$ denotes the radix in FFT~\cite{cooley-tukey}.
Each iteration requires ($2^{k+1}\!-\!1$) HRots and ($2^{k+1}\!-\!1$) PMults, each with a different $\mathbf{evk}_{\text{rot}}^{(r)}$ and a different \emph{plaintext}, a polynomial for precomputed DFT constants ($\mathtt{P}_{s,i}$).
The whole process consumes $\text{log}_{2^k}n$ multiplicative levels, so the choice of $k$ creates a trade-off between computation and level consumption.

\begin{figure}[tb!]
\vspace{-11pt}
\begin{algorithm}[H]
\caption{FFT-like homomorphic DFT (Radix-$2^k$)}\label{alg:fft-like-hdft}
\begin{algorithmic}[1]
\Require{Precomputed plaintexts (polynomials) $\{\mathtt{P}_{s,i}\}$}
\Procedure{DFT}{$[\![\mathbf{m}]\!], \{\mathtt{P}_{s,i}\}$}
\State{$[\![\mathbf{t}]\!] \leftarrow [\![\mathbf{m}]\!]$}
\For{$s \leftarrow$ 0, 1, $\cdots$, $\text{log}_{2^k}n-1$\label{alg:fft-like-hdft:line:for}}
\State $[\![\mathbf{t}]\!] \leftarrow  \sum_{i=-2^k\!+\!1}^{2^k\!-\!1} \text{PMult}(\text{HRot}([\![\mathbf{t}]\!], i\! \cdot\! 2^{k \cdot s}), \mathtt{P}_{s,i})$\label{alg:fft-like-hdft:line:sum}
\State $[\![\mathbf{t}]\!] \leftarrow \text{HRescale}([\![\mathbf{t}]\!])$ \Comment{consumes one level}
\EndFor
\State{\Return{$[\![\mathbf{t}]\!]$}}
\EndProcedure
\end{algorithmic}
\end{algorithm}
\vspace{-0.32in}
\end{figure}

Line~\ref{alg:fft-like-hdft:line:sum} of Alg.~\ref{alg:fft-like-hdft} can be further improved by computing $[\![\mathbf{t}^\prime]\!]\! =\! \text{HRot}([\![\mathbf{t}]\!], -2^k\cdot2^{k \cdot s})$ and applying the BSGS (Baby-Step, Giant-Step) algorithm~\cite{eurocrypt-2018-heaanboot} as shown in Eq.~\ref{eq:bsgs}. $k_1$ and $k_2$ in Eq.~\ref{eq:bsgs} are integers satisfying $k\!+\!1\! = \!k_1\! + \!k_2$.
Consequently, the number of HRots reduces from $\mathcal{O}(2^k)$ to $\mathcal{O}(\sqrt{2^k})$.
\begin{equation} \label{eq:bsgs}
\underbrace{\sum_{\mathclap{j\!=\!0}}^{\mathclap{2^{k_2}\!-\!1}}  \text{HRot}  \big(  \sum_{\mathclap{i=0}}^{\mathclap{2^{k_1}\!-\!1}}  \text{PMult}(\underbrace{\text{HRot}([\![\mathbf{t}^\prime]\!],i\! \cdot\! 2^{k\cdot s})}_{\text{baby-step}}, \mathtt{P}_{s,i,j}^\prime\!), j\cdot\!2^{k_1\! +\! k\cdot s}  \big)}_{\text{giant-step}}
\end{equation}

In ARK\xspace with $n = 2^{15}$, we use $k = 5$, which consumes 3 levels, and $(k_1, k_2)$ in Eq.~\ref{eq:bsgs} as $(3, 3)$.
With additional optimizations, 40 HRots and 158 PMults are performed for each H-(I)DFT, and
40 distinct $\textbf{evk}_{\text{rot}}^{(r)}$s and 158 plaintexts must be prepared, which amount to 6.4GB (resp., 0.6GB) of data for H-IDFT (H-DFT) (see Table~\ref{tab:params}).

\subsection{Memory bottleneck in bootstrapping}
\label{sec:3_3_off_chip_bottleneck}

The time to load $\mathbf{evk}_{\text{rot}}^{(r)}$s and plaintexts, which are used once per H-(I)DFT, becomes the hard bound of latency for H-(I)DFT.
Even if  we assume that integrating hundreds of MBs of on-chip memory is feasible with the current fabrication technology~\cite{isca-2021-tpuv4i,hcs-2021-graphcore,hcs-2021-sambanova}, we must fetch the single-use data of H-(I)DFT from off-chip memory due to its large aggregate size and limited data reuse opportunities.
With the latest HBM3~\cite{jedec-2022-hbm3}, a system having a 3TB/s off-chip memory bandwidth is feasible~\cite{authorea-2022-h100}, which would allow loading the single-use data in 2.1ms (resp., 0.2ms) for H-IDFT (H-DFT).

We analyze the impact of such hard latency bound on domain-specific architectures by measuring the maximum utilization of FUs in F1.
We scaled F1 to the bootstrappable parameter we use (see Table~\ref{tab:params}) by utilizing NTTUs that feature $ \frac{1}{2}\sqrt{N} \log N $=2,048 modular multipliers to fully support bootstrapping.
As a result, the total number of modular multipliers in the chip increases by $2.22\times$ to 40,960.
We first computed the number of possible modular mults in the scaled F1 in 2.1ms (resp., 0.2ms) by assuming that it runs at 1GHz and is fully-pipelined so that 40,960 modular mults are possible every cycle.
Then, we divided it by the number of modular mults performed in H-IDFT (H-DFT) to derive the maximum achievable utilization of modular multipliers.
Overall, our analysis revealed that the scaled F1 performs poorly when handling off-chip memory bandwidth bottleneck caused by bootstrapping, only achieving an 8.61\% (13.32\%) utilization rate for H-IDFT (H-DFT).

Resolving the off-chip memory bandwidth bottleneck in bootstrapping should precede improving the computational capability to support practical FHE workloads that require frequent bootstrapping, especially for domain-specific architectures that can utilize a massive amount of computational logic.

\section[Algorithms tackling the memory bottleneck]
{Algorithms tackling memory\\ bottleneck}
\label{sec:4_algorithm}

With the advancement of fabrication technology, the scaling of off-die memory bandwidth is slower than that of logic density~\cite{isca-2009-disaggregated}.
We present our fabrication-technology-aware algorithmic innovations that reduce off-chip memory accesses, alleviating FHE's memory bandwidth bottleneck.

\subsection{Multi-hop HRot}
\label{sec:4_1_multi_hop}

We devise a novel algorithm, multi-hop HRot, which resolves the memory bottleneck of the two memory-intensive computational patterns (Eq.~\ref{eq:pattern1} and Eq.~\ref{eq:pattern2}) in HE ops.
\vspace{-0.05in}
\begin{equation} \label{eq:pattern1}
    \sum_{i=0}^{m}\text{PMult}\left(\text{HRot}\left([\![\mathbf{x}]\!], i \cdot r, \textbf{evk}_{\text{rot}}^{(i \cdot r)}\right), \mathtt{P}_{i}\right)
\end{equation}
\vspace{-0.2in}
\begin{equation} \label{eq:pattern2}
    \sum_{i=0}^{m}\text{HRot}\left([\![\mathbf{x}_i]\!], i \cdot r, \textbf{evk}_{\text{rot}}^{(i \cdot r)}\right)
\end{equation}
Eq.~\ref{eq:pattern1} involves rotating the same ciphertext, and Eq.~\ref{eq:pattern2} involves rotating and accumulating different ciphertexts, each with rotation amounts showing an \emph{arithmetic progression} ($i\cdot r$).
Each pattern invokes off-chip memory access to multiple ($m$) $\mathbf{evk}_{\text{rot}}^{(i \cdot r)}$s which becomes the hard memory bound.

Alg.~\ref{alg:pattern1} and Alg.~\ref{alg:pattern2} show how multi-hop HRot can be applied to Eq.~\ref{eq:pattern1} and Eq.~\ref{eq:pattern2}, respectively.
Multi-hop HRot sequentially reuses the previous HRot results to compute HRots by $i\cdot r$ rotation amounts.
The amount of computation stays the same while the required number of $\mathbf{evk}_{\text{rot}}^{(r)}$ for each pattern reduces to only one.
The baby-step and the giant-step in Eq.~\ref{eq:bsgs} respectively correspond to Eq.~\ref{eq:pattern1} and Eq.~\ref{eq:pattern2}.
Thus, by applying Alg.~\ref{alg:pattern1} and Alg.~\ref{alg:pattern2} to Eq.~\ref{eq:bsgs}, we only need two distinct $\mathbf{evk}_{\text{rot}}^{(r)}$s, reducing the aggregate key size needed for H-(I)DFT to 15\% of the baseline.
We can also apply multi-hop HRot to the most common HE ops in HE ML workloads, such as homomorphic convolution and homomorphic matrix mult, which
also share the same computational patterns.
We apply the multi-hop HRot to such HE ops in Section~\ref{sec:6_evaluation}.

\begin{figure}[tb!]
\vspace{-11pt}
\begin{algorithm}[H]
\caption{Multi-hop HRot applied to Eq.~\ref{eq:pattern1}}\label{alg:pattern1}
\begin{algorithmic}[1]
\State $[\![\mathbf{t}]\!] \gets [\![\mathbf{x}]\!]$
\State $[\![\mathbf{s}]\!] \gets \text{PMult}\left([\![\mathbf{t}]\!], \mathtt{P}_{0}\right)$
\For{$i$ $\leftarrow$ 1, 2, $\cdots$, $m$}
\State $[\![\mathbf{t}]\!] \gets \text{HRot}([\![\mathbf{t}]\!], r, \textbf{evk}_{\text{rot}}^{(r)})$
\State $[\![\mathbf{s}]\!] \gets [\![\mathbf{s}]\!] + \text{PMult}([\![\mathbf{t}]\!], \mathtt{P}_{i})$
\EndFor
\State{\Return{$[\![\mathbf{s}]\!]$}}
\end{algorithmic}
\end{algorithm}
\vspace{-0.22in}
\end{figure}

\begin{figure}[tb!]
\vspace{-5pt}
\begin{algorithm}[H]
\caption{Multi-hop HRot applied to Eq.~\ref{eq:pattern2}}\label{alg:pattern2}
\begin{algorithmic}[1]
\State $[\![\mathbf{s}]\!] \leftarrow [\![\mathbf{x}_{m}]\!]$
\For{$i \leftarrow$ $m\! -\! 1$, $m\! -\! 2$, $\cdots$, 0}
\State $[\![\mathbf{s}]\!] \leftarrow \text{HRot}([\![\mathbf{s}]\!], r, \textbf{evk}_{\text{rot}}^{(r)}) + [\![\mathbf{x}_i]\!]$
\EndFor
\State{\Return{$[\![\mathbf{s}]\!]$}}
\end{algorithmic}
\end{algorithm}
\vspace{-0.22in}
\end{figure}

\subsection{On-the-fly limb extension (OF-Limb)}
\label{sec:4_2_of_limb}

We propose \emph{on-the-fly limb extension} (OF-Limb), which generates the limbs of plaintexts used in PMult and PAdd on-the-fly to save off-chip memory access.
The plaintext $\mathtt{P}_{\mathbf{m}^\prime}$ used in PMult($ [\![ \mathbf{m} ]\!] $, $\mathtt{P}_{\mathbf{m}^\prime}$) is precomputed before executing an HE application in the same way as in Eq.~\ref{eq:encode}.
Precomputed $\mathtt{P}_{\mathbf{m}^\prime}$ has the same number of limbs ($\ell\!+\!1$) as the polynomials composing $ [\![ \mathbf{m} ]\!] $ and is loaded from off-chip memory for PMult or PAdd.
However, we identify that it is possible to precompute only the $q_0$-limb of $\mathtt{P}_{\mathtt{m}^\prime}$ and extend it on-the-fly using the following equation so that $\mathtt{P}_{\mathtt{m}^\prime}$ has $(\ell\!+\!1)$ limbs :
 \begin{equation*}
     [\mathtt{P}_{\mathbf{m}^\prime}]_\mathcal{C} = \left\{ \text{NTT}([\mathtt{P}_{\mathbf{m}^\prime}]_{q_0} \text{ mod } q_i)\right\}_{q_i \in \mathcal{C}}
 \end{equation*}

OF-Limb reduces off-chip memory access of PMult and PAdd to $\sfrac{1}{(\ell\! +\! 1)}$ of the original method.
When assuming that the $\mathbf{evk}$s and plaintexts are loaded from off-chip memory and the rest of the working set fits into on-chip memory, plaintexts take up 27.5\% (resp., 40.9\%) of off-chip memory access of H-IDFT (H-DFT).

OF-Limb incurs extra computation.
NTT needs to be performed on the extended limbs of a plaintext to convert them into their evaluation representation.
The increased computation accounts for 22.9\% (resp., 24.1\%) of the entire H-IDFT (H-DFT) computation.
However, we identify that the performance gain from decreased off-chip memory access outweighs the overhead (in Section~\ref{sec:6_3_performance}).

\subsection[Impact of optimizations on the arithmetic intensity of homomorphic (I)DFT]
{Impact of optimizations on the arithmetic intensity of homomorphic (I)DFT}
\label{sec:4_3_algorithm_impact}

\begin{figure}
\centering
  \subfigure[Homomorphic IDFT\label{fig:idft_intensity}] {\includegraphics[height=1.22in]{Figure/idft_intensity.pdf}}
  \hfill
   \subfigure[Homomorphic DFT\label{fig:dft_intensity}] {\includegraphics[height=1.22in]{Figure/dft_intensity.pdf}}
    \vspace{-0.05in}
    \caption{Data loaded from off-chip memory and arithmetic intensity (ops/byte) change when applying multi-hop HRot and OF-Limb algorithms to (a) homomorphic IDFT and (b) homomorphic DFT. \label{fig:off_chip_access_and_intensity}}
    \vspace{-0.1in}
\end{figure}

Applying the two algorithms significantly reduces off-chip memory accesses, which eventually increases the arithmetic intensity (ops/byte) of H-(I)DFT.
We computed the arithmetic intensity of H-(I)DFT with and without applying the two algorithms. The arithmetic intensity is defined as the number of modular mult ops divided by the off-chip memory access bytes.
Applying multi-hop HRot results in a $2.6\times$ (resp., $2.0\times$) increase in the arithmetic intensity of H-IDFT (H-DFT).
Applying OF-Limb to multi-hop HRot results in additional $4.0\times$ ($2.9\times$) increase in arithmetic intensity, which reaches 11.1 (9.6) ops/byte.
The algorithms remove 88\% (78\%) of off-chip memory access, as shown in Fig.~\ref{fig:off_chip_access_and_intensity}.

There are other optimizations that lower the compute cost of H-(I)DFT, such as hoisting~\cite{eurocrypt-2021-efficient}, a technique to merge repeated computation, but they do not reduce the single-use data.
As they are not compatible with multi-hop HRot and only further reduce the arithmetic intensity of the baseline H-(I)DFT algorithm, we excluded them from the analysis.

\section[ARK\xspace architecture]{ARK\xspace architecture}
\label{sec:5_architecture}

\begin{figure*}[tb!]
    \centering
    \subfigure[ARK\xspace floorplan\label{fig:floorplan}]{\includegraphics[height=1.47in]{Figure/floorplan.pdf}}
    \hfill
    \subfigure[BConv unit\label{fig:bconvu}]{\raisebox{0.16in}{\includegraphics[height=1.13in]{Figure/bconvu.pdf}}}
    \vspace{-0.05in}
    \caption{(a) Overview of ARK\xspace. ARK\xspace is composed of 4 clusters, where a cluster includes a BConv unit (BConvU), an NTT unit (NTTU), an automorphism unit (AutoU), two multiply-add units (MADUs), a scratchpad memory, and distributed register files (RFs). A cluster in ARK\xspace can be divided into two regions: the coefficient region and the evaluation region. \ \ \ \ \ \ \ \ (b) Organization of a BConv unit (BConvU). 8 BConv lanes are shown for simplicity. Each lane has 6 multiply-accumulate (MAC) units. Central broadcast units (BrUs) hold base tables.\label{fig:floorplan_and_bconvu}}
    \vspace{-0.1in}
\end{figure*}

\begin{figure}[tb!]
    \centering
    \vspace{0.05in}
    \includegraphics[width=0.95\columnwidth]{Figure/ntt_bconv_ratio.pdf}
    \vspace{-0.1in}
    \caption{Computational breakdown, in terms of modular mult, of HRot with a ciphertext at the max level for different $\mathtt{dnum}$s. $(N,L)\!=\!(2^{16},23)$. \label{fig:ntt_bconv_ratio}}
    \vspace{-0.1in}
\end{figure}

We propose ARK\xspace, an architecture-algorithm co-design for FHE that can manage the increased arithmetic intensity and capitalize on the reduced working set by applying the two algorithms.
ARK\xspace utilizes 512MB of on-chip scratchpad memory, enabled by the scaling in fabrication technology~\cite{irps-2018-intel}, to hold a couple of $\mathbf{evk}$s and temporary data of HE ops.
Also, a high NTT computation throughput is required for OF-Limb, so we leverage a high-throughput NTTU similar to that of F1~\cite{micro-2021-f1}.
The NTTU works on a vector of $\sqrt{N}$ elements every cycle, so ARK\xspace also has a form of a vector processor having $ \sqrt{N} \!=\!256$ lanes.
ARK\xspace utilizes 4 clusters, each having an NTTU and other FUs for primary functions of CKKS: a BConv unit (BConvU), an automorphism unit (AutoU), and two multiply-add units (MADUs).

For practical bootstrappable parameters, F1's design targeting the maximum $\mathtt{dnum}$\xspace is no longer applicable.
When using max $\mathtt{dnum}=24$ under $(N, L) = (2^{16}, 23)$, for instance, the size of an $\mathbf{evk}$\xspace reaches 600MB, which hinders data reuse as it becomes impossible to fit a single $\mathbf{evk}$\xspace into on-chip memory.
Moving to practical parameters using smaller $\mathtt{dnum}$s\xspace (see Table~\ref{tab:params}) introduces new data access patterns and computational requirements.
Thus, we design ARK\xspace to handle these new challenges in an FHE accelerator for bootstrappable parameters while using multi-hop HRot and OF-Limb to relieve the off-chip memory bottleneck.

\subsection[Specialized BConv unit for low dnums]
{Specialized BConv unit for low dnums}
\label{sec:5_2_bconvu1}

Computational characteristics of HE ops change for lower $\mathtt{dnum}$s\xspace. Prior FPGA and ASIC HE accelerators~\cite{asplos-2020-heax, micro-2021-f1, hpca-2019-roy} targeted small problem sizes with a low $N$, making it inevitable to use a max $\mathtt{dnum}$\xspace to secure more multiplicative levels. When using max $\mathtt{dnum}$\xspace, (I)NTT takes up 73.3\% of the total computation (Fig.~\ref{fig:ntt_bconv_ratio}), so accelerating (I)NTT could directly improve the performance. Therefore, most prior FPGA and ASIC HE accelerators focused on accelerating (I)NTT. However, for our practical bootstrappable parameters with a large $N$ and a small $\mathtt{dnum}$\xspace, the portion of (I)NTT drops to 54.8\%, and BConv takes up 34.2\% of the total computation.

We design a specialized BConv unit (BConvU) to handle increased BConv computation and relieve the register file (RF) pressure of BConv.
BConv consists of two steps of modular mult.
First, it multiplies each limb by $(\hat{p}_j^{-1}\! \text{ mod }\! p_j)$.
Then, it multiplies each limb by $(\hat{p}_j\!\text{ mod }\!q_i )$ and accumulates the results (see Eq.~\ref{eq:bconv}).
BConvU targets the second step, which accounts for 96\% of the computation of BConv.
The second step can be regarded as a matrix mult between a $ (\ell + 1) \times \alpha $ matrix composed of $ (\hat{p}_j\!\text{ mod }\!q_i)$s, which we refer to as a \emph{base table}, and an $ \alpha \times N$ matrix $[\mathtt{P}_{\mathbf{m}}]_\mathcal{B}$.
A na\"ive matrix mult algorithm would require accessing $[\mathtt{P}_{\mathbf{m}}]_\mathcal{B}$ along the column direction while $[\mathtt{P}_{\mathbf{m}}]_\mathcal{B}$ is stored in memory in row-major order.
To access the matrix along the row direction, our BConvU uses a block matrix mult algorithm~\cite{1992-ics-matmul}, where each of multiple \emph{BConv lanes} processes a column of the polynomial in parallel.

We utilize multiple modular mult-accumulate (MAC) units in each lane to form an output-stationary systolic array~\cite{2018-arxiv-systolic, isca-2017-tpu} to reduce the RF pressure.
Fig.~\ref{fig:bconvu} depicts a Bconv lane composed of 6 MAC units organized as a $1\times6$ systolic array.
A MAC unit passes the input coefficient of a polynomial to the next MAC unit to reuse input data.
A MAC unit is entirely in charge of computing an inner product between a row in a base table and a column in a polynomial.
The $2^8$ MAC units in the same horizontal position receive $2^8$ elements from the $2^8$ lanes of a cluster every cycle.
The $2^8$ elements are each from a different column of the polynomial and are multiplied by the same value in the base table.

BConvU features central \emph{broadcast units} (BrUs) that broadcast the base table elements to the MAC units.
In each MAC unit, we time-multiplex mults between a row of the base table and 4 columns of the polynomial.
This reduces the broadcasting frequency of BrUs to once in every 4 cycles.
We sweep over the number of MAC units to deploy in a lane and choose 6, which is the same as $\alpha$ and provides sufficient throughput (see Section~\ref{sec:6_4_alternative}).
By using 6 MAC units in a BConvU lane, the entire BConv process can be regarded as partitioning the base table into
$6 \times \alpha$-shaped blocks and the polynomial into $\alpha \times (4 \cdot 2^8)$-shaped blocks and performing the block matrix mult between the two.

Each MAC unit has a multiplier, an input buffer to hold 4 elements, and an output buffer to hold 4 accumulated results.
Modular reduction logic is placed on the write-back path and is shared by the 6 MAC units.
Instead, the output buffer of each MAC unit needs to hold accumulated results in 128 bits.

\subsection{Access pattern oriented data distribution}
\label{sec:5_3_coeff_wise_distribution}

BConv has a different data access pattern compared to other functions.
Because an NTTU requires an entire limb of a polynomial to compute using a deep pipeline, and multiple (I)NTTs can be performed on separate limbs of a polynomial, it is natural to distribute a polynomial to multiple clusters \emph{limb-wise}, which is also the data distribution policy F1 used.
However, the limb-wise distribution becomes problematic for performing BConv, where multiple limbs of a polynomial are required to produce an output.
This was not the case when using max $\mathtt{dnum}$\xspace because the input set of limbs was composed of only a single limb when performing BConv, corresponding to $\mathcal{C}_i = \{q_{\alpha i}\} $ or $\mathcal{B} = \{p_0\}$ in Alg.~\ref{alg:key-switching}.

To perform BConv in parallel while distributing data evenly among the clusters, we utilize a \emph{coefficient-wise} data distribution, where $N$ coefficients of a limb in the coefficient representation are distributed evenly across the clusters.
Because BConv can be performed in parallel across the coefficients by splitting the $ \alpha \times N $ matrix (polynomial) into $ \alpha \times \frac{N}{4} $ matrices distributed across the 4 clusters, coefficient-wise distribution is suitable for the data access pattern of BConv.

We devise a data distribution policy that switches between limb-wise and coefficient-wise distribution and corresponding dataflow for performing HE ops in ARK\xspace.
We focus on a BConvRoutine (Alg.~\ref{alg:bconvroutine}).
A polynomial limb passes through an NTTU that executes INTT.
Each cluster prepares an INTT-applied limb and performs an all-to-all data exchange through the network-on-chip (NoC), switching to the coefficient-wise distribution.
When each cluster holds a quarter of $\alpha$ limbs of a polynomial, BConv is executed by a BConvU.
The reverse process is carried out for NTT.
Other than BConv, limb-wise distribution is used.

Switching between limb-wise and coefficient-wise distribution requires all-to-all data communication between clusters.
$ (\alpha\! +\! L\! +\! 1) \cdot N $ words need to be transferred for every BConvRoutine, requiring a total of $ (\mathtt{dnum}\! +\! 2) \cdot (\alpha\! +\! L\! +\! 1) \cdot N $ words to be transferred in a single key-switching (see Alg.~\ref{alg:key-switching}).
Meanwhile, a limb-wise-only distribution incurs more data communication.
For comparison, we implement the limb-wise-only data distribution by assigning $[\mathtt{P}_\mathbf{m}]_{\mathcal{C}_i}$ to cluster $i$ ($i\! =\! 0, 1, \cdots, \mathtt{dnum}\!-\! 1$) to eliminate data communication for BConvRoutine in Alg.\ref{alg:key-switching}.
This implementation enables the clusters to perform the loop in line \ref{alg:key-switching:line:for} of Alg.~\ref{alg:key-switching} independently.
However, we need to redistribute the data for accumulation (line~\ref{alg:key-switching:line:accum}), which causes more data communication involving $ 2 \cdot \mathtt{dnum} \cdot (\alpha\! +\! L\! +\! 1) \cdot N $ words when $\mathtt{dnum} > 2$.
We compare the performance of two data distribution policies
in Section~\ref{sec:6_4_alternative}.

\subsection{Two regions in a cluster and NTT unit}
\label{sec:6_2_nttu}

\begin{figure}[tb!]
    \centering
    \includegraphics[width=0.9\columnwidth]{Figure/nttu.pdf}
    \vspace{-0.05in}
    \caption{Organization of an NTT unit (NTTU). The logic for 64-point (I)NTT is shown for simplicity.\label{fig:nttu}}
    \vspace{-0.1in}
\end{figure}

Our NTTU, depicted in Fig.~\ref{fig:nttu}, has unique components that focus on minimizing data movement and RF pressure.
First, we distribute RFs across a cluster and organize the NTTU to have the opposite dataflows for INTT and NTT.
In Fig.~\ref{fig:nttu}, data flows rightward for NTT and leftward for INTT.
We partition the RF~\cite{1995-micro-tta, 1996-vector, hpca-2000-register} and place RFs at both ends of the NTTU: RF (Coeff) and RF (Eval).
Then, an NTTU splits a cluster into two regions: the \emph{coefficient region} holding polynomials in the coefficient representation and the \emph{evaluation region} holding polynomials in the evaluation representation.
BConvs and data exchanges on the NoC are executed on the coefficient region and the other functions are executed on the evaluation region.
Such a bidirectional dataflow simplifies the floorplan, which comes down to the organization shown in Fig~\ref{fig:floorplan}, and enables a power-efficient design by reducing data movement that induces expensive wire traversal~\cite{isscc-2014-horowitz, ieee-2001-wire}.
RF (NoC) is added to communicate with the NoC and the BConvU.
Revisiting the dataflow of a BConvRoutine, it can be observed from Fig.~\ref{fig:floorplan} that data movement is minimized by flowing through RF (Eval) $\rightarrow$ NTTU $\rightarrow$ RF (Coeff) $\rightarrow$ NoC $\rightarrow$ RF (NoC) $\rightarrow$ BConvU.
For INTT, NTTU also performs the first step of BConv using the \emph{BConv mult unit} in Fig.~\ref{fig:nttu}.

Moreover, we devise \emph{on-the-fly twisting factor generation} (OF-Twist) on our NTTU, which nearly eliminates memory traffic for loading \emph{twisting factors}.
F1 uses the 4-step FFT in \cite{1990-4stepntt}, which performs $N$-point (I)NTT using 2D-FFT by first performing $\sqrt{N}$-point (I)NTTs, multiplying elements with \emph{twisting factors}, transposing the $\sqrt{N}\times\sqrt{N}$ matrix of elements, and finally performing additional $\sqrt{N}$-point (I)NTTs.
F1 implements the entire process as a single long pipeline by utilizing multiple butterfly units, twisting units, and a transpose unit.
The benefit of this implementation is that the twiddle factors multiplied in the butterfly units stay the same for the whole process.
By contrast, different twisting factors are multiplied for every element, so $N$ twisting factors need to be loaded, of which the amount is as large as the actual data to perform NTT.
We observe that the twisting factors
show \emph{geometric progression}.
Thus, we design the twisting units to generate twisting factors on-the-fly by multiplying the common ratios of the geometric progression inside.
Using OF-Twist, we only have to load the starting values and the common ratios, reducing the total data loaded during (I)NTT to roughly half.
Also, OF-Twist can reduce the storage usage for twisting factors by 99\%, saving 30MB of space ($2\cdot (\alpha\!+\!L\!+\!1)\cdot N$ words) on our parameter set (Table~\ref{tab:params}).

\subsection{Automorphism unit (AutoU)}
\label{sec:6_2_autou}

\begin{figure}[tb!]
    \centering
    \includegraphics[width=0.9\columnwidth]{Figure/autou.pdf}
    \vspace{-0.05in}
    \caption{Organization of an automorphism unit (AutoU). The logic for 8 words is shown for simplicity. Datapath is shown in solid lines and the control path in dotted lines. \label{fig:autou}}
    \vspace{-0.1in}
\end{figure}

Automorphism moves data according to the mapping $ \psi_r $ in Eq.~\ref{eq:automorphism}.
Automorphism may look like an irregular permutation between all $N$ coefficients, but it can be performed through internal permutations between $2^8$ coefficients.
Each $2^8$ vector lane consumes a coefficient every cycle, where the $2^8$ consumed coefficients have a stride of $2^8$; i.e., the coefficients with indices $(i, i+1 \cdot 2^8, i+2\cdot 2^8, \cdots , i+(2^8-1)\cdot 2^8)$ are consumed every cycle ($ 0 \leq i < 2^8 $). After automorphism, the $2^8$ coefficients are again mapped to a set of $2^8$ coefficients with a stride of $2^8$; i.e., $(j, j+1 \cdot 2^8, j+2 \cdot 2^8, \cdots , j+(2^8-1)\cdot 2^8)$ for $ j = \psi_r(i) \text{ mod } 2^8 $. Thus, automorphism can be done by (i) loading $2^8$ words (one from each lane), (ii) performing an internal permutation of $2^8$ words according to $ \psi_r $, and (iii) storing $2^8$ words (one to each lane).

We devise an automorphism unit (AutoU)\footnote{Our AutoU actually targets a polynomial in evaluation representation. $\psi_r$ and the data address mapping change for polynomials in evaluation representation, but the same property still holds.} implementing the internal permutation logic (see Fig.~\ref{fig:autou}).
An AutoU is composed of 8 stages, recursively permuting the data according to the $ \psi_r $ index calculation results.
Each stage is pipelined and merges two chunks of data by swapping the order of the two chunks depending on the index calculation results.

\section{Evaluation}
\label{sec:6_evaluation}

\subsection{Modeling ARK\xspace}
\label{sec:6_1_modeling}

\noindent\textbf{Hardware modeling:} We modeled the FUs of ARK\xspace using ASAP7~\cite{mj-2016-asap7}, a predictive process design kit for a 7nm technology node.
In NTTUs and BConvUs, logic units for modular reduction use Montgomery reduction~\cite{1985-montgomery}, and in MADUs, the logic units use Barrett reduction~\cite{eurocrypt-1986-barrett}.
We used FinCACTI~\cite{isvlsi-2014-fincacti} to model the scratchpad and RFs as SRAM components because an SRAM compiler is not included in ASAP7.
Single-ported SRAMs are utilized.
We updated the model of FinCACTI to match the information provided by ASAP7, the IRDS roadmap~\cite{whitepaper-2018-irds}, and other published works on 7nm technology nodes~\cite{isca-2021-tpuv4i, iedm-2017-gf7nm, iedm-2016-tsmc7nm, iedm-2017-intel10nm, isscc-2017-7nm-sram, isscc-2018-7nm-sram-euv, isscc-2020-5nm-sram, vlsit-2018-samsung}.
We separately modeled the overhead of expensive semi-global wires (80nm pitch~\cite{intel-2008-wire45nm}), where their length cannot be ignored.
We used \cite{isca-2021-tpuv4i, micro-2017-finegrainedDRAM} to evaluate the HBM memory.
Two 500GB/s HBM2 stacks~\cite{jedec-2021-hbm2, jedec-2022-hbm3} are used, providing a total of 1TB/s off-chip memory bandwidth.
The scratchpad memory provides a bandwidth of 20TB/s and
distributed RFs provide 72TB/s in total.
NoC is implemented as a simple multiplexer network connecting the same lane from each cluster and provides a modest bandwidth of 8TB/s.
The peak power and area costs of ARK\xspace's components are shown in Table~\ref{tab:area_power}.
Each peak power value is estimated with a usage case where the component can be most active.
ARK\xspace is sized 418.3mm\textsuperscript{2} and consumes up to 281.3W of power.

\noindent\textbf{Performance modeling:} We implemented a cycle-accurate simulator of ARK\xspace to evaluate its performance.
The simulator takes an HE program written at a high level as an input and converts it to a data dependence graph of primary HE functions.
The simulator schedules the functions and data movements while checking the data dependency and structural hazards.
Static scheduling is possible as HE programs do not feature any dynamic control flows~\cite{micro-2021-f1}.
In that sense, our simulator acts as a VLIW compiler for ARK\xspace.
The simulator collects the utilization rates of the components, combined with the power model to derive power consumption.

\renewcommand{\arraystretch}{1.0}
\begin{table}[bt!]
\caption{Area and peak power consumption of ARK\xspace.\label{tab:area_power}}
\centering
\begin{tabular}{p{3cm}rr}
\toprule
\textbf{Component}          & \textbf{Area (mm\textsuperscript{2})} & \textbf{Peak power (W)} \\
\midrule
4 BConvUs          & 9.3       & 18.9          \\
4 NTTUs            & 57.2      & 95.2         \\
4 AutoUs           & 20.6      & 4.6           \\
8 MADUs            & 8.9       & 24.7          \\
Register files     & 42.8      & 25.1          \\
Scratchpad memory  & 229.2     & 54.0          \\
NoC                & 20.6      & 27.0          \\
HBM                & 29.6      & 31.8          \\
\midrule
\textbf{Sum}                & \textbf{418.3}     & \textbf{281.3}         \\
\bottomrule
\end{tabular}
\vspace{-0.08in}
\end{table}

\subsection{Experimental setup}
\label{sec:6_2_setup}
To reflect the difference in parameters used by each work, we compared the performance of ARK\xspace with previous works using a metric called \emph{amortized mult time per slot}~\cite{tches-2021-100x} ($\textbf{T}_{A.S.}$\xspace).
$\textbf{T}_{A.S.}$\xspace combines the cost of HMult with the cost of bootstrapping by adding HMult time for $1, \cdots, L\! -\! L_{boot}$ levels and adding the bootstrapping time, then dividing it by $L\! -\! L_{boot}$ and $n$.
$\textbf{T}_{A.S.}$\xspace reflects various effects of parameter choice on performance: execution time, remaining levels after bootstrapping ($L\! -\! L_{boot}$), and throughput achieved by using many slots ($n$).

We evaluated the performance of ARK\xspace using complex FHE workloads that require bootstrapping.
HELR~\cite{aaai-2019-helr} trains a binary logistic regression classifier model for the MNIST dataset.
In each iteration, it trains the model with a mini-batch containing 1024 14$\times$14-pixel images.
We measured the performance of HELR for 30 iterations.
We also performed CNN inference using Lee et al.~\cite{arxiv-2021-resnet20}'s implementation of the ResNet-20 model for the CIFAR-10~\cite{techreport-2009-cifar10} dataset.
The model shows 92.43\% accuracy.
We additionally used the channel packing method of \cite{usenixsec-2018-gazelle}, which packs multiple channels of a feature map in a single message to fully utilize the slots.
When using the channel packing method, homomorphic evaluation of convolution layers in the model involves a series of HRots and PMults with weight plaintexts, so we applied both multi-hop HRot and OF-Limb.
Finally, we performed sorting~\cite{tifs-2021-sorting}.
Sorting in HE requires a complex evaluation of non-linear functions approximated by high-degree polynomials.
OF-Limb was used for every PMult in each workload.

For comparison, we used the state-of-the-art CKKS implementations of \textbf{Lattigo}~\cite{github-lattigo}, \textbf{100x}~\cite{tches-2021-100x} and \textbf{F1}~\cite{micro-2021-f1}, each representing a single-thread CPU, GPU (NVIDIA V100~\cite{techreport-2017-v100}) and ASIC implementation.
We used the reported execution times in ~\cite{tches-2021-100x, micro-2021-f1}.
We measured the performance of \cite{github-lattigo} by running the workloads on an Intel CPU system, Xeon Platinum 8358 with 512GB of DDR4-3200 memory.
We used the parameters in Table~\ref{tab:params} for each work.

\subsection{Performance of ARK\xspace}
\label{sec:6_3_performance}

\renewcommand{\arraystretch}{1.1}
\setlength{\tabcolsep}{4pt}
\begin{table}[tb!]
\caption{Comparison of performance between ARK\xspace and other prior works~\cite{github-lattigo, tches-2021-100x, micro-2021-f1} for $\textbf{T}_{A.S.}$\xspace and HELR.
We report the average execution time per iteration for HELR.}
\centering
\begin{tabular}{l|rrr|rr}
\hline
 & \textbf{Lattigo}  & \textbf{100x} & \textbf{F1} & \textbf{ARK\xspace} & vs. \textbf{100x} \\
\hline
\hline
$\textbf{T}_{A.S.}$\xspace ($\mu$s) & 88  & 8   &  260  & 0.014 & \textbf{$572\times$} \\
\textbf{HELR} (ms) & 23293  & 775   &  1024 & 7.717 & \textbf{$100\times$} \\
\hline
\end{tabular}
\label{tab:amort_performance}
\vspace{-0.05in}
\end{table}
\setlength{\tabcolsep}{5.4pt}

\begin{table}[tb!]
\centering
\caption{Performance of ARK\xspace for ResNet-20 inference and sorting compared to the baselines~\cite{arxiv-2021-resnet20, tifs-2021-sorting}.\label{tab:workload_performance}}
\begin{tabular}{l|rrr}
\hline
 & Baseline  & ARK\xspace & Speedup\\
\hline
\hline
ResNet-20~\cite{arxiv-2021-resnet20} ($s$) & 10,602  & 0.294 & \textbf{36,122$\times$} \\
Sorting~\cite{tifs-2021-sorting} ($s$) &  23,066 & 0.709 & \textbf{32,511$\times$} \\
\hline
\end{tabular}
\vspace{-0.05in}
\end{table}

ARK\xspace outperforms all other prior works in terms of $\textbf{T}_{A.S.}$\xspace.
ARK\xspace shows 572$\times$ lower (better) $\textbf{T}_{A.S.}$\xspace than \textbf{100x}, which shows the best performance among the prior works (Table~\ref{tab:amort_performance}).
ARK\xspace shows 6,268$\times$ and 18,520$\times$ lower $\textbf{T}_{A.S.}$\xspace compared to \textbf{Lattigo} and \textbf{F1}, respectively.
\textbf{F1} uses single-slot ($n=1$) bootstrapping for CKKS that is severely limited in throughput, which is why \textbf{F1} shows even worse $\textbf{T}_{A.S.}$\xspace than CPU (\textbf{Lattigo}).

ARK\xspace also outperforms prior works for HELR training.
ARK\xspace shows 4,801$\times$, 100$\times$, and 133$\times$ better performance than \textbf{Lattigo}, \textbf{100x}, and \textbf{F1}, respectively.
\textbf{F1} only reported the execution time for a single iteration with 256 images excluding bootstrapping due to the parameter set's limited size.
We estimated the time required for \textbf{F1} to learn 1024 images and incorporated the cost of bootstrapping by assuming that \textbf{F1} trains 1024 images over 4 iterations
and performs $14\times14\!=\!196$ times of single-slot bootstrapping after each iteration.
We ignored the cost for conversion between a packed ciphertext and 196 single-slot ciphertexts in favor of \textbf{F1}.
The performance gap between ARK\xspace and other works decreases compared to $\textbf{T}_{A.S.}$\xspace, which is due to the reduced slot usage ($n\!=\!256$) in bootstrapping for HELR.
The cost of H-(I)DFT becomes less significant for small $n$ (see Alg.~\ref{alg:fft-like-hdft}), reducing the speedup of ARK\xspace over prior works.

ARK\xspace drastically reduces the execution time for complex workloads: ResNet-20~\cite{arxiv-2021-resnet20} and sorting~\cite{tifs-2021-sorting}  (Table~\ref{tab:workload_performance}).
We used the reported performance of multi-threaded CPU implementation in each work as the baseline.
ARK\xspace shows 36,122$\times$ and 32,511$\times$ speedup compared to the baselines.
In particular, ARK\xspace performs a real-time CNN inference in 0.294s.

\begin{figure}[tb!]
    \centering
    \subfigure[Bootstrapping\label{fig:alg_speedup_a}]{\includegraphics[height=1.45in]{Figure/speedup_bootstrapping.pdf}}
    \hspace{0.02in}
    \subfigure[Other workloads\label{fig:alg_speedup_b}]{\includegraphics[height=1.45in]{Figure/speedup_workload.pdf}}
    \vspace{-0.05in}
    \caption{Execution time and speedup of (a) bootstrapping and (b) other workloads while applying our algorithmic optimizations incrementally.\label{fig:alg_speedup}}
    \vspace{-0.1in}
\end{figure}

\noindent\textbf{Impacts of algorithmic optimizations:}
Our fabrication-technology-aware algorithms improve the overall performance of bootstrapping even with increased computational overhead.
Multi-hop HRot improves the performance of H-IDFT (resp., H-DFT) by 2.66$\times$ (1.72$\times$).
OF-Limb further improves the performance by 1.29$\times$ (1.06$\times$), resulting in an aggregated performance improvement of 3.43$\times$ (1.82$\times$).
The overall performance of bootstrapping is improved by 2.45$\times$.
Applying multi-hop HRot resulted in a performance gain similar in ratio to the increase in ops/byte, whereas the performance improvement by additionally applying OF-Limb falls short of the increase in ops/byte due to the increased computation.

Our algorithmic optimizations show different degrees of performance improvement for each workload.
The portion of bootstrapping in the baseline workloads affects the overall performance improvement ratio.
Bootstrapping occupies only 44.6\% of the baseline HELR, whereas in ResNet-20 and sorting, bootstrapping accounts for a relatively high portion of 76.2\% and 91.9\%, respectively.
For this reason, in ResNet-20 and sorting, speedups of 1.84$\times$ and 2.21 $\times$ are achieved, close to the bootstrapping performance speedup.
In HELR, only 1.26$\times$ of speedup is achieved.

The speedup of ResNet-20 attributes partially to the speedup in the convolution layers of ResNet-20.
We additionally applied multi-hop HRot and OF-Limb to the convolution layers, which resulted in a 1.17$\times$ speedup compared to when only applying to bootstrapping.
However, for HELR and sorting, only OF-Limb was applied besides bootstrapping, showing a minor performance improvement of less than 1\%.
Other than bootstrapping, these workloads do not feature a computation pattern where multi-hop HRot is applicable.

\subsection{Alternative designs of ARK\xspace}
\label{sec:6_4_alternative}

We measured the performance and power of alternative designs of ARK\xspace to assess how each factor of ARK\xspace affects the performance and power.
The results are shown in Fig.~\ref{fig:alternative_design}.
The baseline ARK consumes power ranging from 92W to 139W, 44\% of the peak power in geometric mean (gmean).

\noindent\textbf{Limb-wise-only distribution:}
Using limb-wise-only distribution (see Section~\ref{sec:5_3_coeff_wise_distribution}) causes an overall performance drop.
In the limb-wise-only implementation, the data transfer for accumulation after multiplying the evaluation key is the only data transfer.
We designed an NoC suited for the pattern of this data transfer, which deploys adders inside the NoC to support the on-transit accumulation of data, preventing data concentration.
However, even with this additional optimization, performance in bootstrapping, HELR, ResNet-20, and sorting degrades by 0.77$\times$, 0.87$\times$, 0.68$\times$, and 0.73$\times$, respectively.
Power consumption decreases to 0.79$\times$ of the baseline in gmean as the components became less busy.

\noindent\textbf{More clusters:}
Because ARK\xspace has eliminated the off-chip memory bandwidth bottleneck, populating more computational logic leads to better performance.
Here we double the number of clusters to 8, leading to twice aggregate scratchpad and RF bandwidth while fixing the total scratchpad size to 512MB.
The 8-clustered design can perform bootstrapping 1.44$\times$ faster and thus shows 1.29$\times$ and 1.26$\times$ improved performance in ResNet-20 and sorting, respectively.
Due to the lower portion of bootstrapping in total execution time, performance improvement in HELR is relatively modest with 1.07$\times$ enhancement.
However, this performance improvement comes at the cost of 1.27$\times$ more power consumption on average and 1.39$\times$ larger chip area.
In particular, NoC power increases by 2.68$\times$ on average.
When considering the energy-delay-area product (EDAP)~\cite{isca-2008-cacti} to evaluate efficiency, the 8-clustered design shows 1.11$\times$ higher EDAP, suggesting that the baseline ARK\xspace is more efficient.

\noindent\textbf{Doubling HBM bandwidth:}
Because the proposed multi-hop HRot and OF-Limb greatly diminish the off-chip memory bandwidth bottleneck, doubling the off-chip memory bandwidth to 2TB/s has little impact on the performance of bootstrapping, which improves by 1.07$\times$.
ResNet-20 and sorting, the performance of which highly depends on bootstrapping, also show a similar degree of performance improvements by $1.08\times$ and $1.07\times$, respectively.
By contrast, HELR has memory-bound parts where multiple HRots must be performed, and multi-hop HRot is not applicable as their rotation amounts do not show arithmetic progression.
Multiple $\textbf{evk}_{\text{rot}}^{(r)}$s need to be loaded from memory, so the design with doubled HBM bandwidth shows 1.58$\times$ better performance for HELR.

\begin{figure}[tb!]
    \subfigure[Time - boot\label{fig:alternative_design_a}]{\includegraphics[height=1.34in]{Figure/speedup_alt_bootstrapping.pdf}}
    \hspace{0.01in}
    \subfigure[Time - other workloads\label{fig:alternative_design_b}]{\includegraphics[height=1.34in]{Figure/speedup_alt_workload.pdf}}
    \vskip\baselineskip\vspace{-0.22in}
    \subfigure[Power - boot\label{fig:alternative_design_c}]{\includegraphics[height=1.23in]{Figure/power_alt_bootstrapping.pdf}}
    \hspace{0.01in}
    \subfigure[Power - other workloads\label{fig:alternative_design_d}]{\includegraphics[height=1.23in]{Figure/power_alt_workload.pdf}}
    \vspace{-0.05in}
    \caption{Comparison of execution time and average power, while executing (a)(c) bootstrapping and (b)(d) other workloads, with alternative designs of ARK\xspace, including the one using only limb-wise distribution (limb-wise-only), an 8-clustered design (8 clusters), and the other with doubled HBM bandwidth (2TB/s bandwidth).\label{fig:alternative_design}}
    \vspace{-0.10in}
\end{figure}

\begin{figure}[tb!]
    \centering
    \includegraphics[height=1.1in]{Figure/BConvLane.pdf}
    \label{fig:bconv_sweep}
    \vspace{-0.05in}
    \caption{Performance of bootstrapping with various numbers of MAC units per BConv lane.}
    \vspace{-0.1in}
\end{figure}

\noindent\textbf{Organization of a BConv lane:}
As the number of MAC units in a BConv lane increases, the performance of bootstrapping improves to a certain level and then saturates.
Increasing the number of MAC units from one to six leads to $1.78\times$ higher performance (see Fig.~\ref{fig:bconv_sweep}).
After six, the performance improvement saturates with less than 1\% enhancement.

\section{Introduction}
\label{sec:1_introduction}

Homomorphic encryption (HE) is a cryptographic method that enables third parties to process data blindly.
HE is regarded as a game-changer in privacy-preserving cloud computing because HE does not leak information about the data to anyone other than the owner, allowing secure offloading of computation to untrusted servers.

HE schemes are based on the learning with errors~\cite{jacm-2009-lwe} paradigm, a computational problem using noise for security.
Noise accumulates as we continuously apply computation to encrypted data, which can eventually explode and make the data impossible to decrypt.
Leveled HE (LHE) schemes only allow a restricted number of operations (ops) to prevent noise explosion, thus only supporting simple workloads.~\cite{siam-2014-bfv1, iacr-2015-guide}.
An HE op called \emph{bootstrapping} can be used to lower the noise level and allow an unbounded number of ops over encrypted data.
Due to its significance, this class of HE schemes capable of bootstrapping is referred to as \emph{fully HE} (FHE)~\cite{stoc-2009-gentry-fhe}.
Among numerous FHE schemes~\cite{iacr-2012-bfv2, siam-2014-bfv1, toct-2014-bgv, jc-2020-tfhe, 2015-eurocrypt-fhew}, CKKS~\cite{asia-2017-ckks}, our primary target FHE scheme, is gaining popularity because it supports real arithmetic and is thus more suitable for machine learning (ML) services.

Despite its potential, however, the high memory and computational costs of HE are preventing its wide adoption.
When data is encrypted using an HE scheme, the size expands by more than an order of magnitude, and an op as simple as multiplication transforms into a complex sequence of ops.
Prior works have tried to mitigate such overhead by using CPU extensions~\cite{access-2021-demystify, wahc-2021-hexl}, GPUs~\cite{tches-2021-100x, access-2020-privft, tetc-2019-bfv, tches-2018-fv}, FPGAs~\cite{asplos-2020-heax, hpca-2019-roy, fccm-2020-sunwoong-ntt, reconfig-2019-sunwoong-modmult}, and ASICs~\cite{micro-2021-f1, hpca-2021-cheetah}.
Recent proposals suggesting accelerated ASIC/FPGA designs, F1~\cite{micro-2021-f1} in particular, have proven successful for LHE schemes.

However, the parameters used in these designs were not large enough to effectively support CKKS bootstrapping.
Bootstrapping is an expensive op, dominating the execution time of complex FHE workloads~\cite{arxiv-2021-resnet20, tches-2021-100x}.
Moreover, FHE schemes supporting bootstrapping must use large parameters~\cite{iacr-2021-cheon-practical, eurocrypt-2021-efficient}, which correspondingly increase the data size.
In fact, the working set of bootstrapping can reach several GBs, which must be fetched from off-chip memory due to limited on-chip memory capacity~\cite{authorea-2022-h100, vlsit-2018-samsung}.
Given such constraints of FHE schemes, previous hardware accelerators are severely bottlenecked by the low off-chip memory bandwidth.

In this work, we present a set of algorithmic innovations that addresses the off-chip memory bandwidth bottleneck of FHE workloads: \emph{multi-hop HRot} and \emph{on-the-fly limb extension}.
The key benefits of our proposed algorithms are twofold. First, our proposed algorithms drastically reduce the memory bottleneck of FHE, eliminating 88\% of off-chip memory access
in evaluating a major op composing bootstrapping. Second, our proposal enables
the deployment of a massive amount of computational logic and on-chip memory, large enough to hold the reduced working set on-chip, a feature lacking in conventional systems.

We design ARK\xspace, an FHE accelerator that effectively exploits the increased arithmetic intensity enabled by our proposed algorithmic innovations, addressing new computational and data movement requirements arising from FHE support.
ARK\xspace features novel functional units (FUs) tailored to the primary functions of FHE, including base conversion (BConv), number-theoretic transform (NTT), and automorphism.
The FUs lower the bandwidth pressure on register files (RFs) and minimize data movement.
For seamless operation across the FUs, ARK\xspace utilizes a data distribution policy that adapts to two data access patterns of the primary functions and simplifies the dataflow by splitting the chip into two logical regions.

By co-designing the algorithms and the architecture, ARK\xspace outperforms the state-of-the-art HE accelerator~\cite{micro-2021-f1} in multiplicative throughput~\cite{tches-2021-100x} by 18,520$\times$ and in logistic regression training~\cite{aaai-2019-helr} by 133$\times$.
ARK\xspace also enables a real-time CNN inference on encrypted data; ARK\xspace performs an inference with the ResNet-20 model in 0.294 seconds, 36,122$\times$ faster than the baseline CPU implementation~\cite{arxiv-2021-resnet20}.
ARK\xspace is sized 418.3mm\textsuperscript{2} and consumes up to 281.3W of power.

In summary, our major contributions are as follows:
\begin{itemize}[noitemsep,leftmargin=0.135in]
  \item We analyze CKKS bootstrapping in detail and identify the memory bottleneck in hardware acceleration of FHE induced by excessive single-use data during bootstrapping.
  \item We propose two novel algorithms to mitigate the memory bottleneck of FHE by substantially reducing the single-use data in bootstrapping.
  \item We design ARK\xspace, an architecture-algorithm co-designed accelerator for FHE. We deploy specialized FUs that minimize on-chip data movement on ARK\xspace. ARK\xspace adapts data and job distribution based on the data access patterns and fixes dataflow for streamlined execution across the chip.
\end{itemize}

\section{Related Work}
\label{sec:7_related}
\noindent\textbf{CPU/GPU acceleration of HE:} Prior works have tried to accelerate HE by better use of general-purpose processors: CPUs and GPUs.
\cite{access-2021-demystify} and \cite{wahc-2021-hexl} accelerate HE using CPU SIMD extensions.
Meanwhile, Lattigo~\cite{github-lattigo} exploits the latest algorithmic optimizations~\cite{eurocrypt-2021-efficient, crypto-2018-linear} to accelerate HE ops.
However, the relatively low computational power of CPUs has resulted in limited success.
To overcome the limitations of CPUs, prior works~\cite{tches-2021-100x,access-2020-privft,tetc-2019-bfv,tches-2018-fv} have attempted to accelerate HE by utilizing the high parallel computing power of GPUs.
However, most prior GPU works~\cite{access-2020-privft,tetc-2019-bfv,tches-2018-fv} accelerate part of primitive HE ops or do not include acceleration of bootstrapping.
100x~\cite{tches-2021-100x} accelerates all HE ops including CKKS bootstrapping.
100x mitigates the off-chip memory bottleneck in a GPU by performing kernel fusion, which leads to $242\times$ faster bootstrapping over a CPU.
However, the limited on-chip storage capacity of the GPU hinders kernel fusion for some functions~\cite{iiswc-2020-ntt}.
Yet there are other GPU acceleration works~\cite{github-cufhe, github-nufhe, host-2020-cpuandgpufhe, asap-2017-gpufhew} targeting boolean HE schemes~\cite{jc-2020-tfhe, 2015-eurocrypt-fhew}.

\noindent\textbf{FPGA/ASIC designs for HE:}
Prior works have implemented hardware logic for HE ops using FPGAs~\cite{asplos-2020-heax, hpca-2019-roy, fccm-2020-sunwoong-ntt, reconfig-2019-sunwoong-modmult} and ASICs~\cite{hpca-2021-cheetah, micro-2021-f1}.
However, they either do not cover entire HE ops~\cite{fccm-2020-sunwoong-ntt, reconfig-2019-sunwoong-modmult}, target LHE~\cite{asplos-2020-heax,hpca-2019-roy, hpca-2021-cheetah}, or is limited in CKKS bootstrapping throughput~\cite{micro-2021-f1}.
HEAX~\cite{asplos-2020-heax} shows the best performance among FPGA CKKS implementations by accelerating HMult with specialized NTT cores and deep pipelines.
However,
it only supports non-bootstrappable, limited parameters.
Cheetah~\cite{hpca-2021-cheetah} targets privacy-preserving ML using LHE, based on the implementation of Gazelle~\cite{usenixsec-2018-gazelle}, and designs an ASIC accelerator for such workloads.
Gazelle performs ML inference by combining LHE and multi-party computation (MPC)~\cite{sfcs-1986-garbled}.
Such implementation requires frequent server-client communication and needs re-encryption to support complex workloads inducing expensive network overheads.
F1~\cite{micro-2021-f1} is the first ASIC work that reports the performance of CKKS bootstrapping.
By using 64MB of on-chip memory and high throughput FUs, including NTTU, F1 improves the performance of primitive HE ops; however, it does not target the parameters suitable for practical bootstrapping.

redpajama_set_name: RedPajamaArXiv


text:
Pleurota modestella is een vlinder uit de familie van de sikkelmotten (Oecophoridae). De wetenschappelijke naam van de soort is voor het eerst geldig gepubliceerd in 1916 door Rebel.

modestella

redpajama_set_name: RedPajamaWikipedia


Beijing Jingcheng Machinery Electric Co Ltd operates in the Printing trades machinery sector. In addition to historical fundamental analyses, the complete report available to purchase compares Beijing Jingcheng Machinery Electric Co Ltd with three other paper industry machinery manufacturers in Asia: Daejin Dmp Co., Limited. of South Korea (2018 sales of 100.15 billion Korean Won [US$88.24 million] ), Tokyo Kikai Seisakusho, Ltd. of Japan (13.19 billion Japanese Yen [US$117.75 million] ), and Mutoh Holdings Company Limited which is also based in Japan (20.05 billion Japanese Yen [US$179.03 million] of which 31% was Information image related equipment-Asia).
Sales Analysis. Beijing Jingcheng Machinery Electric Co Ltd reported sales of 1.12 billion Chinese Renmimbi (US$167.17 million) for the year ending December of 2018. This represents a decrease of 6.8% versus 2017, when the company's sales were 1.20 billion Chinese Renmimbi.