text:
\section*{Acknowledgments}
\label{sec:acknowledgment}
This work was performed in the framework of the Horizon 2020 program for the project ``Near-Memory Computing (NeMeCo)''. It is funded by the European Commission under Marie Sklodowska-Curie Innovative Training Networks European Industrial Doctorate (Project ID: 676240). Special thanks to Florian Auernhammer and Raphael Polig for providing support with the IBM systems. We appreciate valuable discussions with Kaan Kara and Ronald Luijten. \gagannn{We would also like to thank Bruno Mesnet and Alexandre Castellane from IBM France for help with the SNAP and OC-Accel framework.} This work was partially supported by the H2020 research and innovation programme under grant agreement No 732631, project OPRECOMP.
\juang{We also thank {Google, Huawei, Intel, Microsoft, SRC, and VMware} for their funding support \om{to the SAFARI Research Group}.}
\bibliographystyle{ACM-Reference-Format}

\section{Background}
\label{sec:background}
\juang{In this section, we first provide}
an overview of the \juang{\texttt{vadvc} and \texttt{hdiff}} {compound stencils,
\juang{which represent} a large fraction of the overall computational load of the COSMO \gagannn{weather prediction} model}.
\gtrets{Second, we introduce the OC-Accel \gagan{(OpenCAPI Acceleration)} framework} that we use to connect our \namePaper~accelerator to an IBM POWER9~system.

\subsection{\juangg{Representative} COSMO Stencils}
A stencil operation updates values in a structured multidimensional grid based on the values of a fixed local neighborhood of grid points. \gtrets{ In weather and climate modeling, several stencil operations are compounded together that operate on multiple input elements to generate one or more output results.}
Vertical advection (\texttt{vadvc}) and horizontal diffusion (\texttt{hdiff}) from the COSMO model are two such compound \juang{stencil} kernels, which represent the typical code patterns found in the \emph{dycore} of COSMO.
Algorithm~\ref{algo:hdiffKernel} shows \gagan{the} pseudo-code for \texttt{vadvc} and \texttt{hdiff}  kernels.
The horizontal diffusion kernel iterates over a 3D grid, performing \textit{Laplacian} and \textit{flux} to calculate different grid points, as shown in Figure~\ref{fig:hdiffMemory}.
\gtrets{A single \textit{Laplacian} stencil accesses
the input grid at five memory offsets, the result of which is used to calculate the \textit{flux} \gonur{stencil}. \hdiff has purely horizontal access patterns and does not have dependencies in the vertical dimension. {Thus,} it can be fully parallelized in the vertical dimension. Figure~\ref{fig:hdiffMemory_layout} shows the memory layout for the horizontal diffusion kernel. We observe that the indirect memory accesses of the input grid domain can {severely} impact cache efficiency on our current CPU-based systems.}
\input{images/hdiff}
\gtrets{Vertical advection has a higher degree of complexity since it uses the Thomas algorithm~\cite{thomas} to solve a tridiagonal matrix of \gonur{weather data (called \emph{fields}, such as air pressure, wind velocity, and temperature)} along the vertical axis. \vadvc consists of a forward sweep that is followed by a backward sweep along the vertical dimension. \vadvc requires access to \gonur{the weather data,} which are stored as array structures while performing forward and sweep computations.
Unlike the conventional stencil kernels, vertical advection has dependencies in the vertical direction, which leads to limited available parallelism and irregular memory access patterns.  For example, when the input grid is stored by \emph{row}, accessing data elements in the \emph{depth} dimension typically results in {many} cache {misses}~\cite{xu2018performance}. }

\input{algorithms/cosmo.tex}

Such compound kernels are dominated by memory-bound operations with complex memory access patterns and low arithmetic intensity. This poses a fundamental challenge to acceleration. \juanggg{CPU implementations of these kernels~\cite{xu2018performance} suffer from limited data locality and inefficient memory usage, as our roofline analysis in Figure~\ref{fig:roofline} exposes}.
\gtrets{In Figure~\ref{fig:copy_stencil} we implement a \emph{copy stencil} from the COSMO weather model to evaluate the performance potential of our HBM-based FPGA platform for the weather prediction application. A \emph{\cop stencil} performs an element-wise copy operation over the complete input grid. It is the simplest stencil found in the COSMO model and, hence, serves as a benchmark to characterize the achievable peak performance on a platform for weather kernels. {To implement a \cop stencil, we divide the 3D grid data among the processing elements (PEs), where each PE performs an element-wise copy operation.} We were able to enable only 24 HBM memory channels because adding more HBM channels leads to {timing constraint violations}. From the figure, we make two observations. First, as we increase the number of processing elements (PEs), we can exploit data-level parallelism because of dedicated HBM channels serving data to a PE. Second, the maximum achievable performance tends to saturate after 16 PEs. Since we implement our design in a dataflow pipeline manner, all the functions run in parallel, and the overall latency is equal to the maximum latency out of all the functions. After 16 PEs, for \cop, \grev{we observe that most of the time is spent in the FPGA \gcameraa{computation} logic rather than the transfer of data from an HBM memory channel.   }}

\begin{figure}[h]
  \includegraphics[width=0.45\linewidth,trim={0cm 0cm 0cm 0cm},clip]{images/copy_stencil.pdf}
  \vspace{-11pt}
     \caption{  \gtrets{Performance of copy stencil on our HBM-based FPGA platform.}   \label{fig:copy_stencil}}
\end{figure}

\subsection{OC-Accel {Framework}}
The OpenPOWER Foundation Accelerator Workgroup~\cite{open_power} created the OC-Accel framework,
\juanggg{an open-source environment for FPGA programming productivity}.
OC-Accel \gagan{provides three key benefits}~\cite{wenzel2018getting}: (i) it enables an improved developer productivity for FPGA acceleration and eas\gagan{es the} use of CAPI's cache-coheren\gagan{ce mechanism},  (ii) it places
\juanggg{FPGA-}accelerated compute engines,
\juanggg{also known as} FPGA \textit{actions}, closer to \gagan{relevant} data to achieve better performance, and \gtrets{(iii) access to FPGA memory via user-level DMA (Direct Memory Access) semantics.} OC-Accel provides a simple API to invoke an accelerated \emph{action} and provides programming methods to instantiate customized accelerated \emph{actions} on the FPGA side.  These accelerated \emph{actions} can be specified in C/C++ code that is then compiled to the FPGA target using the Xilinx Vivado High-Level Synthesis (HLS) tool~\cite{hls}.

\gtrets{The benefits of employing such cache-coherent interconnect links for attaching FPGAs to CPUs, as opposed to the traditional DMA-like communication protocols (e.g., PCIe), are not only the ultra lower-latency and the higher bandwidth of the communication, but most importantly, the ability of the accelerator to access the entire memory space of the CPU coherently, without consuming excessive CPU cycles. Traditionally, the host processor has a shared memory space across its cores with coherent caches. Attached devices such as FPGAs, GPUs, network, and storage controllers are memory-mapped and use a DMA to transfer data between local and system memory across an interconnect such as PCIe. The attached devices can not see the entire system memory but only a part of it. Communication between the host processor and attached devices requires an inefficient software stack, including user-space software, drivers, and kernel-space modules, in comparison to the communication scheme between CPU cores using shared memory. Especially when DRAM memory bandwidth becomes a constraint, requiring extra memory-to-memory copies to move data from one address space to another is cumbersome \om{and low performance}~\cite{seshadri2013rowclone,Fang2020}. This is the driving force of the industry to push for coherency and shared memory across CPU cores and attached devices, like FPGAs. This way, the accelerators act as peers to the processor cores. \grev{Note that CAPI2 is built on top of PCIe. However, CAPI2 provides the following two advantages. First,  a CAPI-attached device, unlike a PCIe device, can perform Direct Memory Access (DMA) to application memory \emph{without} calls to a device driver or underlying operating system kernel, resulting in a reduction in latency. Avoiding these unnecessary memory calls improves performance significantly compared to the traditional PCIe I/O model~\cite{stuecheli2015capi}. Second, CAPI2 provides cache-coherent access to the CPU memory, allowing the FPGA to \emph{directly} access the host memory. Such direct cache-coherent access reduces the FPGA developer's burden and debugging time by overcoming read after write (RAW) and write and read (WAR) dependencies, which is typically the FPGA developer's responsibility. OCAPI is a new technology built from the ground up. It includes a faster PHY layer \gcameraa{(BlueLink 25Gb/s x8 lanes~\cite{openCAPI}) } than its CAPI predecessors, providing double the bitwidth between the host and an accelerator.}}

\section{Conclusion}
\label{sec:conclusion}
\juang{We introduce} \namePaper, the first design and implementation on a reconfigurable fabric \juang{with \gagann{high-bandwidth memory} (HBM)} to accelerate representative weather prediction kernels, i.e., vertical advection (\texttt{vadvc}) and horizontal diffusion (\texttt{hdiff}), from a real-world weather prediction application.
These kernels are compound stencils that are found in various weather prediction applications, including the COSMO  model. \gagannn{We show that c}ompound kernels do not perform well on conventional architectures due to their complex data access patterns and low data reusability, which \gagann{make} them memory-bounded.
\juang{Therefore, they greatly benefit from our near-memory computing solution \gagannn{that} takes advantage of the high \gagann{data transfer} bandwidth of HBM.} \gtrets{We use a heterogeneous system comprising of IBM POWER9 CPU with field-programmable gate array (FPGA) as our target platform.
We create a heterogeneous domain-specific memory hierarchy using on-chip {URAM}s and  {BRAM}s, \gcamera{and on-package HBM} on an FPGA. Unlike \om{conventional} fixed CPU memory hierarchies, which perform poorly with irregular access patterns and suffer from cache pollution effects, application-specific memory hierarchies are shown to improve \om{both} energy and latency by tailoring the cache levels and cache sizes to an application's memory access patterns.  }

\juang{\namePaper's implementations of \texttt{vadvc} and \texttt{hdiff} outperform the optimized software implementations on a 16-core POWER9 with \gagann{4-way multithreading} by $5.3\times$ and $12.7\times$, with $12\times$ and $35\times$ less energy consumption, respectively.
We conclude that hardware acceleration on \gagann{an} FPGA+HBM fabric is a promising solution for compound stencil\gagann{s} present in weather prediction applications. \gagann{We hope that o}ur \gagann{reconfigurable near\gagannn{-}memory} accelerator inspire\gagann{s} developers of different high-performance computing applications that suffer from \gagann{the} memory bottleneck.
}

\section{Results}
\label{sec:results}

\subsection{Experimental Setup}
{We evaluate our accelerator designs for  \vadvc, and \hdiff  in terms of performance, energy consumption, and FPGA resource utilization on two different FPGAs, and two different external data communication interfaces between the CPU and the FPGA board.}
We implement our accelerator designs for \vadvc, and \hdiff
on  \sr{both 1)} \gonur{an} Alpha-Data ADM-PCIE-9H7 card~\garxiv{\cite{ad9h7} }featuring the Xilinx Virtex Ultrascale+ XCVU37P-FSVH2892-2-e~\garxiv{\cite{vu37p}} with 8GiB HBM2~\cite{hbm} \sr{and 2)} \gonur{an} Alpha-Data ADM-PCIE-9V3 card~\garxiv{\cite{ad9v3}} featuring the Xilinx Virtex Ultrascale+ XCVU3P-FFVC1517-2-i with 8GiB DDR4~\garxiv{\cite{vu37p}}, connected to an IBM POWER9 host system. For the external data communication interface, we use both CAPI2~\garxiv{\cite{stuecheli2015capi}} and the state-of-the-art OCAPI (OpenCAPI)~\garxiv{\cite{openCAPI}} interface.
We compare these implementations to execution on a POWER9 CPU with 16 cores \juan{({using all} 64 hardware threads).}
Table~\ref{tab:systemparameters}
provides our system parameters.
\grev{We co-design our hardware and software interface around the OC-Accel framework~\cite{ocaccel} while using the HLS design flow. Our development machine is x86 Intel\textsuperscript{\sffamily\textregistered} Xeon\textsuperscript{\sffamily\textregistered
7.9.2009~\cite{centos} distribution with GNU Compiler Collection (GCC) version 4.8.5~\cite{gnu}. We use Xilinx Vivado 2019.2~\cite{vivado} suite to develop our accelerator designs.  }
  \input{tables/system.tex}
\subsection{Performance Tuning}

We
\juanggg{run} our experiments using a $256\times256\times64$-point domain similar to the grid domain used by the COSMO \gagannn{weather prediction} model.  We employ an auto-tuning technique to determine a Pareto-optimal solution (in terms of performance and resource utilization) for our 3D window dimensions.
{The auto-tuning \juanggg{with OpenTuner} exhaustively searches for every tile size in the x- and y-dimensions for \texttt{vadvc}.\footnote{\texttt{vadvc} has dependencies in \gagann{the} z-dimension; therefore, it cannot be parallelized in the z-dimension.} For \texttt{hdiff}, we consider sizes in all three dimensions. We define our auto-tuning as a multi-objective optimization with the goal \gagannn{of maximizing} performance with  minimal resource utilization.} Section~\ref{sec:design} provides further details on our design. \grev{We evaluate frequency values between 50-400 MHz, with an increment of 50MHz, utilizing the complete spectrum of compatible frequency configurations supported by the OC-Accel framework~\cite{ocaccel}. }
Figure~\ref{fig:single_afu} shows hand-tuned and auto-tuned \gagann{performance and FPGA resource utilization} results for {\texttt{vadvc}}\gagann{, as a function of the chosen tile size. From the figure, we draw two observations.}

 \begin{figure}[h]
  \centering
  \includegraphics[width=1\linewidth,trim={0.4cm 0.3cm 0.35cm 0.2cm},clip]{images/singleAFU.pdf}
  \vspace{-10pt}
      \caption{Performance \gagann{and FPGA resource utilization} \gagannn{of} single \texttt{vadvc} PE\gagann{, as a function of tile-size,} using hand-tuning and auto-tuning for (a) single-precision (32-bit) and (b) half-precision (16-bit). We highlight the Pareto-optimal solution that we use for our \texttt{vadvc} accelerator \gagan{(with a red circle). Note that }the Pareto-optimal solution changes with precision.
  \label{fig:single_afu}}
 \end{figure}

First, by using the auto-tuning approach and our careful FPGA microarchitecture design, we can get \gagan{P}areto-optimal results with a tile size of $64\times2\times64$ for single-precision {\texttt{vadvc}}\gagan{, which} gives us a peak performance of 8.49 GFLOP/s. For half-precision, we use a tile size of $32\times16\times64$ to achieve a peak performance of 16.5 GFLOP/s. We employ a similar strategy for {\texttt{hdiff}} to attain a single-precision performance of 30.3 GFLOP/s with a tile size of $16\times64\times8$ and a \gagan{half-precision} performance of 77.8 GFLOP/s \gagan{with} a tile size of $64\times8\times64$.

Second, in FPGA acceleration, designers usually rely on expert judgement to find the appropriate tile-size and often adapt the design to \gagannn{use} homogeneous tile sizes. However, as shown in Figure~\ref{fig:single_afu}\gagan{,} such hand-tuned implementations lead to sub-optimal results in terms of either resource utilization~or~performance.

\gagann{We conclude that} the \gagann{Pareto-optimal} tile size depends on the data precision used\gagan{:} a \gagan{good} tile-size for single-precision might lead to poor results when used with half-precision.
\subsection{Performance Analysis}
\label{subsection:evaluation}

Figure~\ref{fig:perf} shows \gagann{single-precision} performance results for the (a) vertical advection \juann{(\texttt{vadvc})} and (b) horizontal diffusion kernels \juann{(\texttt{hdiff})}.
For both kernels, we implement our design \juanggg{on} an HBM- and \juanggg{a} DDR4-based FPGA board.
\juann{For the DDR4-based design, we use CAPI2 (\texttt{DDR4+CAPI2} in Figure~\ref{fig:perf}.
For the HBM-based design, we use CAPI2 (\texttt{HBM+CAPI2}) and OCAPI. We evaluate two versions of the HBM-based design with OCAPI: (1) one with a single channel per PE (\texttt{HBM+OCAPI}), and (2) one with multiple channels (i.e., 4 HBM pseudo channels) per PE (\texttt{HBM\_multi+OCAPI}).}
To compare the performance
\juann{of these four versions}, we scale the number of PEs and analyze the change in execution time.  \gtrets{ We also tested different domain sizes, varying from $64\times64\times64$-point to $1024\times1024\times64$-point and observe that the runtime scales linearly and the overall performance (GLOP/s) remain constant. This shows the scalability of our accelerator design. }

 \begin{figure}[h]
  \centering
  \includegraphics[width=1\linewidth,trim={13.4cm 0.3cm 0.35cm 0.4cm},clip]{images/TRETS_perf_common_abc_ocapi_multi_update_b_w.pdf}
  \vspace{-10pt}
   \caption{ \gagann{Single-precision} performance for (a) \texttt{vadvc} and (b) \texttt{hdiff}\gagann{, as a function of accelerator PE count on the HBM- and DDR4-based FPGA boards}.
   \gagan{We also show \gagann{the}} single socket (64 threads) performance of \gagan{an} IBM POWER9 host system for both \texttt{vadvc} and \texttt{hdiff}. \gtrets{For HBM-based design, we implement our accelerator with \gonur{both the} CAPI2 interface and the state-of-the-art OpenCAPI (OCAPI) interface (with both single channel and multiple \sr{channels} per PE).}
  \label{fig:perf}}
 \end{figure}

We draw \juann{five} observations from the figure.

\juann{First, the maximum number of PEs that we can fit on the FPGA boards varies for different versions of our design.
For the DDR4-based design, we can accommodate only 4 PEs/8 PEs for \texttt{vadvc}/\texttt{hdiff} on the 9V3 board.
For the HBM-based design, we can fit 14 PEs/16 PEs for \texttt{vadvc}/\texttt{hdiff} for both \texttt{HBM+CAPI2} and \texttt{HBM+OCAPI} versions before exhausting the on-board resources.
The \texttt{HBM\_multi+OCAPI} version can only fit 3 PEs (i.e., 12 HBM channels) for both \texttt{vadvc} and \texttt{hdiff} because adding more HBM channels leads to timing constraint violations.}

\juann{Second, the full-blown \texttt{HBM+OCAPI} versions (i.e., with the maximum number of PEs) of \texttt{vadvc} and \texttt{hdiff} outperform the 64-thread IBM POWER9 CPU version by $5.3\times$, and $12.7\times$, respectively. We achieve 37\% and 44\% higher performance for \texttt{vadvc} and \texttt{hdiff}, respectively, with \texttt{HBM+OCAPI} than \texttt{HBM+CAPI2} due to the following two reasons:
(1) OCAPI provides double the bitwidth (1024-bit) of the CAPI2 interface (512-bit), which provides a higher bandwidth to the host CPU, i.e., 22.1/22.0 GB/s R/W versus 13.9/14.0 GB/s; and
(2) with OCAPI, memory coherency logic is moved onto the IBM POWER CPU, which provides more FPGA area and allows us to run our accelerator logic at a higher clock frequency (250MHz for OCAPI versus 200MHz for CAPI2).} \grev{We observe that when implementing our accelerator designs with targets above those frequencies, the respective timing report results in the worst negative slack (WNS) higher than 200ps, which OC-Accel developers regard as dangerous for system stability. At lower frequencies, we achieved lower performance, regardless of the number of PEs.}  \gtrets{ Our single-precision \texttt{HBM+OCAPI}-based FPGA implementations {provide} 157.1~GFLOP/s and 608.4~GFLOP/s for \vadvc and \hdiff, respectively. For half-precision, if we use
the same amount of {PE}s as in single precision, {our accelerator}
{reaches} a performance of 329.9~GFLOP/s for \texttt{vadvc} ($2.1\times$ {the} single-precision performance) and 1.5~TFLOP/s for \texttt{hdiff} ($2.5\times$ {the} single-precision performance).}

\juann{Third, for a single PE, \texttt{DDR4-CAPI2} is faster than \texttt{HBM-CAPI2} for both \texttt{vadvc} and \texttt{hdiff}. This  higher performance is because the HBM-based design uses one HBM channel per PE, and the bus width of the DDR4 channel (512 bits) is larger than that of an HBM channel (256 bits).
Therefore, the HBM channel has a lower transfer rate of 0.8-2.1 GT/s (Gigatransfers per second) than a DDR4 channel (2.1-4.3 GT/s), resulting in a theoretical bandwidth of 12.8 GB/s and 25.6 GB/s per channel, respectively.
One way to match the DDR4 bus width is to have a single PE fetch data from multiple HBM channels in parallel. In Figure~\ref{fig:perf}, our multi-channel setting (\texttt{HBM\_multi+OCAPI}) uses 4 HBM pseudo channels per PE to match the bitwidth of the OCAPI interface.
We observe that by fetching more data from multiple channels, compared to the single-channel-single PE design (\texttt{HBM+OCAPI}), \texttt{HBM\_multi+OCAPI} achieves $1.2\times$ and $1.8\times$ performance improvement for \texttt{vadvc} and \texttt{hdiff}, respectively.}

\juann{Fourth, as we increase the number of PEs, we divide the workload evenly across PEs.
As a result, we observe linear scaling in the performance of HBM-based designs, where each PE reads and writes through a dedicated HBM channel.
For multi-channel designs, we observe that the best-performing multi-channel-single PE design (i.e., using 3 PEs with 12 HBM channels for both workloads) has $4.7\times$ and $3.1\times$ lower performance than the best-performing single-channel-single PE design (i.e., 14 PEs for \texttt{vadvc} and 16 PEs for \texttt{hdiff}, respectively).
This observation shows that there is a tradeoff between (1)~enabling more HBM pseudo channels to provide each PE with more bandwidth, and (2)~implementing more PEs in the available area.
For both \texttt{vadvc} and \texttt{hdiff}, data transfer and computation take a comparable amount of time. Therefore, we are able to achieve a linear execution time reduction with the number of PEs.}

\juann{Fifth, the performance of the DDR-based designs scales non-linearly for \texttt{vadvc} and \texttt{hdiff} with the number of PEs, as all PEs access memory through the same channel.
Multiple PEs compete for a single memory channel, which causes frequent memory stalls due to contention in the memory channel.}

\subsection{Energy Efficiency Analysis}
We compare the energy consumption of our accelerator to \gagan{a} \juanggg{16-core} POWER9 host system.  We use the AMESTER\footnote{https://github.com/open-power/amester} tool to measure the active power\footnote{Active power denotes the difference between the total power of a complete socket (including CPU, memory, fans, I/O, etc.) when an application is running compared to when it is idle.} consumption.
\juanggg{We measure}~99.2~Watt\gagan{s} for \texttt{vadvc} and ~97.9~Watt\gagan{s} for \texttt{hdiff} by monitoring \gagannn{built} power sensors in
\juanggg{the POWER9}~system.
\grev{For {\texttt{vadvc}} and {\texttt{hdiff}}
\juanggg{on the HBM-} and DDR4-based designs, Figure~\ref{fig:power_consumption} and Figure~\ref{fig:energy_eff} shows the active power consumption and the energy efficiency (GFLOPS per Watt), respectively.}

We make \juann{five} observations from Figure~\ref{fig:power_consumption} and Figure~\ref{fig:energy_eff}.

 \begin{figure}[h]
  \centering
  \includegraphics[width=1\linewidth,trim={0cm 0cm 0cm 0cm},clip]{images/power_consumption_b_W.pdf}
  \vspace{-10pt}
     \caption{Active Power Consumption for (a) \texttt{vadvc} and (b) \texttt{hdiff} on HBM- and DDR4-based FPGA boards. \gtrets{For HBM-based design, we implement our accelerator with \gonur{both the} CAPI2 interface and the state-of-the-art OpenCAPI (OCAPI) interface (with both single channel and multiple \sr{channels} per PE).}

  \label{fig:power_consumption}}
 \end{figure}
 \begin{figure}[h]
  \centering
  \includegraphics[width=1\linewidth,trim={13.4cm 0.3cm 0.35cm 0.4cm},clip]{images/TRETS_energyEfficiency_common_efg_ocapi_multi_3_b_w.pdf}
  \vspace{-10pt}
     \caption{Energy efficiency for (a) \texttt{vadvc} and (b) \texttt{hdiff} on HBM- and DDR4-based FPGA boards. \gagann{We also show \gagann{the}} single socket (64 threads) energy efficiency of \gagan{an} IBM POWER9 host system for both \texttt{vadvc} and \texttt{hdiff}. \gtrets{For HBM-based design, we implement our accelerator with \gonur{both the} CAPI2 interface and the state-of-the-art OpenCAPI (OCAPI) interface (with both single channel and multiple \sr{channels} per PE).}

  \label{fig:energy_eff}}
 \end{figure}

\juann{First, the full-blown \texttt{HBM+OCAPI} designs (i.e., 14 PEs for \texttt{vadvc} and 16 PEs for \texttt{hdiff}) achieve energy efficiency values of 1.61~GFLOPS/Watt and 21.01~GFLOPS/Watt for \texttt{vadvc} and \texttt{hdiff}, respectively.
These represent improvements of $12\times$ and $35\times$ compared to the IBM POWER9 system for \texttt{vadvc} and \texttt{hdiff}, respectively.}

\juann{Second, the \texttt{DDR4-CAPI2} designs for \texttt{vadvc} and \texttt{hdiff} are slightly more energy-efficient ($1.1\times$ to $1.5\times$) than the \texttt{HBM-CAPI2} designs when the number of PEs is small.
This observation is in line with our discussion about performance with small PE counts in Section~\ref{subsection:evaluation}.
However, as we increase the number of PEs, the \texttt{HBM-CAPI2} designs provide higher energy efficiency since they make use of multiple HBM channels.}

\juann{Third, the energy efficiency of the HBM-based designs (\texttt{HBM+CAPI2}, \texttt{HBM+OCAPI}) for \texttt{hdiff} increases with the number of PEs until a saturation point (8 PEs).
This trend is because every additional HBM channel increases power consumption by $\sim$1 Watt (for the HBM AXI3 interface operating at 250MHz with a logic toggle rate of $\sim$12.5\%).}

\juann{Fourth, \texttt{HBM+OCAPI}, \texttt{HBM+CAPI2}, and \texttt{DDR4+CAPI2} versions of \texttt{vadvc} achieve their highest energy efficiency at a number of PEs that is smaller than the maximum possible.
There is a large amount of control flow in \texttt{vadvc}, which leads to large resource utilization. As a result, as shown in Figure~\ref{fig:power_consumption}, increasing the PE count increases power consumption dramatically, causing lower energy efficiency.}

\juann{Fifth, the multi-channel-single PE designs (\texttt{HBM\_multi+OCAPI}) are more energy-efficient than the single-channel-single PE designs (\texttt{HBM+OCAPI}) for the same number of PEs.
However, \texttt{HBM+OCAPI} designs achieve higher energy efficiency for higher numbers of PEs, which are not affordable for \texttt{HBM\_multi+OCAPI} designs.}

\subsection{\gagann{FPGA Resource Utilization}}
Table~\ref{tab:utilization} shows the \gagan{resource} utilization \gagan{of \texttt{vadvc}} and \texttt{hdiff} on the AD9H7 board. \gagann{We draw two observations. First, there is a high BRAM consumption compared to other FPGA resources. This is because we implement input, field, and output signals as \texttt{hls::streams}. In \gagannn{high-level synthesis}, by default, streams are implemented as FIFOs that make use of BRAM. Second,  \texttt{vadvc} has a much larger resource consumption  than \texttt{hdiff} because \texttt{vadvc} has higher computational complexity and requires \gagannn{a larger} number of fields to \gagan{perform} the \gagan{compound} stencil calculation. \gagan{Note that for} \texttt{hdiff}, we can \gagan{accommodate} more {PE}s, but in this work, we  make use of only \gagan{a} single HBM stack. Therefore\gagan{,} we \gagan{use} 16 {PE}s \gagan{because} a single \gagan{HBM} stack offers} \juann{up to 16 memory channels.}

\input{tables/utilization}

\section{Discussion and Key Takeaways}
\label{sec:discussion}
A wide range of application domains have emerged with the ubiquity of computing platforms in every aspect of our daily lives. These modern workloads (e.g., machine
learning, graph processing, and bioinformatics) demand high compute capabilities within strict power constraints~\cite{ghose2019processing}. However, today's computing systems are getting constrained by current technological capabilities, making them incapable of delivering the required performance. \juan{This paper presents our \gonur{recent} efforts to leverage \gonur{near-memory computing capable FPGA-based accelerators} \gagan{to accelerate} two major kernels from the weather prediction application in an energy-efficient way. We summarize the most important insights and takeaways as follows.}

\juan{First, our evaluation shows that \gonur{High-Bandwidth Memory}-based \gonur{near-memory} \gagann{FPGA} accelerator designs can improve performance by \gonur{ 5.3$\times$-12.7$\times$ and energy efficiency by 12$\times$-35$\times$ over} a \gcamera{single-socket} high-end \gagann{16-core} \gonur{IBM} POWER9 CPU.}

\juan{Second, our HBM-based \gagann{FPGA accelerator} designs employ a dedicated HBM channel per PE. Such a design avoids memory access congestion, which is typical in DDR4-based FPGA designs, and ensures memory bandwidth scaling with the number of PEs. As a result, \gagann{in most of the data-parallel applications,} performance scales linearly with the number of PEs. \gtrets{Therefore, HBM provides an attractive solution for scale-out computation.}}

Third, the data needs to be adequately mapped to each HBM channel's address space.   A data mapping scheme should map data in such a way that the data required by the processing unit is readily available in the vicinity (data and code co-location).  An inefficient data mapping mechanism can severely hamper the benefits of processing close to memory.

Fourth, we make use of OCAPI in a coarse-grained way, since we offload the entire application to the FPGA. In this case, OCAPI ensures that the FPGA accelerators access the entire CPU memory with the minimum number of memory copies between the host and the FPGA, e.g., avoiding the intermediate buffer copies that a traditional PCIe-based DMA invokes~\cite{10.1145/2897937.2897972}. However, depending on the application, the CAPI protocol can be employed in finer-grained algorithm-hardware co-design, like the \textit{ExtraV}  \cite{10.14778/3137765.3137776}, where the authors aggressively utilize the fine-grained communication capability of OCAPI to boost graph analytics performance.

\juan{Fifth, \sr{the} maximum performance of our HBM-based design is \gonur{reached using} the maximum PE count that we can fit in the reconfigurable fabric, with each PE having a dedicated HBM channel.} \grev{ However, adding more PEs could lead to timing constraint violations for HBM-based designs. As shown with our multi-channel setting (Section~\ref{subsection:evaluation}), where we can fit only 3 PEs for both \vadvc and \hdiff, enabling more HBM channels leads to timing constraint violations.} HBM-based FPGAs consist of multiple super-logic regions (SLRs)~\garxiv{\cite{hbm_slr}}, where an SLR represents a single FPGA die. All HBM channels are connected only to SLR0, while other SLRs have indirect connections to the HBM channels. Therefore, if a PE is implemented in a non-SLR0 region for a large design, it might make timing closure difficult. A possible way to alleviate timing issues is by running the AFU at a lower frequency, which eases the place and route.

\gtrets{
Sixth, the energy efficiency of our HBM-based designs tends to saturate (or even \gonur{reduces}) as we increase the number of PEs \gonur{beyond some point}. The highest energy efficiency is achieved with a PE count that is smaller than the highest-performing PE count. The \gonur{major} reason \gonur{for a decrease in the energy efficiency is the increase in power consumption with} every additional HBM channel.

Seventh, the emerging cache-coherent  interconnects standards like CXL~\cite{sharma2019compute}, CCIX~\cite{benton2017ccix}, and OCAPI~\cite{openCAPI} could be vital in improving the performance and energy efficiency of big data workloads running on FPGA-based devices because they avoid having multiple data copies. However, a very \om{small number} of works, such as \cite{10.14778/3137765.3137776}, leverage the \emph{coherency} aspect of these interconnects. More quantitative exploration is required to analyze the advantages and disadvantages of using these interconnects.}

Eighth, we are witnessing an enormous amount of data being generated across multiple application domains~\cite{nair2015active,singh2021modeling} like weather prediction modeling, radio astronomy, bioinformatics, material science, chemistry, health sciences, etc. The processing of the sheer amount of generated data is one of the biggest challenges to overcome. In this paper, we demonstrate the capabilities of near-memory reconfigurable accelerators in the domain of weather prediction, however, there are many other high-performance computing applications  where such near-memory architectures can alleviate the data movement bottleneck.}

\section{Introduction}
{Accurate weather prediction \om{and climate modeling} using detailed weather models is essential to \gagan{make} weather-dependent \om{and climate-related} decisions in a timely manner.}
These models are based on physical laws that describe various components of the atmosphere~\cite{schar2020kilometer}.
The Consortium for Small-Scale Modeling (COSMO)~\cite{doms1999nonhydrostatic}
\juang{built} one such weather model
to meet the high-resolution forecasting requirements of weather services. The COSMO model is a non-hydrostatic atmospheric prediction model currently being used by a dozen nations for meteorological purposes and research applications.

The central part of the COSMO model (\juan{called \emph{dynamical core} or \emph{dycore}}) solves the Euler equations on a curvilinear grid and applies implicit discretization in the vertical dimension \gagann{(i.e., parameters are dependent on each other at the same time instance~\cite{bonaventura2000semi})}  and
explicit discretization in the horizontal dimension \gagan{(i.e.,
\juanggg{a solution is dependent on the previous system state}~\cite{bonaventura2000semi})}. The use of different discretizations leads to three computational patterns~\cite{cosmo_knl}: \juang{1)} horizontal stencils, \juang{2)} tridiagonal solvers in the vertical dimension, and \juang{3)} point-wise computation. These computational kernels are compound stencil kernels that operate on a three-dimensional grid~\cite{gysi2015modesto}.
\emph{Vertical advection} (\gagan{\texttt{vadvc}}) and  \emph{horizontal diffusion} (\gagan{\texttt{hdiff}})  are such compound kernels found in the \emph{dycore} of the COSMO \gagannn{weather prediction} model. {These kernels} are representative \gagan{of} the data access {patterns and algorithmic} complexity of the entire COSMO model. \gagan{They} are similar to the kernels used in other weather and climate models~\cite{kehler2016high,neale2010description,doi:10.1175/WAF-D-17-0097.1}.  Their performance is dominated by memory-bound operations with unique irregular memory access patterns
\juang{and} low arithmetic intensity that often results in $<$10\% sustained floating-point performance on current CPU-based systems~\cite{chris}.

Figure~\ref{fig:roofline} shows the roofline plot\gagan{~\cite{williams2009roofline}} for {an} IBM 16-core POWER9 CPU (IC922).\footnote{IBM and POWER9 are registered trademarks or common law marks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies.}
After optimizing the \texttt{vadvc} and \texttt{hdiff} kernels for the POWER architecture\footnote{\grev{We use single instruction, multiple data (SIMD)~\cite{flynn1966very,illiac_simd_1968}, and simultaneous multithreading (SMT)~\cite{smt} techniques to fill the hardware pipelines. We use the same tiling size for both CPU and FPGA-based designs. While compiling these kernels, we use the IBM XLC~\cite{ibmxlc} 16 C/C++ compiler that is optimized for IBM POWER~\cite{POWER9} machines with the following flags: qarch=pwr9, qtune=pwr9, O3, q64,  qprefetch=aggressive, qsmp=omp, and qsimd=auto.}} by following the approach in~\cite{xu2018performance},
\juan{they} achieve {29.1}~GFLOP/s and 58.5~GFLOP/s, \juan{respectively}, \juang{for 64 threads}.  {Our roofline analysis
\juang{indicates} that these kernels are constrained by the host DRAM bandwidth.}
\juan{Their} low arithmetic intensity limits
\juang{their performance, which is one order of magnitude smaller than the peak performance,} and results in a fundamental memory bottleneck that
standard CPU-based optimization techniques \juan{cannot overcome}.

 \begin{figure}[h]
  \centering
  \includegraphics[width=0.75\textwidth,trim={1cm 0.8cm 1.5cm 0.6cm},clip]{images/roofline_fpl20_gagan_b_w.pdf}
    \caption{Roofline~\cite{williams2009roofline} for POWER9 (1-socket) showing vertical advection {(\texttt{vadvc})} and horizontal diffusion {(\texttt{hdiff})} kernels for single-thread and 64-thread
    implementations.
    \juang{The plot shows also the rooflines}
    of the FPGAs used in our work with peak DRAM and on-chip BRAM bandwidth.}

 \label{fig:roofline}
\end{figure}

\gtrets{{Heterogeneous computing} has emerged as an answer to improve the system performance in an energy-efficient way. Heterogeneous computing entails complementing processing elements with different compute capabilities, each to perform the tasks to which it is best suited. In the HPC domain, coupling specialized compute units with general-purpose cores can meet the high-performance computing demands with the ability to realize exascale systems  needed to process data-intensive workloads~\cite{nair2015active}. The graphics processing unit (GPU) is one of the most popular acceleration platforms. GPUs have been used to accelerate workloads like computer graphics and linear algebra~\cite{volkov2008benchmarking} because of their many-core architecture. However, GPUs are power-hungry due to high transistor density and, depending on the power constraints, may not always be the ideal platform for implementation. Recently, the use of field-programmable gate array (FPGA) in accelerating machine learning workloads with high energy efficiency has inspired researchers to explore the use of FPGAs instead of GPUs for various high-performance computing applications~\cite{caulfield2016cloud,duarte2018fast}.  FPGAs provide a unique combination of flexibility and performance without the cost, complexity, and risk of developing custom application-specific integrated circuits (ASICs). } Modern FPGAs show four key trends.
\begin{itemize}
    \item The advancements in the stacking technology with  high-bandwidth memory ({HBM})~\cite{hbm,xilinx_utlra,hbm_specs,intel_altera,lee2016smla} blends DRAM on the same package as an FPGA. This integration allows us to implement our accelerator logic in close proximity to the memory with a lower latency and much higher memory bandwidth
    than the traditional DDR4-based FPGA boards. Memory-bound applications on FPGAs are limited by \juang{the relatively} low DDR4 bandwidth (72 GB/s
\gagan{for four independent dual-rank DIMM interfaces}\gagann{~\cite{VCU}}).
HBM-based FPGAs can overcome this limitation with
 a  peak bandwidth of 410~GB/s~\cite{kara2020hbm}.

    \item New cache-coherent interconnects, such as  Open Coherent Accelerator Processor Interface (OCAPI)~\cite{openCAPI}, Cache Coherent Interconnect for Accelerators (CCIX)~\cite{benton2017ccix}, and Compute Express Link (CXL)~\cite{sharma2019compute},  allow tight integration of FPGAs with CPUs at high bidirectional bandwidth {({on} the order of tens of GB/s)}. This integration {reduces programming effort and} enables us to coherently access the host system's memory through a pointer rather than having multiple copies of the data.
     \item The introduction of UltraRAM (URAM)~\cite{uram} along with the BlockRAM (BRAM) that offers massive \emph{scratchpad}-based on-chip memory next to the logic. \gtrets{URAM is more denser than BRAM, but is not as distributed in the FPGA layout as the BRAM.}
    \item FPGAs are being manufactured with an advanced technology node of 7-14nm FinFET technology~\cite{gaide2019xilinx} that offers higher performance.

\end{itemize}

These above trends suggest that {modern FPGA architectures with near-memory compute capabilities can alleviate the \emph{memory bottleneck} of real-world data-intensive applications~\cite{singh2021fpga}.
However, a study of their advantages for
real-world memory-bound applications is still missing. In this work, our goal is to overcome the memory} bottleneck {of weather prediction} kernels by {exploiting near-memory {computation} capability on} FPGA accelerators with high-bandwidth memory (HBM)~\cite{6757501,hbm,lee2016smla} {that are attached} {to the host CPU}. Figure~\ref{fig:roofline} shows the roofline model{s} of the two FPGA cards {(AD9V3~{\cite{ad9v3}} and AD9H7~{\cite{ad9h7}})} used in this work.
FPGAs \juang{can} \gagan{handle}
irregular memory access patterns \juang{efficiently} and
offer significantly
\juang{higher} memory bandwidth \juang{than the host CPU with}
their on-chip URAMs \gagan{(UltraRAM)}, BRAMs \gagan{(block RAM)}, and \gcamera{on-package} HBM (\gagan{high-bandwidth memory} for the AD9H7 card).
However, taking full advantage of
FPGAs for accelerating a workload is not a trivial task. To compensate \gagan{for} the higher clock frequency of the \gagan{baseline} CPUs,
\gagan{our} FPGAs must exploit at least one order of magnitude more parallelism in a target workload.
\juang{This is challenging, as it requires sufficient FPGA programming skills to \gagan{map the workload and} optimize the design for the FPGA microarchitecture.
}

\juan{We aim} to answer the following \juan{research question}: \textbf{Can FPGA-based \gagan{accelerators} with HBM \gagan{mitigate} the performance bottleneck of \juang{memory-bound} compound weather \gagan{prediction} kernels
in an energy-efficient way?} As an answer to this question\gagann{,} we present \namePaper, a \underline{ne}ar-HBM accelerator for weathe\underline{r} predicti\underline{o}n.
\juan{We design and implement} \namePaper
\juan{on an FPGA with HBM to} optimize two
kernels (vertical advection and horizontal diffusion),
\juan{which notably} represent
\juan{the} spectrum of {computational diversity} found in the COSMO \gagann{weather prediction} application.
\juang{We co-design a hardware-software framework and provide an optimized API to interface efficiently with the rest of the COSMO model, which runs on the CPU}.
Our \gagan{FPGA-based} solution \gagan{for \texttt{hdiff} and \texttt{vadvc}} leads to
\gagan{performance improvements of $5.3\times$ and $12.7\times$ and}
\juang{energy reductions} of $12\times$ and $35\times$, \juanggg{respectively,} \juang{with respect to optimized CPU implementations~\cite{xu2018performance}}.

\gagan{T}he major contributions of \namePaper are \juang{as follows}:
\begin{itemize}
\item \juang{We perform \gagan{a detailed} roofline analysis to show that representative weather \gagan{prediction} kernels are constrained by memory bandwidth on \gagannn{state-of-the-art} CPU systems.}

\item {We propose \namePaper, \gagan{the first} near-HBM FPGA-based accelerator} for representative kernels from a real-world weather prediction application.

\item \juang{We optimize \namePaper~with} a data-centric caching scheme with precision-optimized tiling for a heterogeneous memory hierarchy \gagan{(consisting of URAM, BRAM, and HBM)}.

\item \juan{We evaluate the performance and energy consumption of our accelerator and perform a scalability analysis.}
We show that an FPGA+HBM-based design
outperforms a complete 16-core POWER9 \gagann{system}~
\juang{running} 64~threads)~by~$5.3\times$ for
\juang{the} {vertical advection} \juanggg{(\texttt{vadvc})} and $12.7\times$ for
\juang{the} {horizontal diffusion} \juanggg{(\texttt{hdiff})} kernels with energy \gagann{reductions of $12\times$ and $35\times$, respectively. Our design provides an energy efficiency of 1.61 GLOPS/Watt and 21.01 GFLOPS/Watt for \texttt{vadvc} and \texttt{hdiff} kernels,} \juanggg{respectively}.

\end{itemize}

\gcamera{This work extends our previous work~\cite{singh2020nero} as follows. First, we add new results using a state-of-the-art OpenCAPI (OCAPI) interface~\om{\cite{openCAPI}}. OCAPI provides two key opportunities compared to our previous CAPI2 implementation: (1) OCAPI has double the bitwidth (1024-bit) of our previously used CAPI2 interface, and (2) the memory coherency logic has been moved to the host CPU side, which provides more area and allows us to run our design at a higher frequency. Our implementation and evaluation with state-of-the-art OCAPI improves the performance for our two main workloads from weather modeling (vadvc and hdiff) by 37\% and 44\%, respectively, compared to a CAPI-based  HBM  design~\cite{singh2020nero}. All the functions in our design now operate on a 1024-bit \gcameraa{per clock} rather than 512-bit \gcameraa{per clock, utilizing the maximum processing throughput of OCAPI  (POWER9 cache line is 1024-bit). Thus, we can build a dataflow accelerator with wide AXI  streams of 1024-bit.} Second, we develop \om{\texttt{HBM\_multi+OCAPI}}-based versions of our workloads that make use of multiple channels per processing element (PE). This multi-channel implementation allows the PEs to exploit significantly higher bandwidth. As a result, the workloads achieve an average speedup of 1.5x over the single-channel version for a single PE.
Third, we provide a discussion section (Section~\ref{sec:discussion}) that provides various insights and takeaways while designing HBM-based FPGA accelerators, which we believe would be useful for future FPGA architects \om{and programmers}. Fourth, we implement and evaluate \om{the \cop} stencil~\cite{cosmo_knl} \om{(Figure~\ref{fig:copy_stencil})}, a stencil from the COSMO model to benchmark the peak performance on a platform. Fifth, we use the OC-Accel framework\footnote{https://github.com/OpenCAPI/oc-accel} instead of the SNAP framework for OCAPI accelerator development and deployment. Sixth, we provide a comparison of state-of-the-art works in stencil acceleration (Table~\ref{tab:compare_works}).

}

\section{Design Methodology}
\label{sec:design}
\subsection{NERO, A Near HBM Weather Prediction Accelerator}
The low arithmetic intensity of real-world weather \gagan{prediction} kernels limits the attainable performance on current multi-core systems. This sub-optimal performance is due to \gagan{the kernels'} complex memory \juanggg{access} patterns and their inefficiency in exploiting a rigid cache hierarchy, \gagan{as quantified in \juangg{the} roofline plot in} Figure~\ref{fig:roofline}.
These kernels \gagannn{cannot} fully utilize the available memory bandwidth, which leads to
high
\juanggg{data movement overheads} in terms of latency and energy consumption. We address
\juanggg{these inefficiencies} by developing an architecture \gagan{that} combines fewer off-chip data \gagan{accesses} with higher throughput for the loaded data. \gagan{To this end}, our accelerator design \gagan{takes} a data-centric approach~\cite{mutlu2019,ghose2019processing,teserract,singh2019near,NAPEL,hsieh2016accelerating,7551394,ahn2015pim,googleWorkloads,kim2018grim,mutlu2021intelligent,boroumand2021google,mutlu2020modern} \gagan{that exploits} near high-bandwidth memory acceleration.

Figure~\ref{fig:system} shows a high-level overview of our integrated system. \gagan{An HBM-based} FPGA is connected to a server system based on an IBM POWER9 processor using the \juangg{Open Coherent Accelerator Processor Interface}  (OCAPI).
The FPGA consists of two HBM stacks\footnote{In this work, we enable only a single stack based on our resource and power consumption analysis \gagan{for \gagannn{the} \texttt{vadvc} kernel.}}, each with 16 \emph{pseudo-memory channels}~\cite{axi_hbm}. \gagan{A} channel is exposed to the FPGA as a 256-bit wide port, and in total,
\juanggg{the FPGA} has 32 such ports. The HBM IP provides 8 memory controllers \juanggg{(per stack)} to handle the data transfer to and from the HBM memory ports. Our design consists of an \emph{accelerator functional unit} (AFU) that interacts with the host system through the TLx (Transaction Layer) and the DLx (Data Link Layer), which are the OCAPI endpoint on the FPGA. An AFU comprises of multiple \emph{processing elements} (PEs) that perform compound stencil computation. Figure~\ref{fig:complete_flow} shows the architecture overview of \namePaper. As vertical advection is the most complex kernel, we depict \gagan{our architecture design flow for} vertical advection. We use a similar design for the \gagan{horizontal diffusion} kernel.

\begin{figure*}[h]
\begin{subfigure}[t]{0.5\textwidth}
 \includegraphics[width=1\linewidth,trim={0.5cm 1.2cm 0.5cm 1cm},clip]{images/TRETS_system.pdf}
   \caption{
  \label{fig:system}}
\end{subfigure}%
\begin{subfigure}[t]{0.4\textwidth}
  \includegraphics[width=0.95\linewidth,trim={0cm 0cm 0cm 0cm},clip]{images/TRETS_Execution_timeline.pdf}
  \caption{
  \label{fig:execution}}
\end{subfigure}
\vspace{-10pt}
\caption[Two numerical solutions]{(a) Heterogeneous platform with an IBM POWER9 system connected to an HBM-based FPGA board via~OCAPI. We also show components of an FPGA: flip-flop (FF), lookup table (LUT), UltraRAM (URAM), and Block RAM (BRAM).
 (b)  Execution timeline with data flow sequence from the  host  DRAM  to the onboard  FPGA  memory.\label{fig:complete_hbm_flow}}
\end{figure*}

\begin{figure}[h]
 \centering
  \includegraphics[width=0.95\linewidth,trim={0.5cm 0.8cm 0cm 0cm},clip]{images/TRETS_complete_flow_new.pdf}
  \vspace{-2pt}
   \caption{Architecture overview of~\namePaper~with data flow sequence from the host DRAM to the on-board FPGA memory via POWER9 cachelines. We {depict a} single {processing element (PE)} fetching data from a dedicated HBM port. The number of HBM ports scales linearly with the number of {PEs}. Heterogeneous partitioning of on-chip memory blocks \juangg{reduces} read \juangg{and} write latencies across the FPGA memory hierarchy.
  \label{fig:complete_flow}}
\end{figure}

The weather data, based on the atmospheric model resolution grid, is stored in the DRAM of the host system \gagan{(\circled{1} in Figure~\ref{fig:complete_flow})}.  We employ \juangg{the} double buffering technique  between the CPU and the FPGA to hide the PCIe \gagan{(Peripheral Component Interconnect Express~\cite{pcie})} \gagannn{transfer} {latency}.  {By configuring a buffer of 64 cache lines, between the AXI4 interface of OCAPI/TLx-DLx and the AFU, we can reach the theoretical peak bandwidth of OCAPI (\juanggg{i.e.,} 32 GB/s).} We create a specialized memory hierarchy from the heterogeneous FPGA memories \gagan{(i.e., URAM, BRAM, and HBM)}. By using a greedy algorithm, we determine the best-suited hierarchy for our kernel. The memory controller (shown in Figure~\ref{fig:system}) handles the data placement to the appropriate memory type \gagan{based on the programmer's directives}.

\gagan{On the FPGA, following the initial buffering (\circled{2}), the \gagan{transferred} grid data is mapped onto the HBM memory (\circled{3}).  As \gagan{the} FPGA ha\gagan{s} limited resources, we propose a 3D window-based grid transfer from the host DRAM to the FPGA, facilitating a smaller, less power-hungry deployment. The window size represents the portion of the grid a processing element ({PE} in Figure~\ref{fig:system}) would process. \gagan{Most FPGA developers manually optimize for the right window size. However, manual optimization is tedious because of the huge design space, and \juanggg{it} requires expert guidance.
Further, selecting an inappropriate window size lead\gagan{s} to sub-optimal results.
\juangg{Our experiments (in Section~\ref{subsection:evaluation}) show that}: (1) finding the \juangg{best} window size is critical in terms of \juangg{the} area vs. performance trade-off, and (2) the \juangg{best window} size depends on the datatype precision. Hence, instead of pruning the design space manually, we formulate
\juanggg{the search for} the \juangg{best} window size as a multi-objective auto-tuning problem taking into account the datatype precision. We make use of OpenTuner~\cite{opentuner},
\juanggg{which} uses machine-learning techniques to guide the design-space~exploration~\cite{singh2021modeling}}.}

Our design consists of multiple PEs (shown in Figure~\ref{fig:system}) that exploit data-level parallelism in COSMO \gagannn{weather prediction} kernels. \gagan{A dedicated HBM memory port is assigned to a specific PE}; therefore, we enable as many HBM ports as the number of PEs. This allows us to use the high \gagan{HBM} bandwidth \gagannn{effectively} because each PE fetches from an independent port.
In our design, we use a switch, which provides the capability to
\juanggg{bypass} the HBM, \juangg{when the} grid size \juangg{is small}, and map the data directly onto the FPGA's URAM \gagan{and BRAM}. The HBM port provides 256-bit data, \gagannn{which} is a quarter of the size of \juangg{the} OCAPI \gagannn{bitwidth} (1024-bit). Therefore, to match \gagannn{the} OCAPI bandwidth, we introduce a stream converter logic (\circled{4}) that converts \juangg{a} 256-bit HBM stream to \juangg{a} 1024-bit stream (OCAPI compatible) or vice versa.
From HBM, \gagan{a PE} reads a single stream of data that consists of all the fields\footnote{Fields represent atmospheric components like wind, pressure, velocity, etc. that are required for weather calculation.} \juangg{that} are needed for a specific COSMO kernel computation. \gagan{The PEs} use a fields stream splitter logic (\circled{5}) that splits a single HBM stream to multiple streams (1024-bit each), one for each field.

\gagann{To optimize a PE\gagan{,} we apply various optimization strategies. First, we exploit the inherent parallelism in \gagann{a given} algorithm through hardware pipelining. \gagann{Second}, we partition on-chip memory to avoid the stalling of our pipelined design, since the on-chip BRAM/URAM has only two read/write ports. \gagann{Third}, all the tasks execute in a dataflow manner that enables task-level parallelism. \texttt{vadvc} is more \gagan{ computationally complex} than \texttt{hdiff} because it involves forward and backward sweeps \gagan{with dependencies in \juangg{the} z-dimension}. While \texttt{hdiff} performs \gagan{only} \juangg{Laplacian} and flux calculations with dependencies in \juangg{the} x- and y-dimensions.
Therefore, we demonstrate our design flow by means of \gagannn{the} \texttt{vadvc} kernel (Figure~\ref{fig:complete_flow}).
Note \juangg{that} we show \juangg{only} a single port-based PE operation.  \juanggg{However}, for multiple PEs, we enable multiple HBM ports.}

We make use of memory reshaping techniques to  configure our memory space with multiple parallel BRAMs or URAMs~\cite{dioFPT}. \gagann{We form an intermediate memory hierarchy by decomposing (or slicing) 3D window data into a 2D grid. } This allows us to bridge the latency gap between the HBM memory and our accelerator. Moreover, it allows us to exploit the available FPGA resources efficiently. Unlike traditionally-fixed CPU memory hierarchies, which perform poorly with irregular access patterns and suffer from cache pollution effects \om{and cache miss latency}, application-specific memory hierarchies \juangg{are} shown to improve \juangg{energy and latency} by tailoring the cache levels and cache sizes to \juangg{an} application's memory access patterns~\cite{jenga}.

The main computation pipeline (\circled{7}) \gagann{consists of a forward and a backward sweep logic}. The forward sweep \gagannn{results} are stored in \juangg{an} intermediate buffer to allow for backward sweep calculation. \juangg{Upon} completion of \juangg{the} backward sweep, results are \gagan{placed in} an output buffer \gagan{that is followed by a degridding logic (\circled{6}). The degridding logic} converts the calculated results to a 1024-bit wide output stream (\circled{8}). As there is only a single output stream (both in \texttt{vadvc} and \texttt{hdiff}), we do not need extra logic to merge the streams. The 1024-bit \gagan{wide} stream goes through an HBM stream converter logic (\circled{4}) that converts the stream bitwidth to HBM port size (256-bit).

\gagan{Figure~\ref{fig:execution} shows the execution timeline from our host system to the FPGA board for a single PE. The host offloads the processing to an FPGA and
\juangg{transfers} the required data \juangg{via DMA (direct memory access)} over \juangg{the} OCAPI interface. The OC-Accel framework allows for parallel execution of the host and our FPGA PEs while exchanging control signals over the AXI lite interface\gagan{~\cite{axilite}}. On task completion, the AFU notif\gagannn{ies} the host system via \gagannn{the} AXI lite interface and
\juangg{transfers} back the results \juangg{via DMA}. }

\subsection{\namePaper~Application Framework}
Figure~\ref{fig:snap_api} shows the \namePaper~application framework \gagan{to support our architecture}. \gcamera{Our previous work~\cite{singh2020nero,narmada} describes the corresponding application framework using SNAP-CAPI2.}  A software-defined COSMO API (\circledWhite{1}) handles offloading jobs to \namePaper~with an interrupt-based queuing mechanism.  This allows for minimal CPU usage (and, hence, power \gagan{usage}) during FPGA \gagan{operation}. \namePaper~employs an array of processing elements to compute COSMO kernels, such as vertical advection or horizontal diffusion. Additionally, we pipeline our PEs to exploit the available spatial parallelism.
By accessing the host memory through
\juanggg{the OCAPI} cache-coherent link, \namePaper~acts as a peer to the CPU.  This is enabled \gagannn{through} \gagan{the} TLx (Transaction Layer) and the DLx (Data Link Layer) (\circledWhite{2}). \gagan{ OC-Accel (\circledWhite{3}) allows for seamless \juangg{integration} of the COSMO API with our OCAPI-based accelerator.} The job manager (\circledWhite{4}) dispatches jobs to streams, which are managed in the stream scheduler (\circledWhite{5}). The execution of a job is done by streams that determine which data is to be read from the host memory and sent to the PE array through DMA transfers (\circledWhite{6}). The pool of heterogeneous on-chip memory is used to store the input data from the main memory and the intermediate data generated by \juangg{each} PE.

\begin{figure}[h]
  \centering
 \includegraphics[width=0.7\linewidth,trim={0cm 0cm 0.1cm 0cm},clip]{images/TRETS_snap_accelerator_4.pdf}
 \vspace{-10pt}
  \caption
  \juangg{\namePaper~application framework.}
  We co-design our software and hardware using \gagannn{the} OC-Accel framework. COSMO API allows the host to offload kernels to our FPGA platform.
 \label{fig:snap_api}}
 \end{figure}

\section{Related Work}
\label{sec:relatedWork}
\juang{To our knowledge, this is the first work to evaluate the benefits of using FPGAs equipped with \gagann{high-bandwidth memory (HBM) to accelerate} {real-world weather \om{modeling} stencils.}} \gagannn{We exploit the near-memory capabilities of such FPGAs to accelerate important weather prediction kernels.} \om{Exploiting the high-bandwidth memory in FPGAs,} we answer the following questions with our work. First, do real-world weather prediction applications benefit from HBM-based FPGAs? Second, how can we scale the processing in terms of not only run-time but also energy efficiency? Third, what does the system look like regarding computation and data movement with an HBM-enabled FPGA when integrating accelerators for real-world weather prediction workloads?

\gagannn{Modern workloads exhibit limited locality and operate on large amounts of data, which causes frequent data movement between the memory subsystem and the processing units~\cite{ghose2019processing,mutlu2019,googleWorkloads,mutlu2019enabling,mutlu2021intelligent,boroumand2021google,mutlu2020modern}.  This frequent data movement has a severe impact on overall system performance and energy efficiency. \gtrets{ For example, in the domain of climate and weather modeling,  there is a data avalanche due to large atmospheric simulations~\cite{schar2020kilometer}. Major efforts are currently underway towards refining the resolution grid of climate models that would generate \emph{zettabytes} of data~\cite{schar2020kilometer}. These high-resolution simulations are useful to predict and address events like severe storms. However, the sheer amount of generated data is one of the biggest challenges to overcome. We find another relevant example in radio astronomy. The first phase of the Square Kilometre Array (SKA) aims to process over 100 terabytes of raw data samples per second, yielding of the order of 300 petabytes of SKA data produced annually~\cite{6898703,singh2018review}. { Recent biological disciplines such as genomics have also emerged as one of the most data-intensive workloads across all different sciences wherein just a single human genome sequence produces hundreds of gigabytes of raw data. With the rapid advancement in sequencing technology, the data volume in genomics is projected to surpass the data volume in all other application domains~\cite{navarro2019genomics}.}  }

A way to alleviate this \emph{data movement bottleneck}~\cite{singh2019near, mutlu2019, ghose2019processing,mutlu2019enabling,googleWorkloads,mutlu2020modern,hajinazar2021simdram,seshadri2019dram,mutlu2021intelligent,boroumand2021google,mutlu2020modern} is \emph{near-memory computing} (NMC), which consists of placing processing units closer to memory.
NMC is enabled by new memory technologies, such as 3D-stacked memories~\cite{7477494,6757501,6025219,hbm,lee2016smla,fimdram2021ISSCC,fimdram2021ISCA,mutlu2021intelligent,boroumand2021google,mutlu2020modern}, and also by cache-coherent interconnects~\cite{openCAPI, benton2017ccix, sharma2019compute},  which allow close integration of processing units and memory units.
Depending on the applications \gagannn{and systems} of interest (e.g.,~\cite{nai2017graphpim,7056040,lee2018application,kang2013enabling,hashemi2016continuous,akin2015data,babarinsa2015jafar,lee2015bssync,chi2016prime,kim2016neurocube,asghari2016chameleon,boroumand2016lazypim,seshadri2015gather,liu2017concurrent,gao2015practical,morad2015gp,googleWorkloads,teserract,ahn2015pim,hsieh2016accelerating,hashemi2016accelerating,mutlu2020modern,senolcalimicro2020,oliveira2021pimbench,upmem2021,ke2020recnmp,fernandez2020natsa,ConTutto_2017_MICRO,simon2020blade,giannoula2021syncron,gao2019computedram,seshadri2017simple,angizi2019aligns,mutlu2021intelligent,boroumand2021google,mutlu2020modern,gomez2021benchmarking,seshadri2017ambit,seshadri2015fast,li2016pinatubo,seshadri2016buddy,besta2021sisa,park2021trim,wu2021sieve}), prior works propose different types of near-memory processing units, such as general-purpose CPU cores~\cite{lee2018application,alian2018application,de2017mondrian,koo2017summarizer,7927081,teserract,nair2015active,6844483,googleWorkloads,boroumand2016lazypim,boroumand2019conda,gu2016biscuit,liu2018processing,oliveira2021pimbench,upmem2021}, GPU cores~\cite{zhang2014top,7756764,7551394,ghose2019demystifying}, reconfigurable units~\cite{7446059, jun2015bluedbm,istvan2017caribou,narmada}, or fixed-function units~\cite{ahn2015pim,hsieh2016accelerating,gu2016biscuit,nai2017graphpim,liu2018processing,kim2018grim,hashemi2016accelerating,hashemi2016continuous,boroumand2021google}}.

FPGA accelerators are promising to enhance overall system performance \gagann{with low power consumption.}
Past works~\cite{jun2015bluedbm, kara2017fpga, alser2019shouji, giefers2015accelerating,8373077,10.14778/3137765.3137776,alser2017,chai_icpe19,chang2017collaborative,jiang2020,alser2019sneakysnake,alser2020accelerating,alser2020technology} show that FPGAs can be employed {effectively} for a wide range of applications. \gtrets{FPGAs provide a unique combination of flexibility and performance without the cost, complexity, and risk of developing custom application-specific integrated circuits (ASICs).
The researchers at CERN, for example, are using FPGAs to accelerate physics workload in CERN's \dd{exploration} of dark matter~\cite{duarte2018fast}. Microsoft's Project Catapul
~\cite{caulfield2016cloud} is another example of how FPGAs can be used in the data center infrastructure. \dd{Driven by Catapult's promising research results, Microsoft further deployed the architecture on the Azure cloud marketplace~\cite{FPGA_in_azure}. } Such integration {for certain workloads} can even offer more energy efficiency than CPU or GPU-based systems.} The recent addition of HBM to \juang{FPGAs}
\juang{presents an opportunity to exploit}
high \juang{memory} bandwidth with \gagannn{the} low-power FPGA fabric.
The potential of high-bandwidth memory~\cite{hbm,lee2016smla} has been explored in many-core processors~\cite{hbm_joins,ghose2019demystifying} and GPUs~\cite{hbm_gpu_data_intensive,ghose2019demystifying}.
\juang{Recent benchmarking works~\cite{wang2020, kara2020hbm} show the potential of HBM for FPGAs.}

\juang{\namePaper} is the first work to accelerate a real-world HPC weather prediction application using \juang{the} FPGA+HBM fabric. {Compared to a previous work~\cite{narmada} that \juang{optimizes only the} horizontal diffusion kernel \juang{on an FPGA with DDR4 memory}, our analysis reveals \juang{that the} vertical advection kernel has a much lower compute intensity with little to no regularity.
Therefore, our work \juang{accelerates} both kernels that together represent the algorithmic diversity of the entire COSMO \gagannn{weather prediction} model.
\grev{Our current work differs \om{from~\cite{singh2020nero}} in the following aspects. First, we design and evaluate both horizontal diffusion and vertical advection stencils. Vertical advection  is the most complex stencil in the entire COSMO application. Second, we integrate and implement our accelerator design with an HBM-based FPGA. The bus width of the DDR4 channel (512 bits) is larger than that of an HBM channel (256 bits). Therefore, the HBM channel has a lower transfer rate of 0.8-2.1 GT/s (Gigatransfers per second) than a DDR4 channel (2.1-4.3 GT/s), resulting in a theoretical bandwidth of 12.8 GB/s and 25.6 GB/s per channel, respectively. However, HBM exposes 32 memory channels that provide 4x more bandwidth (410 GB/s for HBM~\cite{kara2020hbm}) compared to traditional DDR4 bandwidth (72 GB/s for four independent dual-rank DIMM interfaces~\cite{VCU}). Therefore, the use of HBM imposes an architectural shift. We evaluate and demonstrate the use of HBM for scaling an accelerator design with different channels provided by HBM. Third, we use an auto-tuning framework to find the right window size (Figure 6) that demonstrates the importance of finding the right window size. Fourth, we provide new results using a state-of-the-art OpenCAPI (OCAPI) interface with the OC-Accel framework. OCAPI provides two key opportunities compared to our previous CAPI2 implementation: (1) OCAPI has double the bitwidth of our previously used CAPI2 interface, (2) a major component of the memory coherency logic is moved to the host CPU side, which provides more FPGA area and enables designs with higher frequency. Due to the above optimizations, we improve the performance for horizontal diffusion by 1.2x on a DDR4-based board and 4.7x on an HBM-based board compared to our previous work NARMADA~\cite{narmada}. }}

\gagan{Enabling higher performance for stencil computations has been a subject of optimizations
across the \juang{whole} computing stack~\cite{sano2014multi,7582502,chi2018soda,de2018designing,christen2011patus,  datta2009optimization, meng2011performance, henretty2011data,fu2011eliminating, strzodka2010cache,zohouri2018combined,waidyasooriya2019multi, tang2011pochoir,gonzalez1997speculative,armejach2018stencil,gysi2015modesto}}. \gtrets{Stencil computation is essential for numerical simulations of finite difference methods~(FDM)~\cite{ozicsik2017finite} and is applied in iterative solvers of linear equation systems. We use stencil computation in a wide range of applications, including computational fluid dynamics~\cite{huynh2014high}, image processing~\cite{hermosilla2008non}, weather prediction modeling~\cite{doms1999nonhydrostatic}, etc.}

\grev{
Unlike stencils found in the literature~\cite{waidyasooriya2019multi,singh2019low,de2021stencilflow,sano2014multi,7582502,chi2018soda,de2018designing}, real-world compound stencils consist of a collection of stencils that perform a sequence of element-wise computations with complex interdependencies. Such compound kernels have complex memory access patterns and low arithmetic intensity because they have limited operations per loaded value. Our work is the first work to accelerate both horizontal diffusion and vertical advection stencils, which are representative of data access patterns and the algorithmic complexity found in the entire COSMO weather model.  }

Table~\ref{tab:compare_works} lists recent works (including \namePaper) that use FPGA to accelerate stencil-based application. We also mention works that accelerate elementary stencils (\texttt{7-point}, \texttt{25-point} Jacobi, \texttt{Hotspot}, and \texttt{Diffusion}). \grev{We make the following three observations. First, the elementary stencils can achieve much higher performance on comparable FPGA devices than complex weather stencils (such as \hdiff) even without using HBM. This high performance is because elementary stencils have a higher arithmetic intensity than weather stencils. Due to their data-parallel nature, these elementary stencils can further take advantage of the increased bandwidth provided by HBM in an energy-efficient way. Second, weather stencils can reach only 2\%-17\% of the peak theoretical performance of an FPGA board.  This low peak performance is because weather stencils have several elementary stencils cascaded together with data interdependencies that lead to complex memory access patterns. Third, compared to NARMADA~\cite{narmada}, which uses a DDR4-based design, our HBM-based design achieves $4.7\times$ performance improvement by exploiting the high bandwidth provided by the HBM. }

\input{tables/relatedWork}

\juang{Szustak~\textit{et al.}} accelerate the MPDATA advection scheme on multi-core CPU~\cite{szustak2013using} and computational fluid dynamics kernels on FPGA~\cite{mpdata}. \gtrets{
Singh~\textit{et al.}~\cite{singh2019low} explore the applicability of different number formats and exhaustively search for the appropriate bit-width for memory-bound stencil kernels to improve performance and energy efficiency with minimal loss in the accuracy.}  Bianco~\textit{et al.}~\cite{bianco2013gpu} \juan{optimize the COSMO \gagannn{weather prediction} model} for GPUs.  \grev{ Thaler~\textit{et al.}~\cite{cosmo_knl}, in a collaboration work between the Swiss National Supercomputing Centre (CSCS) and the Federal Institute of Meteorology and Climatology (MeteoSwiss),  discuss the importance of horizontal diffusion and vertical advection kernels in the entire COSMO model. These kernels together represent the algorithmic diversity of the entire COSMO weather prediction model~\cite{cosmo_knl,gysi2015modesto,bianco2013gpu}. They {port} COSMO to a many-core system.} \grev{Compared to their Intel KNL~\om{\cite{intelknl}} (or NVIDIA P100~\om{\cite{nvidiap100}}) implementation, we observe that our FPGA-based \vadvc and \hdiff design provides $1.5\times$ (or $1.4\times$) and $3.2\times$ (or $2.1\times$) performance improvements, respectively. } Several works~\cite{de2021stencilflow,lai2019heterocl, li2020heterohalide,wang2017comprehensive} propose frameworks for generating optimized stencil code for FPGA-based platforms.
Wahib~\textit{et al.}~\cite{wahib2014scalable} \juang{develop} an analytical performance model for choosing an optimal GPU-based execution strategy for various scientific applications, including COSMO.
Gysi~\textit{et al.}~\cite{gysi2015modesto} provide guidelines for optimizing stencil kernels for CPU--GPU systems.

redpajama_set_name: RedPajamaArXiv


GREAT STUFF! The Bass Rock in the Firth of Forth, the home of the Solan goose or gannet, is a familiar sight to visitors to the popular resort of North Berwick. This photograph shows one of the "Harts" ("Kestrel") of No. 603 (City of Edinburgh) (Bomber) Squadron flying past the Bass Rock.
READY FOR THE FRAY: A "Bulldog" of No. 17 (Fighter) Squadron waiting at Kenley to go up on night patrol.
SOVIET MISSION TO PARIS: Two of the A.N.T.14 machines, which flew from Russia to Paris, at Le Bourget.
A PURSUIT TYPE: The Boeing P-26. The engine is a Pratt and Whitney "Wasp" of about 600 h.p.
FOR RECONNAISSANCE: The Mureaux 113.R2 (Hispano-Suiza 12 Ybrs engine) high-wing monoplane. It can also be used for light bombing.
A SIX-SEATER: The Bellanca "Senior Skyrocket" cabin monoplane.
A SINGLE-SEATER FIGHTER: The Fairey "Firefly" (Rolls-Royce "Kestrel").
The Martin B-10. It has two Wright "Cyclones" of 650 h.p. each.
"SITTING DOWN": The latest type of Autogiro (7-cyl. Siddeley "Genet Major") touches down with its tail wheel first when coming down steeply.
UNSTALLABLE: The Cierva "Autogiro" built by A. V. Roe & Co., Ltd., will be exhibited at Copenhagen, and another machine will be available at the aerodrome. The engine is a 7-cyl. Siddeley "Genet Major."
FOR "STAFF" TRANSPORT. The first of the Avro C.30a Autogiros ready for delivery to the Royal Air Force for Array co-operation work. Note the upper identification mark painted on the top of the fuselage.
TO TAKE LANDING SHOCKS. Autogiros can land vertically, hence this wide undercarriage with compression legs having a long travel to take up the shock. The engine is a 140 h.p. 7-cyl. Siddeley "Genet Major."
IN THE COCKPIT: The flying controls are confined to a single "hanging" stick, the foot bar being used for operating the tail wheel when the machine is taxying.
FOR A SMALL HANGAR. Two of the blades may be folded on their hinges to enable the machine to be housed in a small space.
DIRECT CONTROL: The way in which the rotor head is tilted should be clear from these diagrammatic views.
WELDED CONSTRUCTION. The rear of the fuselage with the tail-wheel mounting.
FUSELAGE DETAILS. The letters show the position of the enlarged details on the fuselage.
WITH "PEGASUS" ENGINE: The Anbo IV two-seater military monoplane. This photograph was taken during a recent visit to Croydon.
BUSINESSLIKE: Showing the installation of the Bristol "Pegasus" engine in the Anbo IV and the wide undercarriage.
THE BLACKBURN T.S.R.: The letters stand for torpedo, spotting, reconnaissance and the engine is a Siddeley "Tiger."
The Fairey Torpedo Spotter reconnaissance ship-plane (Bristol "Pegasus"). An order will probably be placed shortly for machines of this type to equip a squadron of the Fleet Air Arm.
NOT AN ACCIDENT: The Avro 626 is variously used for training in gunnery, bombing, photography and wireless.
FOR THE ROYAL AIR FORCE: The Fairey night bomber (2 Rolls-Royce "Kestrels") is now in production and will be the first monoplane, apart from amphibian flying boats, to be issued to the R.A.F. The machine as here seen was being piloted by Fit. Lt. C. Staniland, Fairey's chief test pilot.
A FRENCH FEEDER LINE MACHINE: The new Potez 56 has a retractable undercarriage, and its two Potez radial engines are supercharged. Six passengers are carried. Its resemblance to the Airspeed "Envoy" is quite marked.
95 H.P. - 145 M.P.H. The Ryan S-T two-seater with Menasco B.4 engine of 95 h.p.
BRAZILIAN CLIPPER TAKES OFF: Pan-American Airways' first Sikorsky S-42 leaves Biscayne Bay for Buenos Ayres. It is reported that Short Bros, are building a large trans-oceanic boat for Imperial Airways.
IN CANADA: The Trail of the Caribou photographed before the start of the flight with her two pilots, L. G. Reid and J. R. Ayling.
COMFORT AND COMPOSURE AT 110 M.P.H.: Not the interior of a railway carriage, but of a modern air liner - Horatius, of Imperial Airways. There are two separate armchair seats on each side of the tables, on which meals are served.
ABOVE THE CLOUDS: In this view the fine lines of the British Klemm "Eagle" (130 h. p. "Gipsy Major") are particularly noticeable, and some idea can be gained of the pilot's range of vision, both above and below. The "Eagle's" cruising speed, with "Gipsy Major" engine, is 130 m.p.h.
The slotted and flapped "Puss Moth."
"OSPREY" ON AUSTRALIAN TRIP: The Hawker "Osprey" seaplane (Rolls-Royce "Kestrel" II.M.S.) on the catapult of H.M.S. "Sussex."
WITH WHEELS UP: The Airspeed "Envoy" (2 Wolseley A.R.9 engines) has the same type of retractable undercarriage as the "Courier."
G.A. drawing of the single-seater "Altair."
TO ASSIST SALES: The "Leopard Moth" recently acquired by Morris Motors Ltd.
A Boeing B-9, fitted with two Pratt and Whitney "Hornets" of 600 h.p. each.
A NIGHT RAID: A "Virginia" making its way to its objective. The photograph is taken from the cockpit in the tail of the bomber.
A TWO-SEATER PURSUIT TYPE: The Consolidated P-25 is being followed up by a later version, the P-30, with Curtiss "Conqueror" engine.
A BOMBARDMENT TYPE: The Douglas B-7 is powered by two Curtiss "Conqueror" engines of 650 h.p. each.
THE DOUGLAS VERSION: Like the Fokker O-27, the Douglas O-35 is an Army Observation type. It has the same type of engines, but is strutbraced and has the "gull's wing" formation.
The Douglas O-31 with the Curtiss "Conqueror" engine.
The Douglas O-43 with the Curtiss "Conqueror" engine.
A FORCED LANDING WITH REASON: Just in case there may be some pilots who underrate the weather with which our English pilots have to contend, we publish this photograph of a Short "Scion" taken by Mr. A. Irwin just after he had decided that discretion was the better part of valour. The black clouds behind the machine are right down on the ground and completely block the way to Aberdeen, a journey which on this occasion took seven hours instead of the usual three or so.
FOR ARMY OBSERVATION: The Fokker O-27 is designed for long-range reconnaissance. It is fitted with two Curtiss "Conqueror" engines of 650 h.p. each.
The Fokker B-8 with two Curtiss "Conquerors."
FOR JAPANESE COMMUNICATIONS: Built by the Kawasaki Dockyard Co., Ltd., of Hyogo, Kobe, the K.D.C.-5 shown here is a three-seater monoplane intended for rapid communication work. Fitted with a Kawasaki IX liquid-cooled engine of about 800 h.p., the top speed is claimed to be well over 200 m.p.h., the ceiling more than 20,000 feet and the endurance about 7 hours.
THE STINSON "MODEL A": This feeder line machine, with three 240 h.p. Lycoming engines, was described in Flight of March 29 this year. Eight passengers are carried at a cruising speed of 150 m.p.h. at 5,000 ft.. and the landing gear is of course retractile.
The first Blackburn Machine: This monoplane was designed, built and flown by Mr. Robert Blackburn at Saltburn in 1909. The machine is here seen on the sands.
THE "RUBBERNECK" OF THE FUTURE? This "interior" view (looking towards the tail) of the Nieuport-Delage 590 Col. 3, General Purpose Colonial monoplane, in which the aft portion of the cabin is left open, suggests a possible use for this form of arrangement for aerial sight-seeing chars-a-banc. The all-round view thus obtained would be excellent.
SEEING FOR THEMSELVES: Last week members of the Parliamentary Air Committee paid a visit to the Filton Works of the Bristol Aeroplane Company, flying down in the Short "Syrinx" (shown arriving). On the left, Mr. Herbert Thomas, of the Bristol Company, is welcoming members of the party.
A machine in the army observation class, the Curtiss O-40B, but having a 700 h.p. Wright "Cyclone" radial air-cooled engine.
The Caudron "Rafale," which, piloted by Lacombe, won the Deauville-Cannes leg of the race and finished second in the final classification. This machine won "Les Douze Heures d'Angers."
FOR AMERICAN WORLD FLIGHT: The Sikorsky seaplane which, piloted by Fit. Com. William H. Alexander and with about five others aboard, is making a "Goodwill Flight" round the world.
HIGH EFFICIENCY. Mr. Warren Eaton's Bowlus du Pont sailplane. With wing flaps and a single landing wheel, this machine can be landed in very small fields at 21 m.p.h. It has a span of 62 ft., an aspect ratio of 18.8 to 1, a gliding angle of 30 to 1 at 34 m.p.h. and a sinking speed of 1.6 ft./sec.
LATEST CURTISS-WRIGHT AMPHIBIAN: Designed by and built under the supervision of Capt. Frank Courtney, this new amphibian, constructed by the Curtiss Wright Co., has a top speed of 151 m.p.h. and cruises at 125 m.p.h. Three wheels are employed in the undercarriage, one wheel being in the nose of the hull. Thus the machine lands in the same position on land as on water. A Wright "Whirlwind" engine of 330 h.p. is mounted as a pusher some distance away from the propeller, which is driven by an extension shaft. Four passengers, a pilot, and 240 lb. of baggage are carried.
AN AUSTRALIAN PRODUCTION: The Wicko "Wizard" (Cirrus II engine) was designed by Mr. G. N. Wikner and does 120 m.p.h. Mr. Wikner is now looking for suitable opportunities in this country.
THE CURTISS "SWIFT": Note the enclosed cockpit. The engine is a "Conqueror."


would transform him into a nationwide punchline. Thanks for the latter goes to MSNBC's Rachel Maddow, who last night zinged Coffman in hilarious (unless you're him) fashion.
To recap: Toward the end of his remarks at a May 12 fundraiser in Elbert County, Coffman said, "I don't know whether Barack Obama was born in the United States of America. I don't know that. But I do know this, that in his heart, he's not an American. He's just not an American." He subsequently issued an apology, saying, "I misspoke" -- a popular excuse among politicians.
A bad moment, no doubt -- but this error was compounded when 9News anchor/reporter Kyle Clark pigeonholed Coffman at an event earlier this week and tried to get something more out of him than a canned statement. But no: Coffman, looking like a remote-control automaton, told Clark, "I stand by my statement that I misspoke and I apologize." And then he said it again. And again. And again. And again, bringing the total to five -- after which Clark, who was studiously polite throughout, mercifully let Coffman run away.
This self-perpetuating meme is ideal for ridicule, and Maddow spent much of her program last night building up to it -- including an intro during which she likened the 9News clip in advance to the moment when Caitlin Upton, the 2007 Miss South Carolina Teen USA, answered a pageant question by rambling that "people out there in our nation don't have maps and, uh, I believe that our, uh, education like such as in South Africa and, uh, the Iraq...."
Will other progressives join the fun and transform Coffman into a poster child for all that's wrong with today's Republican Party? It's too early to know for certain yet -- but bet plenty of them try. Check out the intro and the Coffman video below.
More from our Politics archive: "John Hickenlooper, Mike Coffman could see political fallout, pollster says."


Up until about three weeks ago, my experience with Italy and its coastline consisted of Naples, Capri, and Venice – not exactly the most budget-friendly destinations in some respects.
As part of my ongoing work with the Emilia Romagna Tourism Board and the BlogVille project, I was back in Italy two weeks ago for World Ducati Week – which coincidentally was being held in part on the Italian seaside.
We were based in the beach town of Rimini and also had the chance to party in the nearby town of Riccione – both excellent options for budget-conscious travellers looking to experience the calm of the Adriatic during the day and the all-night party atmosphere this region is known for.
We were quite lucky to have beachfront accommodations provided by the Emilia Romagna Tourism Board, but with over 1,000 hotels and lodging options to choose from in the city, there are options at every budget level, including those seeking hostels.
During the day the vibe is quite casual and laid back, but as the sun begins to set, it is not uncommon to see everyone dressed for the club, even if they are just grabbing a gelato and coffee at a nearby café.
While there is a historic city center and ancient sights worth checking out, it is immediately evident the draw to Rimini is the beach and nightlife. You can't walk a block near the beach without someone asking "Disco tonight?" With as many bars, clubs, and restaurants as there are hotels, there is no shortage of nightlife whether you are looking for a quiet local pub or a discothèque.
Food is relatively inexpensive and if you go with the local specialty, the piadina, you can stuff yourself for only several euros a meal. Piadina is a thin flatbread that they fill with a variety of meats and cheeses, or even sweet versions like Nutella. I mistakenly assumed I would tire of eating piadina after a few days, but I've been gone nearly two weeks and I find myself craving it again. Thankfully, I'm back in Rimini until July 14th so I can get my fill (and possibly a traditional dish to make piadina at home).
The people are friendly, and while Italian and Russian visitors seem to be most prevalent, we did encounter a number of English speakers throughout the week. From our favorite barista to the guys we hung out with nearly every night that worked at one of the local pubs, I found Rimini to be one of the most welcoming cities in my Italian travels.
Riccione is quite close to Rimini – about 10-15 minutes by car or you can take the train as well.
We were in Riccione for the party and concert for World Ducati Week, which was one of the most amazing events in itself with all the fans, legendary racers, and the Ducati All-Star Band (who I randomly got to talk to the following day at the Bologna Airport) coming together to celebrate the world's best motorcycle.
After the event, we headed to one of the most popular spots in Riccione – the all night beach bars. There are four or five bars that charge no cover and they are all packed – literally breathing room only. Being diligent, we checked out each of them until we found one that had our preferred vibe and mix of dance music.
No all-nighter is complete without a good sunrise and my last morning at BlogVille proved to be a winner. We left the clubs in time to drive back to Rimini and catch the sunrise from our beach and it definitely didn't disappoint.
Carriers like Ryan Air fly into Rimini, although flights tend to be more expensive than other airports. I recommend flying elsewhere in Italy (I flew in and out of Bologna) and then take the train to Rimini and on to Riccione. The train from Bologna to Rimini takes anywhere from one hour for the high-speed to just under two hours for the regular train. You can also rent a car, but there can be a lot of traffic during peak times so taking the train and buses may be the best option.
I was a guest in Emilia Romagna at the invite of the tourism board and Ducati Motorcycles but all views and opinions expressed are my own.


text:
Q: PyGTK custom signals send a list as parameter I'm trying to add a custom signal to a class -
class TaskBrowser(gobject.GObject):
    __list_signal__ = (gobject.SIGNAL_RUN_FIRST, gobject.TYPE_NONE, (<List datatype>,))
    __gsignals__ = {'tasks-deleted': __list_signal__}

    ...

    def on_delete_tasks(self, widget=None, tid=None):
        ...
        gobject.idle_add(self.emit, "tasks-deleted", deleted_tasks) #deleted_tasks is of type 'list'
        ...

    ...

In the __gsignals__ dict, when I state list as parameter type, I get the following error traceback -
File "/home/manhattan/GTG/Hamster_in_hands/GTG/gtk/browser/browser.py", line 61, in <module>
  class TaskBrowser(gobject.GObject):
File "/usr/lib/python2.7/site-packages/gobject/__init__.py", line 60, in __init__
  cls._type_register(cls.__dict__)
File "/usr/lib/python2.7/site-packages/gobject/__init__.py", line 115, in _type_register
  type_register(cls, namespace.get('__gtype_name__'))
TypeError: Error when calling the metaclass bases
  could not get typecode from object

I saw the list of possible parameter types, and there's no type for list
Is there a way I can pass a list as a signal parameter ?

A: The C library needs to know the C type of the parameters, for Gtk, Gdk, Gio and GLib objects the types in the wrappers will work as they mirror the C types in the Gtk and family libraries.
However, for any other type you need to pass either object or gobject.TYPE_PYOBJECT. What that means is that the on the C side a "python object" type is passed. Every object accessible from a python script is of that type, that pretty much means anything you can pass through your python script will fit an object parameter.
That also means, of course, that this feature doesn't work in python! Python relies on duck typing, that means we figure out if an object is of a type when we do stuff with it and it works. Passing the type of the parameter works for C as a way to make sure the objects passed are of the type the program needs them to be, but in python every object is of the same type in the C side so this feature becomes completely useless on the python side.
But that doesn't means it is completely useless overall. For example, in python int is an object. But not in C. If you are using property bindings, which were coded in the C side of the Gtk library, you will want to specify the appropriate type as bindings of different property types don't work.
Using C side wrapped signal handlers with object parameter types is also bound not to work, since the C side needs a specific type to function.

A: In pygtk3 this error has been occured for me because import gobject directly.
and fixed this error by from gi.repository import GObject.
you can see details in this link.

redpajama_set_name: RedPajamaStackExchange


← Can you regain standing?
The Progress of Political Thought Influenced by… Satire?
Analyzing the texts of Rand and Kemmis gave perspective on what constitutes American civic republican and classic liberal political thought. But, how did this political thought come about? How is it able to define our modern perspective of democracy today? Of course, there are many reasons why political thought emerges throughout history, but what mechanisms allow such thought to be progressed? Before our contemporary idea of civic republicanism, civic republican political thought emerged from Machiavelli's ideas about civic virtù, which he defines as the "ideal citizenship which requires people to put their country first" or "acting to promote the common good" (Civic Republicanism, Honohan). During the Age of Enlightenment, (which many may know was an important period during English history for political change) ideas that characterize modern democratic thought not only emerged, but began to grow in response to England's corrupt, monarchical government. It was also during this time that satire was a mechanism influencing political thought.
After reading many satirical works during the time of the Age of Enlightenment, it is evident that many of the satirists wrote very politically, which sometimes led them to portray classic liberal and civic republican ideals in their writing. For example, Jonathan Swift represented his political ideals in Gulliver's Travels. In portraying his political opinions of England's government corruption, he progresses Machiavelli's concept of civic virtù, and he begins to supportively represent characteristics of what is now modern civic republicanism. Satire was the mechanism used to urge and promote a more politically democratic thought, which eventually led to the characteristics of future modern, American political thought and democracy.
Because satire was a mechanism leading to the progression of political thought during the eighteenth century in England, I began to wonder if, perhaps, satirical works of today also have the same affect in influencing public opinion and the progress of political thought. I guess another question would be what defines a political satire today. It is obvious that television in general plays an important role in the influencing the daily decisions of many individuals during this time. What I constitute as a political satire of this time, and even go as far as saying influences the political thought and opinions of its viewers, is The Daily Show with Jon Stewart. Jon Stewart is not just a comedian, but he is a political satirist. Of course, I would assume that many of the show's viewers watch the show mainly for entertainment purposes; however, the opinions and mockery given by Jon Stewart are not only entertaining, but also insightful in producing public opinion on the current state of American government decision-making and of the political world in general. Unfortunately, Youtube did not have a good selection of clips from the show, but, if accessible, here is a good clip from Hulu that I found from a recent episode of the show.
This Hulu clip from The Daily Show with Jon Stewart is called "Superbad," which focuses on Congress's decisions in dealing with the budget deficit. Stewart ridicules Congress for their inefficiency and incompetence in dealing with this issue. Although Jon Stewart's mockery is comedic, I would say that he, in the process of entertaining, does manage to influence some of his viewers to develop a political opinion on the issue. Would Jon Stewart's influence ultimately lead people to change their votes and their association with political parties? Could this possibly lead to a change in political thought if people see an inefficient democratic government? It could even be argued that Jon Stewart's comedic representations and opinions of the American government are wrong and do not aid in promoting political, democratic change. Here is a clip in which an individual questions if The Daily Show with Jon Stewart is actually good for democracy.
Do you believe shows such as The Daily Show with Jon Stewart are good for democracy? Do you believe that political satires of today in the media have a role in influencing political thought, and even perhaps influence in changing perspectives of American democracy?


Sugatsune's unique and modern cabinet door hardware designs and other specialty door systems will give you more space by replacing traditional cabinet swinging doors in a stylish & elegant way.
LIN-X corner cabinet door hinge and FAD lateral swing open door hardware are an excellent alternative to pie cut corner hinges and are ideal for accessing hard-to-utilize spaces such as kitchen and garage corners.
RB-600 reversible cabinet system rotates 180° allowing effortless access to both sides cabinet door: designed for storage and display installations in high-end residential and retail settings. Our award-winning cabinet hardware not only reduces "lost" cabinetry space but adds a touch of understated elegance to your cabinetry.