text:
\section{Introduction}
\glsunset{mimo}

\IEEEPARstart{I}{n} \gls{mami} an unconventionally high number of \gls{bs} antennas (hundreds or even higher) is employed to serve \emph{e.g., } a factor of ten less \glspl{ue}.
Due to the excess number of \gls{bs} antennas, linear signal processing may be used to spatially focus energy with high precision, allowing to separate a multitude of \glspl{ue} in the spatial domain while using the same time/frequency resource~\cite{Marzetta2010}.
\Gls{mami} theory promises a variety of gains, \emph{e.g., } increase in spectral and energy efficiencies as compared with single antenna and traditional MU-MIMO systems\cite{rusek2013,Rate}, thereby tackling the key challenges defined for 5G.

Although \gls{mami} is a promising theoretical concept, further development requires prototype systems for proof-of-concept and performance evaluation under real-world conditions to identify any further challenges in practice.
Because of its importance, both industry and academia are making efforts in building \gls{mami} testbeds, including the Argos testbed with 96-antennas\cite{Argos1}, Eurecom's 64-antenna \gls{lte} compatible testbed, Samsung's Full-Dimension (FD) \gls{mimo} testbed and Facebook's Project Aries.
Nevertheless, publications systematically describing the design considerations and methodology of a \gls{mami} testbed are missing and real-time real-scenario performance evaluation of \gls{mami} systems using testbeds have not been reported yet.
At Lund University, the first real-time \gls{mami} testbed, the \gls{lumami} testbed, showing successful \gls{mami} transmission on the \gls{ul}, was built\cite{Vieira2014a}.
Ever since, many testbeds have been constructed based on identical \gls{hw} utilizing the same generic design principle, \emph{e.g., } the \gls{mami} testbeds at the University of Bristol\cite{Bristol1}, Norwegian University of Science and Technology in Trondheim and University of Leuven in Belgium.
The \gls{lumami} testbed provides a fully reconfigurable platform for testing \gls{mami} under real-life conditions.
To build a real-time \gls{mami} testbed many challenges have to be coped with.
For example, shuffling data from 100 or more antennas, processing large-scale matrices and synchronizing a huge number of physically separated devices.
All this has to be managed while still ensuring an overall reconfigurability of the system allowing experimental hardware and software solutions to be tested rapidly.

This paper discusses how implementation challenges are addressed by first evaluating high-level \gls{hw} and system requirements, and then setting up a generic framework to distribute the data shuffling and processing complexity in a \gls{mami} system based on the given \gls{hw} constraints for interconnection network and processing capabilities.
Taking into account the framework and requirements, a suitable modular \gls{hw} platform is selected and evaluated.
Thereafter, a thorough description of the \gls{lumami} testbed is provided including system parameters, base-band processing features, synchronization scheme and other details.
The \gls{lumami} testbed constitutes a flexible platform that supports prototyping of up to 100-antenna \SI{20}{\mega\hertz} bandwidth \gls{mami}, simultaneously serving 12 \glspl{ue} in real-time using \gls{ofdm} modulation in \gls{tdd} transmission mode.
\Glspl{ber} and constellations for real-time \gls{ul} and \gls{dl} uncoded transmission in a static indoor and static outdoor scenario are presented.
Our first real-life proof-of-concept measurement campaigns show, that \gls{mami} is capable of serving up to 12 \glspl{ue} in the same time/frequency resource even for high user density per unit area.
The gathered results suggest a significant increase in spectral efficiency compared to traditional point-to-point \gls{mimo} systems.
By building the \gls{lumami} testbed we now have a tool which supports accelerated design of algorithms\cite{DBLP:journals/corr/VieiraREMLT16} and their validation based on real measurement data, with the additional benefit of real-world verification of digital base-band solutions.

Our main contributions can be summarized as follows:
\begin{itemize}
	\item We provide overall and thorough analysis for \gls{mami} systems, especially from a signal processing perspective, and identify design requirements as well as considerations on building up a \gls{mami} testbed.
	\item We propose signal processing breakdown and distribution strategy to master the tremendous computational complexity in a \gls{mami} system and introduce general hardware architecture for a \gls{mami} testbed.
	\item We present the world's first real-time 100-antenna \gls{mami} testbed, built upon \gls{sdr} technology.
	\item We validate the \gls{mami} concept and its spatial multiplexing capability in real-life scenarios (both indoor and outdoor) with over-the-air transmission and real-time processing.
\end{itemize}

\section{Massive MIMO Basics}
\label{sec:MaMi_Theory}
\begin{figure}[!t]
	\centering
	\includegraphics[width=\linewidth]{mami_theory_sys.pdf}
	\caption{A \gls{mami} system model. Each antenna at the BS (left side) transmits a linear combination of $K$ user-intended data symbols ${u_k}_{k=1}^K$. After propagation through the DL wireless channel $\bm{B}$, each user antenna receives a linear combination of the signals transmitted by the M BS antennas. Finally, each of the K users, say user k, produces an estimate of its own intended data symbol, \emph{i.e., } $u_k$. Similar operation is employed for UL data transmission. Here, reciprocity for the propagation channel is assumed, \emph{i.e., } $\bm{B}=\bm{B}^\mathsf{T}$.}
	\label{fig:system_model}
\end{figure}
In this section, the basic key detection and precoding algorithms utilized in \gls{mami} are presented.
Implementation specific details required to apply these algorithms, such as \gls{csi} estimation, are discussed in Sec.~\ref{sec:lumami_testbed_implementation}.
A simplified model of a \gls{mami} \gls{bs} using $M$ antennas while simultaneously serving $K$ single antenna \glspl{ue} in \gls{tdd} operation in a propagation channel $\bm{B}$ is shown in \figurename~\ref{fig:system_model}.
To simplify notation, this discussion assumes a base-band equivalent channel and expressions are given per subcarrier, with subcarrier indexing suppressed throughout.

\begin{table}[t]
	\centering
	\renewcommand{\arraystretch}{1.3}
	\setlength{\abovecaptionskip}{0pt}
	\footnotesize
	\caption{Linear Precoding/Detection Matrices}
	\scalebox{0.95}{
	\noindent\begin{tabular}{cccc}
		\toprule
		& \textbf{MRT/MRC} & \textbf{ZF} & \textbf{RZF} \\
		\midrule
		DL & $\bm{C}\bm{G}^*$ & $\bm{C}\bm{G}^*(\bm{G}^\mathsf{H}\bm{G})^{\mathsf{-T}}$ & $\bm{C}\bm{G}^*(\bm{G}^\mathsf{H}\bm{G} + \beta_{\mathrm{reg}_\mathrm{pre}}\bm{I}_{K})^{\mathsf{-T}}$\\
		UL & $\bm{G}^\mathsf{H}$ & $(\bm{G}^\mathsf{H}\bm{G})^{-1}\bm{G}^\mathsf{H}$ & $(\bm{G}^\mathsf{H}\bm{G} + \beta_{\mathrm{reg}_\mathrm{dec}}\mathbf{I}_{K})^{-1}\bm{G}^\mathsf{H}$ \\
		\bottomrule
	\end{tabular}}
	\label{tab:Linear_Det_Pre_Schemes}
\end{table}

\subsection{Up-link}
The \gls{ul} power levels used by the $K$ \glspl{ue} during transmission build the $K \times K$ diagonal matrix $\bm{P}_\mathrm{ul}$. By collecting the transmitted \gls{ue} symbols in a vector $\bm{z} \triangleq (z_1,\ldots,z_K)^\mathsf{T}$, the received signals $\bm{r} \triangleq (r_1,\ldots,r_M)^\mathsf{T}$ at the \gls{bs} are described as
\begin{align}\label{eq:ul_model}
    \bm{r} = \bm{G}\sqrt{\bm{P}_\mathrm{ul}}\bm{z} + \bm{w},
\end{align}
where $\bm{G}$ is the $M\times K$ \gls{ul} channel matrix\footnote{$\bm{G}$ is the up-link radio channel capturing both, the propagation channel $\bm{B}^\mathsf{T}$ and the up-link hardware transfer functions.}, $\sqrt{\bm{P}_\mathrm{ul}}$ an elementwise square-root, and $\bm{w} \sim  \mathcal{CN}(0,\mathbf{I}_M)$ is \gls{iid} circularly-symmetric zero-mean complex Gaussian noise.
The estimated user symbols $\widehat{\bm{z}} \triangleq (\hat{z_1},\ldots,\hat{z_K})^\mathsf{T}$ from the $K$ \glspl{ue} are obtained by linear filtering of the received vector $\bm{r}$ as
\begin{align}
\label{eq:Detector}
    \widehat{\bm{z}} = f_\text{eq}(\bm{G})\bm{r},
\end{align}
where  $f_\text{eq}( \cdot )$ constructs an appropriate equalization matrix.

\subsection{Down-link}
On the \gls{dl}, each \gls{ue} receives its corresponding symbol $\hat{u}_k$ which are collected in a vector $\widehat{\bm{u}} \triangleq  (\hat{u}_1, \ldots ,\hat{u}_K)^\mathsf{T}$, representing the symbols received by all \glspl{ue}. With this notation, the received signal becomes
\begin{align}
\label{eq:DLchannel}
\widehat{\bm{u}} = \bm{H}\bm{x} + \bm{w}'
\end{align}
where the $K \times M$ matrix $\bm{H}$ is the \gls{dl} radio channel\footnote{$\bm{H}$ is the down-link radio channel capturing both, the propagation channel $\bm{B}$ and the down-link hardware transfer functions.},  $\bm{w}' \sim  \mathcal{CN}(0,\mathbf{I}_K)$ is an \gls{iid} circularly-symmetric zero-mean complex Gaussian receive noise vector with covariance matrix $\mathbf{I}_K$, and ${\bm{x} \triangleq  (x_1, \ldots , x_M)^\mathsf{T}}$ is the transmit vector.

As explicit \gls{dl} channel estimation is very resource consuming, it is not considered practical in a \gls{mami} setup\cite{Marzetta2010}.
Taking into account that the propagation channel $\bm{B}$ is generally agreed on to be reciprocal\cite{DBLP:journals/corr/VieiraREMLT16}, the estimated \gls{ul} channel matrix $\bm{G}$ can be utilized to transmit on the \gls{dl}.
However, differences due to analog circuitry in the \gls{ul} and \gls{dl} channels, $\bm{G}$ and $\bm{H}$, need to be compensated.
Thus, a possible construction for $\bm{x}$ is of the form
\begin{align}
\label{eq:SteffensAss}
\bm{x} =   f_\text{cal}(  f_\text{pre}(\bm{G}))\bm{u},
\end{align}
where $\bm{u} \triangleq  (u_1, \ldots ,u_K)^\mathsf{T}$ is a vector containing the symbols intended for the $K$ \glspl{ue}, $f_\text{pre}( \cdot )$ is some precoding function, and $f_\text{cal}( \cdot )$ is a reciprocity calibration function to be discussed next.

\subsection{Reciprocity Calibration}
\label{subsec:rec_cal}

In most practical systems, the \gls{ul} and \gls{dl} channels are not reciprocal, i.e. $\bm{G} \neq \bm{H}^T$. This is easily seen by factorizing $\bm{G}$ and $\bm{H}$ as
\begin{equation}
\label{eq:Gfact}
\bm{G} = \bm{R}_{\rm B}  \bm{B}^\mathsf{T} \bm{T}_{\rm U}, \; \; \; {\rm and} \; \; \; \bm{H} = \bm{R}_{\rm U}  \bm{B} \bm{T}_{\rm B},
\end{equation}
where the two $M\times M$ and $K\times K$ diagonal matrices $\bm{R}_{\rm B}$ and $\bm{R}_{\rm U}$ model the non-reciprocal hardware responses of \gls{bs} and \gls{ue} \glspl{rx}, respectively, and the two $M\times M$ and $K\times K$ diagonal matrices $\bm{T}_{\rm B}$ and $\bm{T}_{\rm U}$ similarly model hardware responses of their \glspl{tx}. Thus, in order to construct a precoder based on the \gls{ul} channel estimates, the non-reciprocal components of the channel have to be calibrated. Previous calibration work showed that this is possible by using
\begin{equation}
\label{eq:Calibration}
\bm{C} f_\text{pre}(\bm{G}) = f_\text{cal}(  f_\text{pre}(\bm{G})),
\end{equation}
where $\bm{C}=\bm{R}_{\rm B} \bm{T}^{-1}_{\rm B}$ is the, so-called, calibration matrix which can be estimated internally at the \gls{bs}\cite{DBLP:journals/corr/VieiraREMLT16}. Such calibration is sufficient to cancel inter-user interference stemming from non-reciprocity\cite{6760595}.

\subsection{Linear Detection \& Precoding Schemes}

Table \ref{tab:Linear_Det_Pre_Schemes} shows a selection of weighting matrices used in linear precoding and detection schemes, with non-reciprocity compensation included in the form of the $M \times M$ diagonal matrix $\bm{C}$ as defined above.
The \gls{mrt} precoder and the \gls{mrc} decoder maximize array gain without active suppression of interference among the \glspl{ue} \cite{Marzetta2010}.
The \gls{zf} precoder and \gls{zf} combiner employ the pseudo-inverse, which provides inter-user interference suppression with the penalty of lowering the achievable array gain.
A scheme that allows trade-off between array gain and interference suppression is the \gls{rzf} precoder and \gls{rzf} combiner.
This is achieved by properly selecting the regularization constants $\beta_{\mathrm{reg}_\mathrm{pre}}$ and $\beta_{\mathrm{reg}_\mathrm{dec}}$.
If $\beta_{\mathrm{reg}_\mathrm{pre}}$ and $\beta_{\mathrm{reg}_\mathrm{dec}}$ are selected to minimize \gls{mse} $E{\|\bm{u} - \tfrac{1}{\sqrt{\rho}}\hat{\bm{u}}\|^2}$, where $\rho$ is a scaling constant, we obtain the \gls{mmse} precoder/detector \cite{Bjoernson2014a}.

\section{System Design Aspects}
\label{sec:System_Design_Aspect}

Having discussed the \gls{mami} basics, we move on to system design aspects.
These include modulation scheme, frame structure and hardware requirements.

\subsection{Modulation Scheme}
While many different modulation schemes can be used with \gls{mami}, this paper focuses on \gls{ofdm}, employed in many modern wireless communication systems.
Properly designed \gls{ofdm} renders frequency-flat narrowband subcarriers, facilitating the single channel equalization strategy used here.

For ease of comparison and simplicity, \gls{lte}-like \gls{ofdm} parameters, as shown in Table~\ref{table:SystemParam}, are used throughout this discussion.
The more common parameters with \gls{lte}, the easier it is to evaluate how \gls{mami} as an add-on would influence current cellular systems.
\begin{table}[t]
	\footnotesize
	\caption{High-level system parameters}
	\noindent\begin{tabular*}{\columnwidth}{@{\extracolsep{\stretch{1}}}*{7}{l}@{}}
		\toprule
		\textbf{Parameter} & \textbf{Variable} & \textbf{Value} \\
		\midrule
		Bandwidth & $W$ & 20\,MHz \\
		%
		Sampling Rate & $F_\text{s}$ & 30.72\,MS/s \\
		FFT Size & $N_\text{FFT}$ & 2048 \\
		\# Used subcarriers & $N_\text{used}$  & 1200 \\
		Cyclic prefix & $N_\text{cp}$  & 144 samples \\
		OFDM symbol length & $t_\text{\tiny{OFDM}}$ & \SI{71.4}{\micro\second}\\
			%
		%
		%
		\bottomrule
	\end{tabular*}
	\label{table:SystemParam}
\end{table}

\subsection{TDD versus FDD}
Current cellular systems either operate in \gls{fdd} or \gls{tdd} mode.
\gls{fdd} is, however, considered impractical for \gls{mami} due to excessive resources needed for \gls{dl} pilots and \gls{csi} feedback. \gls{tdd} operation relying on reciprocity only requires orthogonal pilots in the \gls{ul} from the $K$ \glspl{ue}, making it the feasible choice \cite{7402270}. For this reason, we focus entirely on \gls{tdd} below.

\subsection{Reciprocity}
To allow operation in \gls{tdd} mode, differences in the \gls{tx} and \gls{rx} transfer functions on both, the \gls{bs} and \glspl{ue} have to be calibrated as discussed in Sec.~\ref{subsec:rec_cal}.
Drifts over time are mainly caused by \gls{hw} temperature and voltage changes, and thus, the calibration interval depends on the operating environment of the \gls{bs}.

\subsection{Frame Structure}
The frame structure defines among other things, the pilot rate which determines how well channel variations can be tracked and, indirectly, the largest supported UE speed.

\subsubsection{Mobility}
The maximum supportable mobility, \emph{e.g., } the maximum speed of the \glspl{ue} is defined by the \gls{ul} pilot transmission interval.
In order to determine this constraint, a 2D wide-sense stationary channel with uncorrelated isotropic scattering is assumed.
For the contributions from the different \gls{bs} antennas to add up coherently high channel correlation is required and, as an approximation to formulate the final requirement, a correlation of $0.9$ was used to ensure sufficient channel coherency.
Further discussions on such modeling assumption are found in\cite{molisch2010wireless}.
Although these assumptions may not be completely valid for \gls{mami} channels, they allow an initial evaluation based on a maximum supported Doppler frequency, $\nu_\text{max}$, by solving
\begin{align}
J_\text{0}(2\pi\nu_\text{max}T_\text{p}) = 0.9,\label{eq:bessel}
\end{align}
for $\nu_\text{max}$, where $J_\text{0}(\cdotp)$ is the zeroth-order Bessel function of the first kind, stemming from a standard Jakes' fading assumption, and $T_\text{p}$ the distance between pilots in time.
Hence, the maximum supportable speed of any \gls{ue} may be evaluated using
\begin{align}
\text{v}_\text{max}= \frac{c\nu_\text{max}}{f_\text{c}},\label{eq:Mobility}
\end{align}
once a specific frame structure is provided.
In (\ref{eq:Mobility}) $\text{v}_\text{max}$ is the maximum supported speed of a \gls{ue}, $c$ the speed of light and $f_c$ the chosen carrier frequency.
\subsubsection{Processing latency}
The frame structure has to be designed for the highest speed of \glspl{ue} to be supported which requires a high pilot rate for high mobility scenarios.
Within two consecutive \gls{ul} pilot symbols, all \gls{ul} data, \gls{dl} data and guard symbols have to be accommodated	 which in turn decreases the available time between UL pilot reception and DL transmission.
In a high mobility scenario this poses tight latency requirements for \gls{tdd} transmission as \gls{csi} has to be estimated in order to produce the precoding matrix to beamform the \gls{dl} data.

To formulate the \gls{tdd} precoder turnaround time, $\Delta$, all \gls{hw} units introducing a delay must be taken into account.
This includes the analog front-end delays for the \gls{tx} $\Delta^\text{rf,\tiny{TX}}$ and \gls{rx} $\Delta^\text{rf,\tiny{RX}}$, the processing latency for \gls{ofdm} modulation/demodulation (including \gls{cp} and guard band operation) $\Delta^\text{\tiny{OFDM}}$, the time for processing \gls{ul} pilots to estimate \gls{csi} $\Delta^\text{\tiny{CSI}}$, and the processing latency for precoding $\Delta^\text{precode}$ including reciprocity compensation.
Additional sources of latency include overhead in data routing, packing, and unpacking, \emph{i.e., } $\Delta^\text{rout}$ such that the overall \gls{tdd} precoder turnaround time may be formulated as
\begin{equation}
\Delta = \Delta^\text{rf,\tiny{TX}}+\Delta^\text{rf,\tiny{RX}}+\Delta^\text{\tiny{OFDM}}+\Delta^\text{\tiny{CSI}} + \Delta^\text{precode} + \Delta^\text{rout}.
\label{eq:latency}
\end{equation}
Depending on the specific arrangement of the \gls{ofdm} symbols and the pilot repetition pattern in the frame structure, base-band processing solutions, especially $\Delta^\text{\tiny{CSI}}$ and $\Delta^\text{precode}$, have to be optimized to not violate the given constraint, \emph{i.e., } $\Delta$.

\subsubsection{Pilot pattern}
In general, to acquire \gls{csi} at the \gls{bs}, the $K$ \glspl{ue} transmit orthogonal pilots on the \gls{ul}.
Different approaches are, \emph{e.g., } distributed pilots over orthogonal subcarriers\cite{deliverable3_2} or sending orthogonal pilot sequences over multiple subcarriers\cite{6783987,Pilot1,Pilot2} but also semi-blind and blind techniques have been proposed\cite{7339665}.

\figurename~\ref{fig:frame_structure_generic} shows a generic frame structure capturing the aforementioned aspects in a hierarchical manner assuming all \glspl{ue} transmit their pilots within one dedicated pilot symbol.
\begin{figure}[!t]
	\centering
	\includegraphics[width=0.9\columnwidth]{frame_structure_generic}
	\caption{Generic frame structure of a \gls{lte} like \gls{tdd}-based \gls{mami} system. Within one BS reciprocity cycle the BS operates using the same reciprocity calibration coefficients. A certain number of DL pilot cycles are integrated as UEs suffer from faster changing environments. Each control cycle contains a control layer to perform, for example over-the-air synchronization and within these the data transmission slots are encapsulated.}
	\label{fig:frame_structure_generic}
\end{figure}
At the beginning of each \gls{bs} reciprocity cycle, reciprocity calibration at the \gls{bs} is performed and within these a certain number of \gls{dl} pilot cycles are encapsulated where precoded \gls{dl} pilot symbols are transmitted.
The length of the BS reciprocity cycle is determined by the stability of the transceiver chains in the \gls{bs}.
As the reciprocity calibration at the \gls{bs} side only compensates for \gls{bs} transceivers, \gls{dl} pilots are necessary to compensate for transceiver differences at the \gls{ue} side.
Their frequency depends on the stability at the \gls{ue} side and can be considered significantly smaller than for the \gls{bs} as \glspl{ue} are subject to faster changes in their operational environment, \emph{e.g., } thermal differences when having the \gls{ue} in a pocket or using it indoors or outdoors.
To be able to send precoded pilots on the \gls{dl}, transmission of \gls{ul} pilots is required beforehand.
Several control cycles are embedded inside each \gls{dl} pilot cycle carrying a certain number of data time slots.
Time slots contain five different \gls{ofdm} symbol types for physical layer implementation.
These are \emph{(i)} UL Pilot where the \glspl{ue} transmit orthogonal pilots to the \gls{bs}, \emph{(ii)} UL Data where all \glspl{ue} simultaneously send data to the \gls{bs}, \emph{(iii)} DL Pilot where the \gls{bs} sends precoded pilots to all \glspl{ue}, \emph{(iv)} DL Data where the \gls{bs} transmits data to all \glspl{ue} and \emph{(v)} Switch Guard, which idles the RF chains to allow switching from RX to TX or vice versa.

\subsection{Hardware Requirements}
To illustrate the required \gls{hw} capabilities for the testbed, the values from Table~\ref{table:SystemParam} are used to estimate the \SI{}{Gops\per\second}
\footnote{\SI{}{Gops\per\second} is used here, but these can be seen as \SI{}{GMACs\per\second}, \emph{i.e., } the number of multiply-accumulate operations, as almost all operations involve matrix-matrix and matrix-vector calculations.}
and the data shuffling on a per \gls{ofdm} symbol basis for the general case and a specific case assuming $M=100$ and $K=12$.

\subsubsection{Processing Capabilites}
\label{subsubsec:Processing_Capabilities}
Table~\ref{tab:hw_requirements} summarizes the overall number of real-valued arithmetic operations.
For the processing estimates, it is assumed that each complex multiplication requires four real multiplications.
Close to the antennas, $M$ \glspl{fft} or \glspl{ifft} are needed equating to \SI{126}{Gops\per s}.
Data precoding and detection as well as reciprocity compensation require large matrix and vector multiplications, for instance, an $M\times K$ matrix with a $K\times 1$ vector leading to up to \SI{80}{Gops\per s}.

Finally, when using ZF, the pseudo-inverse matrix is required which includes the calculation of the Gram matrix requiring $MK^2$ multiplications with the $K\times K$ matrix inversion adding another $K^3$ in complexity assuming a Neumann-Series approximation\cite{Prabhu2013} or a QR decomposition. The last multiplication of the inverse with the Hermitian of the channel matrix $\bm{H}$ needs another $MK^2$ multiplications which combined with a requirement of finishing within two \gls{ofdm} symbols leads to approximately \SI{1}{Tops\per\second} for the overall pseudo-inverse calculation.
\begin{table}[t]
	\centering
	\footnotesize
	\caption{Processing Requirements in a MaMi system}
	\noindent\begin{tabular}{llll}
		\toprule
		\textbf{Function} & \textbf{General} & \textbf{Specific} \\
		\midrule
		& \SI{}{Gops\per s} & \SI{}{Gops\per s}\\
		FFT/IFFT & $4 M \log_2(N_\text{FFT})N_\text{FFT}/t_\text{\tiny{OFDM}}$ & \num{126} \\
		Detection & $4 M K N_\text{used}/t_\text{\tiny{OFDM}}$ & \num{80} \\
		Precoding & $4 M K N_\text{used}/t_\text{\tiny{OFDM}}$ & \num{80} \\
		Recip. Cal. & $4 M K N_\text{used}/t_\text{\tiny{OFDM}}$ &\num{80} \\
		Pseudo-inv. & $4 N_\text{used}\left(2 MK^2 + K^3\right)/\left(2t_\text{\tiny{OFDM}}\right)$ & \num{1080}\\
		\bottomrule
	\end{tabular}
	\label{tab:hw_requirements}
\end{table}

\subsubsection{Data Shuffling Capabilities}
Table~\ref{tab:data_shuffling_requirements} summarizes required interconnect bandwidth and number of links.
Communication paths to each antenna transfer at the sampling rate of %
$F_\text{s}=\SI{30.72}{\mega S\per\second}$ which is decreased to the subcarrier rate %
$F_\text{sub}=\SI{16.8}{\mega B\per\second}$ by performing \gls{ofdm} processing $\left(F_\text{s}\cdot N_\text{used}/(N_\text{FFT}+N_\text{cp})\right)$.
Considering $M$ antennas, the overall subcarrier data rate is $M\cdot w\cdot\SI{16.8}{\mega B\per\second}$, with $w$ being the combined wordlength for the in-phase and quadrature components in bytes. %
The information rate in an \gls{ofdm} symbol carrying data is $K\cdot\SI{16.8}{\mega B\per\second}$ assuming $8$ bit per sample, \emph{i.e., } $256-$QAM as highest modulation.
Assuming separate links between centralized processing and the antenna units on \gls{ul} and \gls{dl}, $2M$ \gls{p2p} links\footnote{In this discussion, each interconnection transferring data between physically separated devices is denoted a \glsentryfull{p2p} link.} are needed between the antennas and the centralized \gls{mimo} processing.
\begin{table}[t]
	\centering
	\footnotesize
	\caption{Data Shuffling Requirements in a MaMi system}
	\noindent\begin{tabular}{lll}
		\toprule
		\textbf{Purpose} & \textbf{General} & \textbf{Specific} \\
		\midrule
		& \# & \# \\
		Links to cent. proc & $2M$ & 200\\
		\midrule
		& \SI{}{MB\per s} & \SI{}{MB\per s}\\
		Antenna Rate & $w_\text{ant}MF_\text{s}$ & $w_\text{ant}$ 3,072\\
		Subcarrier Rate & $wMF_\text{sub}$ & $w$ 1,680\\
		Information rate & $K\cdot F_\text{sub}$ & $201.6$ \\
		\bottomrule
	\end{tabular}
	\label{tab:data_shuffling_requirements}
\end{table}

\subsubsection{Reconfigurability}
The testbed has to be reconfigurable and scalable, to support different system parameters, different processing algorithms and adaptive processing.
It is also crucial to have the possibility to integrate in-house developed \gls{hw} designs for validation and performance comparison of algorithms.
Variable center frequencies, run-time adjustable \gls{rx} and \gls{tx} gains as well as configurable sampling rates are highly desirable to be able to adapt to other parameters than the ones presented in Table~\ref{table:SystemParam}.

\section{Generic Hardware and Processing Partitioning}
\label{sec:Generinc_Partitioning}
In this section a generic \gls{hw} and processing partitioning is presented
to explore the parallelism in \gls{mami}, which needs consideration of processing together with data transfer requirements (throughput, latency, \# of \gls{p2p} links), and at the same time provides scalability.

\subsection{Hierarchical Overview}
To be able to build a \gls{mami} testbed with modular \gls{hw} components, a hierarchical distribution as shown in \figurename~\ref{fig:Hierarchical_Overview} is proposed.
The main blocks are detailed as follows:
\begin{figure}
	\centering
	\includegraphics[width=0.85\columnwidth]{Hierarchical_Overview.pdf}
	\caption{Hierarchical overview of a \gls{mami} \gls{bs} built from modular \gls{hw} components.}
	\label{fig:Hierarchical_Overview}
\end{figure}

\subsubsection{SDR} \glspl{sdr} provide the interface between the digital and \gls{rf} domain as well as local processing capabilities.

\subsubsection{Switches} Switches aggregate/disaggregate data between different parts of the system, \emph{e.g., } between \glspl{sdr} and the co-processors.

\subsubsection{Co-processing modules}
Co-processing modules provide a centralized node to perform \gls{mimo} processing.

\subsubsection{Higher Layer Processing}
Higher layer processing controls the system, configures the radios, and provides run-time status metrics of the system.

\subsection{Processing and Data Distribution}
For proper base-band processing partitioning, throughput constraints of \gls{hw} components have to be taken into account.
Assuming each \gls{sdr} supports $n_\text{ant}$ antennas, the required number of \glspl{sdr} becomes $\lceil M/n_\text{ant}\rceil$ for an $M$-antenna system.

\subsubsection{Subsystems}
As shown in \figurename~\ref{fig:subsystem}, RF-Front End, \gls{ofdm} processing and reciprocity compensation are performed on a per-antenna basis using the \glspl{sdr}.
This distributes a large fraction of the overall processing and reduces the data rate before transferring the acquired samples over the bus.
Still, the number of direct devices on a bus is limited, and thus, setting up $2M$ \gls{p2p} links directly to the co-processors would most likely exceed the number of maximum \gls{p2p} links for any reasonable number of \gls{mami} antennas.
To reduce this number, data can be aggregated using the concept of grouping.
The different data streams from several \glspl{sdr} are interleaved on one common \gls{sdr} and then sent via one \gls{p2p} link.
Therefore, subsystems are defined, each containing $n_\text{sub}$ \glspl{sdr}.
Data from all antennas within a subsystem is aggregated/disaggregated on the outer two \glspl{sdr} and distributed to the $n_\text{co}$ co-processors using high-speed routers.
\begin{figure}
	%
	\centering
	%
	\hspace*{0pt}\input{subsystem.tikz}
	\caption{A subsystem consisting of $n_\text{sub}$ \glspl{sdr} where the two outer \glspl{sdr} implement an antenna combiner / BW splitter and an antenna splitter / BW combiner, both implemented using high-speed \glspl{fpga} routers. Inter-SDR and SDR to central processor connections utilize a bus for transferring the samples.}
	\label{fig:subsystem}
\end{figure}

At closer look, \figurename~\ref{fig:subsystem} reveals that the \glspl{sdr} on the outer edges which realize the $(n_\text{ant}n_\text{sub}) \ \text{to}\ (n_\text{co})$ and $(n_\text{co}) \ \text{to}\ (n_\text{ant}n_\text{sub})$ router functionalities, require the highest number of \gls{p2p} links, and thus have to deliver the highest throughput.
Hence, the following inequalities have to be fulfilled for the subsystems not to exceed the constraints for maximum number of \gls{p2p} links ($\text{P2P}_\text{SDR,max}$) and maximum bidirectional throughput ($R_{\text{SDR}_\text{max}}$):
\begin{align}
R_{\text{SDR}_\text{max}} > R_{\text{SDR}_\text{out}}=R_{\text{SDR}_\text{in}}=n_\text{ant}\cdot n_\text{sub}\cdot w\cdot F_\text{sub} \label{eq:R_sdr}\\
\text{P2P}_\text{SDR,max} > \text{P2P}_\text{SDR} = n_\text{co}+n_\text{sub}\label{eq:P2P_sdr}
\end{align}
where it is assumed that if an \gls{sdr} employs more than one antenna, the data is interleaved before it is sent to the router on the outer \glspl{sdr}.
The constraints given in equation (\ref{eq:R_sdr})-(\ref{eq:P2P_sdr}) can be used to determine the maximum number of \glspl{sdr} per subsystem ($n_\text{sub}$) such that hardware constraints are not exceeded.

\subsubsection{Co-processors}
As shown in \figurename~\ref{fig:Co_processors}, detection, precoding, \gls{csi} acquisition, symbol mapping and symbol demapping are integrated in the centrally localized co-processor modules which collect data from all \glspl{sdr}.
Using \gls{csi} estimated from \gls{ul} pilots, \gls{mimo} processing as discussed in Sec.~\ref{sec:MaMi_Theory} and symbol mapping/de-mapping is performed.

Based on the selected \gls{ofdm} modulation scheme the subcarrier independence can be exploited allowing each of the $n_\text{co}$ co-processors to work on a sub-band of the overall \SI{20}{\mega\hertz} bandwidth.
This efficiently circumvents issues with throughput and latency constraints in the \gls{mimo} signal processing chain.
The co-processors aggregate/disaggregate data from all the antennas in the system using reconfigurable high-speed routers, as shown in \figurename~\ref{fig:Co_processors} for a system having $\lceil M/(n_\text{sub}n_\text{ant})\rceil$ subsystems and $n_\text{co}$ co-processors.
\begin{figure}
	%
	\centering
	%
	\hspace*{0pt}\input{co_processor.tikz}
	\caption{Shuffling data from the $\lceil M/(n_\text{sub}n_\text{ant})\rceil$ subsystems to the $n_\text{co}$ co-processors. The routers use a simple round robin scheme to combine/distribute the data from/to corresponding subsystems.}
	\label{fig:Co_processors}
\end{figure}

Similarly to the \glspl{sdr}, the two main constraints for the co-processors are the maximum number of \gls{p2p} links denoted $\text{P2P}_\text{CO,max}$ and the maximum throughput denoted $R_{\text{CO}_\text{max}}$.

The following inequalities have to hold for the co-processor not to exceed these constraints:
\begin{eqnarray}
R_{\text{CO}_\text{max}} &>& R_{\text{CO}_\text{out}}=R_{\text{CO}_\text{in}}=\nonumber\\
&=&\left(\frac{M\cdot w+K}{n_\text{co}}\right)\cdot F_\text{sub}\label{eq:R_co}\\
\text{P2P}_\text{CO,max} &>& \text{P2P}_\text{CO} = 2\cdot \lceil M/n_\text{sub}\rceil + 2. \label{eq:P2P_co}
\end{eqnarray}

Using this modular and generic system partitioning, \gls{hw} platforms built using modular components can be evaluated.
Note, that expressions (\ref{eq:R_sdr}) - (\ref{eq:P2P_co}) may also be used with other system parameters, \emph{e.g., } by redefining $F_\text{s}$ and $F_\text{sub}$.

\begin{table*}[t]
	\centering
	\footnotesize
	\caption{Selected Hardware from National Instruments}
	\noindent\begin{tabular}{lll}
		\toprule
		\textbf{Type} & \textbf{Model} & \textbf{Features} \\
		\midrule
		\multirow{2}{*}{Host}&\multirow{2}{*}{PXIe-8135} & 2.3 GHz Quad-Core PXI Express Controller \\
		&& Up to 8 GB/s system and 4 GB/s slot bandwidth \\
		\midrule
		\multirow{3}{*}{SDR}&\multirow{3}{*}{USRP RIO 294xR / 295xR} & 2 RF Front Ends and 1 Xilinx Kintex-7 \gls{fpga} \\
		&& Center frequency variable from \SI{1.2}{\giga\hertz} to \SI{6}{\giga\hertz} \\
		&& \SI{830}{\mega B\per\second} bidirectional throughput on up to 15 DMA channels\\
		\midrule
		\multirow{2}{*}{Co-Processor}&\multirow{2}{*}{FlexRIO 7976R} & 1 Xilinx Kintex-7 410T \gls{fpga} \\
		&& \SI{2.4}{\giga B\per\second} bidirectional throughput on up to 32 DMA channels\\
		\midrule
		\multirow{4}{*}{Switch}&\multirow{4}{*}{PXIe-1085} & Industrial form factor 18-slot chassis \\
		&& \SI{7}{\giga B\per\second} bidirectional throughput per slot\\
		&& 2 switches per chassis with inter-switch traffic up to \SI{3.2}{\giga B\per\second} \\
		&& Links between chassis bound to \SI{7}{\giga B\per\second} bidirectional \\
		\midrule
		\multirow{3}{*}{Expansion Module}&\multirow{3}{*}{PXIe-8374} & PXI Express (x4) Chassis Expansion Module \\
		&& Software-transparent link without programming\\
		&& Star, tree, or daisy-chain configuration\\
		\midrule
		\multirow{2}{*}{Reference Clock Source}&\multirow{2}{*}{PXIe-6674T} & \SI{10}{\mega\hertz} reference clock source with $<\SI{5}{ppb}$ clock accuracy \\
		&& 6 configurable I/O connections\\
		\midrule
		Ref. Clock Distribution&OctoClock & \SI{10}{\mega\hertz} 8-channel clock and timing distribution network \\
		\bottomrule
	\end{tabular}
	\label{table:Selected_Hardware}
\end{table*}

\section{\Gls{lumami} Testbed Implementation}
\label{sec:lumami_testbed_implementation}
In this section the \gls{lumami} specific implementation details are discussed based on the aforementioned general architecture.
The \gls{lumami} system was designed with $100$ \gls{bs} antennas and can serve up to $12$ \glspl{ue} simultaneously.
Based on these parameters, the selected modular \gls{hw} platform is presented and given constraints are evaluated.
Consequently, the specific frame structure and other features of the system including base-band processing, antenna array, mechanical structure and synchronization are briefly described.
Before providing details, the authors would like to emphasize, that this is the initial version of the \gls{lumami} testbed and that add-ons and further improvements are planned for the future.

\subsection{Selected Hardware Platform}
The hardware platform was selected based on requirements discussed in Sec.~\ref{sec:System_Design_Aspect}.
Table~\ref{table:Selected_Hardware} shows the selected off-the-shelf modular hardware from National Instruments used to implement the \gls{lumami} testbed.
The \glspl{sdr} \cite{usrp} allow up to 15 \gls{p2p} links ($\text{P2P}_\text{SDR,max}=15$) with a bidirectional throughput of $R_{\text{SDR}_\text{max}}=\SI{830}{\mega B\per\second}$, support a variable center frequency from  \SI{1.2}{\giga\hertz} to \SI{6}{\giga\hertz} and have a \gls{tx} power of \SI{15}{dBm}.
Each \gls{sdr} contains two \gls{rf} chains, \emph{i.e., } $n_\text{ant}=2$, and a Kintex-7 \gls{fpga}.
Selected co-processors \cite{flex} allow a bidirectional \gls{p2p} rate of $R_{\text{CO}_\text{max}}=\SI{2.4}{\giga B\per\second}$ with up to $\text{P2P}_\text{CO,max}=32$ \gls{p2p} links and employ a powerful Kintex-7 \gls{fpga} with a reported performance of up to \SI{2,845}{GMACs\per\second} \cite{Kintex7}.
This is sufficient for a 100 \gls{bs} antenna \gls{mami} testbed due to the fact that $n_\text{co}$ co-processors can be utilized in parallel.
Interconnection among devices is achieved using 18-slot chassis \cite{chassis} combined with per-slot expansion modules \cite{expansion_module}.
Each chassis integrates two switches based on \gls{pcie} using \gls{dma} channels which allow inter-chassis traffic up to \SI{7}{\giga B\per\second} and intra-chassis traffic up to \SI{3.2}{\giga B\per\second}.

The host \cite{host} is an integrated controller, running LabVIEW on a standard Windows operating system and is used to configure and control the system.
The integrated hardware/software stack provided by LabVIEW provides the needed reconfigurability as it abstracts the \gls{p2p} link setup, communication among all devices and allows \gls{fpga} programming as well as host processing using a single programming language.
An additional feature of LabVIEW is the possibility to seamlessly integrate \gls{ip} blocks generated via Xilinx Vivado platform paving a way to test in-house developed \gls{ip}.

To be able to synchronize the full \gls{bs}, a Reference Clock Source \cite{timing_mod} and Reference clock distribution network \cite{clock_distribution} are required.
Their functionalities will be later discussed when presenting the overall synchronization method.

\subsection{Subsystems and Number of Co-processors}
To build the \gls{lumami} testbed with $M=100$ antennas, $50$ \glspl{sdr} are necessary.
The maximum possible subsystem size is chosen to minimize the utilization of available \gls{p2p} links at the co-processors.
By using (\ref{eq:R_sdr}) and an internal fixed-point wordlength of $w=3$ corresponding to a 12-bit resolution on the I- and Q-components, $n_\text{sub}$ is found to be $8$.
As this is not an integer divider of $50$, the last subsystem only contains two \glspl{sdr}.

Based on Table~\ref{tab:data_shuffling_requirements}, the combined subcarrier rate for all antennas is $wMF_\text{sub}=\SI{5}{\giga B\per\second}$ and another $K\cdot F_\text{sub}=\SI{200}{\mega B\per\second}$ are needed for information symbols.
To not exceed $R_{\text{CO}_\text{max}}$ at least three co-processors must be utilized.
To further lower the burden on the design of the low-latency \gls{mimo} signal processing chain, $n_\text{co}=4$ is chosen such that each co-processor processes $300$ of the overall 1200 subcarriers.

Table~\ref{tab:lumami_parameters} summarizes the \gls{lumami} testbed parameters and shows that constraints are met according to (\ref{eq:R_sdr})-(\ref{eq:P2P_co}).
It can also be seen that the design is still within the constraints if scaling up the number of \gls{bs} antennas to $M=128$, which has been done in subsequent designs based on the same hardware, \emph{e.g., } \cite{Bristol1}.

\begin{table}[t]
	\centering
	\footnotesize
	\caption{System Parameters and validation of constraints in the LuMaMi testbed.}
	\noindent\begin{tabular}{llll}
		\toprule
		\multicolumn{2}{c}{\textbf{Parameters}} & \textbf{Rates \SI{}{\mega B\per\second}} \\
		\midrule
		$M$ & $100$ & $R_{\text{SDR}_\text{max}}=830 > R_{\text{SDR}_\text{out}}=R_{\text{SDR}_\text{in}}=806.4$ \\
		$K$ & $12$ & $R_{\text{CO}_\text{max}}=2,400 > R_{\text{CO}_\text{out}}=R_{\text{CO}_\text{in}}=1,460$\\
		\cmidrule(lr){3-3}
		$n_\text{ant}$ & $2$ & \textbf{P2P Links} \\
		\cmidrule(lr){3-3}
		$n_\text{sub}$& $8^\text{a}$ & $\text{P2P}_\text{SDR,max}=15 > \text{P2P}_\text{SDR} = 12$\\
		$n_\text{co}$& $4$ & $\text{P2P}_\text{CO,max}=32 > \text{P2P}_\text{CO} = 18$ \\
		%
		%
		%
		\bottomrule
	\end{tabular}
	\begin{tablenotes}
		\small
		\item[a] $^\text{a}$ Note, that the last subsystem only consists of two SDRs.
	\end{tablenotes}
	\label{tab:lumami_parameters}
\end{table}
\begin{figure}[!t]
	\centering
	\includegraphics[width=0.9\columnwidth]{radio_frame_new}
	\caption{The default frame structure used in the \gls{lumami} testbed.}
	\label{fig:frame_structure}
\end{figure}
\begin{figure*}
	\center
	\begin{tabular}{cc}
		$\hspace{-.3cm}\vcenter{\hbox{\includegraphics[width=0.8\columnwidth]{Racks_side}\hspace{1.5cm}}}$
		$\vcenter{\hbox{\includegraphics[width=.5\columnwidth]{Mimo_helbild.png}}}$
	\end{tabular}
	\caption{Left: Side view of the mechanical assembly of the \gls{bs}. The two racks sit side by side (not as shown) with the \glspl{sdr} facing the same direction (towards the antenna array). Two columns of USRP \glspl{sdr} are mounted in each rack, totaling 50 of them. Right: The assembled \gls{lumami} testbed at Lund University, Sweden.}
	\label{fig:RackMountedSystem}
\end{figure*}
\subsection{Frame Structure}
The default frame structure for the \gls{lumami} testbed is shown in \figurename~\ref{fig:frame_structure}.
One frame is $T_\text{f}=\SI{10}{\milli\second}$ and is divided in ten subframes of length $T_\text{sf}=\SI{1}{\milli\second}$.
Each subframe consists of two slots having length $T_\text{slot}=\SI{0.5}{\milli\second}$, where the first subframe is used for control signals, \emph{e.g., } to implement over-the-air synchronization, \gls{ul} power control and other control signaling.
The 18 slots in the other nine subframes encapsulate seven \gls{ofdm} symbols each.
Comparing to \figurename~\ref{fig:frame_structure_generic}, a reciprocity calibration cycle is defined over the whole run-time of the \gls{bs} for simplicity and due to the fact that there is no large drift after warming up the system in a controlled environment\cite{Vieira2014a}.
The \gls{dl} pilot cycles and control cycles are both set to be the length of one frame.
Each frame starts with one control subframe followed by one subframe with one \gls{dl} pilot and one \gls{dl} data symbol whereas all others use two DL data symbols.

\subsection{Mobility}
The pilot distance in time in the default frame structure given in \figurename~\ref{fig:frame_structure_generic} is $T_\text{p}\approx \SI{430}{\micro\second}$ or six \gls{ofdm} symbols.
Thus, $\nu_\text{max}\approx\SI{240}{\hertz}$ for a correlation of 0.9.
Due to availability from a network operator, a carrier frequency of $f_\text{c}=\SI{3.7}{\giga\hertz}$ is selected.
Using (\ref{eq:Mobility}), $\text{v}_\text{max}=\SI{70}{\kilo\meter\per\hour}$ is found as maximum supported speed.

\subsection{\gls{tdd} Turnaround Time}
The pre-coding turnaround time requirement for the implementation can be analyzed based on (\ref{eq:latency}).
The analog front-end delay of the \glspl{sdr} was measured to be about $\SI{2.25}{\micro\second}$.
Taking the frame structure in \figurename~\ref{fig:frame_structure} (assuming $\Delta^\text{rf,\tiny{TX}}=\Delta^\text{rf,\tiny{RX}}$ which is not necessarily true), the latency budget for base-band processing is as follows:
Overall time for pre-coding after receiving the UL pilots is $\SI{214}{\micro\second}$ ($3$ \gls{ofdm} symbols).
The 2048 point \gls{fft}/\gls{ifft} (assuming a clock frequency of \SI{200}{\mega\hertz}) requires around $\SI{35}{\micro\second}\times 2 = \SI{70}{\micro\second}$ in total for TX and RX (including sample reordering).
As a result, the remaining time for channel estimation, \gls{mimo} processing, and data routing is around $\SI{140}{\micro\second}$, which is the design constraint for this specific frame structure.

An analysis of the implemented design showed that the latency is far below the requirement for the default frame structure which makes it possible to use the testbed for higher mobility scenarios from this point of view\cite{sips_steffen}.

\subsection{Implementation Features}
\subsubsection{Base-band Processing}
On the \gls{lumami} testbed, each \gls{ue} sends pilots on orthogonal subcarriers, \emph{i.e., } each \gls{ue} uses every $K$-th subcarrier with the first \gls{ue} starting at subcarrier 0, the second at subcarrier 1 etc., overall utilizing a full \gls{ofdm} symbol.
It was shown that performance does not suffer significantly compared to a full detector calculated for each subcarrier using this method\cite{deliverable3_2}.
Moreover, it efficiently remedies processing requirements and reduces the required memory for storing estimated \gls{csi} matrices by a factor of $K$.
A least-square \gls{csi} estimation algorithm with zeroth-order hold over $K=12$ subcarriers was implemented, however, better estimates could be obtained by on-the-fly interpolation between the estimated subcarriers.
Overall, utilizing this approach reduces the required detection matrix throughput to one matrix every 12 subcarriers, \emph{i.e., } \SI{16.8e6}{subcarriers\per\second/12} = \SI{1.4e6}{Detection Matrices\per\second}.

Two versions for detection were implemented.
The first one based on a QR decomposition of the channel matrix augmented with the regularizations factors to a matrix of size $2M\times K$.
This is then formulated into a partial parallel implementation employing a systolic array\cite{1285069}.
The latter one based on a Neumann-series\cite{Prabhu2013}.
In the QR decomposition, each column is processed using the discrete steps of the modified Gram-Schmidt algorithm.
The logic on the co-processors can be reconfigured so that the same hardware resources that provide the \gls{rzf} decoder can also provide the \gls{zf} and \gls{mrc} decoders, \emph{i.e., } the detection / precoding schemes discussed in Sec.~\ref{sec:MaMi_Theory} are supported with run-time switching.
The Neumann-series based \gls{zf} detector utilizes the unique property that in \gls{mami}, the Gramian matrix shows dominant diagonal elements if \glspl{ue} use \gls{ul} power control, or if scheduling is performed to serve \glspl{ue} with similar power levels in the same time/frequency block to mitigate the influence of path loss differences. This, allows the matrix inversion to be approximated with low overall error \cite{Prabhu2013}.
The utilizations for the two \gls{fpga} designs are shown in Table~\ref{tab:FPGA_resources}.
\begin{table}[t]
	\centering
	\small
	\caption{FPGA Utilization for two different MIMO processing implementations}
	\noindent\begin{tabular}{ccccc}
		\toprule
		Implementation & Registers & LUT & RAMs & DSP48\\
		\midrule
		\multirow{2}{*}{QRD} & 46470  & 49315  & 171 & 596 \\
		& (9.1\%) & (20.3\%) & (21.5\%) & (38.7\%) \\
		\midrule
		\multirow{2}{*}{Neumann-Series} & 16000  & 28700 & 6 & 176 \\
		& (3.1\%) & (11.8\%) & (0.75\%) & (11.4\%) \\
		\bottomrule
	\end{tabular}
	\label{tab:FPGA_resources}
\end{table}
Clearly, overall processing complexity and resource utilization can be significantly reduced by exploiting the special properties of \gls{mami}.

At this point, the regularizations factors $\beta_{\mathrm{reg}_\mathrm{pre}}$ and $\beta_{\mathrm{reg}_\mathrm{dec}}$ are not run-time optimized but set manually, however, implementation of this feature is planned in future.
For a more detailed discussion of the low-latency signal processing implementation on the testbed we refer to\cite{sips_steffen}.

\subsubsection{Host-based visualization and data capturing}
The available margin of \SI{1}{\giga B\per\second} and 14 \gls{p2p} links to the corresponding maximum values on the co-processors are used for visualization and system performance metrics.
The host receives decimated equalized constellations and raw subcarriers for one \gls{ul} pilot and one \gls{ul} data symbol per frame.
These features add another
\begin{equation}
\frac{300\cdot 2\text{bytes}+2\cdot 300\cdot 4\text{bytes}}{\num{10}\text{ms}}=\SI{300}{\mega B\per\second}\nonumber
\end{equation}
of data flowing in and out of the co-processor.
The raw subcarriers are used to perform channel estimation and \gls{ul} data detection on the host computer with floating point precision and allow fast implementation of different metrics, like constellation, channel impulse response, power level per antenna and user.
Another 12 \gls{p2p} links available are utilized to transmit and store real-time \glspl{ber} for all 12 \glspl{ue}.

Moreover, to be able to capture dynamics in the channel for mobile \glspl{ue}, \gls{csi} can be stored on a \SI{}{\milli\second} basis.
An integrated \SI{2}{\giga B} \gls{dram} buffer on each of the co-processors was utilized for this since direct streaming to disk would exceed the \gls{p2p} bandwidth limits.
Snapshots can either be taken for \SI{60}{\second} in a \SI{5}{\milli\second} interval or over \SI{12}{\second} in a \SI{1}{\milli\second} interval, both corresponding to \SI{2}{\giga B} of data for $300$ subcarriers per co-processor.
\subsubsection{Scalability/Reconfigurability}
Before startup, the number of deployed \gls{bs} antennas can be arbitrarily set between $4$ and $100$.
This is achieved by introducing zeros for non-existing antennas within the \gls{lut}-based reconfigurable high-speed routers on the co-processors, thereby allowing to evaluate effects of scaling the \gls{bs} antennas in real environments\cite{sips_steffen}.
Additionally, all $140$ \gls{ofdm} symbols in a frame can be rearranged arbitrarily before start-up while each frame always repeats itself.
For instance, we can choose to set the first symbol as UL pilots and all others as UL data in a static UL only scenario.

\subsubsection{Reciprocity Calibration}
Estimation of the reciprocity calibration coefficients was implemented on the host, mainly for two reasons: \emph{(i)} the host can perform all operations in floating-point which increases precision and \emph{(ii)} the drift of the hardware is not significant once the system reached operating temperature \cite{Vieira2014a}.
Estimated reciprocity coefficients are applied in a distributed manner on the \glspl{sdr}\cite{sips_steffen}.

\subsection{Mechanical structure and electrical characteristics}
\label{subsec:MecDesign}
Two computer racks containing all components measuring $0.8 \times 1.2 \times 1$\,m were used, as shown \figurename~\ref{fig:RackMountedSystem}.
An essential requirement for the \gls{lumami} testbed is to allow tests in different scenarios, \emph{e.g., } indoor and outdoor.
Therefore, the rack mount is attached on top of a 4-wheel trolley.

\subsection{Antenna Array}
\label{subsec:AntArray}
The planar T-shaped antenna array with 160 dual polarized $\lambda$/2 patch elements was developed in-house.
A 3.2\;mm Diclad 880 was chosen for the printed circuit board substrate.
The T upper horizontal rectangle has $4\times25$ elements and the central square has $10\times10$ elements (see \figurename~\ref{fig:RackMountedSystem} right).
This yields 320 possible antenna ports that can be used to explore different antenna array arrangements, for example $10\times 10$ or $4\times 25$ with the latter one being the default configuration.
All antenna elements are center shorted, which improves isolation and bandwidth.
The manufactured array yielded an average \SI{10}{\decibel}-bandwidth of \SI{183}{\mega\hertz} centered at \SI{3.7}{\giga\hertz} with
isolation between antenna ports varying between \SI{18}{\decibel} and \SI{28}{\decibel} depending on location in the array.

\subsection{User Equipment}
Each \gls{ue} represents a phone or other wireless device with single antenna capabilities.
One \gls{sdr} serves as two independent \glspl{ue} such that overall six \glspl{sdr} are required for the 12 \glspl{ue}.
The base-band processing, \emph{i.e., } \gls{ofdm} modulation/demodulation and symbol mapping/demapping are essentially identical to the \gls{bs} implementation.
A least-square \gls{csi} acquisition is performed on precoded DL pilot followed by a \gls{zf}-equalizer.
The \gls{dl} pilots occupy a full \gls{ofdm} symbol.
The \glspl{ue} may be equipped with any type of antenna using SMA connectors.
\subsection{Synchronization}
A \gls{mami} \gls{bs} requires time synchronization and phase coherence between each RF chain.
This is achieved using the \SI{10}{\mega\hertz} reference clock source and the reference clock and trigger distribution network (see Table~\ref{table:Selected_Hardware}).
The reference clock is used as the source of each radio local oscillator, providing phase coherence among devices.
The trigger signal is used to provide a time reference to all the radios in the system.
A master provides an output digital trigger that is amplified and divided among all the radios.
Upon receipt of the rising edge of the event trigger, all \glspl{sdr} are started.
The basic structure can be identified in \figurename~\ref{fig:RackMountedSystem} on the left.

To synchronize the \glspl{ue} with the \gls{bs} \gls{ota}, the \gls{lte} Zadoff-Chu \gls{pss} is used, which occupies the center \SI{1.2}{\mega\hertz} of the overall bandwidth.
\gls{ota} synchronization and frequency offset compensation are achieved by employing a frequency-shifted bank of replica filters.
The process follows a two step procedure: finding a coarse candidate position by scanning over the whole radio frame followed by tracking the \gls{pss} in a narrowed window located around the coarse candidate position.
Additionally, by disciplining the \gls{ue} \glspl{sdr} with \gls{gps}, frequency offset compensation may be avoided by lowering the frequency offset to $<\SI{300}{\hertz}$.

\begin{figure}[!t]
	\centering
	\includegraphics[width=.85\columnwidth]{Room_2311_indoor}
	\caption{The indoor measurement setup in a lecture room including the positions of the 12 \glspl{ue}. The \gls{bs} is shown at the right-hand side and is situated at the front of the lecture hall. The terminals are placed in groups of four on three different tables and distances to the \gls{bs}.}
	\label{fig:room2311}
\end{figure}
\begin{figure}[!t]
	\centering
	\includegraphics[width=0.85\columnwidth]{indoor_user_positions_zoomed}
	\caption{One group of four \glspl{ue} with a high user density per unit area to validate the spatial multiplexing capabilities of \gls{mami}.}
	\label{fig:indoor_UE_group}
\end{figure}
\begin{figure*}[!t]
	\centering
	\input  {Grouped_BERs.tikz}%
	\caption{UL and DL \glspl{ber} for 12 \glspl{ue} with \gls{zf} decoder/precoder.}
	\label{fig:BERs}
\end{figure*}

\begin{figure}[!t]
	\centering
	\defDL_one_user_three_const_UE11.dat{UL_all_users_qpsk_with_awgn_user_4.dat}
	\input  {BER_plot_AWGN.tikz}%
	\caption{Comparing the \gls{ber} of \gls{ue}4 to \gls{awgn} and Rayleigh fading channels.}
	\label{fig:BER_AWGN}
\end{figure}

\section{Proof-of-concept Results}
\label{sec:Proof_of_Concept_and_Results}
This section describes two experiments performed to validate our testbed design, the \gls{mami} concept and its performance.
The first test is performed indoors with high density of users per area unit to stress the spatial multiplexing capabilities of the system.
The second test is conducted outdoors with less dense deployment of \glspl{ue} and is primarily designed to test the range and multiplexing capabilities outdoors.
For all tests, the default antenna configuration, \emph{i.e., } $4\times 25$ was used on the \gls{bs} side whereas the \glspl{ue} were equipped with linear polarized ultra-wideband antennas.
It has to be noted that all results shown in this section are obtained from real-time operation without \gls{ul} power control.

\subsection{Indoor Test}
In this test real-time uncoded BER curves are measured, employing \gls{mrc}/\gls{mrt} and \gls{zf} as decoders/precoders.
The \gls{ul} \gls{ber} curves are obtained by sweeping all \gls{ue} \gls{tx} \gls{pa} gains synchronously, and for the \gls{dl} \gls{ber} curves the \gls{pa} gains of the \gls{bs} \gls{tx} chains while keeping other system parameters constant.
Note that the initial parameterization of the system is chosen empirically, so it allows smooth \gls{ber} curves starting at about $0.5$.
Each gain step is held constant for about \SI{4}{\second} corresponding to about \num{36e6} and \num{108e6} transmitted bits per step for QPSK and 64-QAM modulation, respectively.

\subsubsection{Scenario}
Twelve \glspl{ue} are set up in a lecture hall at Lund University with the \gls{bs} at the front as shown in \figurename~\ref{fig:room2311} including the respective \gls{ue} placements.
All \glspl{ue} are packed in groups of four resulting in a high density of \glspl{ue} per area unit.
One of these groups can be seen in \figurename~\ref{fig:indoor_UE_group}.
\subsubsection{UL \glspl{ber}}
\label{subsubsec:indoor_UL_BERs}
\figurename~\ref{fig:BERs}, (a) and (b), show the \glspl{ber} for all 12 \glspl{ue} using \gls{zf} detector for QPSK and 64-QAM modulation, respectively.
For both constellation sizes, the \glspl{ue} furthest away, \gls{ue}0 to \gls{ue}3 show highest \gls{ber}.
\gls{ue}0 and \gls{ue}1 even show a sudden increase for the \gls{ber} to 0.5 which was diagnosed to be due to saturation of their respective \glspl{pa}.
Moreover, their performance shows severe limitation compared to the other \glspl{ue}, giving a clear indication that their performance is interference rather than power limited.
The group closest to the \gls{bs}, \gls{ue}9-\gls{ue}12, shows best performance although the variation within the group is still quite significant.
Overall, the expected trend, increasing performance with increased transmit gain is clearly noticeable with the \gls{ber} curve shapes resembling those of \gls{awgn} channels.
Comparing the amplifier gain settings for QPSK and 64-QAM to achieve the same \gls{ber} the differences are found to be in the range of \SI{10}{\decibel} to \SI{16}{\decibel} whereas a difference of \SI{9}{\decibel} is expected for \gls{awgn}.
Overall, it can be seen that all \glspl{ue} except \gls{ue}0 and \gls{ue}1 achieve \gls{ber} below \SI{10}{\percent} at an amplifier gain of \SI{15}{\decibel} for QPSK and \SI{25}{\decibel} for 64-QAM, respectively.

\subsubsection{DL \glspl{ber}}
\figurename~\ref{fig:BERs}, (c) and (d), show the DL \glspl{ber} using \gls{zf} precoder for QPSK- and 64-QAM modulation, respectively.
Using QPSK modulation, the group closest to the \gls{bs}, \gls{ue}9-\gls{ue}12, achieves a considerably better performance than the other two groups.
Using 64-QAM, all \glspl{ue} show an error-floor towards higher \gls{tx} gain values
which is likely a result of imperfect reciprocity calibration combined with leakage among \glspl{ue} due to non-perfect channel knowledge resulting in interference among \glspl{ue}.
However, for the QPSK modulation case all \glspl{ue} experience better \gls{ber} rates which can be explained by the significantly higher available transmit power on the \gls{bs} side, utilizing 100 active \gls{rf}-chains.
Comparing again the difference in amplifier gain setting for QPSK and 64-QAM, their differences are about \SI{12}{\decibel} to \SI{16}{\decibel}.
The tests performed were mainly to prove functionality, and thus, no special care was taken to achieve best possible accuracy for the reciprocity calibration.
However, individual parts are continuously tested to be improved.

\subsubsection{Performance Evaluation}
While the \glspl{ber} plots in Fig.~\ref{fig:BERs} nicely show the trend with increasing transmit power, they do not provide a real performance indication against \gls{snr}.
The current implementation of the testbed does not provide \gls{snr} estimates in real-time such that the data presented in Fig.~\ref{fig:BERs} can be seen as the raw data provided during measurements.
To provide an indication of the system performance the \gls{snr} of \gls{ue}4 was estimated based on the received \gls{ul} channel estimates.
Estimated subcarriers at different time instances (about \SI{200}{\milli\second} apart) were subtracted / added to extract the noise / signal plus noise level which was then used to calculate the \gls{snr} value.
However, this practice has limits as for close users interference may be stronger than the noise whereas for far away users the signal level may be too low.
Therefore, \gls{ue}4 was chosen which due to its placement during the measurement allowed a relatively good \gls{snr} estimation.
Fig.~\ref{fig:BER_AWGN} shows the \gls{ber} of \gls{ue}4 in comparison with the theoretical performance in \gls{awgn} and Rayleigh fading channels.
It is visible that due to the excess amount of \gls{bs} antennas the performance is close to the \gls{awgn} channel.
To be more specific, due to the channel hardening the performance is only about \SI{3}{\decibel} worse than for a \gls{awgn} channel which would be achieved for perfect channel hardening.
On the \gls{dl} the \glspl{snr} are affected by several factors including the higher overall transmit power from the 100 active \gls{rf}-chains and possible inaccuracies in the reciprocity calibration coefficients.
As \gls{dl} precoding is performed based on \gls{ul} channel estimates, \gls{snr} estimation is practically not feasible.

As all shown \glspl{ber} curves closely resemble the shape of an \gls{awgn} channel it can be claimed that the \gls{mami} concept works and is capable of serving 12 \glspl{ue} on the same time/frequency resource even with a high \gls{ue} density which in turn significantly improves the spectral efficiency compared to current cellular standards.

\subsubsection{MRC/MRT versus ZF}
\begin{figure}[!t]
	\centering
	\subfloat[]{\hspace{-0.5cm}
		\defDL_one_user_three_const_UE11.dat{UL_one_user_three_const_UE11.dat}
		\defMRT{MRC}
		\input  {MRC_vs_ZF.tikz}%
		\label{subfig:UL_both_det}}\\
	\subfloat[]{\hspace{-0.5cm}
		\defDL_one_user_three_const_UE11.dat{DL_one_user_three_const_UE11.dat}
		\defMRT{MRT}
		\input  {MRC_vs_ZF.tikz}%
		\label{subfig:DL_both_pre}}
	\caption{\glspl{ber} for \glspl{ue}7 using QPSK, 16-QAM and 64-QAM modulation. \protect\subref{subfig:UL_both_det} on the \gls{ul} for \gls{zf} and \gls{mrc} detector and \protect\subref{subfig:DL_both_pre} on the \gls{dl} for \gls{zf} and \gls{mrt} precoder.}
	\label{fig:MRC_vs_ZF}
\end{figure}
To compare the performance of \gls{mrc}/\gls{mrt} and \gls{zf} it is beneficial to isolate the analysis to one \gls{ue}.
\figurename~\ref{fig:MRC_vs_ZF}a and  \figurename~\ref{fig:MRC_vs_ZF}b show the \gls{ber} for \gls{ue}7 for QPSK, 16-QAM and 64-QAM modulations while the \gls{bs} employs either \gls{mrc}/\gls{mrt} or \gls{zf} on the \gls{ul} and \gls{dl}, respectively.

Overall, \gls{zf} shows an superior performance trend with increasing \gls{pa} gains, while the performance of \gls{mrc} appears to level off\footnote{This is expected from theory, as inter-user interference is the main source of error during data detection. The high density users setup adopted in this experiment highly contributes to this phenomena.}.
Looking in more detail, \gls{zf} is capable of achieving more than an order of magnitude lower \glspl{ber}, compared to \gls{mrc}.
Using higher constellation sizes, 16-QAM or 64-QAM, the results for \gls{mrc} show an even more significant deterioration.
On the \gls{dl}, \gls{zf} also outperforms \gls{mrt} by far, the latter shows a significant error floor towards higher gains as in the \gls{ul} case.

Unfortunately, direct comparison between \gls{ul} and \gls{dl} results shown here is not easy to perform.
This is due to the fact that on the \gls{ul}, the performance is isolated to the \gls{ul} transmit power only whereas on the \gls{dl} a combination of \gls{ul} channel estimate quality, \gls{dl} transmit power and reciprocity accuracy determines overall performance.

\subsection{Outdoor Test}
For the outdoor test, the testbed was placed on the rooftop of one of the wings of the department building while the \glspl{ue} where placed on the opposite wing utilizing scaffolding mounted to the building.
Up to eight \glspl{ue} were served simultaneously in a distance of about 18 to 22 meters, six on the second floor and two on the first floor while the testbed was situated on the third floor (rooftop).
The scenario is shown in \figurename~\ref{fig:outdoor_scenario}.

\begin{figure}[!t]
	\centering
	\includegraphics[angle=0,width=0.95\columnwidth]{outdoor}
	\caption{Scenario for the outdoor tests. \gls{bs} placed on the rooftop of the building (third floor) serving eight \glspl{ue} on the opposite wing, with six \glspl{ue} on second floor and two \glspl{ue} on first floor.}
	\label{fig:outdoor_scenario}
\end{figure}
\figurename~\ref{fig:Outdoor_figures} shows the \gls{bs} placed on the rooftop of the department building facing towards the opposite wing. The placement for \glspl{ue} 0 and 1 is also marked.
\begin{figure}[!t]
	\centering
	\includegraphics[angle=0,width=0.75\columnwidth]{BS_outdoor}
	\caption{The outdoor test scenario setup with the \gls{bs} deployed on the rooftop of the department building marked with two \glspl{ue} on the opposite building wing.}
	\label{fig:Outdoor_figures}
\end{figure}

\figurename~\ref{fig:UL_const_MRC_ZF} shows a screenshot of the received \gls{ul} QPSK constellations for this test setup when using \gls{mrc} and \gls{zf}, respectively.
Using \gls{mrc} without \gls{ecc} for this test, the six \glspl{ue} show significant interference.
Therefore, focus is put on the results obtained with \gls{zf} which is capable of separating up to eight \glspl{ue} and shows very clear constellations, due to the interference suppression.
\begin{figure}[!t]
	\centering
	\subfloat[]{\includegraphics[width=.85\columnwidth]{BS_8user_MRC_ove}%
		\label{subfig:UL_const_MRC}}\\
	\subfloat[]{\includegraphics[width=.85\columnwidth]{BS_8user_ZF_ove}%
		\label{subfig:UL_const_ZF}}
	\caption{\gls{ul} constellations for the outdoor experiment: \protect\subref{subfig:UL_const_MRC} when using \gls{mrc} with 6 \glspl{ue} and \protect\subref{subfig:UL_const_ZF} when using \gls{zf} to serve 8 \glspl{ue}.}
	\label{fig:UL_const_MRC_ZF}
\end{figure}

Considering \gls{zf} on the \gls{dl}, the constellations for all 8 \glspl{ue} can be seen in \figurename~\ref{fig:DL_const_outdoor}.
\begin{figure}[!t]
	\centering
	\savebox{\myimage}{\hbox{\includegraphics[width=0.45\columnwidth]{LuMaMi_T01.pdf}}}%
	\subfloat[]{\usebox{\myimage}
		\label{subfig:T01}}
	%
	\subfloat[]{\raisebox{\dimexpr.5\ht\myimage-.5\height\relax}{\includegraphics[width=0.45\columnwidth]{LuMaMi_T02.pdf}%
			\label{subfig:T02}}}\\
	\hspace{0.1cm}\savebox{\myimage}{\hbox{\includegraphics[width=0.45\columnwidth]{LuMaMi_T03.pdf}}}%
	\subfloat[]{\usebox{\myimage}
		\label{subfig:T03}}
	%
	%
	\subfloat[]{\raisebox{\dimexpr.5\ht\myimage-.5\height\relax}{\includegraphics[width=0.45\columnwidth]{LuMaMi_T04.pdf}%
			\label{subfig:T04}}}
	\caption{Received DL constellations using \gls{zf}: \protect\subref{subfig:T01} UE0 \& UE1 \protect\subref{subfig:T02} UE2 \& UE3 \protect\subref{subfig:T03} UE5 \& UE8 and \protect\subref{subfig:T04} UE9 \& UE10.}
	\label{fig:DL_const_outdoor}
\end{figure}
Although in-detail analysis is not provided for this test, it is clearly visible that \gls{zf} outperforms \gls{mrc} which is often claimed to be sufficient in literature when analyzing performance based on \gls{iid} channel models\cite{Marzetta2010}.
The results observed in this experiment are representative for most tests performed so far, \emph{i.e., } \gls{dl} always showed to be the more challenging duplex case.

The \gls{lumami} testbed was also utilized to perform the first \gls{mami} outdoor mobility measurements involving moving pedestrians and cars as \glspl{ue}, however, a discussion of this is out of scope of this paper.
Results and analysis from the mobility tests can be found in\cite{2017arXiv170108818H}.

\section{Conclusion}
\label{sec:conclusion}
This paper presented the \gls{lumami} testbed, which is the first fully operational real-time testbed for prototyping massive MIMO.
Based on massive MIMO system requirements, system parameters were discussed and defined.
Further, a detailed generic hardware partitioning to overcome challenges for data shuffling and \glsentrylong{p2p} link limitations while still allowing scalability, was proposed.
By grouping \glsentrylongpl{sdr} and splitting overall bandwidth, implementation of massive MIMO signal processing was simplified to cope with challenges like \glsentrylong{tdd} precoding turnaround time and limited \glsentrylong{p2p} bandwidth enforcing strict design requirements when scaling the number of \glsentrylong{bs} antennas up to 100 or higher.
Based on the generic system partitioning and system requirements, a hardware platform was selected and evaluated.
It was shown that internal system configuration is within throughput and processing capabilities before the complete \glsentryshort{lumami} testbed parameters were described.
Finally, field trial results including \glsentrylong{ber} performance measurements and constellations were presented from both indoor and outdoor measurement campaigns.
The results showed that it is possible to separate up to 12 \glsentrylongpl{ue} on the same time/frequency resource when using massive MIMO.
Having established a flexible platform for testing new algorithms and digital base-band solutions we are able to take \glsentrylong{mami} from theory to real-world tests and standardization for next generation wireless systems.

\section*{Acknowledgment}
This work was funded by the Swedish foundation for strategic research SSF, VR, the strategic research area ELLIIT, and the EU Seventh Framework Programme (FP7/2007-2013) under grant agreement n 619086 (MAMMOET).

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{MyIEEEtran}

redpajama_set_name: RedPajamaArXiv


ST. CLOUD -- Stearns County is getting into the holiday spirit by holding a county wide food drive.
The food drive is being held by employees of each of the county departments, and they are inviting you to participate as well.
You can bring in your food items to the Stearns County Administration Center or at any of the county buildings.
Donations will be accepted through next Friday (December 21st), when staff will then deliver the items to several local food drives.


Download the song Radha Zee Bangla Wikipedia on irmob. You can download mp3 ringtone for free at here. To see the song details click button download on the matching titles, then for the download link is on the next page.
Gopi Murders Radha For Rashi!


2009 - A motorcyclist was struck by a car that failed to yield the right of way, causing an injury to his leg. The car carried $100,000 insurance limits. Larkin, Ingrassia & Tepermayster, LLP insisted that their client was entitled to the entire policy and after negotiating with the insurance company for two months the insurance company was eventually persuaded to offer their entire policy to our client.


It's about that time of the week again, humans—Friday! Kick back and kick off your Memorial Day weekend with some of the most pup-lifting news from the Internet this week.
This pup is redefining pet parenting.
Meet the pups who tried to out-dress the new Duchess.
Cecil has a real Cinderella story.
Louie's walk turned into a surprise family reunion.
Paul the Corgi's got the moves.


text:
Jorge Heilpern Official Site
Biografia en español
"South American Feelings"
Jorge Heilpern, musician, singer songwriter and entertainer was born in Buenos Aires to Romanian and Polish parents. At the age of 13 he got his first guitar and dedicated himself to singing at parties and school functions. At 18, he enrolled in the Municipal Conservatory of Buenos Aires and joined a trio of classical baroque music. Years later he formed a pop group, released his first single on CBS Argentina, giving Heilpern his first hit. Also, following his passion for music and performing he joined the crew of Hair, the Broadway musical in Buenos Aires.
Simultaneously he received a degree as an economist from the University of Buenos Aires, and worked as a professional public servant, and later as diplomat in Washington DC and a business consultant for international institutions.
In New York City he studied at The Julliard School of Music and later he established a music production company and composed music for NBC prime time shows like Cocaine: End of the Line, Fast Forward and American Promises for which his work won an Emmy.
With his Tangoman band Heilpern performed at Trumps, Joyous Lake, Bearsville Theatre, Mid Civic Center Ulster, Woodstock '94 Festival and the River to River Summer Festival New York.
He also performed in Paris, Madrid, and Cologne as well as in South America. Presently Heilpern is performing with his group of musicians and dancers in cultural venues in different cities in the US such as the Den Performing Art Center, The Sunrise Theatre, Murray Studio Theater at Ruth Eckerd Hall, Deering State at Cutler, Norton Museum of Art, Mar-a-Lago . Heilpern also does solo performances, private parties and milongas singing in Miami, Tulsa, Chicago and New York City.
Tangoman at Lydia's cafe Stone Ridge August Sat 18th
Tangoman & Friends February 25th Open Stage Club Miami
Saboreo la Vida Luna Star Cafe Nov. 10th
Tangoman at the Harmony Cafe Woodstock NY
Friday May 13/16 Only Tangos New Paltz
© 2013 COPYRIGHT to Tangomanmusic

redpajama_set_name: RedPajamaCommonCrawl


text:
Сервіс обміну криптовалют, також криптобіржа, криптовалютна біржа, біржа цифрових валют — це онлайн-сервіс, який дозволяє клієнтам обмінювати криптовалюти на інші активи, наприклад, звичайні фіатні гроші або інші цифрові валюти.

Концепція
Сервіс обміну цифрових валют проводить обмін електронних активів. Для підтримки депозитів у різних національних валютах сервіси повинні мати банківські рахунки в кількох країнах. Це дозволяє сервісам обміну приймати кредитні картки, банківські перекази чи інші форми оплати. Отримані кошти зараховуються на депозитний рахунок клієнта для подальших операцій або відразу обмінюються на обраний актив.  регулювання сервісів обміну криптовалют у багатьох країнах залишається невирішеним.

Зазвичай сервіси зараховують криптовалюту на свій внутрішній рахунок клієнта, з якого потім користувач може виконувати інші обмінні операції, або перерахувати на свій зовнішній криптовалютний рахунок. Деякі сервіси можуть надавати клієнтам можливість отримати анонімні передплачені картки, які можна використовувати для отримання звичайних грошей в банкоматах.

Децентралізовані сервіси обміну
Децентралізовані сервіси обміну (такі як Etherdelta, IDEX, HADAX) не отримують і не зберігають кошти користувачів, а лише полегшують пряму торгівлю між поточними власниками. Децентралізовані сервіси стійкіші до проблем безпеки, але менш оперативні щодо виконання угод обміну.  обсяг операцій на таких сервісах був низьким.

Регулювання
До 2016 року кілька сервісів обміну, що працюють в Європейському Союзі, отримали ліцензії відповідно до Директиви про платіжні послуги ЄС та Директиви ЄС про електронні гроші. Європейська рада та Європейський парламент оголосили, що вони прийматимуть положення про введення більш жорстких правил, що стосуватимуться обмінних платформ.

У 2018 році Комісія з цінних паперів та бірж США (SEC) заявила: «Якщо платформа пропонує торгівлю цифровими активами, які є цінними паперами і працює як "біржа", як це визначено федеральними законами про цінні папери, тоді платформа повинна зареєструватися в SEC в якості національної біржі цінних паперів або бути звільнена від реєстрації». Комісія з торгівлі товарними ф'ючерсами дозволяє торгувати деривативами на криптовалюти.

Серед азійських країн Японія має нормативно-правові акти, які передбачають необхідність отримання спеціальної ліцензії Органу фінансових послуг для роботи в якості криптобіржі.

Найбільші сервіси обміну криптовалют
На початку 2018 року Bloomberg News повідомив про найбільші сервіси обміну криптовалют на основі даних про обсяг угод та оцінені доходи, зібрані CoinMarketCap. Аналогічна статистика повідомляла Statista під час опитування Encrybit, щоб зрозуміти проблеми обміну криптовалюти. Згідно з опитуванням, трійка найбільших сервісів обміну криптовалют — це Binance, Huobi та OKEX. Усі ці найбільші сервіси є приватними та відносно новими. Кілька з них не повідомляють базову інформацію: імена власників, фінансові дані або навіть місце розташування.

 Binance
 Upbit
 Huobi
 Bittrex
 Bithumb
 OKEx
 Bitfinex
 Coinbase
 Bitstamp
 Kraken

Binance активно працює в Україні із 2020 року — перекладено українською інтерфейс та новини, є платіжний шлюз для гривні. Але є сервіси, які орієнтовані на клієнтів саме з України:
 BTC TRADE UA
 KUNA

Див. також
 Криптовалюта
 Біткойн
 Цифрова валюта
 Криптовалютний гаманець

Примітки

Посилання

Сервіси обміну цифрових валют
Неофіційні валюти

redpajama_set_name: RedPajamaWikipedia


text:
Home > Inquiry > Vol. 11 (2010) > Iss. 1
Factors Influencing Breastfeeding in a Hospital Setting
Mary Smith, University of Arkansas, Fayetteville
Breastfeeding is promoted across the globe as the optimum method of infant feeding (World Health Organization, WHO, 2003). Though there is a plethora of published benefits of breastfeeding for baby and mother, breastfeeding rates are below the national goal of 75% for Healthy People 2010 during early postpartum and considerably lower at six months and at one year (Li et al., 2005; HHS, 2000). Arkansas is significantly below the national average and goals, with only 60% of mothers ever breastfeeding versus 74% nationally (Centers for Disease Control and Prevention, CDC, 2008). To meet these goals nationally and on a state level, it is important to examine the factors associated with a mother's choice of breastfeeding. Part I of this study examined the relationship between breastfeeding at discharge and the mother's initial feeding preference, method of delivery, time of birth, birth-to-breast time, and any formula supplementation. It also identified which factor had the strongest relationship with rates of breastfeeding upon hospital departure. Part II of this study examined hospital nurses' attitudes, knowledge, and confidence about breastfeeding instruction as related to their self-reported assistance with and promotion of breastfeeding in the past year. Results: Exclusive breastfeeding at discharge was related to type and time of delivery, birth–to-breast time, and use of formula supplementation. Knowledge and attitude scores did not yield significant correlations with the assistance and promotion of breastfeeding, though a weak positive correlation was noted between confidence in breastfeeding promotion and the nurses' assistance and promotion of breastfeeding.
Smith, M. (2010). Factors Influencing Breastfeeding in a Hospital Setting. Inquiry: The University of Arkansas Undergraduate Research Journal, 11(1). Retrieved from https://scholarworks.uark.edu/inquiry/vol11/iss1/17
Maternal, Child Health and Neonatal Nursing Commons, Nursing Midwifery Commons
All Issues Vol. 21, Iss. 1 Vol. 20, Iss. 1 Vol. 19, Iss. 1 Vol. 18, Iss. 1 Vol. 17, Iss. 1 Vol. 16, Iss. 1 Vol. 15, Iss. 1 Vol. 14, Iss. 1 Vol. 13, Iss. 1 Vol. 12, Iss. 1 Vol. 11, Iss. 1 Vol. 10, Iss. 1 Vol. 9, Iss. 1 Vol. 8, Iss. 1 Vol. 7, Iss. 1 Vol. 6, Iss. 1 Vol. 5, Iss. 1 Vol. 4, Iss. 1 Vol. 3, Iss. 1 Vol. 2, Iss. 1 Vol. 1, Iss. 1
Inquiry is licensed under a Creative Commons Attribution-NonCommercial 3.0 International License .


text:
Q: Weird VLOOKUP issue in a macro heavy worksheet I have quite a weird issue with a workbook I built that is very macro dependent. It centers around a simple VLOOKUP().
Currently my macro creates a line item on one sheet (Sheet A), then copies said line item to another sheet (Sheet B). After that line item is copied, I use a VLOOKUP() function in Sheet B to look up the date and time of the line item back in Sheet A to confirm that it is still in the workbook.
Normally this whole scenario works fine. The VLOOKUP() clearly can find the unique ID in Sheet B and trace it back to Sheet A.
The issue I'm having is that if anyone enters the cell (using F2, double-clicking the cell, etc) that contains the Date and Time in Sheet B, the VLOOKUP() function errors out. This is resolved by hitting F2 in the Date and Time cell in Sheet A.
This is causing a litany of problems because I have a macro that deletes line items if they error out, so this temporary error is throwing everything off.
I've tried matching formats and what not but it seems to be a pretty unique and weird issue.
Has anyone encountered this previously? Can you share any fixes?

Sheet A (note that the Date & Time booked is a unique ID)

Sheet B

The formula in A3 is
=IFERROR(VLOOKUP($C3, 'Sheet A'!$A:$A,1,0),"Not in Book")

This is how Sheet B looks when the formula errors out:

A: This issue is caused by your macro placing a text value in the Date&Time Booked cells, but setting the number format of the cells to either General or a custom date format (deliberately or otherwise).
When you press F2 and then Enter in the Date&Time Booked cell of Sheet B, Excel converts the text datetime to a serial datetime and adjusts the number format to a custom date format (so it still looks the same). Your VLOOKUP() function is now trying to compare a text datetime with a serial datetime (i.e. a number), so of course they won't match. Repeating the process on the cell in Sheet A converts that value to a serial datetime as well, so the match succeeds once again.
Without seeing your macro and understanding precisely how you subsequently use the booked datetime values, it's hard to make a specific recommendation as to how to fix the problem.
The simplest solution is to set the NumberFormat property of the Date&Time Booked cells to @ (Text).
The other solution is to modify the macro so it writes proper serial datetimes to the worksheets.

redpajama_set_name: RedPajamaStackExchange


Made with molded teeth, #10 YKK zippers. Please see the Fabric Page for material descriptions and the Features Page for complete details of all framebag features.
This Mudlust is a full suspension framebag specifically designed for full suspension mountain bikes. Like all Mudlust framebags, the Fully uses PhotoFit™ to create a perfect fit to the curves of your full suspension "triangle". Mudlust uses quality materials inside and out. X-Pac™ or Cordura® is used for the outer and inner fabric and Ballistic Cordura® is used on the spine where the bag touches the frame.
Variable velcro attachment to the top tube near your stem. This allows compatibility with your choice of top tube bag, including the Cache and Medic bag.
PhotoFit™ Template Creation: A strategic photo of your bike is used to build your Rockgeist full suspension framebag. PhotoFit™ offers several advantages over physical customer-traced templates. The most important advantage is that it eliminates template errors and helps avoid zipper stress, maximizing packing efficiency, and ensures strap compatibility with existing bags/frame features.
I'm very happy with the full experience. I sent a email to Greg not knowing he was on the AZ trail, but with in 2 days he responded to me with a explanation that he was out of reach. After a quick chat he was back to riding. True to his word he contacted me immediately when he was back in the office. To my surprise he had a template in house. We worked out some details and he shipped it faster than I thought possible.
The frame bag got a serious beating on the CTR in July. Relentless rain, lots of mud and over stuffed. The zippers worked as they should, the stitching held together flawlessly, the straps held tight and never needed readjusting. top shelf quality!
I also had a couple of other small bags made that completed a full custom set. I could not be happier with Greg and his business.
I ordered a full set of custom bags for a Santa Cruz Hightower, including the Mudlust Custom for full suspension. Each bag is incredibly well made and fits perfectly. The functionality of the material and design is incredible. Greg is very helpful in the selection process as well as "tweaking" the set-up. I did a lot of research before ordering. After receiving the bags, I cannot imagine a better custom set-up on the market. You will not be disappointed.


Keven Smith makes it all about him in Jay and Silent Bob Strike Back.
I guess you can't blame Kevin Smith for just giving up. After four feature films from his View Askew production company, pundits and naysayers were still slamming the director of Clerks and Chasing Amy as a Gen-X wordsmith who made movies about as interesting to look at as a white wall. Sure, he could turn a phrase, even raise the occasional idea, but how much filmmaking ineptitude did one have to endure to get to them? Kevin Smith wasn't a filmmaker; he was a movie geek with a camera.
It should come as little surprise, then, that Smith's fifth film, Jay and Silent Bob Strike Back, is an expression of complete disdain for anything but his own private View Askew-niverse of industry buddies, axes to grind and childhood pop culture memories. "Fine," Smith seems to be saying, "I'm not even going to bother trying to tell a story. I'm going to make a 98-minute in-joke out of all my other movies, stuff that pisses me off and stuff I always wanted to do in a movie." Who needs an expansive visual repertoire when you can make an entire film that could be captured by one image: a huge middle finger raised at the rest of the world?
Kevin Smith can whip up crudely funny dialogue with the best of them, but Jay and Silent Bob Strike Back has to rank among the most massively self-absorbed comedies ever to earn a laugh. On the surface, it's a farewell to the long-running title characters—foul-mouthed stoner/slacker Jay (Jason Mewes) and his sidekick Silent Bob (Smith himself)—that have appeared in all of Smith's previous films. They're on an episodic cross-country mission from New Jersey to Hollywood to thwart the making of a film loosely based on their lives, but more to the point, they're on a mission to provide Smith with opportunities to indulge himself. Mission accomplished.
Smith has set all of his films in an interconnected universe, and Jay and Silent Bob often looks more like a big class reunion than a Hollywood film. Brian O'Halloran and Jeff Anderson appear as the convenience store drones from Clerks; Jason Lee turns up as his characters from both Mallrats and Chasing Amy (though not his fallen angel from Dogma); Ben Affleck does similar double-duty as his Chasing Amy character and as himself in a very funny, self-deprecating turn with pal Matt Damon. Alums of previous Smith efforts fill nearly every role, with Chris Rock, George Carlin, Joey Lauren Adams and Shannen Doherty among those who stop by for a nod and a cup of coffee.
Even more often than he grinds an axe, Smith uses Jay and Silent Bob to release his inner adolescent. He puts Shannon Elizabeth and Eliza Dushku in leather catsuits (not that you can blame him), brings in Mark Hammill so he can have a light saber duel with Luke Skywalker, and gets to jump on stage with Morris Day and the Time to boogie to "Jungle Love." He quotes Scooby-Doo, Planet of the Apes, Land of the Lost and Scream, and still manages to find time for bonding with a rescued orangutan. It's Kevin's world, and if you find every pop culture reference as funny as he does, you're welcome to stay.
It's frustrating watching a smart guy like Smith spend so much time gazing at his navel, because when Jay and Silent Bob is funny—including a few noteworthy shots at the state of Utah—it's very funny indeed. Even building an entire movie around Jay and Silent Bob—a shudder-inducing notion after their grating appearance in Dogma—doesn't prove disastrous, with both Mewes and Smith showing a lighter touch this time around. There's just too much devotion to ranting and riffing, including an entire third act that plays like a homage to Tim Burton's Pee-Wee's Big Adventure.
That's a rather ironic choice, considering Smith's public, tongue-in-cheek feud with Burton over Burton's rejection of Smith's Superman Lives script (and more recently over whether Burton's Planet of the Apes cribbed from a Smith-penned comic book). But even if Burton thought Smith had dissed him and ripped him off, it's hard to imagine that Burton would make an entire movie dedicated to giving Smith the finger. Sometimes, instead of striking back, you've just got to let it go.
Jay and Silent Bob Strike Back (R) ** Directed by Kevin Smith. Starring Jason Mewes, Kevin Smith and Shannon Elizabeth.


text:
The Weeknd teases new music from soundtrack of James Cameron's 'Avatar: The Way of Water'
Currently, the extent of the singer's involvement will be in the film or the soundtrack
By ANI
Published: Tue 6 Dec 2022, 10:40 AM
Canadian singer-songwriter The Weeknd recently teased new music which will be part of the soundtrack for James Cameron's much-anticipated sequel Avatar: The Way of Water.
According to Variety, the singer took to Instagram and Twitter on Sunday and shared a short clip displaying the film's logo and December 16 release date.
A choir could be heard playing in the background, supported by a booming drum beat, reminiscent of the previous film's grandiose score by James Horner.
This post was retweeted by the film's official account. Avatar: The Way of Water producer, Jon Landau, followed this up by posting a clip of the two together, captioned, "As the Na'vi [the fictional language of the film] say, 'Zola'u niprrte' soaiane Avatar'... Welcome to the Avatar family."
Currently, the extent of the Weeknd's involvement will be in the film or the soundtrack, reported Variety.
Avatar: The Way of Water stars Zoe Saldana, Sam Worthington, Sigourney Weaver, Stephen Lang, Cliff Curtis, Joel David Moore, CCH Pounder, Edie Falco, Jemaine Clement, and Kate Winslet.
20th Century Studios will release the movie in English, Hindi, Tamil, Telugu, Kannada and Malayalam on December 16, 2022.
As per Deadline, behind the scenes, reports have claimed that the production budget alone for the film has reached USD250 million.
A sequel to the Award-winning 2009 epic adventure Avatar, which was released in theatres 13 years ago, the script for Avatar: The Way of Water comes from Cameron and Josh Friedman.
Taylor Swift dominates again at American Music Awards
More news from Entertainment
Why everyone's debating this British star's Best Actress Oscar nod
Andrea Riseborough was unexpectedly nominated for her performance in the scantly seen indie drama 'To Leslie'
The filing disputes the validity of an amendment to Presley's living trust that removed her mother and a former business manager as trustees
She worked with some of Hollywood's most elite directors in a film career that preceded her full-time move to television
In 'Shrinking,' Jason Segel does the work
The Apple TV+ series is a comedy about a therapist crushed by grief and trying to put himself back together
Korean star Song Joong-ki announces marriage, wife's pregnancy
The 'Descendants of the Sun' actor confirmed his wedding has been registered today
Deepika Padukone hides face under hoodie as she visits cinema hall to see public's reaction to 'Pathaan'
She visited Mumbai's Gaiety Galaxy theatre on Sunday, January 29
'Grateful, happy, loved': Shah Rukh Khan greets fans outside Mumbai residence post 'Pathaan' success
Dressed in all-black and a bandanna, Shah Rukh on Sunday made an appearance on the balcony of his sea-facing bungalow 'Mannat'
Bollywood blockbuster: Shah Rukh Khan's Pathaan dialogue goes viral, shows Arabic connect
The brand ambassador of Dubai truly did represent the city in this movie with his action-packed scenes filmed in the emirate


Dylan Haskin is an professional photographer and adventurer who was born in Cape Town, South Africa and currently lives and works in Costa Rica. Dylan shoots a lot of adventure, travel and sport photography.
My goal in life is to live. Live with passion, with enthusiasm, spread positivity, give as much as I can, teach, learn, love, help, and just to be real!


Stuttgart – Investments in startups are on the rise. In 2017, institutional investors and companies invested 147 billion euros in these enterprises – nearly three times as much as in 2012. Bosch, too, is increasing its investment in external startups, providing Robert Bosch Venture Capital GmbH (RBVC) with 200 million euros for a fourth fund. "Shaping the future also means recognizing good ideas early on and helping them achieve a breakthrough. As a leading IoT company, we want to drive forward select technologies in areas of future relevance, such as artificial intelligence," explains Dr. Volkmar Denner, CEO of Robert Bosch GmbH. RBVC currently has a stake in three of the most promising artificial intelligence (AI) startups worldwide: DeepMap, Graphcore, and Syntiant. "We not only invest in startups, we also collaborate with them on open innovation projects. This is one way in which we boost our innovative strength." Open innovation is a concept that brings together customers, researchers, suppliers, and partners and integrates them into a company's innovation activities.
RBVC, one of Europe's largest corporate investors, specializes in innovative technology startups. Its portfolio includes more than 35 companies active in autonomous driving, AI, the internet of things (IoT), and even distributed ledger technologies such as blockchain.
Each year, experts from RBVC examine over 2,000 startups, but only around a hundred of these make it to the shortlist. "We invest in six to ten carefully selected companies each year. In addition to capital, we also offer know-how and operational support. We also connect startups with Bosch operating units," says RBVC managing director Philipp Rose. These alliances benefit the fledgling companies as well as Bosch itself. Startups get the opportunity to become a supplier, technology partner, or even customer of Bosch. Bosch, in turn, receives early access to the latest technologies and can incorporate them into its own innovations. "Open innovation is a win-win for startups and Bosch alike," Rose says. "It also lets us ensure that the startup's expertise stays in the startup." Bosch currently collaborates with more than one hundred companies on open innovation projects.
RBVC's portfolio comprises thriving companies operating in areas such as machine learning, semiconductor development, and the IoT. One example is British processor company Graphcore, which produces AI components designed to accelerate machine learning. In 2018, the company closed the largest venture capital funding round in Europe and is currently valued at 1.7 billion dollars. "We have been supporting Graphcore since its initial financing round, not only financially, but also with our network and expertise," Ramesohl says. Another recipient of early-stage RBVC investment was Sensoro, which today is the leading supplier of urban IoT solutions in China. A third example is the U.S. company DeepMap, which produces highly accurate maps for autonomous vehicles.
Like every other institutional venture capitalist, RBVC ultimately aims to sell its stake in a startup at a profit. For instance, Intel acquired Movidius, a manufacturer of processor chips for computer vision, and cybersecurity expert SecurityMatters is now part of ForeScout Technologies, a provider of network security.


text:
Rev. Joel Walther, Jeannine Page-Tear, Rick Martin
Rev. Joel Walther – Pastor
The Rev. Joel Walther is originally from Minnesota, the son of a United Methodist minister. Following high school, he moved to New York City where he studied acting at the American Musical and Dramatic Academy (AMDA), graduating in 2000. He moved back to Minnesota where he worked a variety of jobs, but eventually ended up as a youth director at a church. After four years as a youth director, Walther reconnected with the call he had received when he was 12 and went back to school at Metropolitan State University in St. Paul, MN where he earned his BA. Walther attended Wesley Theological Seminary in Washington, DC starting in 2008. He graduated with his MDiv in 2011. While at Wesley he met Megan, whom he married in 2011. Megan is also a pastor, currently serving in Clarkston. Pastor Joel seeks to be a fool for Christ (1 Corinthians 1:18-31), understanding that while the world will try to tell you how it has all of the wisdom, true wisdom comes from understanding the foolishness of the cross and how living in the way of the cross will often make us appear foolish to the world. He hopes to share in this foolishness with others.
[email Pastor Joel]
Jeannine Page-Tear – Office Administrator & Youth and Sunday School Director
Jeannine is a long standing member of Goodrich United Methodist Church. As Membership Secretary, Jeannine organizes and compiles the Sunday bulletin weekly. She also tracks Sunday worship attendance and follows membership of the congregation. Jeannine is also the Sunday School Superintendent and has been involved in Christian Education most of her adult life. She, as Sunday School Superintendent, helps to create a fun and safe learning environment for children. Jeannine holds an Associates degree in Applied Science from Baker College. Jeannine enjoys shuttling her three children to their various activities, reading, and is an avid card player.
[email Jeannine]
Rick Martin – Custodian
Rick started in his role in 2017 and has done a great job given his many years of experience in school systems. Rick grew up in the area. He loves his job and helping people he meets on the job. He really enjoys everyone he runs into and feels a part of the GUMC family.
[email Rick]