text:
\section{Introduction}
The progress of machine learning algorithms and accelerators are bidirectionally linked. Algorithms that are well-structured for available computing achieve higher quality at less cost. \cite{tan2020efficientnet} Similarly, accelerators are benchmarked on pre-existing ML algorithms and succeed or fail accordingly.\cite{reddi2020mlperf} In this paper, we try to escape this self-reinforcing system by considering a different form of machine learning, contemplating what kind of accelerator this approach would require, and consider whether the results have consequences for conventional machine learning systems.

Current ML accelerators use matrix multiplication as the basic building block. These matrix multiplication units are primarily:
\begin{itemize}
\item Dense: Dense matrix multiplication is very efficiently implemented in silicon with systolic arrays. \cite{tpu} Algorithmically, density is a result of backpropagation, which tends to fill in the value of every weight with a number, even if that value means little to the end computation.
\item Small: Dense matrix multipliers are easily composable to provide the computation for arbitrarily sized matrices. Therefore, a smaller matrix multiplier can be used as an efficient atom in a big matrix computation, whereas a big matrix multiplier will suffer from lower utilization on small matrix computations.
\item Two-operand: Finally, most matrix multipliers have both the matrix and the vector as stored variables. There is a need to change matrix values during training and in order to enable time-multiplexing of the multiplier unit to support different matrix dimensions. Having two operands can be costly from area and power. The primary power-saving technique for these multipliers is batching, where the matrix values are kept constant while performing multiple vector times matrix computations.
\end{itemize}

Reservoir computing, specifically Echo State Networks as discussed in this paper, uses very large, sparse matrices. These matrices are generated randomly and are never modified by training, and therefore the matrix is fixed for the lifetime of the computation. In order to handle these matrices, conventional ML accelerators perform indexing and tiling of the sparse matrix, which effectively transform the large sparse operation into multiple small dense operations. However, this transformation has compute and storage overhead, and tiling increases the latency of the computation. \textbf{This paper shows that it is both possible and advantageous to implement these large sparse operations directly in programmable hardware, which eliminates the cost of indexing and tiling.} We use Reservoir Computing as a motivating factor, but discuss how this could impact ML inference acceleration in the future.

\subsection{Contributions}

The primary contributions of our paper are as follows:
\begin{itemize}
\item We present a bit-serial architecture for fixed vector-matrix multiplication, which minimizes hardware, energy and latency.
\item We implement very large, sparse matrices using this architecture without paying the cost of indexing or tiling.
\item We present simple cost and power models, which enable the quick estimation of size and power of any fixed matrix on an FPGA.
\item We show how a simple transformation can significantly improve the achievable density.
\item We compare the achievable performance against state of the art sparse libraries for GPUs and DNN accelerators.
\item We discuss how this idea could be applied to a custom circuit that would eliminate the limitations of the FPGA.
\end{itemize}

\subsection{Outline}
The remaining content is structured as follows:
In Section \ref{sec:rescomp}, we describe reservoir computing in detail and enumerate prior art. Section \ref{sec:bitserial} reviews the state-of-the-art in bit-serial machine learning acceleration and describes our low-latency bit-serial matrix multiplier. Section \ref{sec:synthesis} describes synthesis of our architecture and presents a simple area model based on our data. Section \ref{sec:csd} investigates the use of canonical signed digits in our matrices. Section \ref{sec:physical} discusses experiments compiling very large matrices onto a large FPGA and measuring area, frequency and power. Section \ref{sec:eval} compares these results against the state-of-the-art sparse libraries for comparable GPU implementations. We also compare against a sparse DNN accelerator. Finally, in Section \ref{sec:future} we discuss the implications of this approach to implementations of more conventional models and ASICs.

\section{Reservoir Computing}
\label{sec:rescomp}
Reservoir computing is a subset of supervised machine learning which seeks to learn a non-linear transformation from an input sequence to an output sequence. It's conceptually similar to a recurrent neural network, but has one key difference, the recurrent weights are not trained. A reservoir system can be described using the following equations, adapted from \cite{TANAKA2019100}:
\begin{equation}
  \mathbf{x}(n) = f(W^{in}\mathbf{u}(n) + W\mathbf{x}(n-1))
\end{equation}
\begin{equation}
  \mathbf{y}(n) = W^{out}\mathbf{x}(n)
\end{equation}
Where \begin{math}\mathbf{u}(n),\mathbf{y}(n)\end {math} are the input and output at time $n$ respectively, \begin{math} W^{in}, W^{out} \end{math} are the input and output weight matrices, and \begin{math} f() \end{math} is a non-linear activation function. \begin{math}W\end{math} is a very large, fixed matrix, and $x$ is a large vector that carries the state information from one time unit to another.

Ordinarily, \begin{math} W, W^{in}\end{math} are initialized according to some heuristics, and remain fixed.  \begin{math} W^{out} \end{math} is trained via linear regression. It is evident to see why this approach is appealing from a compute perspective - only a linear regressor needs to be trained which completely eliminates the need for error backpropagation and allows the designer to choose the optimizer for the linear layer, which may be gradient-descent, least-squares, or any other optimization technique.

A comparison of various systems for recognition of multivariate time series is presented in \cite{bianchi20}. This paper compares reservoir computing systems against fully trainable recurrent networks. The reservoir solutions train much faster with comparable quality. In this paper, the baseline reservoir computing system consists of a fixed square matrix with a dimension of 800 with 75\% of the elements being 0, which we henceforth refer to as "element sparsity".

A comprehensive summary of physical systems used to implement physical reservoirs \cite{TANAKA2019100} includes a section on FPGA implementations. The earliest citation is \cite{stochastic}, which describes encoding neuron values as "stochastic bitstreams". \cite{res_pca} describes an FPGA implementation of RC and presents heuristics for defining the input matrix \begin{math} W^{in}\end{math}. \cite{cyclic-res} describes an encoding system that allows for multiplication to be represented as shifts.

An FPGA design that includes both the reservoir as well as the output layer with auto-regression was presented in \cite{antonik2015fpga}. This system was used to perform channel equalization, which is ideal for online learning because the known patterns with expected results are presented on a periodic basis. This paper is not easily replicated, but the results state that the design used most of the block RAM on an FPGA, with low logic utilization. This paper did not emphasize performance exploration, as evidenced by the 33MHz clock frequency.

It is shown in \cite{kleyko2020integer} that the elements of these reservoirs can be quantized into integers. The authors show a variety of tasks where a precision of 3-4 bits leads to no accuracy loss. They evaluate their design on a ZedBoard FPGA with an integrated ARM CPU and are able to run a 300x300 reservoir at 100MHz. Studies in \cite{gallicchio2020sparsity} show that sparsity should exceed 80\% to maximize performance and enable rich interaction among neurons.

The idea of using large, sparse matrices has consequences beyond reservoir computing. Numenta~\cite{numenta2019} is a company creating  artificial neural networks by mimicking biological networks, which are all sparse. They are also using large, sparse, random, fixed matrices with a trainable output selection layer for their investigations. Finally, \cite{lth18} showed that special subnetworks with 80-95\% sparsity exist within backpropagation-trained dense networks. These subnetworks have the same or better error rates than full dense networks. In other words, most of the computation performed in inference using the full matrix is wasted.

The remainder of this paper details our approaches to implement a large, fixed matrix multiplication on an FPGA using bit serial arithmetic. Specifically, we accelerate the key primitive in reservoir computing:
\begin{equation}
\label{eqn:prim}
    o = a^{\intercal}V
\end{equation}
Which is often called "gemv" and is an important primitive for many machine learning workloads.

\section{Bit-serial vector-matrix multipliers}
\label{sec:bitserial}
Bit-serial arithmetic was an important technique decades ago when parallel arithmetic was too costly.  Simple VLSI implementations of bit-serial neural networks \cite{bitserialnn} were proposed as far back as 1988.  More recently, DNN accelerators have used bit-serial arithmetic to support variable precision arithmetic.
STRIPES \cite{stripes} presents a bit-serial architecture that enables different precisions on a per-layer basis, quantizing each layer to their required bit-widths.  Bit-fusion \cite{bit-fusion} enables dynamic-precision for inputs as well.  Bit-pragmatic \cite{bit-pragmatic} and Laconic \cite{laconic} exploit bit-level sparsity. They detect the bits that are zero and skip those multiplications. In all the above cases, the DNN dimensions often exceed the compute capability of the hardware, so tiling and striding the inner product is regularly employed.

As discussed earlier, reservoir computing uses a single large, sparse, weight matrix that is fixed for both training and inference. This allows us to optimize the logic implementation by performing constant propagation on the matrix to remove the logic corresponding to all zeros; both zeros as whole terms, and bits that are zeros within individual terms. It would be possible to implement these matrices in custom logic (ASIC), but in this work we use an FPGA. The FPGA consists of an array of Lookup Tables (LUTs), Registers (FFs), and programmable interconnect. FPGAs generally include other programmable resources such as embedded multipliers and SRAMs, which we will not consider in this work. The particular FPGA we are using has the capability to re-purpose some of the LUTs into small RAMs or shift registers which are called LUTRAMs.

Our FPGA design flow takes the content of the matrices and compiles it to a physical design consisting of the programmable logic resources and interconnect. This design flow produces an achievable frequency, area, and power estimation.

\subsection{The Microarchitecture}
Figure \ref{fig:bs-adder} shows the design of a bit-serial adder, which is the basic building block of bit-serial arithmetic units.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/Bit-serial_adder.pdf}
  \caption{Bit-serial adder}
  \label{fig:bs-adder}
\end{figure}

In the bit-serial adder, inputs A and B are shifted into a full-adder one bit each clock cycle, LSb first. The result of the addition (S) is shifted into a result register. The carry is registered as the carry input for the next cycle. This is analogous to manual long addition - add one digit at a time, with any resulting carry being added in the next most significant digit. Rather than chaining the carries through multiple adders "in space", we use a single adder and process the carry "in time". Table \ref{tab:bs_exl} shows an example of a bit-serial addition of \begin{math} 3_{10} + 7_{10} = 10_{10} \leftrightarrow 011_{2} + 111_{2} = 1010_{2} \end{math}

\begin{table}[h]
  \centering
  \caption{Bit-serial Addition Example}
  \label{tab:bs_exl}
  \begin{tabular}{c|cccccc}

    Cycle & $C_{in}$ & A & B & S & $C_{out}$ & Result\\
    \hline
    1 & 0 & 1 & 1 & 0 & 1 & 0000\\
    2 & 1 & 1 & 1 & 1 & 1 & 1000\\
    3 & 1 & 0 & 1 & 0 & 1 & 0100\\
    4 & 1 & 0 & 0 & 1 & 0 & 1010\\
\end{tabular}
\end{table}

Similarly, we can create a bit-serial subtractor that performs $a - b$ by initializing the carry bit to 1, and adding a NOT gate between $b$'s register and the full adder. This scheme is equivalent to negating $b$ in two's complement and then adding it to $a$. In the FPGA, the bit serial adder or subtractor can be mapped to a single 6-input LUT and two registers. The timing path that determines the frequency for this design (the critical path) will be very short: a single stage of logic, the setup to the flop, and any interconnect delay between the components.
\paragraph{Single-bit dot-product} Using these primitives, we now build a single-bit dot-product, as described below:

\begin{equation}
\label{eqn:1-bit_prot_prod}
s =\sum_{i=1}^{N} a_{i}*b_{i}
\end{equation}

Where $a, b$ are N-length vectors whose elements are 1-bit wide, and $s$ is a scalar integer. Because we are multiplying single bits, we can realize the multiplication with a simple AND gate. After multiplying each pair of elements, we reduce the partial sums through a tree of bit-serial adders.

\paragraph{Multi-bit input} We can extend this design to support one of the operands, $a$, being multiple bits wide. We hold the single-bit vector $b$ fixed, and stream the multi-bit vector through the circuit one bit at a time from LSb to MSb using a shift register for each dimension. Similarly, we capture the result in a shift register. One can think of this circuit as summing the elements of $a$ that are selected by a Boolean vector $b$. This design is shown in Figure \ref{fig:mixed-bit_dot_prod}a.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/New_Lg_N_1-bit_times_multi-bit_dot-product.pdf}
  \caption{a)Multi-bit by fixed single-bit dot-product with tree reduction and shift registers (N=4) (trapezoids represent bit-serial adders, which have flip-flips between each stage but are omitted for brevity)\\ b) Setting b = [1 1 0 1] reduces the circuit}
  \label{fig:mixed-bit_dot_prod}
\end{figure}

Note that because this is merely summing the dimensions selected by $b$, this circuit works for both signed and unsigned $a$ inputs. To ensure signed inputs produce the correct sign bit, we sign extend the input $a$ from the shift register until the computation has finished.

Given that our boolean vector $b$ is fixed, consider a given element $b_i$. If this element is set to 1, the input to the AND gate is 1, so the output of the AND gate will just be the associated $a_i$. In this case, we can cull the AND gate entirely and connect the input $a_i$ directly to the bit serial adder. On the other hand, if the element is 0, the AND gate will have a fixed 0 input and therefore its output will always be 0. Furthermore, one of the inputs to the bit-serial adder is now a fixed 0, so the adder is just passing the value forward. In this case, the adder is acting as a D-flip-flop. This means we can not only cull the AND gate, but we can also replace the adder with a single flip-flop, which reduces the cost significantly. Put simply, we can greatly reduce the cost of this circuit by fixing $b$, and the cost should be proportional to the number of bits set in $b$. \textbf{This optimization is the fundamental minimization technique in our design.} An example of this reduction can be seen in Figure \ref{fig:mixed-bit_dot_prod}b. We will revisit this idea in Section \ref{sec:synthesis}.

\paragraph{Multi-bit input and weight} Finally, we can combine multiple of these multi-bit by 1-bit dot-products to create a full multi-bit dot-product. This allows the dot-product of two integer vectors, each with its own bit-width. Because each dot-product in the previous circuit occurs at the bit-level in $b$, we merely need to create such a circuit for each bit position in $b$, and then combine the results for each bit position.

Luckily, the bit-serial adders already have this capability by shifting in time. For example, to make a 2-bit dot-product circuit, we make one circuit for the MSb, and one for the LSb. The results of the two are combined by feeding the LSb result into a bit-serial adder, and then feeding the MSb result, delayed by 1 cycle, into the same bit-serial adder, which then outputs into a shift register to capture the final result. Delaying the MSb result by one cycle effectively multiplies it by 2. To extend this idea to arbitrary bit-width, we just create a chain of bit-serial adders, where the bottom input is the previous link in the chain, and the top input is the corresponding bit. In this case, the result of a bit position is delayed accordingly. A multi-bit example is shown in Figure \ref{fig:multi-bit}. Notice that the MSb is fed into a bit-serial adder along with 0, which becomes a D flip-flop.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/New_Lg_N_multi-bit_dot-product.pdf}
  \caption{Multi-bit Dot-product }
  \label{fig:multi-bit}
\end{figure}

This architecture works for signed inputs and unsigned inputs, but the weights are unsigned. An easy way to implement signed weights is to separate the positive and negative terms of the $b$ vector into two separate unsigned vectors, and simply subtract the two resultant streams. Because the number of ones in the two matrices is conserved by this transform, it makes almost no impact on the total area, and adds a single cycle to the latency.

\paragraph{Vector-matrix product} Now that we have a unit to perform dot-products, we can finish up this matrix-multiplier by creating a dot-product unit for each column in the matrix. Then, we just broadcast the input on each row to all columns, and that results in a matrix multiplier, which takes shape as shown in Figure \ref{fig:arch}. (Full bit-serial circuits are omitted here for brevity)

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/Detailed_Lgn_MM_Drawing.pdf}
  \caption{Full Vector-Matrix Multiplier Architecture \\ Broadcast of inputs shown in green. Multiplication at bit-level of fixed weights shown in blue. Per-column partial sum reduction shown in purple, yielding the final output vector in orange}
  \label{fig:arch}
\end{figure}

Because the computations are parallelized across the matrix columns, and then again across each bit-position, the latency for this design is described in Equation \ref{eqn:latency}:

\begin{equation}
    \label{eqn:latency}
    \text{Latency}=BW_{i} + BW_{w} + \log_2 R + 2
\end{equation}
Where $BW_{i}$ and $BW_w$ are the bitwidth of the input and weights respectively, and $R$ is the number of rows in the matrix. We incur the input width to stream the input in, the output width to stream the output out, and our adder tree is logarithmic in depth. We incur a single cycle to accumulate across bit positions and an additional cycle to subtract the positive and negative weight matrices. For example, given 8-bit inputs and weights and a 1024x1024 weight matrix, we perform the vector-matrix product in $8 + 8 + \log_2(1024) + 2 = 28$ cycles.

\section{RTL Synthesis Results}
\label{sec:synthesis}

In this section, we seek to understand the behaviour of our design by studying it on small matrices. We extend the insights here to larger matrices in Section \ref{sec:physical}. Recall the optimization from Section \ref{sec:bitserial} that if a bit in the weight-matrix is 0, we can cull the AND gate and bit-serial adder, replacing it with a simple D flip-flop.  We coded our design in SystemVerilog and ran synthesis in Xilinx Vivado 2020.2 targeting Xilinx UltraScale+ devices \cite{ultrascale-plus}. If our design behaves as expected, the hardware cost should be proportional to the number of bits set in the weight matrix. To test this, we randomly initialized a series of 64x64 weight matrices at 8-bit precision. For each bit in the weight matrix, we sample from a Bernoulli distribution, where the $p$ parameter is equal to (1 - bit\_sparsity). That is to say, the bit-sparsity of the weight matrix is the number of bits that are 0 out of the total number of bits. 0\% bit-sparse means all bits are 1, 50\% means the bits are uniformly random between 0 and 1, and 100\% means all bits are 0. This gives us a straightforward way to measure the cost of these matrices.

Figure \ref{fig:utilvsparsity} shows synthesis results for sweeping bit-sparsity from 0\% to 100\%. We report utilization of LUTs, FFs, and LUTRAMs.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth, scale=0.7]{figs/UtilBitSparse.pdf}
  \caption{Hardware utilization vs bit-sparsity of a 64x64 matrix}
  \label{fig:utilvsparsity}
\end{figure}

As expected, the total hardware cost of our architecture is linear with respect to the number of bits set in the weight matrix.

This illustrates that our architecture effectively exploits bit-level sparsity to reduce the cost of fixed-matrix multiplication. Recent advances in neural-networks and architecture have similarly sought to exploit element-level sparsity to make computations more efficient. \cite{han2016deep, han2016eie} By element-level sparsity, we mean a portion of the weights being 0, so there is no need to multiply by them. Because bit-level sparsity is a super-set of element-level sparsity, we wanted to know if our design was ineffective given an element-sparse matrix. To test this, we randomly initialized a set of matrices, where the weights are sampled from a uniform distribution of all possible values for the given bit-width. In this case, the matrix is 50\% bit-sparse, as every bit has an equal probability of being 0 or 1. We then randomly replace matrix elements with 0 until we reach a desired level of element-sparsity. Taking these samples, we convert the element-sparse value into a bit-sparse value, and compare the two approaches. This second experiment encourages bits to gather in individual elements, rather than the bit-sparse experiment which encouraged bits to be spread out. Figure \ref{fig:EsBs} shows our findings and plots them against the original experiment.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/Lgn_es_bs.pdf}
  \caption{Cost of Element Sparse Matrix Compared To Bit-Sparse Matrix \\
  Here, (es) and (bs) designate Element Sparse and Bit-Sparse schemes respectively. While results do not match exactly, they are within an acceptable noise margin}
  \label{fig:EsBs}
\end{figure}

The graph shows us that it doesn't matter if the bits are concentrated or not. The two lines are nearly identical, which means that we don't have to make any concessions to support element-sparse designs. Unlike most sparse accelerators, this design exploits sparsity in the elements when matrix entries are zero, and also elicits great benefit from elements being powers-of-two.

Figure \ref{fig:mat_size} shows the LUT and FF utilization vs matrix size. The cost is quadratic with respect to matrix dimension and therefore linear with respect to the number of elements. This implies there is little optimization our RTL flow is doing across weight elements. In other words, large matrices are no more and no less dense than smaller matrices.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth, scale=0.7]{figs/MatrixSize.pdf}
  \caption{Hardware Utilization vs Matrix Size for random 8-bit integers}
  \label{fig:mat_size}
\end{figure}

We also explore the cost function with respect to the bit-width of the weights. Figure \ref{fig:bitwidth} shows the cost of a 64x64 matrix, sweeping the weights bit-width from 1-bit to 32-bit. Because the architecture performs 1-bit dot-products for each bit-position, and then accumulates the results, we would expect there to be little cross-bit optimizations. Indeed, we observe a linear LUT and FF cost with respect to the bit-width of the weights.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/Bitwidth.pdf}
  \caption{Hardware Utilization of 64x64 Random Matrix for Varying Bitwidths}
  \label{fig:bitwidth}
\end{figure}

\section{Canonical Signed Digit}
\label{sec:csd}
In an effort to reduce the cost of the multiplier, we consider Canonical Signed Digit (CSD) representations \cite{signed_digit}. CSD decomposes an unsigned integer into a sum of a positive and negative integers, where the positive and negative integer together have fewer set bits than the original number. For example, consider the following :
\begin{equation*}
    15_{10} = 16_{10} - 1_{10} \leftrightarrow 1111_{2} = 10000_{2} - 00001_{2}
\end{equation*}
Notice that we can decompose a number that had four bits set to the difference of two numbers which each have one bit set, for a total of two. Do also take note that the bit-width of the decomposition is one wider than the original. Recall that the cost function of our multiplier is the number of bits set, so this shows promise in our case. We transform Equation \ref{eqn:prim} using:
\begin{equation}
\label{eqn:csd}
    V \equiv P - N \implies o = a^{\intercal}(P-N) = (a^{\intercal}P) - (a^{\intercal}N)
\end{equation}
To transform V into N and P, we apply the algorithm in Listing \ref{lst:csd} element-wise, placing positive elements in P, and placing the absolute value of negative elements in N. In short, the algorithm searches for strings of consecutive 1 bits, which we call a chain. If a chain is length 1, nothing is done. If a chain is length 2, we flip a coin. On heads, we replace the chain with a +1 in the bit position one-past that of the MSb in the chain and a -1 at the LSb in the chain. On tails, we leave the original representation. For chains of length 3 and greater, we perform the same replacement that we do for heads on length 2. We introduce the random variable to balance the decomposition, since a transformation of a length 2 chain has no benefit and no detriment.

When operating on signed weights, we perform a CSD transform on both the positive and negative weight matrices. Positive elements that result from CSD remain in the original matrix, and negative elements are transferred to the opposite weight matrix.

\begin{lstlisting}[language=Python, caption= CSD Conversion Algorithm, label=lst:csd]
def convert_to_csd(num_bin_list):
  local_list = list(num_bin_list)
  target = [0] * (len(local_list) + 1)
  local_list.reverse()
  chain_start = -1  # are we in a chain?
  for i in range(len(target)):
    if i < len(local_list):
      bit = local_list[i]
    else:
      bit = 0
    if bit == 0:
      if chain_start == -1:  # no chain
        target[i] = 0  # nothing to be done here
      else:
        #  We terminate a chain, how long is it?
        chain_length = i - chain_start
        if chain_length == 1:  # leave it alone
          target[chain_start] = 1
        elif chain_length == 2:  # a chain of two
          if bool(random.getrandbits(1)):
            # do the substitution
            target[chain_start] = -1
            target[i] = 1
          else:
            target[chain_start] = 1
            target[i-1] = 1
        else:  # will get benefit
          target[chain_start] = -1
          target[i] = 1
      chain_start = -1  # not in a chain anymore
    else:  # bit == 1
      if chain_start == -1:
        chain_start = i
  target.reverse()
  return target
\end{lstlisting}

Figure \ref{fig:csd_es} shows the resource utilization when CSD is applied to random matrices of varying element-sparsity. Note that the x-axis is element-sparsity and not bit-sparsity.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/csd_es.pdf}
  \caption{CSD Resource Utilization for 64x64 Element-Sparse matrices}
  \label{fig:csd_es}
\end{figure}

CSD results are strictly better than the naive implementation. In this experiment, all the numbers are uniform random. CSD applies equally to them and reduces the hardware by ~17\% for any level of element-sparsity. In other words, CSD always makes numbers more bit-sparse, which reduces the cost of our architecture. We would expect these savings to improve for larger weight bitwidths.

\section{Large Scale Design Results}
\label{sec:physical}

The prior sections looked at the logic synthesis results of small matrices. In this section, we show the scaling of these techniques to much larger matrices. Our experiments in this section use square matrices with dimensions of 512 and 1024, with 8-bit signed weights. We consider element sparsity from 40\% to 98\%. We use a split matrix to deal with the signed weights, and the split matrices are generated by either splitting positive and negative terms (PN) or by using the CSD technique previously described. The results of these two matrices are fed to an array of final bit-serial subtractors to complete the signed matrix implementation. We "wrap" the matrix multiplier with a small design that feeds inputs from an SRAM, and captures results in that same SRAM. This design wrapper only adds a few extra LUTs and registers. We run synthesis and place and route on these designs and set a timing constraint of 450MHz. Our target FPGA is the Xilinx XCVU13P \cite{ultrascale-plus}, which is a 16nm device containing four chiplets in the package (called Super Logic Regions or SLRs). This device has a capacity of 1.7M 6-input LUTs and 3.4M logic flip-flops.

The LUT and register counts as a function of ones in the matrix is plotted in Figure \ref{fig:largescale-area}. The PN and CSD points are based on the identical original signed matrix. The trend lines show the strong correspondence with the resource counts and the number of ones. LUTs are essentially equivalent to the number of ones, and there are two registers per LUT. CSD reduces both the number of ones in the matrix and the resulting resource counts.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth, scale=0.9]{figs/large_area.png}
  \caption{Large Scale Area Results: The very strong linear relationship between matrix ones and FPGA resources is obvious.}
  \label{fig:largescale-area}
\end{figure}

The process of FPGA design does not always meet the specified timing constraint. Figure \ref{fig:largescale-fmax} shows the achieved frequency (Fmax) of these designs after placement and routing. All the paths within these designs have at most one LUT between flops, which means that the frequency is primarily a result of the interconnect delays between LUTs and flops. Other components, such as the SRAMs have a maximum frequency that exceeds 600MHz. There is some noise in these results, but the trends are that bigger matrices run slower. As the matrices get bigger, two mechanisms impact the routing delay:

\begin{itemize}
    \item The initial layer has a large fanout, approximately corresponding to the dimension times the sparsity. Nets that have a fanout of 100s can have delays of several nanoseconds, which becomes the critical path.
    \item Nets cross the chiplet boundaries, and those routes have significantly slower propagation delays.
\end{itemize}

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth, scale=0.9]{figs/large_fmax.png}
  \caption{Large Scale Frequency Results: Increasing fanout of the first stage and the spanning of multiple chiplets increase the critical path delay and decrease maximum frequency.}
  \label{fig:largescale-fmax}
\end{figure}

Each of the four SLRs within the FPGA have a maximum capacity of 425k LUTs. After about 80\% of LUTs are used the tools can struggle. Figure \ref{fig:largescale-fmax} has tick marks illustrating the 82\% thresholds of SLR capacity. Within one SLR, the frequencies range from 597MHz to 445MHz. Designs requiring 2 SLRs range from 296MHz to 400MHz. Matrices bigger than 2 SLRs seem relatively consistent between 225MHz and 250MHz. Both the fanout and chiplet crossing problems could be addressed by adding registers to perform the fanout and chiplet crossings in multiple cycles. These optimizations are not represented here.

The other limit to the frequency is power. Figure \ref{fig:largescale-pwr} plots the estimated total power consumption of this device scaled to run at the maximum achievable frequency. These results were obtained from the Vivado tool based on the default assumptions about switching activity. Under medium settings for airflow and heatsink, the thermal power limit of this FPGA is approximately 150W, which we approach at high dimension and low sparsity.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth, scale=0.9]{figs/large_power.png}
  \caption{Large Scale Power Results: Note the sublinear increase due to the decreasing acheivable frequency. Under medium cooling assumptions, this FPGA has a limit of about 150W.}
  \label{fig:largescale-pwr}
\end{figure}

\section{Evaluation}
\label{sec:eval}
To evaluate our design, we compare it against state-of-the-art sparse multiplication libraries on GPU, and also against a recent sparse DNN accelerator.

\subsection{GPU Benchmarks}
We compare our results against the Volta V100 GPU from NVIDIA. The V100 is built in 12nm technology, whereas the UltraScale+ FPGA is 16nm. The more recent Ampere Architecture from NVIDIA includes hardware support for sparsity up to 50\%, but is unfortunately unavailable to us for this study. We use two libraries for the GPU benchmarks: cuSPARSE \cite{cusparse}, and a recently published optimized kernel \cite{gale2020sparse}. Neither of these libraries support integer arithmetic, so we are using FP16 as a best-case proxy. We run the FPGA at the maximum frequency achieved after place-and-route, which can be seen in Figures \ref{fig:largescale-fmax} and \ref{fig:largescale-pwr}.

For both of these configurations, we initialize a random weight matrix using the same scheme as described previously. We transfer this to the device and allow the given library to perform any necessary optimizations before timing. Then, we randomly initialize a dense vector, and repeatedly multiply this vector by the matrix so that all caches and memory systems are warm. We do this for 1000 iterations and take the latency to be the mean iteration time. In this case, the latency is measured from the devices memory, through the arithmetic, and back to memory - which is identical to our FPGA implementation.

\paragraph{Sweeping dimension} Our first experiment demonstrates latency performance as a function of array dimensions at 98\% element sparsity. We sweep the matrix dimension from 64x64 to 4096x4096 in powers of two. The results are show in Figure \ref{fig:bench_dim_lat} with actual latency numbers and in Figure \ref{fig:bench_dim_su} as comparative speedup. The "Optimized Kernel" is the library in \cite{gale2020sparse}.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth, scale=0.9]{figs/bench_dim_latency_2.pdf}
  \caption{Latency (ns) of varying 98\% element sparse matrices}
  \label{fig:bench_dim_lat}
\end{figure}

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/bench_dim_su_2.pdf}
  \caption{Speedup of varying 98\% element sparse matrices}
  \label{fig:bench_dim_su}
\end{figure}

Notice that in all cases, our FPGA latency is less than 120ns, whereas the GPU cannot break the 1$\mu$s barrier. When the matrix size is less than 512x512, the GPU performance is nearly constant. This indicates the GPU is underutilized and therefore  latency bound. GPUs possess incredible throughput using techniques like thread-level parallelism, memory latency hiding, double-buffering, and others. However, all these techniques require the GPU to spawn many more threads than the arithmetic can handle, swapping threads in and out to maximize utilization. In the low-latency regime, these techniques introduce overhead which cannot be overcome, even with the Optimized Kernel. Our solution configures the hardware to exactly support the matrix in question, leading to astounding latency. When the GPU is latency-bound, its latency does not increase for larger matrices. Our solution pays a constant cost to stream the inputs and outputs, but the number of cycles spent reducing partial sum increases logarithmically with matrix size. Furthermore, increasing the matrix size decreases our highest achievable clock frequency. In the GPU's latency-bound regime, we see our speedup fall from 86x to 60x. However, at 1024x1024, the GPU is utilized and is no longer latency-bound, so it begins to see linear scaling with respect to matrix size. Because our solution scales logarithmically in cycles, we see our speedup leveling off at 50x due to the slower clock.

\paragraph{Sweeping sparsity} Our next experiment considers the average latency as a function of matrix sparsity at a fixed dimension of 1024x1024. The results are show in Figures \ref{fig:bench_spar_lat} and \ref{fig:bench_spar_su} with average latency and speedup, respectively. Element sparsity is swept from 70\% to 98\%. Recall that our solution's latency in cycles does not depend on sparsity, but we can clock the design faster as sparsity increases. The GPU sees decreasing latency with increasing sparsity because it needs to operate on fewer elements and the cost of indexing is amortized. For cuSPARSE, increasing sparsity from 70\% to 85\% sees large reductions in latency as the library reduces the amount of compute to be done. The optimized kernel comparatively spends less time indexing and has higher performance at lower sparsity. In both cases, the compute reduction from 70\% to 85\% is effective and sees our speedup go from 77x to 72x. As sparsity increases further, the GPU again becomes underutilized and both the latency and speedup level off, yielding a minimum speedup of 60x.  Again, the GPU is unable to break the 1$\mu$s barrier, whereas our solution stays under 120ns.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/bench_spar_lat_2.pdf}
  \caption{Latency of varying sparsity}
  \label{fig:bench_spar_lat}
\end{figure}

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/bench_spar_su_2.pdf}
  \caption{Speedup of varying sparsity}
  \label{fig:bench_spar_su}
\end{figure}

\paragraph{Batching} We have illustrated the advantage of our architecture for latency-sensitive scenarios, which is our targeted case. In this next experiment, we compare throughput of the GPU solution versus our solution. Previous experiments, because of the recurrence, were done with a batch size of 1, which underutilizes the GPU's pipelined resources. In this experiment, we change the GPU's computation to perform matrix-matrix multiplication, where the number of columns in the multiplicand matrix is the "batch-size", borrowing terminology from DNN processing. We again randomly initialize a fixed weight matrix with 95\% sparsity and signed 8-bit integers. We choose 95\% sparsity to give the GPU ample headroom. We sweep the batch size from 1 to 64, reporting the speedup of our solution compared to the GPU. We are still measuring latency, but in the higher batch sizes, the reciprocal of the latency will approach the maximum throughput of the GPU.

Figure \ref{fig:bench_batch_1024} presents our results for a 1024x1024 weight matrix, and multiplying a batch\_size x 1024 random dense matrix.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/bench_batch_su_1024_2.pdf}
  \caption{Speedup against V100 for varying batch size (matrix = 1024x1024, 95\% sparse) Batch-size 1 is a comparison of pure latency. As batch size increases, the speedup is comparing achievable throughput}
  \label{fig:bench_batch_1024}
\end{figure}

Figure \ref{fig:bench_batch_64} presents our results for a 64x64 weight matrix, and multiplying a batch\_size x 64 random dense matrix.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/bench_batch_su_64_2.pdf}
  \caption{Speedup against V100 for varying batch size (matrix = 64x64, 95\% sparse) Batch-size 1 is a comparison of pure latency. As batch size increases, the speedup is comparing achievable throughput}
  \label{fig:bench_batch_64}
\end{figure}

As expected, the latency for the GPU solution scales sub-linearly with respect to batch size. As the GPU becomes more utilized, it's able to overlap computation and memory to take full advantage of the massive parallelism. Because our architecture is built for vector products, we have to stream the columns of the input matrix in one-by-one, which yields to linear scaling. Because our solution begins with such a lead, it's able to stream multiple batches before the GPU can reach the same performance. This means our solution is still lower latency at small batch sizes. In the 1024 case, our solution is still marginally better due to the GPU already being close to utilized with the large matrix. With 64x64, the GPU has more computational intensity to fill before it becomes utilized.

The TDP of our FPGA is 150W, while the V100 is 300W, but these numbers are not reliable for accurate run-time comparisons. Given the disparity between the platforms, it is very difficult to accurately compare power consumption between these two solutions. There is a difference in the process technology. The FPGA and GPU also have a different set of peripherals associated with them, that may or may not be active. The dynamic power will be different based on the specific variations of the random weights and activations, which is exacerbated by the int8 vs fp16 difference. Finally, the different boards have different methods to monitor power consumption on their different supply rails. However, we believe that the efficiency gains shown here are due to fundamental computational simplification, and it would be reasonable to assume that the dynamic power would be correspondingly lower.

\subsection{DNN Accelerator Benchmarks}
We also consider the state-of-the-art in Sparse DNN acceleration: SIGMA \cite{sigma}. SIGMA similarly relies on an input broadcast and reduction tree to perform matrix-multiplication. We reached out to the authors and were given their cycle-accurate simulator. In their paper, they design SIGMA as a 128x128 array of fp16 processing elements (PEs) clocked at 500MHz. To approximate process technology node differences and the change to int8 from fp16, we assume that SIGMA can be clocked at 1GHz. We run SIGMA with the weight matrix stationary and stream the input in to minimize latency.

\paragraph{Sweeping dimension} We repeat a similar suite of experiments as before. We sweep the dimension of 98\% element-sparse matrices. Figures \ref{fig:bench_sigma_dim_lat} and \ref{fig:bench_sigma_dim_su} show the latency and speedup respectively.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/bench_sigma_dim_lat_2.pdf}
  \caption{Latency of FPGA and SIGMA for varying dimension (98\% sparse)}
  \label{fig:bench_sigma_dim_lat}
\end{figure}

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/bench_sigma_dim_su_2.pdf}
  \caption{Speedup against SIGMA for varying dimension (98\% sparse)}
  \label{fig:bench_sigma_dim_su}
\end{figure}

For small dimensions, SIGMA does report nanosecond-scale latency due to its input broadcast and reduction tree. Furthermore, small, sparse matrices easily fit into the PE grid, so there is little overhead from tiling. However, after 1024x1024, the elements no longer fit in the PE grid and the computation must be tiled. This invokes extra SRAM use and transitions SIGMA into the memory-bound region, where it sees linear scaling. This yields a 4.1x speedup for our solution in the worst case, but we quickly gain a 25x advantage as the matrix size increases.

\paragraph{Sweeping sparsity} Moving on, Figures \ref{fig:bench_sigma_spar_lat} and \ref{fig:bench_sigma_spar_su} report the latency and speedup of a 1024x1024 matrix, sweeping element-sparsity.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/bench_sigma_spar_lat_2.pdf}
  \caption{Latency of FPGA and SIGMA for varying sparsity (1024x1024)}
  \label{fig:bench_sigma_spar_lat}
\end{figure}

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/bench_sigma_spar_su_2.pdf}
  \caption{Speedup against SIGMA for varying sparsity (1024x1024)}
  \label{fig:bench_sigma_spar_su}
\end{figure}

The advantage of SIGMA is that it only maps non-zero weight and activation pairs to PEs. If this union can fit into the PE grid, no tiling is done. As we expect, the size of this set is directly related to the sparsity, and SIGMA sees huge latency improvements as sparsity increases, taking it into the nanosecond regime. However, even 90\% sparsity and below is enough to push it back into the microsecond regime, which yields a large advantage to our design.

\paragraph{Batching} Finally, we compare matrix-matrix multiplication between our solution and SIGMA. We repeat the same batching test and the result is shown in Figure  \ref{fig:bench_sigma_batch_1024}.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{figs/bench_sigma_batch_1024.pdf}
  \caption{Speedup against SIGMA of varying batch-size (1024x1024, 95\%) Batch-size 1 is a comparison of pure latency. As batch size increases, the speedup is comparing achievable throughput}
  \label{fig:bench_sigma_batch_1024}
\end{figure}

Because SIGMA is able to fill its PE array with sparse entries at small batches, it doesn't have as much to gain as a GPU when transitioning from vector to matrix products. At batch-size 2, SIGMA does find opportunity to utilize more PEs and our advantage decreases. However, batch-size 4 and beyond quickly see SIGMA in the memory-bound region again, which causes the speedup to saturate at 5.4x.

\section{Discussion and Future Work}
\label{sec:future}

For the given use case, our design has two major limitations: 1) the fanout of the input broadcast saturates the interconnect resources of the FPGA and limits frequency. 2) we are bound by the number of 6-input LUTs in the FPGA, which limits the number of set bits we have in the weight matrix. Creating a CGRA architecture could solve both of these issues. We could employ a broadcast-friendly pipelined interconnect network for the inputs, similar to the Benes network in SIGMA. Furthermore, a 6-input LUT is made using 64 SRAM bits of 6 transistors each, with 64 MUX T-gates of 2 transistors each, which yields a total of 512 transistors for every LUT. A full-adder uses 16 or fewer transistors \cite{full-adder-16t}, which is 1/32 the cost. A CGRA implementation of our design would see a grid of full-adders and flip-flops, with a flexible tree-like interconnect to perform partial sums and broadcast interconnect for the input. This approach would allow for higher compute density at higher frequencies.

Even with these optimizations, there may be instances where the compute matrix cannot entirely fit in hardware and must be tiled similar to DNN accelerators. To amortize the cost of loading and moving weights, DNN inference accelerators often employ batching to perform the multiplication of multiple vectors with the same matrix, which saves power by reducing the movement of matrix weights. Our approach eliminates the movement of matrix weights by programming them into an interconnect matrix and takes further advantage by propagating the constants within the matrix. The time to modify the interconnect matrix of the FPGA is on the order of 200ms, which limits its practicality in moving weights during runtime.

However, the feed-forward topology of this network allows for the approach of pipeline reconfiguration \cite{piperench99}. Pipeline configuration is the ability to configure hardware cycle-by-cycle as the pipeline fills. This concept would allow a reconfigurable platform with nearly zero configuration overhead time to change the matrix weights. Pipeline configuration is not supported in conventional FPGAs, but we could support it in a custom CGRA architecture. As the partial sums travel down the tree, the levels above the current accumulation can be recongifured as their state is no longer needed. One can think of "waves" of configuration travelling down the tree, where some parts of the tree are reconfiguring and some parts are computing. This is analogous to double-buffering in DNN processing.

We plan to explore these optimizations in future work.

\section{Conclusion}
We present an architecture for performing integer matrix-vector multiplication. The cost and power of our implementations scale exactly with the number of non-zero terms in the matrix. Bit serial implementations allow for us to support matrices with up to 1.5 million ones, as large as 1024x1024 eight-bit matrix at a sparsity of 60\%. These products can be produced with nanosecond-scale latencies, which is 2-3 orders of magnitude faster than a GPU, and 4-47x faster than a sparse accelerator. Throughput is also competitive, especially for large matrices. Given an FPGA's slow configuration time, these techniques are primarily suitable to computations where the matrix is essentially fixed, and where low latency is critical, such as reservoir computing. However, a customized programmable device for this approach could pipeline the configuration, thus effectively hiding it, and enable this approach to work for dynamic sparse matrices.

\section*{Acknowledgements}
We would like to thank Peter Matheu for the extensive technical discussion and help with the SystemVerilog implementation. We would also like to thank Amir Yazdanbakhsh and Dan Zhang for paper review, and Guillermo Maturana for code review. This work was completed as an internship project.

\bibliographystyle{IEEEtranS}

redpajama_set_name: RedPajamaArXiv


We (my wife and I) rented in my area of Philadelphia for a year before buying a home around the corner, and I agree with that advice. We could tell the area was changing for the better, and deciding buying would be a really good option for us. We were renting for around $1100 and it was going to go up the next year (2 br, 1 bath). We ended up being able to knock $5k off the price for things that needed to be fixed in the house, and paid around $3k for the repairs. We pay the SAME AMOUNT on our mortgage (for a 2 br, 2.5 bath, twice as much SQFT) as we paid for the apartment, at a fixed 3.5% interest rate.
We rent the other bedroom to friends at $450/ month (GREAT deal in our neighborhood). Other rental properties at the same size as our home in the area go for around $2,000/month today.
So my point is this: Even though we've had to pay for repairs in our home, we are actually saving around $1,500/month by owning instead of renting the area we like to live in, and building a small amount of equity in the process. If you do it smart, and can have a plan going in, you can save a lot of money by owning vs renting. Just my 2 cents.


Today, Senate Majority Leader Mitch McConnell finally achieved a (semi) win with the GOP members barely voting with a majority on healthcare. The vote was for a motion to move to repeal and replace Obamacare. (50-50, with VP Pence casting the tie-breaking yes vote) This does not immediately repeal, or replace anything however. The Senate will now debate for 20 hours over amendments to the bill. Many GOP Senators stated that they will not vote for the current bill, even though they voted to move forward on the debate. It goes unsaid that ZERO of the Democrats will vote for any type of repeal of Obamacare. Democrats have repeatedly said that they are open to working on Obamacare (The Affordable Care Act) to fix the problems they are experiencing. But, so far the Republicans have shown no interest in working with their colleagues across the aisle. The Republicans have a 52 Senator majority. That means that with VP Pence casting a tie-breaking vote, they can only afford to lose 2 Senators.
Subscribe to our weekly email list to stay informed!


Finished the 2012 Buller Gorge Half Marathon Run in 02:25:42 in M5059 class!
He came 1000th in class and 1000th overall!
No photos of Eric Verstappen in 2012 , we may still be uploading/processing, check back later.


Steve Carter, Lead Teaching Pastor of Willow Creek Community Church in Illinois, has announced his resignation—effective immediately. Mentioning new allegations against the church's founder, Bill Hybels, Carter says his decision to leave is due to the way the current leadership of the church is handling the controversy.
"The new facts and allegations that came to light this morning are horrifying… These most recent revelations have also compelled me to make public my decision to leave, as much as it grieves me to go," Carter writes.
Carter made the announcement on his blog Sunday afternoon. While he made the decision to leave "many weeks ago" and communicated that decision to the elder board of the church, the board asked Carter to hold off on announcing his resignation to the congregation. The board wished to determine the best way to announce his leaving. However, new allegations (the details of which were published in an article in the New York Times Sunday morning) involving another former Willow Creek staff member hastened Carter's need to act.
Pat Baranowski's story is definitely hard to read through. She describes unwanted touching and groping by Hybels while she worked as his assistant for more than eight years, starting in 1985. Baranowski also lived with Hybels and his wife, Lynne, for two years. When Baranowski expressed guilt and concern over what she and Hybels had done together, Hybels put a plan in place for her transition off the church staff. Her story includes similar themes to Nancy Beach's and Vonda Dyer's stories, who both claim to have been the object of Hybel's inappropriate advances and then edged out of the church after a time.
When the allegations first came to the public's attention in April, the elder board sided with Hybels, echoing his statement that the allegations against him were untrue and that those bringing them were "colluding" against him. The leaders held this position until they started changing their story beginning in May, saying at that time that they no longer believed the women and other involved in reporting their stories had "colluded" against Hybels to force him into an early retirement and tarnish his legacy. In June, the elder board announced they spoke of the allegations and those who made them in the wrong manner and apologized. However, no statement has yet been made to indicate the board believes Hybels is, in fact, guilty of the inappropriate advances and harassment.
Apparently, though, Carter has been uneasy about the board's response from the beginning. "Since the first women came forward with their stories, I have been gravely concerned about our church's official response, and its ongoing approach to these painful issues," he writes. Carter believes there is a "fundamental difference in judgment between what I believe is necessary for Willow Creek to move in a positive direction" and what the board believes is necessary.
Previous articleIs it Wise to Hire Family Members?


text:
Antonía-Athiná Frey –en griego, Αντωνία-Αθηνά Φράι– (Hamburgo, RFA, 17 de enero de 1976) es una deportista griega que compitió en vela en la clase RS:X. Ganó una medalla de bronce en el Campeonato Europeo de RS:X de 2006.

Palmarés internacional

Referencias

Regatistas de Grecia
Regatistas en los Juegos Olímpicos de Atenas 2004
Regatistas en los Juegos Olímpicos de Pekín 2008
Deportistas de Grecia en los Juegos Olímpicos de Atenas 2004
Deportistas de Grecia en los Juegos Olímpicos de Pekín 2008
Regatistas de la clase RS:X

redpajama_set_name: RedPajamaWikipedia


text:
Signature Consultants Ranked 24th Largest IT Staffing Firm for Second Consecutive Year
In the News in fb tw em
Staffing Industry Analysts Releases List of 2012 Largest U.S. IT Staffing Firms
Signature Consultants, a leading IT staffing firm celebrating its 15th year in business, announced it has been ranked as the 24th largest IT staffing firm in the U.S. by Staffing Industry Analysts, the leading global advisor on contingent work.
"For the past 15 years, Signature Consultants has partnered with hundreds of clients to provide superior IT consultants," said Jay Cohen, Signature's president and CEO. "Our success and company growth have been a direct result of our number one guiding principle to always do the right thing and our preference in building fulfilling partnerships, not just filling positions, for our clients."
Staffing Industry Analysts ranked the companies by their estimated 2011 U.S. revenue generated in the IT staffing segment. The 33 largest firms on the 2012 list represent 63 percent market share.
"The majority of firms on the list achieved double-digit revenue growth in 2011, underscoring the immense demand for information technology staffing," said Research Analyst Timothy Landhuis.
"As technology increasingly undergirds more and more business processes, staffing agencies that can supply IT talent will continue to benefit from this trend," Landhuis said. "We project that the market for information technology staffing in the U.S. will grow by 14 percent in 2012 to reach $23.5 billion."
To make the list of largest IT staffing firms, companies had to have at least $100 million in U.S. staffing revenue in 2011.
Earlier this year Signature Consultants was voted the "#1 Best Staffing Firm to Temp/Contract For" in the United States by Staffing Industry Analysts. This is the second year in a row that the company has received the accolade. Employees were anonymously surveyed to measure their level of engagement within the company, which accounted for Signature's first-place ranking.
About Staffing Industry Analysts
Staffing Industry Analysts is the global advisor on contingent work. Known for its independent and objective insights, the company's proprietary research, award-winning content, data, support tools, publications and executive conferences provide a competitive edge to decision-makers who supply and buy temporary staffing. In addition to temporary staffing, Staffing Industry Analysts also covers related staffing sectors. Founded in 1989, acquired by Crain Communications Inc in 2008, the company is headquartered in Mountain View, California, with offices in London, England.
Download the PDF Here.
Signature Consultants In the News in fb tw em

redpajama_set_name: RedPajamaCommonCrawl


The thing I love about my MacBook is how sleek, elegant and beautiful it is. Every time I take it out from my backpack, I feel a flush of pleasure and pride. There is also a certain aura of authority and professionalism about it, especially if you are out there in Starbucks (or any coffee shop equivalent) for the public eye to judge you. Believe me - I had an old Lenovo before! It's a good computer, but when you put it on the small coffee shop table it feels like just drove an old Volvo into a Ferrari parking lot.
However, as every technology it comes with its own set of inconveniences and problems. One of those for me always was storage upgrade. I edit video (like a lot!) and I need a lot of storage. And yes, I know there are SD cards for that. It is true, but as I mentioned, what I love about my Mac is how elegant and sleek it is.
Having a little piece of plastic sticking it out from the side ruins the beautiful integrity. It is like a pimple on otherwise perfect face, if you pardon the comparison, not crucial, but not appealing as well. You might think I am being nitpicky here, and well yes I am. I am a Mac user and a designer – what do you expect?
Recently I found a perfect solution to this issue and it's named mDrive. So called "stealth card" (sounds like ninja weapon, no?) fits seamlessly to a specific MacBook model. mDrive is fully compatible with Micro SD/SDHC/SDXC, it's ultra lightweight and no software drivers or complicated hardware installation are required — simply stick it in your Mac and you have more space! This little fellow can be used as a regular hard disk drive or as a back-up device with the "Time Machine" application. Oh, and also, even with seamless fit to a Mac it doesn't require any additional tools for removal (which would be a hassle) – it has metallic ridge and can be removed only with your nails. How neat is that!
So, thanks to mDrive the integrity of my MacBook is in its perfection again. If you're anything like me and want to have the best you can and deserve – this is something to look into.


text:
Find sources: "France Inter" – news · newspapers · books · scholar · JSTOR (April 2009) (Learn how and when to remove this template message)
France Inter is a major French public radio channel and part of Radio France. It is a "generalist" station, aiming to provide a wide national audience with a full service of news and spoken-word programming, both serious and entertaining, liberally punctuated with an eclectic mix of music.
Vous êtes bien sur France Inter (You are well on France Inter)
1 January 1947 (1947-01-01)
Former callsigns
Club d'Essai (1947)
Paris-Inter (1947-1957)
France I (1957-1963)
RTF Inter (1963)
franceinter.fr
France Inter broadcasts on FM transmitters across France, and via the internet.
The radio channel France Inter announced during 2016 that the channel would discontinue transmitting on the 162 kHz frequency on the longwave on 1 January 2017, seeking cost savings of approximately €6 million per year. The transmission of the atomic clock generated time signal from Allouis will be continued after this date on the 162 kHz frequency as this time signal is critical for over 200,000 devices, which are deployed within French enterprises and state entities, like the French railways SNCF, the electricity distributor ENEDIS, airports, hospitals, municipalities, et cetera.[1]
France Inter was founded in the reorganization of state broadcasting which followed the end of World War II as "Paris-Inter" and charged with being French public radio's generalist (i.e. "full-service") service. The channel was renamed "France I" in 1958, although three years later one of France's most popular radio and television listings magazines was still showing the station's programmes under the heading "Paris-Inter" with "France I" as a subtitle.[2] In 1963 the France I and France II networks were merged to form "RTF Inter", renamed "France Inter" one month later.
The major challenge faced by France Inter at the time of its reorganization in the 1960s was the private "peripheral stations" (in particular, RTL and Europe 1, broadcasting from powerful transmitters outside France) success in capturing the majority of the French radio audience since the war. They had done so by adopting a modern broadcasting style and earning a reputation for greater freedom from government influence.
As well as rapidly modernizing its style to match its competitors, France Inter stressed its freedom from commercial pressures – although it does carry a limited amount of paid-for advertising – and especially presented itself as intelligent radio accessible to a general audience under the slogan Écoutez la différence ("Listen to the difference").
SlogansEdit
1975: Écoutez la différence (Listen to the difference)
1983: France Inter, pour ceux qui ont quelque chose entre les oreilles (France Inter, for those who have something between the ears)
1987: Plus haut la radio ! (Turn up the radio!)
1995: Écoutez, ça n'a rien à voir (Listen, it has nothing to do)
2001: Au début ça surprend. Après aussi (At first it is surprised. Later too)
2005: Qu'allez-vous découvrir aujourd'hui ? (What will you discover today?)
2008: France Inter, la différence (France Inter, the difference)
2012: La voix est libre (The voice is free)
2013: 50 ans qu'on ouvre la voix (50 years of opening the voice)
2014: InterVenez (InterCom)
2015: Vous êtes bien sur France Inter (You're well on France Inter)
ProgrammesEdit
France Inter programmes, a number of which have been important milestones in the history of French radio, include:
Le Masque et la Plume, arts reviews from journalist critics (on air since 13/11/1955)
Le Jeu des 1000 euros, a general-knowledge quiz programme (since 19/4/1958)
La marche de l'histoire, formerly Deux mille ans d'histoire, an in-depth daily documentary on a specific historical subject (since 1999)
Pop Club (1965-2005, i.e. 40 years with the same presenter José Artur)
Le téléphone sonne, a current affairs discussion and phone-in programme (since 1978)
Là-bas si j'y suis, a reports programme (1989-2014)
Le sept neuf, formerly Inter Matin, morning news sequence (since 1982)
classique avec Dessay, formerly Carrefour De Lodéon, classical music programme (since 1992)
^ ‹See Tfd›(in French) Le signal horaire restera sur le 162 kHz de France Inter
^ La Semaine Radio-Télé 29/41, 8–14 October 1961
France Inter transmitters map ‹See Tfd›(in French)
Official website ‹See Tfd›(in French)
Radio France Online Streaming ‹See Tfd›(in French)
Retrieved from "https://en.wikipedia.org/w/index.php?title=France_Inter&oldid=894462268"


As some of you know, Privilege Service (now part of Infrastructure Services) was designed as an Infrastructure-as-a-Service (IaaS) solution, and after it's relase and due to customer demand, we released a "customer-managed" version designed to be deployed and operated in an on-premises setting.
The goal of the lab is set-up the building-blocks to test the high availability capabilities of Privilege Service by leveraging Windows Server Failover Clustering (WSFC).
Administrative Failover (upgrades, patches, etc).
System, Network or Storage failure.
Disaster Recovery (backup/restore, orphaned or replicated database).
Active Directory - required for management and role computer objects, delegation and name resolution (DNS).
Redundant Shared Storage - WSFC orchestrates the storage switching between active and passive nodes.
Redundant Network Components - this is a requirement of any modern clustering technology (including WSFC) to provide the assurance of communications via multiple paths, backup network cards, etc.
At least 3 Windows Server (2012R2) with Windows Failover Clustering and connected to the Storage and Network layers for the CPS Server and at least 2 to act as Centrify Connectors. This will provide the assurance that even when the Windows servers are being updated (or Centrify's software) there is at least a 2-node cluster in service.
A properly set-up Windows Cluster, passes all the key validation tests during cluster setup.
Note: If this will be a test lab, and data availability is not important, you can use iSCSI virtual disks and virtual network components. Using such a configuration in a production environment does not constitute an effective availability control.
Have a planned hostname and URL for the service (e.g. safe.example.com).
A static IP address for the corresponding role in Windows Failover Clustering.
Obtain an x.509 SSL certificate from a public or enterprise CA with the planned name for the service.
(If planning to use Smart Cards, make sure the service name and the servicename +zso is added in the DNS name of the x.509 certificate. E.g. vault.centrify.vms, vaultszo.centrify.vms). Make sure you have the password if the cert is protected.
Privilege Service software - minimum version 17.7.161 with PostgreSQL back-end.
These are Windows Server 2012 R2 with Quad Cores and 16GB of RAM. Plan for 50-100GB for the data.
At least 2 Windows Servers to be used as Centrify Connectors (to maintain high-availability).
A Windows administrator (that can add/remove systems from the domain and local admin rights).
Understanding of TCP/IP and name resolution, plus the ability to create DNS records and verify name resolution.
Understanding of Active Directory administration and privileges to create computer objects.
Administrative rights and familiarity with Microsoft Management Consoles like the Failover Cluster Manager tool.
Understanding of network attached storage (e.g. iSCSI or FibreChannel) partitions, volumes, logical disks.
Understanding of Public Key Infrastructure concepts.
Is the cluster ready or does it need to be created, are the name and IP address secured?
What will be the name of the service? (e.g. vault.example.com).
Have you secured a static IP address for the service name?
What will be the shared logical disk path for the Centrify Privilege Service database? (e.g. E:\cps-db).
What will be the procedure to handle the configuration file?
This file contains configuration, encryption and recovery data (without it, no new nodes can be added, and recovery from backup or orphaned database is impossible).
Is there a test plan to verify the highly available or recovery test cases? (see subsequent post).
Are the ports required for CPS communication and Centrify Connectors clearly understood?
E.g. HTTP and HTTPS (TCP 80/443), PostgreSQL (TCP/UDP 5432), DirectConnect (TCP 30001).
First Node: Installation and Verification.
Configuring Windows Failover Clustering for Centrify Privilege Service.
These steps are performed to make sure the Windows Failover Clustering components are ready for Centrify Privilege Service.
Verify cluster Active Directory object (e.g. verify that the computer object in AD is alive an well).
Verify cluster DNS name resolution (e.g. ping the cluster name from all systems).
Verify CPS port communications (e.g verify the Windows firewall and check for CPS communications).
Verify Validation Report of the cluster (Right-click Cluster > View Validation Report).
is highly available, or consider adding additional networks to the cluster.
There you have it. I have only one NIC card, this is a single-point of failure. In a truly highly-available scenario, I'd probably have a pair of NIC cards dedicated to communication and/or storage and a heartbeat interface (just to talk to the cluster), this multiplies the risk per each node added.
Make sure you understand the disk layout (e.g. logical letter and database and backups location); e.g. E:\cps-db and E:\backups.
Verify that all cluster members can own the shared storage. (Pause each member and verify the storage switch).
Set all nodes but the first node to maintenance mode and secure the storage folder.
Log on to your First node as a Windows Administrator (e.g. node-1).
Navigate to Cluster > Nodes and right click node-2, select Pause > Drain Roles.
Repeat for all except for the current node (e.g. node-3, node-4 and so on).
Verify that the shared disk designated for the CPS database is mounted in the current system.
E.g. open Windows explorer and navigate to the designated drive (e.g. E:\).
Optional: If you haven't done so, create a folder for the CPS database (e.g. E:\cps-db).
Set name resolution of the service to the first node.
This step is required because during installation the setup the program will need to get a token from the web service, and it thas to be resolvable by name. This step will be undone before the additional nodes are added.
Open your DNS Management tool (e.g. DNS Manager).
Create a new CNAME record in your DNS zone. The record should be called as the service name (e.g. vault). The CNAME should point to your node-1 system.
Verify by pinging the service name (E.g. vault.centrify.vms) from all the systems.
Run the Privilege Service installation program, this will start the setup wizard.
Welcome Page > Press Next.
EULA Page > Check the box and press Next.
License Information > Type the case sensitive Company Name and Key, press Next.
Feature Selection page > Select "Clustered Primary" and press Next.
Destination Folder page > Select the preferred location (typically not the shared drive, this is NOT the database path).
Ready to Install page > Press Install.
Completed page> Press Finish (answer YES to the UAC prompt).
1. What username should the initial administrator account be created with?
use your valid business e-mail address.
Sample answer: Use a strong password or paraphrase, plan to change after setup.
the service because once installed, it will respond just to that URL over HTTPS.
auto-generated cert, may be OK for testing, use a real cert for production.
Sample Answer:  N (if it does not require it), answer Y and provide if needed.
on the cluster shared storage.
can't add any new nodes, restore from backup or orphaned database.
When CPS is ready, it's time to verify that it's set.
Browse to your service address (e.g. https://vault.example.com).
Note that if you did not get a good certificate (e.g. self-signed), your browser may require an exception.
Attempt to log in with the credentials from the previous step (e.g. admin@example.com).
Navigate between the user and admin portals.
Optionally, you can upload the configuration file (clconf.zip) as a secret.
In the admin portal, go to Infrastructure > Secrets.
Press Add File and Name it "CPS config file" and browse to the config file location.
Sign-in to your next node (e.g. node-2) with administrative credentials.
Run Privilege Service GUI Setup and this time, select Secondary Node and advance until PowerShell Configuration.
During PowerShell, you'll get asked the location of the clconfig.zip file created with the first node.
Repeat the process (1-3) in all the additional nodes.
When completed with all nodes, copy the configuration file to a secure thumb drive, make copies and distribute securely and delete the configuration file from the primary node.
Log in to your primary server (the one that is currently running).
Stop the Centrify Identity Service Database Service and the Web Server.
Open Windows Failover Cluster and resume the nodes in maintenance (note that if you choose to fail roles back, this may shift the disks).
Use your DNS Management tool to erase the CNAME record that was created in the previous section. The reason for this is that the new service (a WSFC Role) will have a DNS name controlled by Windows Clustering.
Make sure that the service's DNS record vault is not resolvable by any of the systems. If required, flush the DNS cache using the ipconfig /flushdns command.
Ping request could not find the host vault.  Please check the name and try again.
Verify that the Centrify Identity Platform Database is stopped in the last active node.
Open Failover Cluster Manager and connect to the cluster.
Verify that all cluster nodes where CPS is installed are in the cluster.
Right Click Roles and Select "Configure Role"; this starts the role configuration wizard.
Before you begin page > press Next.
Select Role page > select Generic Script.
Generic Script info page > Type or paste the path below, then press Next.
Client Access Point page > In Name, type the shortname for the service (e.g. vault) and the IP address for the service.
If you get an error stating that the network name is in use, this means that you have not cleaned-up DNS or the system cache.
Storage page > Check the box next-to the logical disk (this will be the same logical letter and drive that has the folder with the CPS database information).
We have been consistently using the E:\ drive in this example. Note that there are other ways to do this setup, one includes installing the cluster AFTER CPS installation, in that case, the "Cluster validation tests" can potentially change the logical lettering of the drive. Make sure this is kept consistent.
Summary page > press Finish.
Now it's time to inspect the role, and verify what is the active node. Click on Roles and refresh. You should see the service name and the current owner node.
Use your browser to connect to the service again. When you sign-in, go to the upper-right menu under the user name.
On Windows failover cluster, connect to the cluster.
Right-click the newly-created role (with the service name, e.g. vault) and elect Move > Best Available Node.
This step will move the cluster ownership to another node. Note the name.
Once you have confirmed the transfer was succesful, refresh your browser and confirm the ownership. Keep testing until all nodes are validated.
At this point, after verifying administrative node change to all systems, we can add our connectors.
Log on to a Windows 2012R2 (and up) 64 bit system.
Go to Settings > Network and press Add Centrify Connector.
Once your connectors are installed, you can authenticate AD users and perform CPS services like Discovery or Import.
Configure SMTP, Google Maps API or Twilio Service settings.
Configure Policies for User Access.
Configure System and Account Security Settings.
Create Roles and assign administrative roles (e.g. Privilege Service User, Privilege Service Administrator).
Configure and run a Network Discovery.
Import systems, accounts via CSV import.
Privilege Service On Premises - High Availablility - Where to next?


The Cherry Ave. branch of Fast Money Loan provides the residents of Long Beach CA with car title loans. The office is located at 3451 Cherry Avenue. If you are a resident of Long Beach and need quick cash, please stop by our Long Beach branch location of Fast Money Loan to talk with us. We take great pride in serving our customers and look forward to talking with you.
Long Beach, CA is one of the most popular and beautiful areas in California, with miles of beach, sunshine, and lots of space to drive around. In this bustling area, you don't want to be without your transportation, especially if you run into trouble making ends meet.
Long Beach CA is growing and attracts more and more newcomers. Not just from the US, but from the whole world. If you are looking for a new place to call home, then Long Beach CA might be that place. It has great infrastructure, pleasant climate, and attractive cultural life. That is why Fast Money Loan has its main location in Long Beach where you can come in and talk to one of our friendly staff to help take care of your money needs. With so much beauty around, we want to help you get the auto loan you need so you can make the most of your life in California.
Fast Money Loan is here to help get you back on track!
Having money troubles? Your vehicle can be the key to solving your problems!
With a fast and easy car title loan from Fast Money Loan, we can help you get the money you need today to cover your bills or help in a financial emergency.
Visit our Fast Money Loan Loan Processing Headquarters on Cherry Ave, in Long Beach, CA.
There are sometimes when the costs of life seem overwhelming and you need a quick source of money to help handle things. When that happens, Fast Money Loan can help you turn your car, truck, van, or boat into the money you need while you still keep the vehicle.
Your credit score doesn't matter – we don't even check with the credit bureaus. As long as you can repay your loan, we usually can do a car title loan for up to 100% of the Manheim value of your vehicle.
The loan is approved based on the specific details of the car that you own and your ability to repay the loan. Therefore, your ability to pay the loan, the value of your car, and the condition of your car are the most important things we will consider. You can apply for the loan online in a few easy steps, or come by our Long Beach location. Bring your auto car title into our office and on approval you can still ride away with your vehicle along with the cash you need to pay your bills. Go ahead and give us a call or Apply Online, You will be very glad that you took this step. We will work with you to make it as hassle-free as possible.
A great way to get quick cash that you may need to get through the month, pay rent, catch up on past due bills or take care of other important expenses is to get a quick loan. The low-interest car title loan that Fast Money Loan offers has one significant advantage – you get to keep and drive your car and use the cash you got for its title, all at the same time. It is such a convenience for someone who depends on their car for the everyday errands or work.
The most popular way to apply for an auto title loan is online. Here, at Fast Money Loan, we also offer an alternative to our customers – they can stop at our location in Long Beach and get the necessary fund as fast and hassle-free as online.
On occasion, money troubles can happen to anyone. While we always would like to be prepared for the worst hardships, life can always throw us curveballs. Hardships can suddenly appear in any shape or form and sometimes it is just not possible to get by without some external help. We, therefore, think that the only way to help people is to provide the best car title loans and alleviate their need.
At Fast Money Loan, based on your ability to repay, we are able to use the value of your vehicle to help you get the money fast you need. We hold your lien-free, clear title for the duration of the loan as security. This helps you get quick cash in hand and hence, making it easier to get a loan regardless of your credit score. Once the loan is paid off, you get your title back. It's really very simple.


text:
TVBEverywhere
Subscribe to TVBEurope
About TVBEurope
Home › Production & Post
NDS's new MTV hybrid breaks cover
NDS demonstrated the world's first hybrid rights and business management platform for mobile TV at the World Mobile Congress. The hybrid platform, which combines content protection and business management for both the DVB-H IPDC and OMA BCAST broadcast profiles in a single system, was shown in conjunction with technologies from NDS partners Thomson, Gemalto and LG, writes Andy Stout.
test ⋅ Feb 18, 2008
NDS demonstrated the world's first hybrid rights and business management platform for mobile TV at the World Mobile Congress. The hybrid platform, which combines content protection and business management for both the DVB-H IPDC and OMA BCAST broadcast profiles in a single system, was shown in conjunction with technologies from NDS partners Thomson, Gemalto and LG, writes Andy Stout.
The MWC demonstration features a single NDS mobile TV platform, providing encrypted streams conforming to both the DVB-H Open Security Framework and the OMA BCAST Smart Card Profile. The NDS system is integrated with the Thomson SmartVision service platform and Electronic Service Guide and is rendered on separate OSF and SCP handsets from LG using SIM cards provided by Gemalto. All four companies contributed significantly to the success of the joint project.
At the heart of the new platform is NDS' Unified Headend (UHE) architecture, which provides a single point of business management for all video broadcasts, irrespective of the delivery networks or the rendering devices. The UHE convergence platform enables operators and broadcasters to centralise the administration of subscriptions, purchasing, transactions, promotions and a wide range of other business functions.
As a result, NDS and Thomson now offer a fully integrated end-to-end DVB-H system for OMA BCAST SCP and OSF. This platform – combining content security from NDS with Thomson DVB-H headend and ESG (Electronic Service Guide) server technology – is now fully available as an off-the-shelf solution.
"The new platform will enable an operator to support multiple mobile broadcast profiles and multiple handsets simultaneously and cost-effectively," said Nigel Smith, NDS chief marketing officer. "It makes possible the coexistence of connected and unconnected devices and opens up a market which has been compartmentalised into standards until now. Both the savings for the operator and the new revenue opportunities are likely to be huge."
"Gemalto has successfully developed a complete range of mobile TV products which are used in this end-to-end demonstration: a SIM card that supports both DVB-H Open Security Framework and OMA BCAST Smart Card Profile, and a micro SD card for unconnected devices," said Gabrielle Bugat, senior vice president, Mobile TV Business Group at Gemalto. She continued: "These products will help our customers to simplify their SIM logistics by supporting two standards on a single card and open the door to additional revenue streams with mobile TV unconnected devices."
The company has also announced the integration of NDS VideoGuard Mobile CA technology with Expway's FastESG Electronic Service Guide (ESG).
The integration supports both the Open Security Framework and the OMA BCAST Smart Card Profile, with additional profiles on the roadmap, and provides the necessary security for monetising mobile TV content while ensuring that users enjoy a personalised mobile TV experience.
"The integration of the NDS CA system with the Expway ESG means that European operators need not be concerned about launching with a specific profile. NDS' ability to support multiple standards concurrently means that our mutual customers are able to migrate to a new standard seamlessly, with no service disruption to customers, while maintaining key vendor relationships," said Nicolas Choquart, NDS director of mobile and new business, EMEA.
www.nds.com
For more stories like this, and to keep up to date with all our market leading news, features and analysis, sign up to our newsletter here.
ATM System's new HD truck sports L-S-B control
Portugal's TVI relies on Snell's ICE for two new channels
Quantel and Snell get together with a new name, new brand and new vision
Finland's MTV chooses Orad brand
BBC S&PP creating new production facilities for Endemol
Telestream's new Vantage point for system management
Anton/Bauer's new motion-sensing battery
Philophon's new Dolby ATMOS facility built by Munro
© 2020 Future Publishing Limited, Quay House, The Ambury, Bath BA1 1UA. All rights reserved. England and Wales company
registration number 2008885.


and even the records from your friends or your opponents.
The program is ideal for sports people, athletes, runners, swimmers, cyclists.
Also perfect for teachers and professors or for athletic trainers.
1. You enter a person's name (yourself, a friend, a friend, a classmate, a student, a world champion, ...).
2. You enter a short acronym that indicates the person's membership group (class, school, Sports Club, friendships, team, Sports Group).
The group code is used for groupings in the report records.
The acronym is converted to uppercase automatically.
You can see the abbreviations in place to type on the keyboard as much as possible.
The acronym will be used for groupings in the report of the records and it is converted to uppercase automatically.
The person's name and sport is editable with the dedicated button.
The code can be changed without problems.
Whereas if you change the name of the person or of the sport, the inserted records for that name will be deleted.
After you create a person and a sport you can add a record.
- Display of all sports and the best result.


Don't it ever get lonesome?
Don't it ever get sad when you go out on the road?
Don't it ever get sad when you go out on a thirty day tour?
Don't it ever give a young man the blues?
Don't it ever make a young man wanna go back home?
What the Road Ladies do to you?
Now . . . now . . . now . . .
Tell me you love me, tell me you love me!
The night you went to the show?
Where did your brassiere go?
The Monster From The U.S.O.
Lift up your dress, if the answer is "no" . . .
Frank Zappa—drum set, wood blocks, temple blocks, boo-bams, tom-toms, etc.
Any you boys not paid up on your cards? Huh?
Wanna get a good look?
Let me pull it right out!
Let me whip it right out!
Any you boys not paid up on your cards?
Send my baby home to me!


text:
OPINION >> Editorial
Western media report on Xinjiang lacks morality
History will evaluate the governance in Xinjiang, and the standard will never be determined by Western public opinion.
Source: Global Times | 2019/11/17 22:28:39
Pompeo once again discredits himself by smearing China
As America's top diplomat, Pompeo probably believes that lies told a thousand times become truth, so he spares no effort to repeat the platitudes against China and take the lies seriously. However, he is destined to go against the wind and sail, which is unbearable.
US politicians ruthless in pushing HK bill
Washington indulges in the mobs' vandalism to destroy Hong Kong and views efforts by the Hong Kong Special Administrative Region (HKSAR) government as suppression on democracy and freedom. How ruthless and hypocritical it is.
1.4 billion people firmly stand with patriotic Hongkongers
The 1.4 billon mainlanders and a powerful country are standing behind the Hongkongers while the mobs are in their final struggle. There is no reason to be afraid of them.
BRICS ties vital in uncertain world
It is hoped that BRICS will open up more areas of cooperation and form more long-term cooperation mechanisms. BRICS countries should have more say in the future world. Will the world become fairer and more balanced? The group's performance is crucial.
To weaken police power is to help rioters
All those who hope for a good condition for Hong Kong should support and cherish the police's endeavor. Any act which makes it difficult for the police and weakens police power adds fuel to the fire.
Biased Western media are fueling HK riots
Western reports on Hong Kong affairs have made the Chinese people see clearly how these media outlets put their standpoint ahead of objectivity without regard for professional ethics.
HK rioters' acts similar to terrorists'
Those who are coerced to be "valiant" should walk away as soon as possible when you still can make the call. Those who stubbornly engage in evil acts will eventually be punished by law and morality.
US cannot break China's supply chain
China is not afraid of any game against the supply chain. Producers without China's supply chain will certainly feel more pain than China.
Pompeo's attacks on CPC show his hysteria
The CPC has led China to prosperity. China has not engaged in a war in 30 years and has insisted on a national defense policy defensive in nature. It is the biggest trading partner of over 100 countries and regions. Is it possible to turn such a country and its ruling party to a "common enemy" of the world? American elites such as Pompeo have become hysterical due to their ideological paranoia, that's why they keep targeting China in a rude and stupid manner.
Source: Global Times | 2019/11/8 22:48:40
India should look to China for its pollution solution
Both China and India are victims of air pollution and two countries should join hands to create clear blue skies. What China has achieved offers valuable experience to India, while China could also learn from India's plight. China and India mirror one another, and there is plenty of room for cooperation.
Source: Global Times | 2019/11/8 3:48:06
Mauling mainlanders shows HK's decline
Universities in Hong Kong have been in a mess, but Hong Kong society is unable to stop the chaos. Hong Kong is in decline. Mainland society has done its best under the "one country, two systems" principle. If the chaotic situation continues, universities in Hong Kong will no longer be suitable for mainland students. It's believed that starting next year, mainland students would think twice before applying.
US adopts blinkered view of TikTok
We hope the US won't go to extremes. Being open is where US interests lie. Even if they have worries about TikTok, they must exercise restraint. Many people are worried that the US might monitor them through various means every day, but they are restrained and rational. The US has no reason not to do likewise.
Rioters must pay price for violent acts
Under continuing violence, Ho is a hero. But he is not alone. A powerful motherland is standing behind him. All rioters must pay the price for their violent acts.
CIIE presence shows Macron's acute vision
The US-launched trade war has spoiled the global economic environment, but offers opportunities for European countries to participate more actively in China's economy. Macron showing up at CIIE at the right time reflects a keen sense of him and France.
US risks being loner in Indo-Pacific Strategy
This is an insult to the wisdom and dignity of Asian countries. In this way, the Indo-Pacific Strategy will be eventually recorded as a shameful story in the history of international relations.
Import Expo to change West's opinion
We should keep holding the CIIE. By doing so, we will see subtle changes in international public opinion and the attitudes of some countries toward China. These changes will be positive and constructive, which we must have confidence in.
Focus on trade wisest choice for ASEAN
China has understood and catered to such interests of ASEAN. China has to a maximum extent coordinated with ASEAN in various fields and let ASEAN play the role of convener in these exchanges.
First << Previous 1 2 3 4 5 6 7 8 9 Next >> Last


strategy, statesmanship and lack of ego. Williams has provided numerous instances wherein he provides ample support to his arguments and facts. In spite of the unnecessary detail and verbosity that Williams sometimes uses, there is no doubt that this book is a remarkable insight into Lincoln's persona. Bibliography Works Cited Dawson, Joseph G. III. "T. Harry Williams." Dictionary of American Biography. Vol. 17, 431-446 Williams, T. Harry. Lincoln and His Generals. New York: Dorset Press, 1952.


text:
Call to chat to our team Find your
With a wealth of industry knowledge developed over many decades, The Apex Group is a solid, reputable and reliable company. We are proud to declare ourselves as the first choice when selecting a steel partner for your organisation.
Forms and Terms & Conditions
Australian Manufacturer & Distributor of Steel Building Products, Steel Long Products, Steel Sheet & Coil. Supplier of COLORBOND® Steel and BlueScope Steel products. Apex prides itself on the quality of our products and services.
Long Products
Bluescope Colours
Apex Group
The Apex Group is one of Australia's largest privately owned steel manufacturers and distributors. We have had significant success since beginning operations in 1994, achieving growth through expanding product ranges and exceptional service quality. We pride ourselves in delivering a personal approach to service for our valued customers.
The Apex Group represents the union of some of Australia's major steel companies, offering a wider variety of products and services to both the public and trade. By combining resources, we have built on our reputation as market leaders in the manufacturing and distribution of steel and steel building products.
Not your average steel company
The Apex team is enriched with a wealth of industry knowledge developed over many decades, making the Apex Group a solid, reputable and reliable steel company. We are proud to declare ourselves the first choice for your organisation's steel needs.
The Apex Group has been the first choice by Australian organisations for over 20 years. We are your trusted steel partner. Our industry knowledge & experience allows us to continue to deliver the highest quality products & services to our customers. We offer custom cut lengths, special requirements and a tailored approach to each and every one of our customers.
Our knowledgeable and dedicated team pride themselves on delivering the best possible customer service. Our success continues to be based on our mutual respect, pride, and emphasis on quality. The Apex Group is committed to safety. Our ongoing investment in training, education & safety procedures for all staff ensures the safety and wellbeing of our entire team, customers, suppliers and extended network.
The Apex Group continues to grow our national network. Expanding into new regional areas such as Geelong & Darwin has further strengthened our delivery network and our ability to service our customers needs. We pride ourselves on delivering the same level of customer service and care across Australia.
The Apex Group is committed to remain the first choice in the industry by investing in our staff and our technology. Our continued investment allows us to further improve our internal procedures and deliver a seamless process for our customers. Advancements in technology and investments in our infrastructure have helped our production and manufacturing facilities become more efficient, and allowed us to continue to deliver a tailored approach to all customers.
Copyright © The Apex Group 2022 Privacy PolicyTerms & Conditions
Website design by Thirst Creative
Find your nearest location
Please select your state:


Can the doctor tell what stage of Peyronie's im in?
Question...i've Always had a congenital curve but say I do have peyronies along with congenital... and because of the congenital curve.. i never noticed the Peyonies. If I were to get an examiniation or ultrasound, would the urologist know what stage Im in? I know I wanna get a surgery this year but if im in the active phase... I know it'll be delayed. Though, how would they be able to know Peyronies stage im in?
Re: Can the doctor tell what stage of Peyronie's im in?
They are professional urologist, they deal with Peyronies day in day out. Most of them can tell you what you have over the phone to be honest. They will know what stage it's at or if you even have it in the first place.
P.s I can't believe you actually made a next post. This is like your 12 post or something. Stop that, you're spamming the form with half answered threads. Keeping all your post on one or two threads would be better. Why? Forum members would be better able to see everything you are saying so they will be more informed if they decide to reply, also new members. New members can read all your posts in one go and get a good understanding of what's going on.
Still in pre-diagnosis stage. Trying to get a head start. Atypical Peyronie's?
In Phase One of Peyronie's, Anything I can do in this stage?