Fast Stencil-Code Computation on a Wafer-Scale Processor

The performance of CPU-based and GPU-based systems is often low for PDE codes, where large, sparse, and often structured systems of linear equations must be solved. Iterative solvers are limited by data movement, both between caches and memory and between nodes. Here we describe the solution of such systems of equations on the Cerebras Systems CS-1, a wafer-scale processor that has the memory bandwidth and communication latency to perform well. We achieve 0.86 PFLOPS on a single wafer-scale system for the solution by BiCGStab of a linear system arising from a 7-point finite difference stencil on a 600 X 595 X 1536 mesh, achieving about one third of the machine's peak performance. We explain the system, its architecture and programming, and its performance on this problem and related problems. We discuss issues of memory capacity and floating point precision. We outline plans to extend this work towards full applications.

Abstract-The performance of CPU-based and GPUbased systems is often low for PDE codes, where large, sparse, and often structured systems of linear equations must be solved. Iterative solvers are limited by data movement, both between caches and memory and between nodes. Here we describe the solution of such systems of equations on the Cerebras Systems CS-1, a wafer-scale processor that has the memory bandwidth and communication latency to perform well. We achieve 0.86 PFLOPS on a single wafer-scale system for the solution by BiCGStab of a linear system arising from a 7-point finite difference stencil on a 600 × 595 × 1536 mesh, achieving about one third of the machine's peak performance. We explain the system, its architecture and programming, and its performance on this problem and related problems. We discuss issues of memory capacity and floating point precision. We outline plans to extend this work towards full applications.
Index Terms-Algorithms for numerical methods and algebraic systems, Computational fluid dynamics and mechanics, Multi-processor architecture and microarchitecture The need for high memory bandwidth is captured by a problem's arithmetic intensity, the number of operations performed on each datum loaded from memory. Solvers of partial differential equations by finite difference, element, or volume methods have low intensity, as they make repeated sweeps over meshes. Performance for them on CPU or GPU based systems suffers due to insufficient bandwidths. For example, on the highperformance conjugate gradient (HPCG) benchmark, the top 20 performing supercomputers achieve only 0.5% -3.1% of their peak floating point performance [1]. For HPCG as for many HPC kernels and real applications, limited memory bandwidth and high communication latency are primary performance limiters.
HPC memory and communication systems struggle to keep up with processing performance. In 2016 the flops to words ratios for both memory and interconnect bandwidth were in the hundreds, and the flops needed to cover the memory or network latencies were in the 10,000 to 100,000 range, with the trend going higher; see Figure 1. The recent introduction of wafer-scale processors promises to change this situation. The Cerebras Systems CS-1, a system whose compute and memory resources are all fabricated in a single 462 cm 2 silicon wafer, can move three bytes to and from memory for every flop. This is achieved in a highly parallel, distributed memory architecture in which all memory is on the same silicon wafer as the processing, providing orders of magnitude more memory bandwidth, single cycle memory latency, and lower energy cost for memory access. There are 18 GB of on-wafer memory. The memory on the wafer is fast, static random-access memory (SRAM). Processing elements are interconnected by an on-the-wafer network with injection bandwidth one fourth of the peak floating point compute bandwidth and with nanosecond per hop message latencies. Thus, on the CS-1, memory bandwidth matches the peak compute rate, and communication bandwidth is only slightly lower. The CS-1, as shown in Figure 1, sits at the desirable bottom on the flops per access scale, a region that becomes ever farther away as single chip systems are pushed towards greater on chip capability with the hard limits of off-chip communication.
Here we demonstrate the potential for wafer-scale systems to achieve breakthrough performance on regular mesh finite difference (stencil) problems that can fit in the on-wafer memory, by giving some examples of the implementation and performance of model problems on the CS-1. We have implemented a BiCGStab solver for a linear system arising from the 7-point discretization of a PDE on a 3D mesh. The solver achieves performance of 0.86 PFLOPS in mixed precision floating point that uses 16-bit for all arithmetic except the inner products and a mixed precision inner product with 16-bit multiply and 32-bit add. The achieved performance per Watt (at 20 kW) and for the size of the machine (1/3 rack) are beyond what has been reported for conventional machines on comparable problems. We describe the system, the processing core architecture, the programming model, and our implementation of BiCGStab in that model. We discuss the implementation of the global reductions required for, among other things, the inner products common to all Krylov subspace linear solvers that are often a bottleneck in large-scale messagepassing implementations. Because on-wafer communication has low latency, our AllReduce (to use the MPI term) for scalars takes under 1.5 microseconds for a system of about 380,000 independent, interconnected processors. We then present and validate a simple performance model, and use it to predict the effect of changing mesh size and shape and of an implementation for a problem arising from a large two-dimensional mesh.
Then we talk about the limitations of this experiment and potential extensions to more complex and realistic applications. We sketch a proposed implementation of the NETL 1 code, MFIX, for modeling combustion or chemical reactions of solid particles (e.g., fuels, sorbents) transported in a fluid. The performance projections for 1 National Energy Technology Laboratory MFIX indicate that real-time, highly resolved simulation will be possible, opening many opportunities for practical applications that demand both high fidelity and speed.
We then consider issues of floating point arithmetic precision and show the accuracy achieved with mixed (16/32) and higher precision for this problem.
Finally, we discuss the limitation of the single-wafer solution (a limited amount of memory), pointing out situations where the speed is critical and the size is adequate, and indicating the evolution of the technology towards greater capacity.

II. THE CS-1 WAFER SCALE ENGINE
Cerebras Systems, formed in 2016, has designed and brought to market the industry's first wafer-scale system [3]. All compute and memory resources of the CS-1 are contained on the Wafer-Scale Engine, a 462 cm 2 silicon wafer. The wafer also contains a powerful communication network; see Figure 2. The system comprises 380,000 processor cores, each with 48 KB of dedicated SRAM memory (for a total of 18GB); up to eight 16-bit floating point operations per cycle; 16 bytes of read and 8 bytes of write bandwidth to the memory per cycle; a 2D mesh interconnection fabric with 16 bytes of injection bandwidth per core per cycle; and a total system power of 20 kW.

A. Architecture
The CS-1 wafer is an MIMD, distributed-memory machine with a 2D-mesh interconnection fabric. The repeated element of the architecture is called a tile. The tile contains one processor core, its memory, and the router that it connects to. The routers link to the routers of the four neighboring tiles. Figure 2 illustrates the layout.
The wafer contains a 7×12 array of 84 identical "die." A die holds thousands of tiles. Ordinary chips are made by cutting the wafer into individual die; in the WSE, the die are instead connected by extending the interconnect across the "scribe lines", the spaces between die.
The memory, functional units, and instruction set are designed for high throughput numerical computation. The roughly 380,000 tiles each have their own fast SRAM memory. There is no shared memory. Local memory is 48 KB, which totals 18 GB across the wafer. The load-to-use latency is one cycle.
The instruction set supports operations on 16-bit integer, 16-bit (IEEE fp16), and 32-bit (IEEE fp32) floating point types. Floating point adds, multiplies, and fused multiply-accumulate (or FMAC, with no rounding of the product prior to the add) can occur in a 4-way SIMD manner for 16-bit operands. The instruction set supports SIMD operations across subtensors of four dimensional tensors, making use of tensor address generation hardware to efficiently access tensor data in memory. These play the role of nested loops and eliminate any loop overhead. There are enough memory banks to provide the bandwidth needed to fetch eight 16-bit words from memory and store four such words per cycle, enough to support SIMD-4, AXPY operations y = y + a × x, where the operand a is a scalar held in a register and x and y are tensors that stream to and from memory. Such an operation can be launched with a single instruction. The tensor operands can have more than four elements, so the instruction executes for multiple cycles.
In mixed precision with multiplications in fp16 and additions performed in fp32, the throughput is two FMACs per core per cycle. Purely 32-bit floating point computations run one FMAC per core per cycle. The theoretical peak performance of the system varies depending on the number of cores configured on the wafer, clock rate and power settings.
The core connects to a local router that has five bidirectional links, one to each of its four nearest neighbors and one to its own core. The router can move data into and out of these five links, in parallel, on every cycle. Even with scalar granularity, communication is efficient. The router has hardware queues for its connection to the core and for each of a set of virtual channels, avoiding deadlock. Communication between potentially distant processors occurs along predetermined routes. Routing is configured offline, as part of compilation; data travel along virtual channels that can be programmatically reconfigured at run time. The fanout of data to multiple destinations is done through the routing; the router can forward an input word to any subset of its five output ports. There is no runtime software involved with communication. Arriving data are deposited by the hardware directly into memory or registers or routed to functional units as specified by the program.
An instruction with tensor operands can run synchronously or, at the discretion of the programmer, as a background thread that shares the datapath with other threads including the main one. A background thread runs a single tensor operation, as a single asynchronously running instruction. There is no context switch overhead. The registers and memory used by an asynchronous thread are those assigned by the programmer or compiler in the instruction, and these may not be overwritten until the thread terminates. Subsequent computation can be delayed until the thread terminates. The core supports nine concurrent threads of execution.
A stream of data to or from the fabric may be used as an input to a tensor operation, or as the destination for one. The hardware directly implements scheduling activities that would normally be performed by an operating system. This allows compact and efficient software implementations. For example, one core can be sending data from its local memory to another core; simultaneously it can receive data from another core while adding it to values stored in its local memory. All of this is accomplished using only two machine instructions that run as independent threads.
Code consists of tasks that react to events. Tasks are triggered by other tasks, or by arriving data words. The channel of the arriving word determines the code that is triggered. There is little delay between the completion of a task and the start of a subsequent task, as this is handled in hardware. Together with the SIMD operations of the instruction set, this efficient scalar-data-triggered method of operation allows us to implement distributed linear algebra kernels, such as those used in machine learning and the iterative linear solver discussed here, with minimal performance impact from message latency and bandwidth. Special purpose Data Structure Registers (DSRs) generate tensor access addresses in hardware eliminating overheads of nested loops.

III. THE BICGSTAB METHOD
Discretized partial differential equations lead to systems of linear equations that are commonly solved using Krylov subspace iterative methods such as the conjugate gradient (CG) method. The Biconjugate Gradient Method [4] extends CG to nonsymmetric systems. The stabilized version of the method, BiCGStab [5] (Algorithm 1), makes it numerically stable (and uses four dot products per iteration instead of two). The kernel operations in the algorithm are sparse matrix -dense vector multiply (SpMV), AXPY (y = y + ax for vectors x and y and scalar a), and inner product. In SpMV, each matrix element is involved in only one multiply-add operation; thus the arithmetic intensity is low. 9: 12: end for 14: end function

IV. MAPPING BICGSTAB TO THE CS-1
The CS-1 architecture provides a good match to the compute, memory, and communication needs of Krylov subspace methods, especially on a regular mesh. We map our test case's regular 3D mesh to the 2D machine in a straightforward domain decomposition manner. Let the mesh be X × Y × Z. Then map X and Y across the two axes of the fabric, with each core handling all of the Z dimension ( Figure 3). The mesh mapping dictates the mapping of the vectors in the BiCGstab method as each vector element is associated with one meshpoint. As to the matrix A, we map the needed portion of its nonzero diagonals to each core. A has seven nonzero diagonals; but with diagonal preconditioning the main diagonal is all ones. Therefore, we only store six other diagonals. In addition to the elements of A, local portions of four vectors must be stored to implement the BiCGstab iteration. The vectors q i and r i+1 reuse the storage of s i and y i , leading to a storage requirement per core of 10Z words. Thus, with Z = 1536 we are using about 31KB out of 48KB for the matrix and vector data.
1) SpMV (3D): The SpMV implementation uses architectural concepts that may be unfamiliar. To convey how it works we provide a pseudocode listing (Linsting 1) and a coordinated diagram showing the data flow ( Figure 4). The code allocates memory for vectors, specifies the communication connectivity between tiles, defines tensor descriptors, and specifies tasks and threads that constitute a specification of data flow that is illustrated in the figure.
With the iterate vector v(x, y, z) mapped as shown in Figure 3, each core stores a local iterate vector in an array of v of length Z. The local result of the SpMV is an array u of length Z computed as the sum of seven vectors. Six of these are elementwise products of a vector of matrix elements and a vector of iterate elements. Four of the iterate vectors stream in from the neighboring cores. Two are shifted-by-one (in the Z direction) copies of the local iterate. Thus, to perform an SpMV we use eight memory vectors. The eight memory vectors, shown in gray, are declared with the float16 keyword in the code. There are the local iterate (v), the result vector (u), and six vectors of matrix elements, denoted by coordinate directions (x, y, z) and p or m to indicate plus or minus, according to their role in the 7-point stencil. In order to make an asynchronous term [1] term [2] term [3] term [4]  connection from multiplication threads to addition threads, the code allocates five in-memory FIFOs to forward the elementwise products of streaming vectors. The instruction set supports hardware-managed, in-memory FIFOs that use memory regions as circular buffers. The core has special hardware registers to manage the state (head and tail location, for example) of each FIFO. These are the boxes labeled term[0] to term [4] in the figure and the objects of fifo type in the listing. Each core exchanges its local iterate vector v with its four neighbors. The broadcast is performed using a single communication channel that fans out to its four neighbors. But the core receives vector segments on four distinct channels, one corresponding to each of its four neighbors, each consumed by a different background thread handling the four incoming streams.
We allocate channel numbers to make all five of these channels different at every tile. The assignment we used is shown with color as a representative of channel in Figure 5. Note that at every tile, the outgoing color (in four directions) differs from each of the four incoming colors.
Tensor descriptors are used in code to specify the geometry of data and the stepping though it during execution of vector instructions. In Figure 4 these de-scriptors are annotations, such as xp _ rx (meaning a descriptor of a vector received from the neighbor in the +x direction) next to the arrows between blocks. Some, like xp _ rx, are fabric tensor descriptors that indicate the channel used to communicate this tensor as well as a vector length or tensor shape; they are declared with the fabric keyword in the code. A memory tensor descriptor such as (xp _ a in the figure and declared with the tensor keyword) can point to a memory address and indicate a length of Z and unit stride for tensors that reside in memory.
So how does it work? There is one outgoing stream (the arrow coming out of v, into the router, the dashed box above), and five incoming streams: xm _ rx, xp _ rx, ym _ rx, yp _ rx, and c _ rx, which is the v data looped back. (We loop back the outgoing local data and route it in for processing the z dimension, as this saves memory bandwidth, saving time.) The assignment statement c _ tx[] = v1[] launches a thread because in the declaration of the fabric tensor c _ tx, there is a thread resource assigned (.thr = 5).
The job is started by a task, called spmv, that first sends the local iterate to the fabric with the assignment mentioned above. It then initializes the result as the product of the zm _ a memory vector and a shift-by-one, v0, of the in-memory local iterate. This is done by a tensor multiply instruction (denoted spmvtask in the white box on top) specified by the statement zm _ acc[] = v0[] * zm _ a[]. Since none of the tensor descriptors has a thread assignment, this runs to completion before spmv proceeds.
The task, spmv, then launches the six background threads (in white boxes, denoted thread _ 0 through thread _ 5) that do elementwise vector multiplication as single SIMD instructions. These threads pull inputs from the fabric; two of them, 4 and 5, pull the loopedback local data. They send results to FIFOs (except for thread 5).
After the main task has launched these background threads, it may safely terminate. The threads it launched continue to multiply and push elements of product vectors into FIFOs.
Thread 5 simply adds its input to the result vector. It doesn't need to multiply, because it is working on the main diagonal of the matrix, which is all ones.
The Cerebras hardware FIFOs have a distinctive feature. They are able to activate tasks, in this case sumtask, whenever they aren't empty. This they do. The hardware task scheduler runs sumtask (while the threads continue to fill the FIFOs). It executes five vector add instructions (as shown in the white box) that each has one FIFO input. Each add pulls as much data as it can from its input FIFO, finishing when empty. Their destination tensor descriptors track their progress, so that they add once to each of the result tensor. So sumtask is invoked repeatedly, pulling from the FIFOs, allowing the upstream threads to continue to push data into them.
The six add operations that increment elements of the result run in a temporally interleaved way. There is no race condition. The additions occur in a nondeterministic order. But there is no danger of a data race, and no locks need be acquired, as the hardware handles the interleaving, working on only one thread at a time.
In order to return control to the invoking task (called bicg here in the code), we need to ascertain that all five adds are done. This is done with a small tree of two-way barriers implemented by unblock and activate actions.
For the 2D problem we map a rectangular region of the mesh of v to each core, and store all elements of the corresponding columns of A. After multiplication of the local v with the local A we have generated products in an output halo that must be sent to neighboring tiles. We accomplish this output-halo exchange with sends of fabric tensors in threads that arrive and feed data into addition threads. We complete a round of send and add in one direction, then a round for the other direction, and Fig. 5. Tessellation routing pattern for SpMV: a single core pushes its content into adjacent cores' fabric router using a single communication channel. Messages from the four neighbors arrive on four distinct channels and are processed by corresponding threads. This allows us to achieve high fabric utilization due to the fact that we can send the data in 4 directions in a single cycle. WSE allows the fabric to be dynamically reconfigured. Such adaptive topology plays a significant role in offloading routing logic from cores, which can be used primarily for computation.
in this way avoid communication along diagonals of the tile grid.
The efficiency of this approach is approximately the same as for the 3D mapping discussed above. On one hand, because all 9 multiplies and adds for a given element of the vector x are performed on the same core, we are able to use the fused multiply-accumulate instruction. The 18 flops performed take place in three machine cycles which increases utilization over the 3D mapping which performed only adds or only multiplies on any given cycle. However, the summation work for the halos are redundant operations that offset this efficiency. Furthermore, although we perform multiplication along the main diagonal, we should not receive performance credit for this operation because most problems will precondition the main diagonal to unity.
The local memory in each core is sufficient to store a matrix, halo, and vector (as well as all terms needed for BiCG) to hold a sub-block up-to 38x38 in size, corresponding to geometries of 22800x22800. Efficiency remains high for smaller problems. When a core holds only an 8x8 region in local memory (4800x4800 meshpoints), the overhead remains less than 20%.
3) AllReduce: BiCGStab requires inner products of vectors distributed across the whole fabric. This requires an AllReduce operation: the results of local dot products need to be summed across all cores and then the result broadcast back to all cores. Because we did not use a communication-hiding variant of BiCGStab, this collective operation is blocking, so we minimized latency.
The routing configuration for this task is illustrated in  Figure 6. The reduction is performed in parallel along fabric rows, then along two central columns. When the reduction starts, each core sends its value toward the center of its row. On each successive cycle the two central cores of that row receive a datum and accumulate it into a local sum. We use two cores in the center, each receiving input from one direction at the rate of one datum per cycle. Similarly, the partial sums are reduced along two columns towards the central four cores that finally reduce their content to a single core. The reason for the use of pairs of cores is core-tofabric injection/extraction bandwidth: a core can add two 32-bit quantities per cycle but can receive only one from the fabric. The broadcast is done in reverse, sending the result along two central columns and then across all rows. The single cycle-per-hop latency of the interconnect allows us to implement the AllReduce operation in a cycle count only about 10% greater than the diameter of the system. To control the growth of roundoff error, we use a hardware inner product instruction that employs mixed 16-bit multiply/32-bit add precision, and we do the AllReduce at 32-bit precision.

4) AXPY:
These operate on core-local fp16 data and use the four-way SIMD capability.

V. MEASURED RESULTS
We implemented BiCGStab for a 600 × 595 × 1536 mesh on a CS-1 machine available to us for experiments that has a 602 × 595 compute fabric. We measured the wall clock time between consecutive BiCGStab iterations; the mean differences across 171 iterations was 28.1 microseconds. The standard deviation is about 0.2% of the mean.
Per Table I, there are 44 operations per meshpoint per BiCGStab iteration. Thus, we perform 44 × 600 × 595 × 1536 operations per iteration floating-point, which works out to an achieved performance of 0.86 PFLOPS. As pointed out in Table II, we are using 16-bit floating point for 40 operations per iteration per meshpoint, and 32-bit floating point for the remaining 4 operations.

A. A strong scaling comparison
To make a comparison to CPU-based clusters, we solved a comparable problem, the BiCGstab solution of a nonsymmetric linear system arising from a 7-point stencil finite volume approximation; this was done within the NETL MFIX code while computing a lid-driven cavity flow. We have data for two problem sizes, both 370 3 and 600 3 . For the larger mesh, the performance for systems with numbers of cores ranging from 1024 to 16384 is shown in Figure 8. These are 64-bit floating point results obtained on Joule 2.0, the NETL supercomputer, which is based on HPE ProLiant servers, Intel Xeon Gold 6148, 20-core, 2.4GHz processors, using the Intel Omni-Path interconnect. (In a Navier-Stokes solver like MFIX, four linear systems are solved at every time step, one for each of the solution variables, three velocity components u, v, w and pressure p.) There are some differences between the behavior for the pressure and the momentum equations, but these differences do not change the main points.
The failure to scale beyond 8K cores on the smaller mesh highlights the difficulty of achieving strong scaling on a cluster.
For the larger mesh, time per BiCGstab iteration on Joule ranges from 75 ms on 1024 cores, and scales down to about 6 ms on 16K cores. This is about 214 It is interesting to try to understand why this striking difference arises. A first explanation is that there are a lot more cores on the CS-1. The cores are not equal in their performance, however, and the peak performance of 16K cores on Joule at 64-bit is about 40 percent of the CS-1 peak at 16-bit. This explains some of the difference, but there is still a large performance gap per core.
The difference in memory technology is clearly important. A single 20-core Xeon 6148 socket has 27.5 MB of last-level (L3) cache, and at 16K cores the aggregate cache is 22.5 GB, substantially more than the memory of the CS-1. But the Xeon caches seem to be less effective at deriving performance from the available SRAM. We suspect that this may be because the L3 cache is shared by the 20 cores on the socket, and does not have enough port bandwidth to satisfy all of their needs when, as in this case, there will be little temporal data reuse in the L1 and L2 caches. The CS-1 SRAM is not shared, and each core has access to an array of small SRAM banks that deliver 16 bytes (read) and 8 bytes (write) per clock to the single core, which is enough to support the full compute rate for an operation like an AXPY that streams two vectors from memory and streams the result vector back.

VI. CFD ON THE CS-1
Real world applications exhibit complexity not found in simple incompressible Euler flow in rectangular geometry; they feature complex geometries with heat, mass, compressibility, stretched meshes, and even structure interactions. These all complicate the steps to form the matrices of the linear systems. They account for 30 to 50 percent of the operation count. And they add to the memory requirement-memory will limit the maximum problem size that can be solved on CS-1. Thus, we have to look at what happens outside the linear solver. We analyze next the work and memory required in a realistic case. Field Update (u, v, w, p) 10: Calculate Residual 11: end for As a case study, we examine the Multiphase Flow with Interphase eXchanges (MFIX) CFD code from the National Energy Technology Laboratory. In particular, we examine the newest variation on the code MFIX-TF, which is a fully vectorized formulation. MFIX is a general purpose, Cartesian mesh, multi-phase CFD code. The code solves the fully compressible Navier-Stokes equations using an adapted version of the Semi-Implicit Method for Pressure Linked Equations (SIMPLE). In order to show that a significant problem can be solved, we discuss a single phase, compressible, viscous fluid problem without energy and species equations. It is straightforward to extrapolate the allowable size and arithmetic intensity at any level of complexity following the methodology outlined below.

A. Performance and accuracy of CFD on the CS-1
The operations needed to construct the coefficient matrix and source vector depend on the discretization scheme. First order upwinding is the most common scheme and was used to determine operation types and counts. The necessary operations can be grouped into vector merge operations, floating point (FLOP) operations (multiply, add, subtract), square root, divide, and neighbor transport operations. The cycle counts for each operation have been estimated and the operations counted for all steps outside of the linear solver in the SIMPLE algorithm per Z meshpoint. The residual calculations were ignored because they involve dot products and a few scalar calculations. The analysis revealed that they could be overlapped with other computations. SIMPLE Step  For typical flow problems, the number of simple iterations ranges from 5-20 per time step, the linear solver is limited to 5 iterations for transport equations and 20 for continuity equation. Based on the performance estimates, the wall time per time step was estimated to be roughly two microseconds per Z meshpoint. Assuming a problem size of 600x600x600 and 15 simple iterations per time step, and we expect to achieve between 80 and 125 timesteps per second. This places the likely performance of CS-1 above 200 times faster than for MFiX runs on a 16,384-core partition of the NETL Joule cluster. Furthermore, our experiments indicate that little more performance can be gained by scaling the cluster with fixed mesh size.

B. Accuracy achieved with mixed precision arithmetic
There is already extensive work on the use of lower precision in linear algebra and other areas. Ongoing research has explored how full precision can be maintained while doing most work in lower precisions [6]- [8]. For linear systems, variants of iterative refinement have been successful.
We have assessed the quality of solution we obtain using mixed 16 and 32-bit precision. We took a linear system from the timestep discretization (in the NETL code MFIX) of the momentum equation for a velocity component on a 100 × 400 × 100 mesh. The measured normwise relative residuals with mixed and 32-bit are shown in Figure 9. Up to iteration 7 the mixed precision implementation tracks the 32-bit, but then fails to reduce the residual further. With this precision, machine precision is about 10 −3 . We have observed this accuracy for very well conditioned systems. Here, the growth of rounding errors during the iterative solve explains the loss of an additional factor of 10, leading to a plateau at a relative residual of 10 −2 . We have not yet tested whether this accuracy is acceptable in the MFIX code. We expect that for some realistic situations, mixed precision solvers are usable as is; in others they may need to be coupled with a correction scheme such as an iterative refinement or an outer iteration that solves a nonlinear system; and in other situations one may need to use higher precision arithmetic. There is interest in finding the limits. In a 2018 Gordon Bell Finalist the authors demonstrate a nonlinear finite-element solver in which linear systems are solved using mixed-precision implementations based on fp16 arithmetic [9]. A largescale example is NASA's Fun3d code. NASA is using Fun3d to simulate the powered supersonic entry of manned space vehicles in the Martian atmosphere. To reduce runtimes, NASA has implemented cast in place kernels to convert fp32 values to fp16 so that they can run the Red-Black Gauss-Seidel linear solver in fp16 and transition to using Tensor Cores. They reported no ill effects on convergence or stability [10].

VII. RELATED WORK
In contrast to this work, there has been extensive work on specialized machine aimed at particular use cases. Examples include Anton [11] and the MDGRAPE systems [12]. This work stands in stark contrast to ours. We explain that a new approach at the hardware layer provides a breakthrough in memory bandwidth that make a fully general purpose system, the CS-1, capable across the spectrum of arithmetic intensity. The efforts we cite provide specialized machines that are ordinary at the hardware level and that draw their advantages from architectural specialization, and that are aimed at molecular dynamics, which lives at the high end of the intensity spectrum.
The issue of memory and fabric communication in PDE and Krylov subspace solvers remains unresolved for conventional systems. Indeed it has spawned research on communication avoiding methods [13], [14]. These can produce incremental improvements, but memory bandwidth remains a performance limiting issue.

A. Applications
We have shown here that the approach of keeping all the processing, data, and communication on one silicon wafer can, when a problem fits on the wafer, eliminate memory bandwidth and greatly reduce communication as performance limits, yielding dramatic speedup and allowing accurate real-time simulation. Before now, practitioners have had to choose between very fast reduced order/fidelity models which often lack the resolution needed to be really useful and high-resolution simulations which lack the speed to be practical.
To our knowledge, this is the first ever system capable of faster-than real-time simulation of millions of cells in realistic fluid-dynamics models. In real-time applications, memory footprints are normally tractable in order to achieve the mandatory real-time performance. To take one example, it is quite difficult and potentially dangerous to land a helicopter on the windy flight deck of an aircraft carrier, due to the complexity and speed of the air flow. In his 2017 PhD thesis [15], Oruc noted that "Especially, when the aircraft flies in the near vicinity of the superstructure of the ship, the pilot workload increases seriously." He examines the potential for semi-automatic flight control with in the loop CFD. He found that modest meshes of in the neighborhood of one million cells can provide adequate accuracy, but that the necessary real-time performance is hard to achieve on a cluster of multicore CPU systems. We give other examples of compact applications in the section below.
Cheap simulations mean that they could be treated as an input to a neural network residing in CS-1 as well and train a model approximating them. It has been shown that combining machine learning with traditional simulation accelerates scientific experiments [16]. Another implication is that we could simulate an event on demand instead of storing previous experiments. This solution makes interactive fluid dynamics a reality.
In NETL's application domain, real-time simulation can enable physics-based online equipment monitoring, cyber-physical security, equipment failure prediction, dynamic baseload power following, fixed asset turn down, and renewable integration. It may become less expensive to rerun simulations than to save and retrieve information from storage. And, since high efficiency operation of energy systems often depends on operating near failure points, the advent of faster than real time, high resolution simulations may make it safe to achieve unprecedented levels of efficiency.

B. Memory capacity
There are compelling HPC use cases for the CS-1, notwithstanding its modest memory capacity. In the helicopter example of the last section and in three examples below, extraordinary performance in a limited memory footprint is of great value. In these examples a great many spatially compact problems must be solved, for purposes that span uncertainly quantification, to long time duration simulations involving perhaps millions of timesteps, to real-time deployments with in-the-loop models. None of these use cases can be addressed using systems based on low-performance, low cost-per-byte memory.
The first use case is in automated exploration of design spaces. In the work of Madsen et al [17], the first attempt at 3D simulation-based shape optimization of whole wind turbine rotors, the wind turbine scale was 10MW. Richardson extrapolation showed that the number of cells needed is in the 14-50M range to get estimates within 0.5% to 2%. In the optimization setting, simulations have to be done sequentially, as the optimizer depends on gradients around simulation runs. There can be hundreds to thousands of simulations to complete a shape optimization. This type and scale of problem is also common in aircraft and automotive design optimization. Today these optimizations are not routinely done due to the compute times.
Second, as computational models play a growing role in industrial and societal decision-making, the confidence worthiness of these models has to be quantified. Uncertainty quantification (UQ) relies on the assessment of the effect of model and parameter uncertainty, usually requiring hundreds or thousands of simulations to sweep relevant ranges of parameters. For example, to predict the performance uncertainty of a 1MW pilot scale carbon capture system providing 90% capture efficiency with 95% confidence [18], a total of 1,505 simulations were run, each needing about 600 seconds of simulation to reach steady state. This kind of simulation campaign is currently achievable only by simplifying the geometry (two-dimensional geometry), and using filter models to account for the coarse meshes.
Finally, consider a study by Jasak et al [19] of commercial ship modeling in which speed rather than capacity is the issue. The solution time on a moderately sized computer system typical in engineering was as much 83 hours for a test case involving 11.7 million finite volume cells. Such a turnaround time is clearly a burden in design optimization and design space exploration.
Our goal is not to tout the CS-1 specifically; rather we want to illuminate the benefits of the wafer-scale approach. Recall that the CS-1 is the first of its kind; we expect that memory limits will recede over time with ongoing technological and engineering developments. A technology shrink from the 16 nm to 7 nm technology node will provide about 40 GB of SRAM on the wafer and further increases (to 50 GB at 5 nm) will follow. These changes will help, and while they don't produce petabyte capacities, there are new directions that the hardware can take, involving systems that go beyond a single wafer and a single wafer type, that can add much more. Solutions involving the clustering, with sufficient bandwidth, of several wafer-scale systems is certainly a possibility.


Embedding Text in Audio Steganography System using Advanced Encryption Standard, Text Compression and Spread Spectrum Techniques in Mp3 and Mp4 File Formats

Hiding text in a digital audio file format has been a major challenge because of the Human Auditory System (HAS) and how the digital audio will be converted into analog form for text to be embedded into it. Several techniques that include but are not limited to Least Significant Bit, Echo Hiding, Phase Coding, Parity Coding and so on have been proposed in both research communities and the academia. Audio steganography hides data in selected audio files. Several audio steganography works exist, but their major limitations include their inability to embed information in multiple audio file formats, high distortion rate and low level of robustness of their resultant stego files. This research attempts to proffer solution to the obvious challenges of the previous works by developing an efficient and robust audio steganography system for the security of information whether in store or on transit across the Internet. Results of performance evaluation of the developed system shows that it has very low level of distortion as revealed by the Signal to Noise Ratio (SNR). The compression ratio obtained is also equal to one (1), which shows that the cover audio file is identical to the resultant stego file.

INTRODUCTION
Secret digital music MP3 and MP4 files, are important and popular audio compression standard in the Internet. MP3 and MP4 use the destructive compression technologies to achieve high compression rate and the original file is shrunk to a very small size. MP3and MP4 compression always consumes time. To protect digital media files, researchers propose and improve many data-hiding algorithms, which are known as steganographic algorithms.
Steganography is the art and science of writing hidden messages in such a way that no one, apart from the sender and intended recipient, suspects the existence of the message, a form of security through obscurity. The main goal of steganography is to communicate securely in a completely undetectable manner and to avoid drawing suspicion to the transmission of a hidden data.
The most crucial parameters of the data-hiding applications are: security, reliability, invisibility, complexity, and data-hiding capacity, these parameters are mostly related to each other [14].
This research work combined both Cryptography and Steganography to hide text file into MP3 and Mp4 digital audio signal. The use of Huffman algorithm and Lempel-Ziv-Welch algorithm is used for text compression in order to reduce the distortion in the audio file and Frequency Hopping Spread Spectrum technique to increase the robustness of the stego file.

RELATED WORKS
In an attempt to address security issue, information hiding techniques like steganography have shown some promising solutions. However, there are some raising concerns when using this approach, for example, in research work presented by [1] and [2], the research works suffered a limitation of high distortion in the stego file. [12] introduce LSB method as a robust method of hiding data in audio signal. However, as the size of the secret message become larger, the embedding process becomes complicated.
[2] explained a novel method for data embedding in the audio stream. But the system can only embed information in wav file.
[15] develop a tool to encrypt a message and hide the message into a digital object. The objectives of the research work are to combine various encryption and steganography techniques in other to provide more powerful message concealing method and to improve the quality of the resultant stego-file. Thereby, reducing suspicion of convert communication. The encryption method used in their research is ElGamal Encryption. The encrypted message is embedded in the homogenous frames of mp3 audio file. Before embedding the message into the file, the encrypted message is enhanced with spread spectrum method and XOR modulation to improve its randomness. The limitations of this research work are: a. the quality of the stego file depends on file size and message length. The research work was only carried out on mp3 audio format and not on other audio format.

METHODOLOGY
Audio file format is a container format for storing audio data and metadata on a computer system. This format is divided into three major groups namely: the uncompressed audio formats, (e.g., WAV), formats with lossless compression, (e.g., WMA) and formats with lossy compression (e.g., MP3). In this research work, the selected cover media which is the audio signal undergoes different stages; the selected audio file irrespective of its format will be converted into .Wav (Wave File Audio) format using a function in MATLAB.
Many types of audio file are available, such as Windows Audio Visual (WAV), Windows Media Audio (WMA), and MPEG (MP3). The type used in this study is WAV file format of type Pulse Code Modulation (PCM), because it is uncompressed audio format, which gives more flexibility for data hiding. A stego object (WAV file) with high sampling rate and sampling resolution may draw suspicion, because of its large size, especially if its subjective quality is not high. Usually, it is easy to hide more secret data in the high quality audio data (for example, the use of least significant bit encoding to embed one bit in each sample, consist of 16 bits, sample has less effect on the stego object than adding one bit in a sample consist of 8 bits). In the developed system the wave files, with 8-bit samples resolution, are used as cover media for hosting the secret data.
There are some basic parameters to understand when discussing on digital audio files. This parameter includes: i.
Sampling rate ( ) is the number of samples of audio carried per seconds. It is measures in Hertz. In this research, 44100Hz is used in wav audio format.
ii. Frame size: The frame size is the amount of bits in each frame.

Frequency Resolution ( ):
It is the scaling factor, which ensures the cover medium is embedded below the audibility threshold. It is measured in Hertz. Human Auditory System (HAS) works dynamically in a wide range of frequencies between 20Hz-20000Hz. It is calculated using equation 2 2

Embedding Module
Embedding module is a module at which the secret message (text file) is embedded into the selected cover medium (audio Signal). This is depicted in the figure 1.

Figure 1: Architecture of the Embedding Module
The embedding module involves three processes which include: plain text (secret message) encryption and compression, audio signal decomposition as well as combined signal construction processes.
The secret message which is in plain text is encrypted in order to enhance the security of the secret message from the intruders by using a public key and private key to avoid unauthorized access of the text. This research work makes use of Advanced Encryption Standard (AES) over other encryption algorithms because it uses higher length key sizes such as 128, 192 and 256 bits for encryption, it uses 128-bit block size and also has 10, 12 or 14 rounds of bits depending on the key size used. Hence it makes AES algorithm more robust against hacking. The general encryption procedure is mathematically represented in equation 4 where represents the Encryption function, M denotes the Plain text (Secret message), stands for Encryption Key, and is the cipher text.

Figure 3: AES encryption algorithm
The encrypted text will be compressed using a two stage compression techniques which are Huffman algorithm and Lempel-Ziv Welch (LZW). Data compression is a method of encoding rules that allows substantial reduction in the total number of bits to store or transmit a file and this will help to reduce distortion in the stego file. The feature of both LZW (Lempel-Ziv-Welch) and Huffman algorithms are combined to improve the compression ratio. The Huffman algorithms is represented as where denotes the length of the cipher text, is the word character and is the number of bit.
Lempel Ziv Welch is represented as = 6 where represents the compressed cipher text using Huffman model, denotes the fixed codes and denotes the LZW compression.
The main advantage of this combined algorithm is that the percentage of data reduction increases more compared to the existing text compression techniques.
The audio format irrespective of its audio file format is converted into .wav format. This is achieved using an audio write function in MATLAB.

Figure 3: Audio format conversion to wav format
The reason for this is that it will be easier to decompose the digital audio signal into analog signal using the One Dimensional Discrete Cosine Transform (DCT).
The DCT is one of the powerful compact transforms. It relocates most of the signal energy into the first transform coefficients, lesser energy or information is relocated into other (i.e., high frequency) coefficients. The frames thus created are queued based on the energies of the frames. DCT is applied on the voiced blocks that have power less than the predefined second threshold value (T'). The block size was taken small to avoid the high computational complexity of DCT calculations which makes the system slow. Thereafter, frames will be selected using frequency frames with low frequency will be selected in order to ensure the secret message does not introduce audible distortion in the audio signal.

=
, 7 for where where represents the original sequence of the audio signal, denotes the Last frame in the audio file, is the Number of frames in an audio file and denotes the frame size.
Spread Spectrum technique was used to hide the encrypted and compressed secret message (Text file) into the digital Audio signal. Spread spectrum is a method by which energy generated in particular bandwidth is deliberately spread in the frequency domain, resulting in a signal with a wider bandwidth. Spread spectrum ( ) systems encode data as a binary sequence which sounds like noise but which can be recognized by a receiver with the correct key. There are two types of spread spectrum techniques which are: Direct Sequence Spread Spectrum (DSSS) and Frequency Hopping Spread Spectrum (FHSS). In Direct Sequence Spread Spectrum, data to be transmitted is divided into small pieces and each piece is allocated to a frequency channel across the spectrum. In this research work, Frequency Hopping Spread Spectrum is used. In Frequency-Hopping Spread Spectrum, the audio file's frequency spectrum is altered so that it hops rapidly between frequencies. Spread spectrum combined the compressed text file with the low frequencies of the audio signal using: . 8 The embedded signal is added to the other frames of high frequency using: 9 The analog signal generated is then converted into digital signal using the Inverse Discrete Cosine Transform (IDCT) as given below: 10 where is the new audio signal (stego file).

RESULT AND DISCUSSION
The system was implemented using MATLAB (R2017a version) programming language on Windows 8.1 Operating System platform with hardware configuration of 3GB RAM, 1.6Hz Intel processor speed and 250GB of hard disk. In this developed system, MP3 and MP4 digital audio file format were used as the cover media and different ranges of secret text were hidden into them for evaluation. The stego files (embedded audio file) were evaluated. The stego file (audio file) retains its initial size after evaluating the proposed approach and the amount of information that the developed system can hide is very high (500KB).
The research work was evaluated using the following performance metrics: computational time, bit per character, compression ratio and signal to noise ratio.

Compression Ratio
This is the ratio of the cover medium before and after the secret message is embedded into it to its ratio when the secret message is embedded into it.
Compression Ratio = 1 From the result carried out in this research work, the system has a compression ratio of 1, which means the size of the audio file before the secret message is embedded into it is still the same size after the secret message is embedded into it.

Signal to Noise Ratio (SNR)
Signal to noise ratio is a parameter used to know the amount by which the signal is corrupted by the noise. It is defined as the ratio of the signal power to the noise power. Alternatively, it represents the ratio of desired signal (say a music file) to the background noise level. It is measured in decibel (db).

Computational Time
This is the time taken for the system to execute its function.

MP3 Audio Format
Mp3 file format, with size 4.58Nb and a length of 5 minutes was used as an experiment to perform the performance metrics. Table 1 shows the result of the evaluation. Table 1

: Mp3 File Evaluation Result
From table 1, the values of the Signal to Noise ratio are more than 50db when the size of the text file to be embedded ranges from 50kb to 200kb, and this indicates that there will be no distortion in the audio. But from 250kb to 500kb, the values of the Signal to Noise Ratio began to decrease; making the values to be less than 50db and this implicates that there will be distortion as the value decreases from 50db. This can be represented graphically in figure 1.
From table 2, the values of the Signal to Noise ratio goes above 50db when the size of the text file to be embedded ranges from 50kb to 100kb, but the value of SNR for 150kb is 49.3 which van be approximately to 50db can also be accepted as good SNR vale. This indicates that there will be no distortion in the audio with a secrete message that's up to 150kb. But from 200kb to 500kb, the values of the SNR began to decrease; making the values to be less than 50db and this implies that there will be distortion as the values decreases from 50db. This is represented graphically in figure 2.

MP4 Audio Format
MP4 file format, with size 4.70 Mb and a length of 5 minutes was used as an experiment to perform the performance metrics. Table 2 shows the result of the evaluation.

CONCLUSION AND RECOMMENDATION
There are a number of proven methods for applying steganography to hide information within audio data. In this research work, an audio steganography system for MP3 and MP4 that uses using Discrete Cosine Transform (DCT) and spread spectrum techniques was developed. It was shown through implementation and subjective experimentation that the developed audio steganography system supports MP3 and MP4 digital audio format. The system developed has the ability to embed a secret message of size that is up to 500kb but the system has the ability to embed a text size of 250kb with respect to the digital audio length or size without any distortion and has the ability to retain the same size after embedding text into it.
The work has been able to develop a robust stenographic system that would be very useful in securing and sharing large amount of sensitive data or information without arousing suspicion. This system is therefore recommended for security agencies and other organization that consider information security as being of uttermost priority. This system is a useful means for transmitting covert battlefield information via an innocuous cover audio signal.  [5] Bhowal, K., Pal, A. J., Tomar, G. S., and Sarkar, P. P. (2010, November). Audio steganography using GA.


Few-Shot Adaptation of Generative Adversarial Networks

Generative Adversarial Networks (GANs) have shown remarkable performance in image synthesis tasks, but typically require a large number of training samples to achieve high-quality synthesis. This paper proposes a simple and effective method, Few-Shot GAN (FSGAN), for adapting GANs in few-shot settings (less than 100 images). FSGAN repurposes component analysis techniques and learns to adapt the singular values of the pre-trained weights while freezing the corresponding singular vectors. This provides a highly expressive parameter space for adaptation while constraining changes to the pretrained weights. We validate our method in a challenging few-shot setting of 5-100 images in the target domain. We show that our method has significant visual quality gains compared with existing GAN adaptation methods. We report qualitative and quantitative results showing the effectiveness of our method. We additionally highlight a problem for few-shot synthesis in the standard quantitative metric used by data-efficient image synthesis works. Code and additional results are available at http://e-271.github.io/few-shot-gan.

INTRODUCTION
Recent years have witnessed rapid progress in Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) with improvements in architecture designs (Radford et al., 2015;Karras et al., 2018;Zhang et al., 2019;Karras et al., 2019a), training techniques (Salimans et al., 2016;Karras et al., 2018;Miyato et al., 2018), and loss functions Gulrajani et al., 2017). Training these models, however, typically requires large, diverse datasets in a target visual domain. While there have been significant advancements in improving training stability (Karras et al., 2018;Miyato et al., 2018), adversarial optimization remains challenging because the optimal solutions lie at saddle points rather than a minimum of a loss function (Yadav et al., 2017). Additionally, GAN-based models may suffer from the inadequate generation of rare modes in the training data because they optimize a mode-seeking loss rather than the mode-covering loss of standard likelihood maximization (Poole et al., 2016). These difficulties of training GANs become even more severe when the number of training examples is scarce. In the low-data regime (e.g., less than 1,000 samples), GANs frequently suffer from memorization or instability, leading to a lack of diversity or poor visual quality.
Several recent efforts have been devoted to improving the sample efficiency of GANs through transfer learning. The most straightforward approaches are finetuning a pre-trained generator and discriminator on the samples in the target domain (Wang et al., 2018;Mo et al., 2020). When the number of training examples is severely limited, however, finetuning the network weights often leads to poor results, particularly when the source and target domains are distant. Instead of finetuning the entire network weights, the method in (Noguchi & Harada, 2019) focuses on adapting batch norm statistics, constraining the optimization problem to a smaller set of parameters. The authors report that this method achieves better results using MLE-based optimization but fails for GAN-based optimization. Although their quality outperforms GAN-based methods in the low-shot setting, the images are blurry and lack details due to maximum likelihood optimization. Invertible flow-based models have shown promising results in data-efficient adaptation (Gambardella et al., 2019), but require compute-and memory-intensive architectures with high-dimensional latent spaces.
In this paper, we propose a method for adapting a pre-trained GAN to generate novel, high-quality sample images with a small number of training images from a new target domain (Figure 1). To accomplish this, we restrict the space of trainable parameters to a small number of highly-expressive parameters that modulate orthogonal features of the pre-trained weight space. Our method first 1 arXiv:2010.11943v1 [cs.CV] 22 Oct 2020 Figure 1: Few-shot image generation. Our method generates novel and high-quality samples in a new domain with a small amount of training data. (Top) Diverse random samples from adapting a FFHQ-pretrained StyleGAN2 to toddler images from the CelebA dataset (with only 30 images) using our method. (Bottom) Smooth latent space interpolation between two random seeds shows that our method produces novel samples instead of simply memorizing the 30 images. Please see the supplementary video for more results.
applies singular value decomposition (SVD) to the network weights of a pretrained GAN (generator + discriminator). We then adapts the singular values using GAN optimization on the target few-shot domain, with fixed left/right singular vectors. We show that varying singular values in the weight space corresponds to semantically meaningful changes of the synthesized image while preserving natural structure. Compared with methods that finetune all weights of the GAN (Wang et al., 2018), individual layers (Mo et al., 2020), or only adapt batch norm statistics (Noguchi & Harada, 2019), our method demonstrates higher image quality after adaptation. We additionally highlight problems with the standard evaluation practice in the low-shot GAN setting.

BACKGROUND
Generative Adversarial Networks (GANs) GANs (Goodfellow et al., 2014) use adversarial training to learn a mapping of random noise to the distribution of an image dataset, allowing for sampling of novel images. GANs optimize a competitive objective where a generator G(Z) maximizes the classification error of a discriminator D(X) trained to distinguish real data p(X) from fake data G(Z). The GAN (Goodfellow et al., 2014) objective is expressed formally as: Recent research reformulated this objective to address instability problems Heusel et al., 2017a;Gulrajani et al., 2017). Improved architecture and training has led to remarkable performance in synthesis (Karras et al., 2020;Brock et al., 2019). Compared to pixel-reconstruction losses (Kingma & Welling, 2014;Higgins et al., 2017;Bojanowski et al., 2018) GANs typically produce sharper images, although strong priors over the latent space can offer competitive quality (Razavi et al., 2019). A high-quality generation has relied on large datasets of high-quality images (10K+) that may be expensive or infeasible to collect in many scenarios. Additionally, GANs can suffer from a lack of diversity, even when large training sets are used because the objective does not penalize the absence of outlier modes (Poole et al., 2016). Data-efficient GAN methods are, therefore, of great utility.
Sample-efficient Image Synthesis Sample-efficient image synthesis methods encourage diverse and high-quality generation in the low-data regime, most commonly through pretraining (Wang et al., 2018;Noguchi & Harada, 2019) or simultaneous training (Yamaguchi et al., 2019) on large image datasets. The main differences among these methods lie in the choice of learnable parameters used for adaptation. Examples include adapting all weights of the generator and discriminator Wang et al. (2018), freezing only lower layers of the discriminator Mo et al. (2020), or changing only channelwise batch statistics Noguchi & Harada (2019). Flow-based methods (Gambardella et al., 2019) show promising results in few-shot adaptation, but their architecture is compute-and memory-intensive and requires latent space of the same dimensionality as the data. Our method uses a smaller but more expressive set of parameters ( Figure 2), resulting in more natural adapted samples.
(b) FSGAN singular value adaptation. One-shot Image Re-synthesis. Recent work in one-shot image synthesis has demonstrated highquality and diverse results by modeling the internal distribution of features from a single image without pretraining (Shaham et al., 2019;Shocher et al., 2019). Our work differs as we transfer external knowledge from a pretrained GAN to a new domain and, therefore, can generate drastically more diverse samples.
Singular Value Decomposition (SVD). SVD factorizes any matrix M ∈ R m×n into unitary matrices U ∈ R m×m ,V ∈ R n×n and diagonal matrix Σ such that M = UΣV , where U,V contain the left and right singular vectors respectively and Σ contains the singular values along the diagonal entries. SVD can be interpreted as a decomposition of a linear transformation x − → Mx into three separate transformations: a rotation/reflection U, followed by rescaling Σ, followed be another rotation/reflection V . The transformation defined by the maximum singular value σ 0 = Σ (1,1) and its corresponding normalized singular vectors represent the maximal axis of variation in the matrix M. This interpretation is commonly used in data science for dimensionality reduction via PCA (Kwak, 2008). PCA can be obtained via SVD on a column-normalized matrix (Golub & Reinsch, 1971). SVD is also used for a wide number of other applications, including regularization (Sedghi et al., 2018), and quantification of entanglement (Martyn et al., 2020), and has also been used to build theoretical background for semantic development in neural networks (Saxe et al., 2019). The work most closely related to ours is GANSpace (Härkönen et al., 2020) for image synthesis editing. GANSpace applies PCA within the latent feature space of a pretrained GAN to discover semantically-interpretable directions for image editing in the latent space. In contrast, our work performs SVD on the weight space of a GAN to discover meaningful directions for domain adaptation. Performing SVD on the weight space enables two critical differences between our work and Härkönen et al. (2020): (i) we edit the entire output distribution rather than one image, and (ii) rather than manual editing, we adapt to a new domain.

OVERVIEW
Our goal is to improve GAN finetuning on small image domains by discovering a more effective and constrained parameter space for adapting the pretrained weights. We are inspired by prior work in GAN adaptation showing that constraining the space of trainable parameters can lead to improved performance on target domain (Rebuffi et al., 2017;Mo et al., 2020;Noguchi & Harada, 2019). In contrast to identifying the parameter space within the model architecture, we propose to discover a parameter space based on the pretrained weights. Specifically, we apply singular value decomposition to the pretrained weights and uncover a basis representing orthogonal directions of maximum variance in the weight space. To explore the interpretation of the SVD representation, we visualize the top three singular values of synthesis and style layers of StyleGAN2 (Karras et al., 2020). We observe that varying the singular values corresponds to natural and semantically-meaningful changes in the output image as shown in Figure 3. Changing the singular values can be interpreted as changing the entanglement between orthogonal factors of variation in the data (singular vectors), providing an style 4 conv 8×8 conv 1024×1024 Original 10σ 0 10σ 1 10σ 2 5σ 0 5σ 1 5σ 2 10σ 0 10σ 1 10σ 2 Figure 3: Effects of singular values. We visualize FSGAN's adaptation space by magnifying the top 3 singular values σ 0 , σ 1 , σ 2 from SVD performed on style and conv layers of a StyleGAN2 (Karras et al., 2019a; 2020) pretrained on FFHQ. In mapping layer 4 (style 4 ), the leading σ s change the age, skin tone, and head pose. In synthesis layer 2 (conv 8×8 ), face dimensions are modified in term of face height/size/width. In synthesis layer 9 (conv 1024×1024 ), the face appearance changes in finer pixel stats such as saturation, contrast, and color balance.
expressive parameterization of the pretrained weights, which we leverage for adaptation as described in the following section.

ADAPTATION PROCEDURE
Our method first performs SVD on both the generator and discriminator of a pretrained GAN and adapts the singular values to a new domain using standard GAN training objectives. A generator layer G ( ) or a discriminator layer D ( ) may consist of either 2D (c in × c out ) fully-connected weights or 4D (k × k × c in × c out ) convolutional filter weights. We apply SVD separately at every layer of the generator G ( ) and discriminator D ( ) . Next, we describe the decomposition process for a single layer of pretrained weights W ( ) 0 . For fully-connected layer W ( ) 0 , we can apply SVD directly on the weight matrix. For 4D convolution weights W ( ) 0 ∈ R k×k×c in ×c out this is not feasible because SVD operates only on a 2D matrix. We therefore reshape the 4D tensor by flattening across the spatial and input feature channels before performing SVD to obtain a 2D matrix W ( ) 0 ∈ R k 2 c in ×c out . Our intuition is that the spatial-feature relationship in the pretrained model should be preserved during the adaptation. We apply SVD over each set of flattened convolutional weights or fully convolution weights to obtain the decomposition: After decomposing the pretrained weights, we perform domain adaptation by freezing pretrained left/right singular vectors in (U 0 ,V 0 ) ( ) and optimizing the singular values Σ = λ Σ 0 using a standard GAN objective to obtain transferred weights ( Figure 2): Effectively, our GAN domain adaptation aims to find a new set of singular values in each layer of a pretrained model so that the generated outputs match the distribution of the target domain.
During forward propagation, we reconstruct weights W Σ using the finetuned singular values at each convolution or fully-connected layer of the generator and discriminator before applying the operation.

TRAINING & INFERENCE
Our experiments use the StyleGAN2 (Karras et al., 2020) training framework, which optimizes a logistic GAN loss (Equation 1) with latent space gradient regularization and a discriminator gradient Train set (10-shot): Figure 4: Problem with FID as a few-shot metric. TGAN (Wang et al., 2018) adaptation from English characters to 10-shot Kannada characters (Bottom) (De Campos et al., 2009). The adaptation process is illustrated by interpolating two random latent vectors at different timesteps (t=20 means 20K images seen during training). We measure FID against a 2K-image Kannada set, from which the 10 images was sampled. The interpolation shows larger timesteps (t) tend to memorize the 10-image training set while yielding lower FID, revealing that FID favors overfitting and is not suitable for the few-shot setting.
penalty. We retrain the singular values Σ for a fixed number of timesteps (20K images or 16K for 5-shot). We find limiting the training time is essential for quality and diversity in the low-shot setting, as longer training often leads to overfitting or quality degradation (examples in Figure 4 & 7). Like Noguchi & Harada (2019), we use the truncation trick (Brock et al., 2019) during inference, but our method works with a less-restrictive truncation parameter of ψ = 0.8, which enables more diversity in the generated images.

EVALUATION IN FEW-SHOT SYNTHESIS
A common adverse outcome in few-shot image generation is overfitting to the target set, such that all generated images look similar to the training data. Evaluation metrics should reflect the diversity of generated images, so that memorization is penalized. The standard evaluation practice used in prior low-shot GAN adaptation work (Wang et al., 2018;Noguchi & Harada, 2019;Mo et al., 2020) is to estimate FID (Heusel et al., 2017b) using a large held-out test set with 1K+ images, from which the low-shot training set was sampled. Standard GAN evaluation typically measures FID with respect to the training set, but in the low-shot setting, this is not desirable because the generator may simply memorize the training set. However, we find that even when measuring FID against a held-out test set, this evaluation still favors overfitted or poor-quality models, as shown in Figure 4. FID between real and fake images is calculated as the Frechet distance between perceptual features p r (X) and p f (Z): where it is assumed features are Gaussian i.e., p f (Z) = N(µ f ,C f ) and p r (X) = N(µ r ,C r ). In the fewshot setting, our n-shot training set T = (x 1 , x 2 , ..., x n ) is sampled from our test set p r (X). Assuming T is chosen at random, its sample mean and varianceμ,σ 2 are unbiased estimators of µ r ,C r . Therefore if the generator memorizes T , its statistics approximate µ r ,C r . This artificially decreases the FID of an overfit model (Figure 4). Consequently, we suggest that FID should be supplemented with additional metrics and extensive qualitative results in the low-shot setting. In high-data settings, a very large number of parameters would be required to memorize the images, so this problem is less likely to occur. Based on these observations, throughout our evaluation, we limit training timesteps rather than select the step with the best FID as we find the latter approach gives more inferior qualitative results.
To address the limitations of standard metrics for GAN evaluation, we also report sharpness (Kumar et al., 2012) and face quality index (Hernandez-Ortega et al., 2019) for human face transfer.

SETTINGS
We adapt a pretrained model to a new target domain using only 5-100 target images, as we focus on scenarios with 1-2 orders fewer number of training samples than standard data-efficient GAN adaptation methods (Wang et al., 2018;Mo et al., 2020;Noguchi & Harada, 2019). As discussed in Section 3.4, we find that the FID score is unsuitable in the low-shot regime due to overfitting bias. However, we still report the FID scores of our experiments for completeness. In addition, we report additional quality metrics and extensive qualitative results.  Datasets. We used FFHQ (Karras et al., 2019a) and LSUN Churches (Yu et al., 2015) pretrained checkpoints from StyleGAN2 (Karras et al., 2019b), and transferred to few-shot single-ID CelebA (30 or 31 images) (Liu et al., 2015), Portraits (5-100 images) (Lee et al., 2018), Anime ID "Rem" (25 images) 2 , and Van Gogh landscapes (25 images) (Zhu et al., 2017). We evaluate FID against a large test set (10K for CelebA) following the evaluation method of Wang et al. (2018). We also evaluate face quality index (Hernandez-Ortega et al., 2019) and image sharpness (Kumar et al., 2012) for face domain adaptation, using 1000 images from each method generated using identical seeds. Full few-shot target sets are shown in Figures 5 & 6, and we will make all few-shot sets available online.  Figure 6: Far-domain adaptation (Photo→Art). Comparing FSGAN with alternative GAN adaptation methods in the photo-to-art setting. (a) FSGAN more effectively alters building layouts and adds landscape in the foreground to match the Van Gogh paintings, maintaining better spatial coherency. (b) FSGAN adopts features from the Portraits dataset (hats, beards, artistic backgrounds), while other methods primarily alter image textures. (c) FSGAN transforms natural hair and facial features to imitate the anime target while retaining spatial consistency. Note the occurrence of pink hair in our generated images, which does not exist in the few-shot target but is visually consistent.

NEAR-DOMAIN ADAPTATION
We first show a near domain transfer setting (adapting FFHQ to single-ID CelebA datatset (Liu et al., 2015)). As both source and target domains contain faces, the pretrained model has useful features for the transfer domain. Figure 5 shows that existing GAN adaptation methods produce artifacts around the eyes/chin and low overall structural consistency. In contrast, our method generates more natural face images with characteristics similar to the training samples (e.g., the head size, position of the faces). Comparing Figure 5 and Table 1 shows that the FID correlates poorly with qualitative evaluation for this setting. In light of this, we report additional metrics of face quality (Hernandez-Ortega et al., 2019) and sharpness (Kumar et al., 2012). On these metrics, our method achieves competitive performance across adaptation settings.

FAR-DOMAIN ADAPTATION
We show far-domain 25-shot transfer, where we define "far" as differing significantly in the distribution of image features such as textures, proportions, and semantics. 1) LSUN Churches→ Van Gogh paintings: The two domains differ in the foreground, building shapes, and textural styles. 2) FFHQ→Art portraits: The main differences between the two domains are low-level styles and facial features. 3) FFHQ→Anime Rem ID: A challenging setting with exaggerated facial proportions and lack of texture details. Figure 6 shows visual comparisons with three state-of-the-art methods. We 5-shot 15-shot 50-shot 100-shot find that the proposed FSGAN can adapt the model to produce more dramatic changes to match the target distributions in terms of semantics, proportions, and textures while maintaining image quality.

N-SHOT SETTINGS
We test the sensitivity of both FSGAN (ours) and FreezeD (Mo et al., 2020) to differing n-shot settings and show the results in Figure 7. We find that FSGAN is more robust to n-shot setting compared to FreezeD. To show this better, we compare two variations of FreezeD. The first FreezeD variant (FD) is limited in timesteps (20K images / 16K on 5-shot) to match FSGAN and the results reported in Figures 5 & 6. Limiting timesteps prevents degradation that occurs at later iterations in the few-shot settings. However, the time-limited FD produces low quality and limited adaptation of textures and semantic features. The second FreezeD variant (FD-FT) is trained for longer (60K images) to demonstrate (1) degradation in fewer n-shot and (2) improvements in quality/adaptation in higher n-shot as seen in (Mo et al., 2020). In contrast, our method (FSGAN) effectively transfers semantic features while preserving quality across all n-shot settings tested in Figure 7. We note variance across n-shot settings for all methods as the data distribution changes.

CONCLUSIONS
We presented Few-Shot GAN, a simple yet effective method for adapting a pre-trained GAN based model to a new target domain where the number of training images is scarce. Our core idea lies in factorizing the weights of convolutional/fully-connected layers in a pretrained model using SVD to identify a semantically meaningful parameter space for adaptation. Our strategy preserves the capability of generating diverse and realistic samples while provides the flexibility for adapting the model to a target domain with few examples. We demonstrate the effectiveness of the proposed method with close-domain and far-domain adaptation experiments and across various n-shot settings. We show favorable results compared with existing data-efficient GAN adaptation methods.