## nsCouette - A high-performance code for direct numerical simulations of turbulent Taylor-Couette flow

Jose Manuel L´pez o a, ∗∗ , Daniel Feldmann b, ∗ , Markus Rampp , c Alberto Vela-Mart´ ın d , Liang Shi , Marc Avila e b

a

Institute of Science and Technology Austria, Am Campus 1, 3400 Klosterneuburg, Austria b University of Bremen, Center of Applied Space Technology and Microgravity (ZARM), Am Fallturm 2, 28359 Bremen, Germany c Max Planck Computing and Data Facility (MPCDF), Gießenbachstraße 2, 85748 Garching, Germany d School of Aeronautics, Universidad Polit´cnica de Madrid, e Plaza del Cardenal Cisneros 3, 28040 Madrid, Spain e Max Planck Institute for Dynamics and Self-Organization (MPIDS), Bunsenstraße 10, 37073 G¨ttingen, Germany o

## Abstract

We present nsCouette , a highly scalable software tool to solve the NavierStokes equations for incompressible fluid flow between differentially heated and independently rotating, concentric cylinders. It is based on a pseudospectral spatial discretization and dynamic time-stepping. It is implemented in modern Fortran with a hybrid MPI OpenMP -parallelization scheme and thus designed to compute turbulent flows at high Reynolds and Rayleigh numbers. An additional GPU implementation ( C CUDA -) for intermediate problem sizes and a version for pipe flow ( nsPipe ) are also provided.

Link to open access journal publication: DOI: 10.1016/j.softx.2019.100395

Keywords: Wall-bounded turbulence, Rotating shear-flow, Thermal convection, Direct numerical simulation (DNS), Hybrid parallelization, GPU

## 1. Motivation and significance

Flows in engineering and nature are often characterized by large Reynolds ( Re ) or Rayleigh ( Ra ) numbers. Examples are the flow of gas in astrophysical disks, atmospheric flows and the cooling of rotating machines. In most cases,

∗ Corresponding author at ZARM: daniel.feldmann@zarm.uni-bremen.de

∗∗ Corresponding author at IST Austria: jlopez@ist.ac.at

it is impossible to resolve all scales of the turbulent flow in a direct numerical simulation (DNS). However, DNS provide reliable data to allow extrapolation to the large Re limit and to enable the development of adequate subgrid-scale models. Taylor-Couette (TC) flow - the flow between two independently rotating concentric cylinders - stands out as a testbed for these purposes [1, 2]. It allows exploring a variety of physical mechanisms, including buoyancy, shear, rotation and boundary layers in the vicinity of curved walls. Our DNS code nsCouette integrates the incompressible Navier-Stokes equations for TC flow forward in time using cylindrical coordinates and primitive variables. Optionally, the cylinder walls can be differentially heated, in which case an additional equation for the temperature is solved. The goal of this paper is to make nsCouette publicly available and thus enable DNS of rotating turbulent shear flows to a wide range of users in the mathematics, physics and engineering communities.

## 2. Software description

## 2.1. Functionality

In nsCouette , the governing equations are discretized using a pseudospectral Fourier-Galerkin ansatz for the azimuthal ( θ ) and the axial ( z ) direction. High-order finite differences (FD) are used in r ; the only inhomogeneous direction. The user can select the distribution of radial grid points at runtime and the stencil-length of the FD scheme is specified at compile time with a default of nine points.

Periodic boundary conditions (BC) are assumed in z to avoid the need for dense grids close to the vertical boundaries. The comparison between DNS with axially periodic BCs and laboratory experiments with solid end-plates is extremely satisfactory for a wide range of Re from laminar to highly turbulent flows [2]. Additionaly, z -periodicity often provides a more accurate model of astrophysical and geophysical flows and prevents misleading physical interpretations due to undesired end-wall-effects [3, 4]. Details of the method and implementation are published in [5].

The temporal integration scheme has been upgraded to a predictor-corrector method [6]. This enables a variable time-step size with dynamic control, which is of advantage if the flow state is suddenly modified (applying disturbances, changing rotation rates) or naturally undergoes strongly transient dynamics.

Another significant upgrade is the extension to heat transfer where a temperature difference between the cylinders is imposed. To this end, a Boussinesq-like approximation [7] has been implemented to account for buoyancy effects. In the distributed version of nsCouette , a negative temperature

gradient in r is considered, whereas gravity is aligned in z . Other scenarios can be easily investigated by changing only a few lines of source code.

Additionally, divergence-free initial conditions can be used to easily excite selected Fourier modes. This enables the user to systematically investigate different transition scenarios.

A single input file defines all relevant parameters (number of points, modes and timesteps, rotation rates etc.) at runtime. At every restart, the spatial resolution can be changed using an automated interpolation and mode padding functionality.

This and many other convenient features, together with a number of example Makefiles for the most common high-performance computing (HPC) platforms, provide newcomers an easy start into the world of highly-resolved and massively-parallel DNS. A user guide is included to help non-expert users get started with nsCouette (compilation, setup, select proper resolution, analyse data, etc.).

## 2.2. Software architecture

Over time, nsCouette has been ported to all major CPU-based HPC platforms. Amongst IBM Power, BlueGene and x86 64 architectures - including a few generations of the prevalent Intel Xeon multi-core processors it has also been ported to Xeon Phi (KnightsLanding), AMD EPYC (Naples, Rome) and ARMv8.1 (Marvell ThunderX2) platforms. Developments for the NEC SX-Aurora vector architecture and multi-GPU clusters are underway.

Building the executable requires a modern Fortran compiler, a standard C compiler and only very few additional libraries: MPI , BLAS LAPACK / , FFTW and optionally HDF5 . All of them are commonly available as high-quality, open-source software (e.g. GCC , OpenMPI [8], FFTW [9], OpenBLAS ) and as vendor-optimized tool chains (e.g. Intel Parallel Studio PSXE ).

Our code runs on laptops and - for large-enough problems - efficiently scales up to the largest HPC systems with tens of thousands of processor cores [10]. The basic architecture of nsCouette and our design choice for a hybrid MPI OpenMP -parallelization scheme (HPS) follows straightforwardly from the Fourier-Galerkin ansatz and from the current technological trend of multi-core processors with ever increasing core counts and stagnating percore performance. The basis of our HPS is a one-dimensional (1d) MPI -only slab decomposition into Fourier modes, which can be treated independently of each other in the computation of the linear terms occurring in the governing equations. For computing the non-linear terms, global data transpositions ( MPI Alltoall ) and task-local transposes are employed to gather all modes locally on each MPI task. Therefore, the number of radial grid points ( N r ), which is typically much smaller than the number of Fourier modes, imposes

a natural limit to the 1d MPI -only approach. We relax this limit by introducing an additional parallelization layer, such that multiple cores per MPI task will be allocated to accommodate multiple OpenMP threads. First, this allows to compute the linear terms in OpenMP -parallel loops over Fourier modes. Second, an OpenMP -coarse-grain parallelism can be exploited during the computations of the non-linear terms by overlapping the global transpositions and the Fourier transformations of state variables. Overall, when confronted with a 2d MPI -only domain decomposition (e.g. [11]), which might be superior for certain setups, our HPS is an excellent compromise between versatility, simplicity of the implementation and achievable peak parallel scalability.

Additionally, we provide a basic GPU version of nsCouette written in C CUDA -, which implements the exact same numerical schemes as the Fortran version, but currently provides less output and functionalities. These will be added in the future. A CUDA -capable GPU device with compute capability 2.0 (or higher), support for double-precision arithmetic, and NVIDIA's CUDA toolkit are required. The GPU version runs on single GPU devices in a massively parallel setup with thousands of GPU threads and highly efficient memory management. It relies on custom CUDA kernels for linear algebra, uses the highly-optimised cuFFT library to perform Fourier transforms without the need for inter-node communication and, therefore, provides a clear speedup with respect to the Fortran code for small problems that fit into the main memory (RAM) of one GPU.

## 2.3. Computational performance

The number of MPI tasks can be selected at program start-up and changed between runs with the only restriction that it must divide N r . The HPS of nsCouette achieves a scalability well beyond the limitations imposed by 1d MPI -only approaches and maps naturally to the prevailing multi-node, multicore HPC architectures. Specifically, the flexibility to choose appropriate combinations for the numbers of MPI tasks per node and OpenMP threads per MPI task has proven key for achieving good performance across a wide range of architectures from 16 to 64 cores per node. On such machines, nsCouette enables highly resolved DNS with N r = O (10 ) using 3 O (10 ) cores [5]. 4 In absolute terms, for an intermediate problem size with N r = 512 and 513 × 1025 Fourier modes (i.e. Re up to O (10 )), 4 the computation of a single timestep takes less than a second on 64 nodes (2560 cores) of a contemporary HPC cluster (Figure 1) and requires roughly 450 GB of RAM. On the SKL platform, this run achieves a performance of 1 5 TFlop/s, which, due to a . rather moderate arithmetic intensity of the algorithm (0 3), is bounded by . the memory bandwidth. When increasing the number of cores for a fixed problem size, the computations of the FFT and linear terms show very good

Figure 1: Runtime per timestep and breakdown into the main algorithmic components (different colours) of a typical nsCouette run ( N r = 512 and 513 × 1025 Fourier modes) computed on 32 and 64 dual-socket nodes of various HPC clusters, using a platformspecific number of MPI tasks/node (TpN). IVB: Intel Xeon E5-2680v2 (IvyBridge), 20 cores/node. BDW: Intel Xeon E5-2698v4 (Broadwell), 40 cores/node. SKL: Intel Xeon 6148 (Skylake), 40 cores/node. KNL: Xeon Phi 7230 (KnightsLanding), 64 cores/node. ARM: Marvell ThunderX2 ARM v8.1, 64 cores/node. The IVB and BDW clusters employ a Mellanox InfiniBand FDR network (56 Gbit / s ), whereas SKL and KNL use Intel OmniPath (100 Gbit / s ). The ARM cluster is interconnected with Cray Aries (80 Gbit / s ). nsCouette was built using platform-optimized software tool chains (i.e. compilers and libraries) but no platform-specific optimization of the source code was performed. Corresponding Makefiles are shipped with the code.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00587v3-figure-e834e5ac-4a97-4ed3-800e-a699f5d791dc.jpg)

strong scalability, whereas the global transposes ultimately limit the total parallel efficiency at large core counts (Figure 1). A comprehensive study of the parallel scalability and efficiency of nsCouette has been presented in [5], and its potential to scale up to extremely high core counts was shown in [10]. The upgraded version presented here exhibits the same scalability and maintains consistent performance over a range of different HPC systems without the need for any platform-specific adaptations of the source code.

The performance of the GPU-accelerated version was tested on two NVIDIA graphics cards based on the Volta architecture: Titan V and Tesla V100 (Figure 2). The runtime per timestep has been found to be similar on both cards over a wide range of problem sizes. The speed-up of the GPU version compared to the HPS version running on one node was shown to vary between a factor of three and 17, depending on the problem size and the particular

Figure 2: Performance of the GPU-accelerated version of nsCouette compared to the MPI OpenMP -version. Runtime per timestep for different numbers of degrees of freedom (DOF). The GPU code ran on a single NVIDIA Titan V and a single Tesla V100 graphics card. It was built using NVIDIA's CUDA toolkit version 10.1. The hybrid code was built using Intel's PSXE2018 and ran on one and 16 nodes of different platforms. BDW: Intel Xeon E5-2620v4 (Broadwell), 16 cores/node, Mellanox InfiniBand FDR network (56 Gbit / s ). SKL: Intel Xeon 6148 (Skylake), 40 cores/node. Solid (dashed) lines represent runs with N r = 64 ( N r = 128) radial points.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00587v3-figure-45a8e1af-65cf-4f3f-b24b-8c33b3563f66.jpg)

choice of platform used as reference. Comparing a single GPU run against an MPI OpenMP -run on a single CPU node is a reasonable choice, since for server-class hardware both set-ups are roughly comparable in terms of price and electrical power consumption. However, 16 nodes (256 cores) were necessary to outperform the GPU version for small problem sizes. Currently, the maximum problem size applicable to the GPU version is limited by the amount of RAM available on the graphics card.

## 2.4. Data analysis and visualization

In nsCouette , the spectral coefficients and the primitive variables are dumped to individual files for each timestep at user-specified output intervals. It implements an easy-to-use checkpoint-restart mechanism based on the coefficients for handling long-running DNS. The primitive variables - velocity ( u r , u θ , u z ), pressure ( p ) and optionally temperature ( T ) - are written in HDF5 format, along with metadata in small xdmf files in order to facilitate analysis with common visualization software like ParaView and VisIt . Both tools allow loading sequences of xdmf files produced by nsCouette . Sample scripts based on the Python interface of VisIt , as well as a custom-made ParaView filter for handling the cylindrical coordinate system are distributed

with the code. A detailed visualization tutorial is included in the user guide. This enables the user to easily perform comprehensive visual and quantitative analysis of the flow field.

## 2.5. Quality assurance

Verification and validation (V&amp;V) of nsCouette is documented in [5]. For maintaining the correctness of the source code, we make extensive use of the continuous integration (CI) functionality of gitlab . Upon every push to the repository, a number of regression tests are automatically triggered, including builds of the code in various configurations and a static code analysis using the Forcheck tool. In addition, a number of short test runs are automatically launched using runtime-checks and the tightest debug settings of the compiler to identify undefined variables, out-of-bounds errors and alike. The numerical results are then rigorously verified against previously recorded reference runs. A final validation run compares the wave speed of a simulated wavy vortex flow with an experimentally determined value [12], which is considered successful if the wave speeds match up to 10 -4 . The entire CI configuration, the V&amp;V results for every push, and an auto-generated source code documentation ( Ford ) - including dependencies and call graphs - are publicly accessible through the web interface of our development site.

## 3. Illustrative Example

The fluid flow between a hot rotating cylinder and a cooled stationary enclosure is a simple model to investigate heat transfer in many engineering applications [13] such as cooling of rotating machinery [14]. At low angular speeds ( Re i ) and small temperature differences ( Ra ), the heat transfer is purely conductive. In this simple case, termed basic state, the governing equations admit analytic solutions for u θ and T , which only depend on r . The heat transfer can be enhanced by either increasing Re i (forced convection) or by increasing Ra (natural convection). In both cases, the basic state exhibits a sequence of distinct instabilities ultimately leading to turbulent heat transfer [15]. A measure of the efficiency is given by the Nusselt number ( Nu i ), which is the ratio of total heat transfer at the inner cylinder, normalized by that of the basic state at the same temperature difference.

Figure 3 summarizes the results of three DNS with increasing temperature difference at a fixed rotation rate, using nsCouette . The first DNS was initialized by applying small single harmonic disturbances to the basic state at Re i = 50 and Ra = 2130. Figure 3a shows that initially Nu i ≈ 1, corresponding to purely conductive heat transfer. However, after roughly two viscous time units, a sharp increase of Nu i is observed, indicating that

Figure 3: Temporal evolution of the Nusselt number ( Nu i ) at the inner cylinder wall and the streamwise velocity component ( u θ ) at a mid-gap position as Ra increases for a fixed inner cylinder rotation ( Re i = 50). The final flow state for each Ra is visualized with instantaneous temperature iso-surfaces ( T = 0), which are color-coded by inwards/outwards (blue/red) facing values of the wall-normal velocity component ( u r ).

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00587v3-figure-b87bfe34-9ca9-4fa2-a71e-20e2f06443c1.jpg)

the basic state has become unstable. This is confirmed by the time-series of u θ at a fixed probe location in the computational domain (Figure 3b). The final state of this run is visualized in figure 3c. It shows a three-dimensional rendering of a T = 0 iso-surface generated with ParaView . The spiral flow pattern rotates at a constant speed without changing its shape; like a barber pole. It passes repeatedly through the probe location, what explains why the u θ signal reaches a periodic state (2 5 . d 2 / ν &gt; t &gt; 3 5 . d 2 / ν ). Its constant shape explains why the integral heat flux ( Nu i ), on the other hand, remains constant at the same time. The second and third DNS were initialized with the final state of the former runs and by increasing the Rayleigh number to Ra = 2840 and 3550, respectively. The time series and final states in figure 3 reveal, that the TC system undergoes a sequence of transitions to different flow states with increasing spatio-temporal complexity as Ra increases. These, and other illustrative examples, are documented in the tutorial section of the user guide.

## 4. Impact

Our software completes the list of publicly available Navier-Stokes solvers for the three most common prototypes of wall-bounded shear flow: plane

Couette flow ( channelflow [16]), pipe flow ( openpipeflow [11]) and TaylorCouette flow ( nsCouette ). It can be quickly installed and productively used by researchers interested in pattern formation and chaos, for which TC flow has long been a paradigm [17]. The example of section 3 can be run in a laptop and is meant to illustrate how easy results can be obtained and analyzed.

We however stress that the main aim of providing nsCouette is to make the full potential of a massively-parallel and highly-scalable DNS code readily usable and, therefore, valuable for the broad scientific community. It enables users with little experience in HPC and DNS to easily perform highlyresolved simulations of turbulence. Among other software design choices, this was achieved by providing a user friendly build process that supports many pre-configured HPC architectures, easy runtime handling of all control parameters, tailored visualization tools, a comprehensive user guide and thorough internal quality assurance. Thus, nsCouette is a powerful tool to study highly-turbulent shear flows. For example, it has already contributed to a better understanding of astrophysical [18] and geophysical [19, 20] flows.

Because of its modular structure, which closely follows the numerical formulation, and its moderate code complexity (in particular the HPS), new functionalities can be easily added to nsCouette , with given algorithmic domain knowledge and basic programming skills. As an example for this, we also provide nsPipe ; a modified version to simulate flows in straight pipes. It follows the numerical formulation of openpipeflow [11] and uses the parallel infrastructure of nsCouette , as detailed in the user guide. Extensions including the modelling of polymer additives [21] and two-phase flows [22] have already been developed and tested and will be released in the future.

## Acknowledgements

This work was supported by the Max Planck Society and partially funded by the German Research Foundation (DFG) through the priority programme Turbulent Superstructures (SPP1881). AVM was supported by the European Research Council (ERC) through the COTURB project (ERC-2014.AdG669505). Computational resources provided by the following institutions are gratefully acknowledged: The Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility (DE-AC02-06CH11357). The Isambard UK National Tier-2 HPC Service operated by GW4 and the UK Met Office, and funded by EPSRC (EP/P020224/1). Further computations were performed on the HPC systems Hydra, Draco and Cobra at the MPCDF in Garching.

## References

## References

- [1] Y. Bazilevs, I. Akkerman, Large eddy simulation of turbulent TaylorCouette flow using isogeometric analysis and the residual-based variational multiscale method, Journal of Computational Physics 229 (9) (2010) 3402-3414. doi:10.1016/j.jcp.2010.01.008 .
- [2] S. Grossmann, D. Lohse, C. Sun, High-Reynolds Number TaylorCouette Turbulence, Annual Review of Fluid Mechanics 48 (1) (2016) 53-80. doi:10.1146/annurev-fluid-122414-034353 .
- [3] E. M. Edlund, H. Ji, Nonlinear stability of laboratory quasi-Keplerian flows, Physical Review E 89 (2) (2014) 021004. doi:10.1103/PhysRevE. 89.021004 .
- [4] J. M. Lopez, M. Avila, Boundary-layer turbulence in experiments on quasi-Keplerian flows, Journal of Fluid Mechanics 817 (2017) 21-34. doi:10.1017/jfm.2017.109 .
- [5] L. Shi, M. Rampp, B. Hof, M. Avila, A hybrid MPI-OpenMP parallel implementation for pseudospectral simulations with application to Taylor-Couette flow, Computers &amp; Fluids 106 (2015) 1-11. doi: 10.1016/j.compfluid.2014.09.021 .
- [6] A. Guseva, A. P. Willis, R. Hollerbach, M. Avila, Transition to magnetorotational turbulence in Taylor-Couette flow with imposed azimuthal magnetic field, New Journal of Physics 17 (9) (2015) 093018. doi:10.1088/1367-2630/17/9/093018 .
- [7] J. M. Lopez, F. Marques, M. Avila, The Boussinesq approximation in rapidly rotating flows, Journal of Fluid Mechanics 737 (2013) 56-77. doi:10.1017/jfm.2013.558 .
- [8] E. Gabriel, G. E. Fagg, G. Bosilca, T. Angskun, J. J. Dongarra, J. M. Squyres, V. Sahay, P. Kambadur, B. Barrett, A. Lumsdaine, R. H. Castain, D. J. Daniel, R. L. Graham, T. S. Woodall, Open MPI: Goals, Concept, and Design of a Next Generation MPI Implementation, in: D. Kranzlm¨ uller, P. Kacsuk, J. Dongarra (Eds.), Recent Advances in Parallel Virtual Machine and Message Passing Interface. EuroPVM/MPI 2004. Lecture Notes in Computer Science, Vol. 3241, Springer, Berlin, Heidelberg, 2004, pp. 97-104. doi:10.1007/ 978-3-540-30218-6\_19 .

| [9]   | M. Frigo, S. Johnson, The Design and Implementation of FFTW3, Pro- ceedings of the IEEE 93 (2) (2005) 216-231. doi:10.1109/JPROC.2004. 840301 .                                                                                                                      |
|-------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [10]  | M. Rampp, J. M. Lopez, L. Shi, B. Hof, M. Avila, Extreme scaling of nsCouette, a pseudospectral DNS code, in: INSIDE: Innovative Super- computing in Germany, Vol. 12, 2014, pp. 48-50.                                                                              |
| [11]  | A. P. Willis, The Openpipeflow Navier-Stokes solver, SoftwareX 6 (2017) 124-127. doi:10.1016/j.softx.2017.05.003 .                                                                                                                                                   |
| [12]  | G. P. King, W. Lee, Y. Li, H. L. Swinney, P. S. Marcus, Wave speeds in wavy Taylor-vortex flow, Journal of Fluid Mechanics 141 (1984) 365- 390. doi:10.1017/S0022112084000896 .                                                                                      |
| [13]  | M. Ali, P. D. Weidman, On the stability of circular Couette flow with radial heating, Journal of Fluid Mechanics 220 (1990) 53-84. doi:10. 1017/S0022112090003184 .                                                                                                  |
| [14]  | D. A. Howey, P. R. N. Childs, A. S. Holmes, Air-Gap Convection in Ro- tating Electrical Machines, IEEE Transactions on Industrial Electronics 59 (3) (2012) 1367-1375. doi:10.1109/TIE.2010.2100337 .                                                                |
| [15]  | J. M. Lopez, F. Marques, M. Avila, Conductive and convective heat transfer in fluid flows between differentially heated and rotating cylin- ders, International Journal of Heat and Mass Transfer 90 (2015) 959- 967. doi:10.1016/j.ijheatmasstransfer.2015.07.026 . |
| [16]  | J. F. Gibson, Channelflow: A spectral Navier-Stokes simulator in C++, Tech. rep., University of New Hampshire (2014). URL www.channelflow.org                                                                                                                        |
| [17]  | M. A. Fardin, C. Perge, N. Taberlet, 'The hydrogen atom of fluid dy- namics' - introduction to the Taylor-Couette flow for soft matter sci- entists, Soft Matter 10 (20) (2014) 3523. doi:10.1039/c3sm52828f .                                                       |
| [18]  | L. Shi, B. Hof, M. Rampp, M. Avila, Hydrodynamic turbulence in quasi- Keplerian rotating flows, Physics of Fluids 29 (4) (2017) 044107. doi: 10.1063/1.4981525 .                                                                                                     |
| [19]  | C. Leclercq, J. L. Partridge, P. Augier, C.-C. P. Caulfield, S. B. Dalziel, P. F. Linden, Nonlinear waves in stratified Taylor-Couette flow. Part 1. Layer formation. arXiv:1609.02885 .                                                                             |

- [20] C. Leclercq, J. L. Partridge, C.-C. P. Caulfield, S. B. Dalziel, P. F. Linden, Nonlinear waves in stratified Taylor-Couette flow. Part 2. Buoyancy flux. arXiv:1609.02886 .
- [21] J. M. Lopez, G. H. Choueiri, B. Hof, Dynamics of viscoelastic pipe flow at low Reynolds numbers in the maximum drag reduction limit, Journal of Fluid Mechanics 874 (2019) 699-719. doi:10.1017/jfm.2019.486 .
- [22] B. Song, C. Plana, J. M. Lopez, M. Avila, Phase-field simulation of coreannular pipe flow, International Journal of Multiphase Flow 117 (2019) 14-24. doi:10.1016/j.ijmultiphaseflow.2019.04.027 .

## Required Metadata

Current code version

Table 1: Code metadata (mandatory)

| Nr.   | Code metadata description                                         | Please fill in this column                                                                                                                                                                                                                                                                                                                                               |
|-------|-------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| C1    | Current code version                                              | 1.0                                                                                                                                                                                                                                                                                                                                                                      |
| C2    | Permanent link to code/repository used for this code version      | https://github.com/dfeldmann/ nsCouette                                                                                                                                                                                                                                                                                                                                  |
| C3    | Legal Code License                                                | GPLv3                                                                                                                                                                                                                                                                                                                                                                    |
| C4    | Code versioning system used                                       | git                                                                                                                                                                                                                                                                                                                                                                      |
| C5    | Software code languages, tools, and services used                 | Fortran , C (for some housekeeping tasks), MPI , OpenMP                                                                                                                                                                                                                                                                                                                  |
| C6    | Compilation requirements, operat- ing environments & dependencies | Developed and tested under Linux and IBM AIX. Compiler: A Fortran 2003 compiler which is OpenMP-3 compliant, a basic C compiler, an MPI library with support for MPI THREAD SERIALIZED , a serial BLAS / LAPACK library, a serial but fully thread-safe FFTW3 installation or equivalent, for output and visu- alization (optional): an MPI -parallel HDF5 installation. |
| C7    | Link to developer documenta- tion/manual                          | https://gitlab.mpcdf.mpg.de/ mjr/nscouette                                                                                                                                                                                                                                                                                                                               |
| C8    | Support email for questions                                       | nsCouette@zarm.uni-bremen.de                                                                                                                                                                                                                                                                                                                                             |
Jose Manuel Lopez Daniel Feldmann Markus Rampp Alberto Vela-Martin Liang Shi Marc Avila
nsCouette -- A high-performance code for direct numerical simulations of turbulent Taylor-Couette flow
We present nsCouette, a highly scalable software tool to solve the
Navier-Stokes equations for incompressible fluid flow between differentially
heated and independently rotating, concentric cylinders. It is based on a
pseudospectral spatial discretization and dynamic time-stepping. It is
implemented in modern Fortran with a hybrid MPI-OpenMP parallelization scheme
and thus designed to compute turbulent flows at high Reynolds and Rayleigh
numbers. An additional GPU implementation (C-CUDA) for intermediate problem
sizes and a basic version for turbulent pipe flow (nsPipe) are also provided.