INCOMPACT3D USER GUIDE version 2.

0
Sylvain Laizet (Imperial College London)

Contents
1 Overview of Incompact3d

2 Installa on
2.1 Compiling incompact3d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Known issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2
3
3

3 The code
3.1 incompact3d.f90 . . . . . . . . . . . . .
3.2 incompact3d.prm . . . . . . . . . . . . .
3.3 parameters.f90 . . . . . . . . . . . . . .
3.4 variable.f90 . . . . . . . . . . . . . . . .
3.5 schemes.f90and derive.f90 . . . . . . . .
3.5.1 Spa al discre za on . . . . . . .
3.5.2 Matrix formula on . . . . . . . .
3.6 tools.f90 . . . . . . . . . . . . . . . . . .
3.6.1 Subrou ne test_speed_min_max
3.6.2 Subrou ne test_scalar_min_max
3.6.3 Subrou ne restart . . . . . . . .
3.6.4 Subrou ne stretching . . . . . . .
3.6.5 Inversion subrou nes . . . . . . .
3.7 decomp_2d.f90 . . . . . . . . . . . . . .
3.8 convdi.f90 . . . . . . . . . . . . . . . .
3.9 navier.f90 . . . . . . . . . . . . . . . . .
3.9.1 Subrou ne in . . . . . . . . . .
3.9.2 Subrou ne corgp . . . . . . . . .
3.9.3 Subrou nes inow and ou low .
3.9.4 Subrou ne ecoule . . . . . . . .
3.9.5 Subrou ne init . . . . . . . . . .
3.9.6 Subrou ne divergence . . . . . .
3.9.7 Subrou ne gradp . . . . . . . . .
3.9.8 Subrou ne corgp_IBM . . . . . .
3.9.9 Subrou ne body . . . . . . . . .
3.9.10 Subrou ne pre_correc . . . . . .
3.10 poisson.f90 . . . . . . . . . . . . . . . .
3.11 visu.f90 . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

3
3
4
6
6
6
7
7
9
9
9
9
9
10
10
10
10
10
10
11
11
11
11
11
11
11
11
12
12

Overview of Incompact3d

Incompact3d is a powerful high order ow solver for academic research. It combines the versa lity of industrial codes with
the accuracy of spectral codes. Thank to a successful so ware development project with NAG and HECToR (UK's old na onal
supercompu ng facility), Incompact3d can be used on up to one million computa onal cores to solve the incompressible NavierStokes equa ons. This high level of parallelisa on is achieved thanks to a highly scalable 2D decomposi on library and a distributed
Fast Fourier Transform (FFT) interface. This library is available at h p://www.2decomp.org and can be freely used for your own
solver.
Incompact3d is based on a Cartesian mesh. The use of such a simplied mesh oers the opportunity to implement high-order
compact schemes for the spa al discre za on whilst a customized Immersed Boundary Method (IBM) allows the implementa on
of any solid wall/blu body geometry inside the computa onal domain. The main originality of the solver is that the Poisson
equa on (to ensure the incompressibility) is fully solved in the spectral space via the modied wave number formalism, no ma er
what the boundary condi ons are (periodic, free-slip, no-slip, inow/ou low, etc.). Note nally that the pressure mesh is staggered from the velocity one by half a mesh to avoid spurious pressure oscilla ons that can be introduced by the IBM.
More informa on about the numerical methods can be found in:
Laizet S.& Lamballais E., High-order compact schemes for incompressible ows: a simple and ecient method with the
quasi-spectral accuracy, J. Comp. Phys., vol 228-15, pp 5989-6015, 2009
Lamballais E., Fortune V. & Laizet S., Straigh orward high-order numerical dissipa on via the viscous term for Direct and
Large Eddy Simula on, J. Comp. Phys., Vol 230-9, pp 3270-3275, 2011
More informa on about the parallel strategy of the code can be found in:
Laizet S.& Li N., Incompact3d, a powerful tool to tackle turbulence problems with up to 0(105 ) computa onal cores, Int.
J. of Numerical Methods in Fluids, Vol 67-11, pp 1735-1757, 2011
Li N. & Laizet S., 2DECOMP&FFT - a highly scalable 2D decomposi on library and FFT interface, Cray User Group mee ng:
Simula on comes of age Edinburgh, Scotland -- 24/05-27/05, 2010
Laizet S., Lamballais E. & Vassilicos J.C., A numerical strategy to combine high-order schemes, complex geometry and
parallel compu ng for high resolu on DNS of fractal generated turbulence, Computers & Fluids, vol 39-3, pp 471-484,
2010
IMPORTANT:
It is strongly recommended to read the previous references before star ng using Incompact3d.
We kindly ask you to cite the previous references (when suitable) in your work based on Incompact3d.

Installa on

When star ng to use Incompact3d on your local worksta on, it is recommended to install gfortran and openmpi (preferably
with an Ubuntu distribu on). It is possible to download the latest archive for openmpi from the address h p://www.openmpi.org/so ware/ompi/v1.10/. See below for the installa on of gfortran and openmpi:
sudo apt-get install build-essential
sudo apt-get install gfortran
tar -xvzf openmpi-***.tar.gz
cd openmpi***
sudo ./configure --prefix=/usr/local F77=gfortran FC=gfortran
sudo make all install
It is also necessary to add a line in the .bashrc le:
export LD_LIBRARY_PATH=/usr/local/lib/

2.1

Compiling incompact3d

A Makele is available and the only thing to do is to select the relevant compiler. For the combina on gfortran+openmpi, it is:
FC = mpif90
OPTFC = -O3 -funroll-loops -ftree-vectorize -fcray-pointer -cpp
CC = mpicc
CFLAGS = -O3
To compile the code, you just need to type make clean and make

2.2

Known issues

Some users have experienced dicul es with the MPI librairy mpich. Other than that Incompact3d has been used on a wide
range of supercomputers with various compilers and MPI librairies.
It is strongly recommended to carefully read the documenta on of the supercomputer you will be using. You should be able
to nd all the informa on you need about the op mum compiler op ons.

The code

In this sec on, only the les related to the Navier-Stokes equa ons are described. The 2D domain decomposi on les are derived
from the open source 2DECOMP&FFT library and are not explained in this user guide.

3.1

incompact3d.f90

This is the main le containing the me loop rela ve to the evalua on of the incompressible Navier-Stokes equa ons.

Figure 1: Stucture of Incompact3d with the 2D domain decomposi on.

The structure of the solver is presented in gure 1. This gure also shows the management of the pencils swaps. For the
Poisson solver in the spectral space, a single division is required and the modied wave numbers combined with the transfer funcons are all independent with each other. Note that, when performing the 3D FFT forward, we are in Z-pencils in the physical
space, then in X-pencils in the spectral space and nally, in Z-pencils again a er the 3D FFT backward, in order to reduce the global
transpose opera ons. When using tri-periodic boundary condi ons, 55 global transpose opera ons need to be performed at each
me step. This number can be up to 69 depending on the boundary condi ons.
As it can be seen, the code is using various modules (for the constants, 1D and 2D arrays). All the modules can be found in
module_param.f90.
The variables t1 and t2 are used to compute the average cost per me step of a simula on.
It is possible to save the sta s cs and the visualiza ons on a coarser mesh (every nstat and nvisu mesh nodes respec vely).
call init_coarser_mesh_statS(nstat,nstat,nstat,.true.)
call init_coarser_mesh_statV(nvisu,nvisu,nvisu,.true.)
The sta s cs and the visualiza ons are dealed with in visu.f90.
The parameters of the simula ons are ini alized with
call parameter()
call schemes ()
Note that because the pressure eld is staggered by half a mesh with respect to the velocity eld it is necessary to dene various
mesh sizes (for the interpola ons) depending on the boundary condi ons of the simula on (variable phG).
The following rou nes are called to ini alize the ow eld. It is possible to use a checkpoint procedure in order to restart a
simula on from a previous one (parameter ilit). The restart procedure can be found in tools.f90 and the restart le is called
sauve.dat. Note that for some supercomputers (usually the IBM ones) and for very large simula ons, you may have to split
sauve.dat in smaller les as some supercomputers do not deal with very large les.
if (ilit==0) call init(ux1,uy1,uz1,ep1,phi1,gx1,gy1,gz1,phis1,hx1,hy1,hz1,phiss1)
if (ilit==1) call restart(ux1,uy1,uz1,ep1,pp3,phi1,gx1,gy1,gz1,&
px1,py1,pz1,phis1,hx1,hy1,hz1,phiss1,phG,0)
call test_speed_min_max(ux1,uy1,uz1)
if (iscalar==1) call test_scalar_min_max(phi1)

3.2

incompact3d.prm

The name of each variable is clearly explained in this le. All the quan

es are non-dimen onalised with a length and a velocity.

The domain size is xlx yly zlz.

There is no check on the me step so it is your responsability to carefully inves gate about the stability condi ons for your simula on. The op mum me step has to be dened with a trial-and-error procedure. It is possible to use explicit Adam-Bashforth
schemes (recommended when Immersed Boundary Methods are used) or explicite Adam-Bashforth schemes (parameter ncheme).
Note that in some versions of the code, a semi-implicit strategy has been implemented in order to relax the stability condi ons
for the me step.
The me step is xed for a given simula on. It is not possible (yet?) to use a dierent me step at each me step.
Dierent boundary condi ons can be used in the three spa al direc on ( nclx,nclynclz) as seen in gure 2:
Periodic condi ons corresponding to ncl = 0
Free-slip condi ons corresponding to ncl = 1
Open Boundary condi ons (Dirichlet condi ons for the velocity for no-slip or inow/ou low condi ons), corresponding to
ncl = 2
4

BC = 0 PERIODIC
BC = 1 FREE SLIP
BC = 2 NO SLIP, INFLOW/OUFLOW
10 different sets of boundary conditions in Incompact3d

0-0-0
1-0-0
2-0-0
0-2-0
1-1-0
2-1-0
1-1-1
2-2-1
1-2-1
2-2-2

Y-direction

Z-direction

X-direction

-Possibility to add a passive scalar

-Possibility to add a polymer subroutine
-Possibility to add a particle subroutine

Non-uniform mesh in the Y-direction

STR = 1 central mesh refinement

STR = 2 near the boundaries refinement
STR = 3 near the bottom boundary refinement

Figure 2: Descrip on of the dierent boundary condi ons.

So far and as can been in the gure above, 12 dierent combina ons can be used in the code, covering a wide range of ow
congura on: (0 0 0), (1 0 0), (2 0 0), (0 2 0), (1 1 0), (2 1 0), (1 1 1), (2 2 1), (2 1 2),
(1 2 1), (1 1 2) and (2 2 2).
For instance the combina on for a temporal channel ow is (0 2 0). For a spa ally evolving ow, with periodic boundary condi ons in the spanwise direc on and free-slip boundary condi ons in the lateral direc on, the combina on is (2 1 0).
All the combina ons have not yet been tested so you may eventually experience some problems.
At the moment, the same boundary condi ons have to be imposed in a given direc on.
Some coding is needed if you want to impose let's say a no-slip and a free slip boundary condi on in a spa al direc on (for a
turbulent boundary layer for example).
It you want to use a passive scalar with dierent boundary condi ons than the velocity eld ones, some coding is needed: you
will have to create new deriva ve subrou nes with the correct set of coecients depending on the boundary condi ons.
It is possible to use a stretched mesh in the ydirec on with the parameter istret. If istret = 0, no stretched mesh is used.
If istret = 1 then a stretching is used with a mesh renement in the centre of the computa onal domain. If istret = 2 then a
stretching is used with a mesh renement at the boundaries of the computa onal domain. Finally, if istret = 3 then a stretching
is used with a mesh renement only at the bo om boundary of the computa onal domain. istret = 2 can be used for a turbulent channel ow, istret = 3 can be used for a turbulent boundary layer. The parameter beta is the stretching parameter (small
values for strong stretching and large values for almost regular mesh).
It is possible to choose dierent ow congura on with the parameter itype but they are not all implemented yet. Dening
a ow congura on (and a ini al and/or inlet condi on) can be done in the subrou nes init and ecoule.
The parameters ifirst and ilast correspond respec vely to the rst and last itera on. As there is a checkpoin ng procedure, it is possible to split a simula on in small ones.

Two dierent formula ons can be used for the convec ve terms of the Navier-Stokes equa on:
(
)
uj
ui
the rota onal formula on Hi = uj x

corresponding to iskew = 0,
x
j
i
the skew-symmetric formula on Hi =

1
2

ui uj
xj

ui
+ uj x
j

)
corresponding to iskew = 1.

We recommend to use the skew-symmetric formula on for a be er modelisa on of the small scales and less aliasing.
It is possible to combine Incompact3d with Immersed Boundary Methods (IBM). For the latest IBM implemented in the solver,
please have a look at:
Gau er R., Laizet S. & Lamballais E., 2014, A DNS study of jet control with microjets using an alterna ng direc on forcing
strategy, Int. J. of Computa onal Fluid Dynamics, 28, 393--410

3.3

parameters.f90

This le contains the subrou ne parameter which is used for the ini alisa on of the simula on (for instance, the me coecients, the mesh sizes x, y and z).
The subrou ne parameter read the parameter le incompact3d.prm. If a stretching is used in the y-direc on then the subrou ne stretching is called to dene the 1D array yp which contains the coordinates of the mesh in the y-direc on.

3.4

variable.f90

The 3D arrays are dynamically dened in this le, depending on the size of the simula on and the number of computa onal cores.
The 3D arrays with a 1 at the end (resp. a 2, a 3) are dened in X-pencils (resp. in Y-pencils, Z-pencils). For the 3D arrays in the
X-pencils, two op ons are available. size can be used if there is no need to use the coecients i, j, k in a subrou ne (for example
in the me advancement subrou ne intt). For instance, to store the velocity uy at the previous me step, the array gy is dened with allocate(gy(xsize(1), xsize(2), xsize(3) whereas uy1 is dened with allocate(uy1(xstart(1) : xend(1), xstart(2) :
xend(2), xstart(3) : xend(3))). In pra ce, both arrays have the same size but the coecients i, j, k will be dierent.
EXAMPLE:
A simula on is based on nx ny nz = 129 64 32 mesh nodes using a 2D mapping prow pcol = 4 8=32 computa onal
cores, ranking from 0 to 31.
For this congura on, xsize(1) = nx , xsize(2) = ny /prow = 16, xsize(3) = nz /pcol = 4.
The size of gy is gy(129, 16, 4) with i = 1, .., 129, j = 1, .., 16 and k = 1, .., 4 for all the ranks.
The size of uy is uy(129, 16, 4) with i = 1, .., 129 for all the ranks; j = 1, .., 16 for ranks 0 to 7; j = 17, .., 32 for ranks 8 to
15; j = 33, .., 48 for ranks 16 to 23 and j = 49, .., 64 for ranks 24 to 31. k = 1, .., 4 for ranks 0/8/16 and 24; k = 5, .., 8 for ranks
1/9/17 and 25; k = 9, .., 12 for ranks 2/10/18 and 26; k = 13, .., 16 for ranks 3/11/19 and 27; k = 17, .., 20 for ranks 4/12/20 and
28; k = 21, .., 24 for rank 5/13/21 and 29; k = 25, .., 28 for ranks 6/14/22 and 30 and nally k = 29, .., 32 for ranks 7/15/23 and
31.
When developing your own subrou ne in the code or before star ng a simula on, it is recommended to check/print the
size, start and end parameters for a be er understanding of the 2D mapping decomposi on.

3.5

schemes.f90 and derive.f90

The coecients for the sixth-order schemes are dened in schemes.f90. 1D arrays are used to dene the dierent matrices related to the computa on of rst and second deriva ves. The deriva ves and interpola ons subrou nes are dened in
derive.f90.
More details are given in the following. The main reference for compact schemes (deriva ve, mid-point interpola on, for
dierent boundary condi ons) is:
K.S. L , 1992, Compact nite dierence schemes with spectral-like resolu on, Journal of computa onal physics, 103(1),
16-42.

3.5.1 Spa al discre za on

Let us consider a uniform distribu on of nx nodes xi on the domain [0, Lx ] with xi = (i1)x for 1 i nx . The approxima on
of values fi = f (xi ) of the rst deriva ve f (x) of the func on f (x) can be related to values fi = f (xi ) by a nite dierence
scheme of the form
fi+1 fi1
fi+2 fi2

fi1
+ fi + fi+1
=a
+b
(1)
2x
4x
By choosing = 1/3, a = 14/9 and b = 1/9, this approxima on is sixth-order accurate while having a so-called ``quasi-spectral
behaviour'' due to its capabili es to represent accurately a wide range of scales.
Analogous rela ons can be established for the approxima on of the second deriva ve values fi = f (xi ) with

fi1
+ fi + fi+1
=a

fi+1 2fi + fi1

fi+3 2fi + fi3
fi+2 2fi + fi2
+c
+b
x2
4x2
9x2

(2)

By choosing = 2/11, a = 12/11, b = 3/11 and c = 0, this approxima on is sixth-order accurate. To control the aliasing errors
via the viscous term, this type of schemes can also be dened to be over-dissipa ve at the highest wave numbers (in the spectral
range where even a high-order nite dierence scheme becomes inaccurate). More details can be found in:
L
E., F
V. L
S., 2011 Straigh orward high-order numerical dissipa on via the viscous term for Direct
and Large Eddy Simula on, J. Comp. Phys., 230(9), 3270-3275.
As already stressed, four dierent boundary condi ons can be considered in each spa al direc on. The periodic and free-slip
(equivalent to symmetric and an symmetric condi ons) boundary condi ons allow the use of schemes (1, 2) for all the nodes
considered. More precisely, the schemes (1, 2) just need to be relevantly modied near the borders i = 1 and i = nx through
the subs tu on of the extra-node (some mes called ``ghost'') values f0 , f1 , fnx +1 , fnx +2 by their counterparts f1 , f2 , fnx 1 ,
fnx 2 . For a periodic boundary condi on, this type of subs tu on for f , f and f can be wri en
f0 fnx , f1 fnx 1 , f0 fn x , f0 fnx ,

(3)

while symmetric and an symmetric condi ons lead to

and

f0 f2 , f1 f3 , f0 f2 , f0 f2 ,

(4)

f0 f2 , f1 f3 , f0 f2 , f0 f2 ,

(5)

respec vely. For simplicity, only rela ons on the le boundary condi on (near i = 1) are given here, their right counterparts (near
i = nx ) being easily deduced.
When no-slip or open (i.e. Dirichlet for velocity) boundary condi ons are used, nothing is assumed concerning the ow outside
the computa onal domain. Single sided formula ons are used for the approxima on of rst and second deriva ves for these types
of boundaries using rela ons of the form
f1 + 2f2

f1 + 11f2

1
(5f1 + 4f2 + f3 )
2x
1
(13f1 27f2 + 15f3 f4 )
x2

(6)

that are third-order accurate. At the adjacent nodes, because a three-point formula on must be used, Pad schemes are employed
with
1
1
f + f2 + f3
4 1
4
1
1
f + f2 + f3
10 1
10

=
=

3 f3 f1
2 2x
6 f3 2f2 + f1
5
x2

(7)

these schemes being fourth-order accurate.

3.5.2 Matrix formula on
The previous rela ons can be wri en as matrices:

Ax f =

1
Bx f
x
7

(8)

1
1

0
1
2

3
3
even function

odd function

Figure 3: symmetric and an symmetric boundary condi ons.

A xf =

where Ax , A x , Bx , B

1
B xf
x2

(9)

are matrices with nx nx coecients. f, f et f are vectors of size nx .

1. periodic case : We have fnx +1 = f1 and f0 = fnx . Ax and Bx can be wri en as:

Ax =

1
.

.
.

.
.
.

.
.
.

.
.

.
1

and

Bx =

0
a

b
a

a
0
a
.

b
a
0
.
.

b
a
.
.
b

b
.
.
a
b

.
.
0
a
b

.
a
0
a

a
b

b
a
0

2. free slip case : We can have two situa ons, either symmetric or an -symetric boundary condi ons. f is either odd or even
as shown in gure 3. When f is even we have:

Ax =

0
1
.

.
.

.
.
.

.
.
.

.
.

.
1
0

and

Bx =

0
a

b
a
.

a
0
.
.

b
a
.
.
b

b
.
.
a
b

.
.
0
a

b
a
0

.
a
b

When f is odd, we have:

Ax =

2
1
.

.
.

.
.
.

.
.
.

.
.

.
1
2

and

Bx =

0
a

2a
b
a
.

2b
a
0
.
.

3. Dirichlet case : In this case we need to use the non-centered schemes and we have:

b
a
.
.
b

b
.
.
a
b

.
.
0
a
2b

.
a
b
2a

b
a
0

1
2

Ax =

Bx =

a1
a2

1
1

2
1
.

.
.

.
.
.

.
.
nx 1

b1
0
a
.

c1
a2
0
.
.

.
1
nx

nx 1
1

a
.
.
b

b
.
.
a

.
.
0
anx 1
cnx

.
a
0
bnx

b
anx 1
anx

IMPORTANT COMMENT: Note that the parameter npaire is only ac ve when ncl=1. Please see below for the values depending
on the velocity component and the spa al direc on:
ux
uy
uz
uxux
uyuy
uzuz
uxuy
uxuz
uyuz
dux/dx
duy/dx
duz/dx
dux/dy
duy/dy
duz/dy
dux/dz
duy/dz
duz/dz

3.6

X
0
1
1
1
1
1
0
0
1
1
0
0
0
1
1
0
1
1

Y
1
0
1
1
1
1
0
1
0
1
0
1
0
1
0
1
0
1

Z
1
1
0
1
1
1
1
0
0
1
1
0
1
1
0
0
0
1

tools.f90

3.6.1 Subrou ne test_speed_min_max

This subrou ne is used to calculate maxima and minima of the velocity eld. Note that the value 1609 does not have a physical
meaning. It corresponds to the date of birth of Sylvain Laizet (16th September) and is used for the ini alisa on of the min/max
values.
3.6.2 Subrou ne test_scalar_min_max
This subrou ne is used to calculate maxima and minima of the scalar eld.
3.6.3 Subrou ne restart
This subrou ne is used to either generate a restart le or to read a restart le (parameter ilit in incompact3d.prm). As already
stated, you may have to split the restart le on IBM supercomputers when the simula on is too large.
3.6.4 Subrou ne stretching
This subrou ne is only ac ve when a stretched mesh is used in the y-direc on. It denes a stretched mesh and some 1D arrays
which are used for deriva ves and interpola ons. More details can be found in
9

L
S. L
E., 2009, High-order compact schemes for incompressible ows: a simple and ecient method with
the quasi-spectral accuracy, J. Comp. Phys., 228(15), 5989-6015.
3.6.5 Inversion subrou nes
Several subrou nes are available for the inversion of pentadiagonal matrices. It is requires when a stretched mesh is used in the
y-direc on.

3.7

decomp_2d.f90

This le is part of the 2D Decomp%FFT library which is freely available at h p://www.2decomp.org/. It implements a generalpurpose 2D pencil decomposi on into Incompact3d. The les related to this library are not explained in this document. As
a communica on library, it implements so-called 2D pencil decomposi on for par oning three-dimensional data sets on distributed systems and performing transpose-based communica on. It also provides a highly scalable and ecient interface to
perform three-dimensional Fast Fourier Transforms (FFT). The library is wri en in Fortran and built on top of MPI. It is recommended not to modify this le.

3.8

convdi.f90

The convec ve and diusive terms of the Navier-Stokes equa ons are calculated in this le. As already stated, two dierent
formula ons can be used for the convec ve terms:
(
)
uj
ui

the rota onal formula on Hi = uj x

xi corresponding to iskew = 0,
j
the skew-symmetric formula on Hi =

1
2

ui uj
xj

ui
+ uj x
j

)
corresponding to iskew = 1.

We recommend to use the skew-symmetric formula on for a be er modelisa on of the small scales and for less aliasing. Note
that at the beginning and at the end of the subrou ne, the data are stored in X-pencils.
Note that when a stretched mesh is used in the y-direc on, the evalua on of the second deriva ve is
2f
2f
=
y 2
s2

ds
dy

)2
+

f d2 s
1 2f
h f
=

s dy 2
h 2 s2
h 3 s

with
y = h(s),

0 s 1,

0 y Ly

where h(s) is the mapping from the equally spaced coordinate s to the stretched spaced coordinate y.
Large-Eddy Simula ons (LES) models should be implemented at the end of the convdiff subrou ne.
Any forcing term should also be implemented here (for instance a Coriolis forcing).

3.9

navier.f90

3.9.1 Subrou ne in
Subrou ne for the explicit me advancement of the Navier-Stokes equa on. Note that the 3D loops are fully vectorised, hence
the (ijk,1,1) nota on. Data are stored in X-pencils.
3.9.2 Subrou ne corgp
Subrou ne for the correc on step in the frac onal step method. The velocity eld is corrected by the pressure gradients in order
to become incompressible. Data are stored in X-pencils.
In the case of a temporal channel ow, it is necessary to modify the ow rate to account for the pressure loss at the wall (subrou ne
channel).

10

3.9.3 Subrou nes inow and ou low

Subrou nes only used if inow/ou low boundary condi ons are used in the streamwise direc on of the ow (nclx=2). The outow condi on is a simple 1D convec on equa on. The inow condi on can be adapted to any kind of inlet condi on via the
subrou ne ecoule.
A random noise can be added to the inlet if necessary.
3.9.4 Subrou ne ecoule
In this subrou ne, the inlet condi on is dened. At the moment, only few dierent ow congura ons are implemented (itype==1
for a constant ow eld, itype==2 for a Poisseuille prole, itype==6 for the Taylor-Green vor ces).
Note that the subrou ne ecoule is called in the subrou ne inflow at each me step.
3.9.5 Subrou ne init
Subrou ne dedicated to the ini alisa on of the simula on via the subrou ne ecoule. Data are stored in X-pencils.
Random noise can be added to the ini al condi on.
3.9.6 Subrou ne divergence
Subrou ne dedicated to the evalua on of .u when nlock=1 and to .un+1 when nlock=2.
Note that the inputs are the three components of the velocity dened in X-pencils and the output is dened in Z-pencil (rst
argument of the subrou ne). The output is dened on the staggered mesh.
3.9.7 Subrou ne gradp
Evalua on on the X-pencils of the three pressure gradients (on the velocity mesh) from the pressure eld (dened on the staggered
mesh) which was evaluated in Z-pencils.
3.9.8 Subrou ne corgp_IBM
Only used when Immersed Boundary Methods are used. It has to do with a pre- and post-correc on by the pressure gradient of
the ow inside the solid regions. More details about the method can be found in:
L
S. L
E., 2009, High-order compact schemes for incompressible ows: a simple and ecient method with
the quasi-spectral accuracy, J. Comp. Phys., 228(15), 5989-6015.
3.9.9 Subrou ne body
Only used when Immersed Boundary Methods (IBM) are used. In this subrou ne, the velocity eld is forced to zero inside the
solid regions.
Note that new IBM are currently under development. The latest method implemented in the solver is described in:
G
R., L
S. L
E. 2014 A DNS study of jet control with microjets using an alterna ng direc on forcing
strategy, Int. J. of Computa onal Fluid Dynamics, 28, 393--410
3.9.10 Subrou ne pre_correc
It is impossible to impose boundary condi ons on uk+1 because of incompressibility (the velocity eld cannot be modifoed a er
the correc on step by the pressure gradients). Therefore the imposi on of boundary condi ons is on u instead of uk+1 .
See example for a channel ow:
uwall = 0

11


uk+1
pk+1
wall = uwall

uk+1
pk+1 = 0
wall =
It is not sa sfactory. It is be er to do the following:
uwall =
pk

uk+1
pk+1
wall = uwall

uk+1
pk
pk+1 0
wall =
As it can be seen, correc ons with the pressure gradients (at the previous me step) are needed so that the boundary condions are more accurate.
Note that boundary condi ons only need to be imposed in the subrou ne pre_correc when ncl=2.
For par cular ow congura ons, it is necessary to check that the ow rate at the ou low is the same as the ow rate at the
inow. This correc on can be performed in the subrou ne pre_correc.

3.10

poisson.f90

This le has to do with the Poisson equa on which is performed in the spectral space even when Dirichlet boundary condi ons
are used for the velocity eld.
Note that the pressure eld is obtained on a staggered mesh, hence the use of a pre and post processing for the Fast Fourier
Transforms.
The modied wave numbers are dened in the subrou ne waves.
Note that the implementa on of the Poisson solver for the stretched mesh is fairly complicated and involved an inversion of
one or two penta-diagonal matrices. The theory can be found in
L
S. L
E., 2009, High-order compact schemes for incompressible ows: a simple and ecient method with
the quasi-spectral accuracy, J. Comp. Phys., 228(15), 5989-6015.
Please pay a en on in par cular to the appendix B.
It is recommended not to modify this le.
12 dierent combina ons can be used for the Poisson equa on, covering a wide range of ow congura on: (000), (100),
(2 0 0), (0 2 0), (1 1 0), (2 1 0), (1 1 1), (2 2 1), (2 1 2), (1 2 1), (1 1 2) and (2 2 2).
For the Poisson solver the case ncl=1 and ncl=2 are dealt in the same way.

3.11

visu.f90

File which contain all the post-processing subrou nes. Several tools have been developed to manage eciently Input/Output
in the code. As previously explained there is a restart procedure in the code (le tools.f90, subrou ne restart). A restart
le sauve.dat is generated every isave me step. Note that it is possible to restart the simula on with dierent number of
computa onal cores. In terms of 2D/3D snapshots, several subrou nes can be used:
2D snapshots (full resolu on):
subrou ne decomp_2d_write_plane(pencil,var,iplane,n,filename):
ipencil is equal to 1,2,3 for X-pencil, Y-pencil, Z-pencil respec vely. var is the name of the 3D array. iplane is equal to
1,2,3 to save a X-plane, Y-plane, Z-plane respec vely. n corresponds to the loca on of the plane. filename is the name of
the output le.

12

EXAMPLE:
call
decomp\_2d\_write\_plane(1,ux1,1,112,'ux2d') is going to write in ux2d a 2D (y-z)-plane of the 3D
array ux1 (dened in X-pencil), for i=112.
3D snapshots (full resolu on):
subrou ne decomp_2d_write_one(nx,ny,nz,ipencil,var,filename):
ipencil is equal to 1,2,3 for X-pencil, Y-pencil, Z-pencil respec vely. var is the name of the 3D array. filename is the
name of the output le.
EXAMPLE:
call
decomp_2d_write_one(nx,ny,nz,2,uy2,'uy2.dat') is going to write in uy2.dat the 3D array uy2 (dened
in Y-pencil).
3D snapshots (coarse resolu on):
subrou ne decomp_2d_write_one(ipencil,var,filename,icoarse):
ipencil is equal to 1,2,3 for X-pencil, Y-pencil, Z-pencil respec vely. var is the name of the 3D array. filename is the
name of the output le. icoarse is equal to nstat or nvisu (see le module_param.f90). The array var is dened on a
coarse mesh (every nstat or nvisu mesh node). Before calling the subrou ne it is necessary to call the subrou ne
fine_to_coarseV(ipencil,var_full,var_coarseV) or
fine_to_coarseS(ipencil,var_full,var_coarseS).
The size of var_coarseV is (xszV(1),xszV(2),xszV(3)).
The size of var_coarseS is (xszS(1),xszS(2),xszS(3)).
EXAMPLE:
call fine_to_coarseV(1,uz1,uvisu) will write in the coarse array uvisu the 3D arrays (full resolu on) uz1 (dened
in Z-pencil).
Then call decomp_2d_write_one(1,uvisu,'uz_coarse.dat',2) will write in uz_coarse.dat the 3D array uvisu
(dened every nvisu mesh nodes in X-pencil).
X-pencils (various resolu ons):
It is obviously possible to save data for only a few numbers of X-pencils.
EXAMPLE:
In this example, a simula on with nx ny nz = 2881 360 360 is running on 3600 computa onal cores with a 2D
mapping praw pcol = 60 60. I want to save the three components of the velocity at each me step for j = k = 181
and every 8 mesh nodes in the xdirec on. I will simply use the following lines:
if (nrank==4095) then
if (itime==ifirst) then
write (filename,923) nrank
open (nrank,file=filename,form='unformatted')
endif
write (nrank)(((ux1(i,j,k),i=1,xsize(1),8),j=1,xsize(2),2),k=1,xsize(3),2),&
(((uy1(i,j,k),i=1,xsize(1),8),j=1,xsize(2),2),k=1,xsize(3),2),&
(((uz1(i,j,k),i=1,xsize(1),8),j=1,xsize(2),2),k=1,xsize(3),2)
if (itime==ilast) close (nrank)
endif
To iden fy which computa onal core corresponds to j = k = 181, it is recommended to use the arrays xstart and xend.
A similar procedure can be used to save data in Y-pencils and Z-pencils.
sub-3D domains:
subrou ne decomp_2d_write_subdomain(ipencil,var,imin,imax,jmin,jmax,kmin,kmax,filename):
This subrou ne is only available in the latest version of the solver.
EXAMPLE:
call decomp_2d_write_subdomain(1,ux1,851,1600,311,411,311,511,'ux_sub.dat') will write in ux_sub.dat
part of the 3D array ux1 for i = 851, 1600, j = 311, 411 and k = 311, 511

13