JID: CAF

ARTICLE IN PRESS [m5G;April 18, 2016;19:59]

Computers and Fluids 0 0 0 (2016) 1–10

Contents lists available at ScienceDirect

Computers and Fluids

journal homepage: www.elsevier.com/locate/compfluid

Parallel partitioned coupling analysis system for large-scale

incompressible viscous fluid–structure interaction problems
Tomonori Yamada a, Giwon Hong b,∗, Shunji Kataoka c, Shinobu Yoshimura b
a
RACE, The University of Tokyo, 5-1-5 Kashiwanoha, Kashiwa-shi, Chiba 277-8568, Japan
b
School of Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan
c
JGC Corporation, 2-3-1 Minato Mirai, Nishi-ku, Yokohama 220-6001, Japan

a r t i c l e i n f o a b s t r a c t

Article history: Fluid–structure interaction (FSI) affects the dynamic characteristics and behaviors of the fluid and struc-
Received 2 October 2015 ture, and being able to understand and solve FSI problems is very important in engineering, science,
Revised 1 March 2016
medicine, and everyday life. Generally, FSI problems are simulated by either monolithic or partitioned
Accepted 28 March 2016
methods. There are still many challenges in the development of better numerical methods for FSI prob-
Available online xxx
lems in terms of accuracy, problem scale, stability, robustness, and efficiency. We have focused on par-
Keywords: titioned methods because they allow the use of existing flow and structural analysis solvers without
Fluid–structure interaction (FSI) elaborate modification. This paper describes the development of a parallel partitioned coupling analysis
Iterative partitioned techniques system for large-scale FSI problems. In this study, we employed the existing flow and structural analy-
Parallel coupling analysis system sis solvers FrontFlow/blue (FFB) and ADVENTURE_Solid, respectively, both of which are general-purpose
Flapping motion of elastic plate codes used to solve large-scale analysis models ranging from millions to billions of degrees of freedom
(DOFs). In addition, we developed a parallel coupling tool called ADVENTURE_Coupler to efficiently han-
dle the exchange of interface variables in various parallel computing environments. To achieve the ro-
bust and fast convergence of the fixed-point iteration, we employed Broyden’s method, which is a quasi-
Newton method, to update the interface variables. We verified the accuracy and fundamental perfor-
mance of the developed FSI analysis system by using it to solve a FSI benchmark problem: the vortex-
induced oscillation of a flexible plate in the wake of a square column. The results agreed quantitatively
well with other researchers’ results. Finally, we successfully applied the system to the analysis of the
three-dimensional flapping motion of an elastic rectangular plate, with the objective of furthering the
research and development of micro air vehicles (MAVs) with flapping wings.
© 2016 Published by Elsevier Ltd.

1. Introduction freedom (DOFs) of models used to analyze real-world problems.

For the above reasons, the development of an efficient and robust
Fluid–structure interaction (FSI) is an interdependent phe- analysis system to solve large-scale FSI problems in parallel com-
nomenon between a fluid and structural parts that affects the dy- puting environments is considered a challenging task.
namic characteristics and behaviors of the fluid and structure. As Generally, coupled phenomena can be simulated by either
there are many FSI problems in engineering, science, medicine, and monolithic or partitioned methods. In monolithic methods, the
everyday life, we recognize that it is very important to understand equations describing the fluid and the structure are formed into
and solve FSI problems. Many researchers have attempted to de- a single system of equations, allowing the equations to be solved
velop numerical methods of solving FSI problems with improved simultaneously. Conversely, the equations are solved separately in
accuracy, problem scale, stability, robustness, and efficiency (e.g., partitioned methods.
[1–32,33,34–56,57,58–66]). Although previous studies have made Monolithic approaches [34–39] are generally known to be ac-
significant progress, there are still demands for further studies on curate and robust and can be applied to solve strongly coupled
how to decrease computational time and increase the degrees of problems, especially large added-mass effect problems. However,
the DOFs of equations solved in the monolithic methods tend to
∗
be larger than those in the partitioned methods. Hence, algebraic
Corresponding author. Tel.: +818033788050.
splitting [40,42] or substructuring [36,41] has been proposed to
E-mail addresses: yamada@race.u-tokyo.ac.jp (T. Yamada), g_hong@save.sys.t.u-
tokyo.ac.jp (G. Hong), kataoka.shunji@jgc.com (S. Kataoka), yoshi@sys.t.u-tokyo.ac.jp avoid the larger DOFs of equations. Also, the monolithic methods
(S. Yoshimura). require efficient iterative linear algebraic solvers rather than direct

http://dx.doi.org/10.1016/j.compfluid.2016.03.030
0045-7930/© 2016 Published by Elsevier Ltd.

Please cite this article as: T. Yamada et al., Parallel partitioned coupling analysis system for large-scale incompressible viscous fluid–
structure interaction problems, Computers and Fluids (2016), http://dx.doi.org/10.1016/j.compfluid.2016.03.030
JID: CAF
ARTICLE IN PRESS [m5G;April 18, 2016;19:59]

2 T. Yamada et al. / Computers and Fluids 000 (2016) 1–10

solvers. This necessitates the new development of effective pre-

conditioners, such as the segregated equation solver for FSI (SES-
FSI) proposed by Tezduyar and Sathe [14,28,30], a nested iterative
scheme proposed by Manguoglu et al. [43–45] for FSI problems, an
extension of the balancing domain decomposition (BDD) method
proposed by Minami et al. [46], and an extension of the finite el-
ement tearing interconnection (FETI) method proposed by Mandel
[47] for acoustic FSI (AFSI) problems.
Conversely, the partitioned methods [48–56,57,58–66] use effi-
cient techniques, such as fluid and structural equations, to solve
each phenomenon, possibly without the elaborate modification of
existing analysis solvers. Nowadays, iterative partitioned methods
Fig. 1. Basic FSI problem.
have attracted a great deal of attention because they can im-
prove accuracy and robustness in comparison with simple parti-
tioned methods. In iterative partitioned methods, fluid and struc- reference
 
tural problems are solved separately and iteratively by fixed-point ∂u  
iteration at each time step until the conditions of equilibrium and ρ F
ˆ · ∇ u − ∇ · σ F = fF
+ u−u (1)
∂t
continuity are satisfied.
In our previous study [50], we developed a new parallel dy- and the continuity equation
namic response analysis system of AFSI problems using iterative ∇ · u = 0, (2)
partitioned coupling methods, assuming incompressible inviscid
where ρ F is the density of the fluid, u is the velocity of the fluid,
flow. The system consists of independently developed acoustic flow
ˆ is the mesh velocity, σ F is the stress tensor of the fluid, and fF
u
and structural analysis solvers and a coupler. The coupler supports
is the external force applied to the fluid. Eq. (1) is derived by sub-
data exchange and the execution of several coupling algorithms.
stituting the following stress tensor σ F of the fluid into the Cauchy
Through the analysis of a number of numerical examples, it was
momentum equation:
demonstrated that the system could be applied to large-scale AFSI
problems with more than 10 million DOFs. σ F = −pI + 2μD, (3)
In this paper, we extend the above mentioned parallel-coupled where p is the pressure of the fluid, I is the unit tensor, μ is the
analysis system to the analysis of incompressible viscous FSI prob- viscosity of the fluid, and D is the strain rate tensor given by
lems with new developments. To solve problems with moving
1 
boundaries and the accompanying mesh distortion, we imple- D= ∇ u + ∇ uT . (4)
mented the ALE method and mesh control into the parallel flow 2
analysis solver FrontFlow/blue developed by Kato et al. [67,68]. 2.2. Structure domain
The employed parallel structural analysis solver is the open-source
software ADVENTURE_Solid developed by Yoshimura et al. [69,70]. The structure domain is governed by the Cauchy momentum
We employed Broyden’s method [72], which is a quasi-Newton equation of the Saint Venant–Kirchhoff model considering geomet-
method, as the iterative partitioned algorithm, together with a ric nonlinearity:
line-search technique.
∂ 2d
To verify the performance of the developed FSI analysis sys- ρS − ∇ · σ S = fS , (5)
tem, we analyzed a benchmark FSI problem: the vortex-induced
∂t2
oscillation of a flexible plate in the wake of a square column. We where ρ S is the density of the structure, d is the displacement of
then applied the system to the analysis of the three-dimensional the structure, σ S is the stress tensor of the structure, and fS is the
flapping motion of an elastic rectangular plate with the objective external force applied to the structure. σ S is given by
of furthering the research and development of micro air vehicles σ S = E : S, (6)
(MAVs) with flapping wings.
where E is the elasticity tensor of the structure and S is the strain
The remainder of this paper is organized as follows. In Chapter
tensor of the structure.
2, the governing equations of FSI problems are explained. The it-
erative partitioned coupling algorithm is described in Chapter 3. In 2.3. Interface between fluid and structure domains
Chapter 4, we present the parallel coupling system for large-scale
FSI problems. In Chapter 5, we verify the accuracy and fundamen- At the FSI interface  FSI , the following conditions of continuity
tal performance of the developed system by solving a benchmark and stress equilibrium must be fulfilled:
problem. In Chapter 6, we describe the application of the system 
to a practical problem. Finally, we provide conclusions in Chapter udt = d on F SI (7)
7.

σ F nF + σ S nS = 0 on F SI , (8)
2. Governing equations
where andnF nS
are the unit normal vectors of the fluid and the
Fig. 1 shows a typical FSI problem. The problem consists of two structure, respectively, at the FSI interface.
domains, a fluid domain F and a structure domain S . The two
3. Iterative partitioned coupling method
domains share an FSI interface  FSI .
The iterative partitioned coupling method divides an original
2.1. Fluid domain FSI problem into three components: fluid analysis F, structure anal-
ysis S, and mesh control M. In the problem, the governing equa-
The fluid domain is governed by the following Navier–Stokes tions of the fluid, structure, and mesh control must satisfy the con-
equation in an arbitrary Lagrangian–Eulerian (ALE) frame of ditions of continuity and equilibrium at the FSI interface.

Please cite this article as: T. Yamada et al., Parallel partitioned coupling analysis system for large-scale incompressible viscous fluid–
structure interaction problems, Computers and Fluids (2016), http://dx.doi.org/10.1016/j.compfluid.2016.03.030
JID: CAF
ARTICLE IN PRESS [m5G;April 18, 2016;19:59]

T. Yamada et al. / Computers and Fluids 000 (2016) 1–10 3

3.1. Redefinition of FSI problem This method requires a matrix of only size M × N. In addition, cal-
culation is easily parallelized because only vector operations are
Each component is treated as a nonlinear problem as follows: required in Broyden’s method and the memory requirement is lim-
  ited in the restart version. The algorithm for Broyden’s method is
Xk+1 = M dk (9)
summarized below.
Broyden’s method
    Step 0: Give an initial guess d−1
uk+1 , pk+1 = F Xk+1 (10)   .
   
Step 1: Calculate s0 = − d−1 −1
 − S F M d .
−1
  Step 2: Set d0 = d + w0 s0 , k = 0.
   
dk+1 = S uk+1 , pk+1 , (11) Step 3: Calculate rk = dk − S F M dk .
where X is the mesh coordinate and p is the pressure of the fluid. Step 4: If converged, exit with the solution dk .
The superscript k and subscript  represent the kth iteration and If not, continue to Step 5.
the variables at the FSI interface, respectively. At each time step, Step 5: Calculate the increment sk using the following process.
Eqs. (9)–(11) are solved sequentially and iteratively until the final Step 5.0: Set q ← −rk .
solution, which satisfies the following nonlinear equation, is ob- Step 5.1: For i = 0, 1, ..., k − 1:
tained: a = wi /wi+1 , b= wi − 1 
q ← q + si , q /si 2 · asi+1 + bsi .
d = S (F (M (d ) ) ). (12)
Step 5.2:
 Find the increment
   
The residual r of the above nonlinear equation is defined as sk ← q − 1 − wk sk / 1 − wk sk , q /sk 2 .
r(d ) ≡ d − S (F (M (d ) ) ). (13) Step 6: Set dk+1 = dk + wk sk .
Step 7: If k = t, restart.
Thus, an iterative partitioned method can be considered as the pro-
If not, set k = k + 1 and return to Step 3.
cess of seeking the displacement d of the structure at the FSI in-
In these steps, wk is a relaxation parameter and is defined by
terface that minimizes the above residual.
Backtrack [49], which is a line-search technique. w0 is the initial
value of relaxation factor for Backtrack. Because no relaxation is
3.2. Coupling algorithms
expected at the first Broyden’s step, it is set to 1.0.
The line-search technique was applied to stabilize the nonlinear
To improve the stability and accuracy of the partitioned meth-
analysis. The relaxation parameter wk was reduced by a reduction
ods, iterative coupling methods are applied. In this study, a fixed-
factor s if the flow analysis solver failed to converge due to bad
point iteration was introduced at each time step. One robust and
mesh quality or excessive movement of the FSI interface. The algo-
fast partitioned coupling method is Newton’s method, where the
rithm of the technique is as follows:
solution of the kth fixed-point loop dk is updated by sk to reduce
Backtrack
the residual:
Step 0: Set wk to the default step length wmax .
sk = −Bk rk , (14) Step 1: If the criterion given below is satisfied or wk is less than
where Bk is the Jacobian of the function related to the nodes at the the minimum step length wmin , exit.
FSI interface. However, calculating the Jacobian requires the fluid If not, continue to Step 2.
and structure solvers to be significantly modified. The Jacobian of Step 2: Reduce wk by the reduction factor s and return to Step
the function Bk is often approximated with the approximation of 1.
the Jacobian of the function Hk as The following simple criterion is considered in Step 1:
The residual norm is less than that
sk = −Hk rk . (15) in the previous iteration, i.e.,
The approximation of the Jacobian of the function Hk does not
rk  ≤ rk−1 . (16)
need to be computed directly but can instead be obtained using
successive calculations of the function. Finally, the convergence of the fixed-point iteration is judged
To solve the FSI equation (Eq. (13)), the block Gauss–Seidel and by comparing the normalized Euclidean norm of the interface dis-
Broyden’s methods can be used as nonlinear algorithms compared placement with a sufficiently small value  tol as follows:
with existing nonlinear algorithms for the following three reasons
[49].
rk /r0  ≤ tol . (17)

1. The Jacobian matrices do not need to be explicitly calculated. 4. Fully parallelized coupling analysis system
2. These algorithms are easy to implement; i.e., little additional
programming is required for the fluid and structure solvers. To develop a system for solving incompressible viscous FSI
3. These algorithms use little additional memory for FSI analysis. problems, we employ a parallel large eddy simulation (LES)-based
In this study, we employed Broyden’s method [72], which is a finite element (FE) flow analysis solver named FrontFlow/blue
family of quasi-Newton methods. In quasi-Newton methods, the (FFB) [67,68], a parallel FE structural analysis solver named AD-
Jacobian of the function does not need to be computed directly. VENTURE_Solid [69,70], and a parallel coupling tool named AD-
Instead, an approximation of the Jacobian can be calculated us- VENTURE_Coupler for parallel data exchange and the execution of
ing successive calculations of the function. The original version of partitioned coupling algorithms.
Broyden’s method needs to hold in memory an N × N matrix dur-
ing the calculation, where N is the number of structural DOFs at 4.1. FrontFlow/blue
the FSI interface. Considering the application of this method to
large-scale problems, we adopt a restart version of this method For the flow analysis solver, we employed FFB, which is
[73] instead of the original version. In the restart version, cal- a general-purpose computational fluid dynamics (CFD) analysis
culation starts again when the iteration number reaches a user- solver with high parallel computing efficiency and computational
specified parameter M using the solution obtained up to that point. accuracy. The finite element method (FEM) and Crank–Nicolson

Please cite this article as: T. Yamada et al., Parallel partitioned coupling analysis system for large-scale incompressible viscous fluid–
structure interaction problems, Computers and Fluids (2016), http://dx.doi.org/10.1016/j.compfluid.2016.03.030
JID: CAF
ARTICLE IN PRESS [m5G;April 18, 2016;19:59]

4 T. Yamada et al. / Computers and Fluids 000 (2016) 1–10

method were adopted for spatial and time discretization, respec-

tively. The fractional step (FS) method was used to solve the gov-
erning equations of the fluid domain. The domain decomposition
technique was used to implement parallel computing. The global
linear system from the equations was solved using the paral-
lel biconjugate gradient stabilized (Bi-CGSTAB) method. Both the
standard Smagorinsky model (SSM) and the dynamic Smagorinsky
model (DSM) were included in LES models to solve turbulent in-
compressible flows. FFB has been successfully tuned for the latest
high-performance computers such as the Earth Simulator and the
K computer [63,71].
In this study, we integrated some additional functions into the
original FFB software. First, the ALE method was implemented to
track the moving boundary of the structure. Here, the mesh ve-
locity in ALE formulation computed by the pseudo-elastic tech-
nique and Jacobian-based stiffening approach [9], where the fluid
mesh motion is governed by the equations of linear elastostatics
in each time step and material property (Young’s modulus) of each Fig. 2. Communication model of coupling analysis using ADVENTURE_Coupler.
finite element is controlled by the Jacobian of it. This approach is
implemented to prevent excessive mesh distortion of the smaller
meshes in fluid domain around the structural mesh. The mesh con-
trol problem was solved using the parallel equation solver of the
FFB.

4.2. ADVENTURE_Solid

For the structural analysis solver, we employed ADVEN-

TURE_Solid, which is one of the modules of the ADVENTURE sys-
tem. The ADVENTURE system is an open-source general-purpose
computational mechanics system based on the FEM and was de-
veloped by Yoshimura et al. [69,70]. The system was designed
with the ability to analyze a three-dimensional FE model of ar-
bitrary shape and 10 million to 1 billion DOFs. The Newmark-β
method was adopted for time integration. Domain decomposition-
based parallel algorithms, e.g., the hierarchical domain decompo-
sition method (HDDM) [74] with a preconditioned iterative solver,
were implemented in ADVENTURE_Solid. The employed precondi-
tioner was the BDD method, which was originally proposed by
Mandel [75]. Each decomposed domain was solved by the conju- Fig. 3. Model of communication among processes of ADVENTURE_Coupler.
gate gradient (CG) method, but the Newton–Raphson method was
used to solve the global nonlinear system derived from the govern-
ing equations of the structure domain. ADVENTURE_Solid has also
socket is used to communicate between each analysis solver and
been successfully tuned for the latest high-performance computers
the coupler.
such as the Earth Simulator and the K computer [76,77].
The data transmission and the interpolation of interface vari-
ables from one mesh to another are executed within the processes
4.3. ADVENTURE_Coupler of the coupler. In the case of nonmatching meshes on the FSI inter-
face, the interface variables are interpolated using the shape func-
For the purpose of conducting coupling analysis together with tions of the elements. To perform coupling analysis, only a few
independent parallel solvers in various parallel computing envi- communication libraries, which are called ADVENTURE_Coupler li-
ronments, we have developed a generalized parallel coupling tool braries, are required for integration into parallel solvers. The com-
named ADVENTURE_Coupler and it was used for developing the munication model within the coupler is schematically illustrated in
analysis system of large-scale AFSI problems [50]. The coupler can Fig. 3.
be applied to the coupling of two independent parallel analysis The overall analysis flow is shown in Fig. 4. In the iterative par-
solvers, even if the solvers use different parallel algorithms suited titioned coupling analysis, fixed-point iteration is executed. At each
to their own phenomenon. To conduct parallel partitioned analy- time step of the analysis, the displacement of the structure is cal-
sis, the interface data should be exchanged in parallel between two culated by the structure analysis solver, and that at the FSI inter-
analysis solvers. The coupling tool is designed to then perform ef- face is transformed through the coupler to the flow analysis solver.
ficient, robust, and rapid computation. The flow analysis solver controls the fluid mesh and calculates the
Fig. 2 shows the communication model of the coupler. Flow velocity and pressure of the fluid. The velocity and pressure at
and structure analysis solvers are assumed to be parallelized using the FSI interface are then transferred back to the structure anal-
MPI libraries. The coupler handles interface variables using sub- ysis solver through the coupler. The structure analysis solver then
processes corresponding to the processes of the flow and struc- again calculates the displacement of the structure. After the fixed-
ture analysis solvers. The coupler runs with NF + NS parallel sub- point iteration converges, the analysis progresses to the next time
processes, where the numbers of processes run by the flow and step, and the coupler provides the initial value of the displacement
structure analysis solvers are NF and NS , respectively. The UNIX of the structure. However, if the iteration does not converge, the

Please cite this article as: T. Yamada et al., Parallel partitioned coupling analysis system for large-scale incompressible viscous fluid–
structure interaction problems, Computers and Fluids (2016), http://dx.doi.org/10.1016/j.compfluid.2016.03.030
JID: CAF
ARTICLE IN PRESS [m5G;April 18, 2016;19:59]

T. Yamada et al. / Computers and Fluids 000 (2016) 1–10 5

Fig. 4. Analysis flow.

downstream end of the column. As the inlet flow moves around

the column and generates vortices, the structural oscillation of
the plate occurs. This problem is usually simulated as a two-
dimensional problem. In this study, this problem was solved using
the developed three-dimensional FSI analysis system. We applied
slip boundary conditions to outer boundaries parallel to the x-axis
except the inlet and outlet ones because we wanted the inlet flow
to stream as uniform flow before it passes the column. In addi-
tion, we applied no-slip boundary conditions to inner boundaries
surrounding the column and plate.
Fluid mesh is shown in Fig. 6(a). The numbers of linear tetra-
hedral elements and nodes in the fluid domain are 156,312 and
31,187, respectively. Structural mesh is shown in Fig. 6(b). We only
partitioned the plate into the structural mesh, considering the col-
umn as a wall in the fluid domain. In the structure domain, the
numbers of quadratic tetrahedral elements and nodes are 14,234
and 24,367, respectively. The number of structural DOFs at the
FSI interface is 10,359. The user-specified parameter M defined in
Fig. 5. Geometric model.
Section 3.2 is 10. The properties of the analysis model are as fol-
lows. The dynamic viscosity and density of the fluid are 0.35 Pa ·
analysis starts again within the current time step by changing the s and 1.18 kg/m3 , respectively. The Young’s modulus and density of
initial value using the line-search technique. the structure are 2.5 × 105 Pa and 100 kg/m3 , respectively.
The personal computer (PC) cluster used for the analysis con-
sisted of four PCs, each of which had 3.4 GHz quad-core Intel i7
5. Verification: Vortex-induced oscillation of a flexible plate in
2600 processor and 16 GB of DDR3 memory.
the wake of a square column
In this analysis, we first conducted only fluid analysis to pre-
pare the initial velocity and pressure fields. When a vortex began
To verify the developed FSI analysis system, we first analyze a
to form, we started the FSI analysis. The time step was set to 5.0
benchmark FSI problem: the vortex-induced oscillation of a flexible
× 10−4 s. For the convergence criterion, tol = 10−3 was used in
plate in the wake of a square column.
the fixed-point iteration, and an average of four iterations were re-
The geometric model of the problem is shown in Fig. 5. A
quired for each time step.
square column is set in place, and a flexible plate is fixed at the

Please cite this article as: T. Yamada et al., Parallel partitioned coupling analysis system for large-scale incompressible viscous fluid–
structure interaction problems, Computers and Fluids (2016), http://dx.doi.org/10.1016/j.compfluid.2016.03.030
JID: CAF
ARTICLE IN PRESS [m5G;April 18, 2016;19:59]

6 T. Yamada et al. / Computers and Fluids 000 (2016) 1–10

Fig. 6. Fluid and structural meshes.

[33]]. Some of the present authors have also been working to solve
this type of problem by using the partitioned coupling approach
[48]. The aim of our series of simulation studies is not to compu-
tationally reproduce the flapping flight of actual birds or insects
but to develop a design support tool for the optimization of the
shape, structure, and motion of the flapping wings of MAVs. In this
study, the developed FSI analysis system was applied to solve the
flapping motion of an elastic rectangular plate.
A geometric model of the structure is shown in Fig. 9. Length
in geometric models is scaled to rule 0.002463 m, which is the av-
erage chord length of the crane fly [78], as 1. As the first step of
this study, we focused on the flapping motion of one wing, omit-
ting the body. The wing consists of two parts, an elastic part and a
rigid one. The deformable flapping wing is located at the center of
the fluid domain. To analyze the aerodynamic performance of hov-
Fig. 7. Snapshot of pressure field and the deformation of the model. ering flight, we fix the node O at the wing root. The flapping of the
wing is initiated by applying a prescribed sinusoidal displacement
to the leading edge. The prescribed motion of the leading edge of
A snapshot of the pressure field and the deformation of the the wing is determined from two design variables: the maximum
model is shown in Fig. 7. angle of flapping and the flapping frequency. The elastic part of
A time history of the vertical displacement at the tip of the the wing deforms, and the wing then passively enters the feather-
plate is shown in Fig. 8(a). The plate started moving with increas- ing mode. No-slip boundary conditions are applied to boundaries
ing amplitude as the vortices approached it, but it entered a steady composed of the wing. There is no inlet flow, and an outlet bound-
state when the amplitude reached 0.01 m. The frequency of the ary condition is applied to the entire outer boundary of the fluid
flapping of the plate in the steady state was approximately 3.2 Hz, domain.
which was close to the frequency of the structure itself (3.03 Hz), Fluid mesh is shown in Fig. 10. The numbers of linear tetra-
whereas the frequency of the vortex shedding was approximately hedral elements and nodes are 404,159 and 68,348, respectively.
10 Hz when the Strouhal number was assumed to be 0.2. The am- The radius of the fluid domain is set to be about 20 times larger
plitude obtained using the present method was almost the same as than the length of the wing. We also design the fluid mesh such
those obtained in the studies by Yamada and Yoshimura [48] and that the mesh size decreases from the outer boundary to the inner
Dettmer and Peric [21], both of which were two-dimensional FSI boundary and the stiffness increases as the mesh size decreases
simulations and obtained amplitudes slightly above 0.01 m. Thus, [9], effectively reducing the distortion of smaller elements near the
we concluded that the developed FSI analysis system is able to wing. Because of this mesh control technique, most of the distor-
solve the dynamic response to a flow excitation force with suffi- tion of the mesh caused by the flapping motion of the wing occurs
cient accuracy. in larger elements far from the wing, whereas smaller elements
near the wing are not significantly distorted. This requires a larger
6. Application: Flapping of an elastic rectangular plate fluid analysis domain but stabilizes the FSI analysis without the
remeshing of the fluid domain.
To demonstrate the practical application of the developed FSI Structural mesh is shown in Fig. 11. In the structure domain, the
analysis system, we analyzed the three-dimensional flapping mo- numbers of quadratic tetrahedral elements and nodes are 154,080
tion of an elastic rectangular plate with the purpose of furthering and 243,243, respectively. The number of structural DOFs at the
the research and development of MAVs. The flapping of a wing, FSI interface is 74,578. The user-specified parameter M defined in
which is realized by birds and insects, is an unsteady aerody- Section 3.2 is 10. The properties of the analysis model are as fol-
namics problem. However, the aerodynamic performance of a flap- lows. The kinematic viscosity and density of the fluid are 1.72 ×
ping wing has not been yet perfectly evaluated by experimental 10−5 m2 /s and 1.19 × 10−2 kg/m3 , respectively. The Young’s mod-
studies. Full-scale three-dimensional simulations of flapping wings ulus of the elastic part of the wing is a design variable, having a
are still challenging to execute, although some studies have per- value less than that of the rigid part. The density of the structure is
formed such simulations by the monolithic methods [[27,31,32], 3.58 kg/m3 . The parameters used in the simulation are given as di-

Please cite this article as: T. Yamada et al., Parallel partitioned coupling analysis system for large-scale incompressible viscous fluid–
structure interaction problems, Computers and Fluids (2016), http://dx.doi.org/10.1016/j.compfluid.2016.03.030
JID: CAF
ARTICLE IN PRESS [m5G;April 18, 2016;19:59]

T. Yamada et al. / Computers and Fluids 000 (2016) 1–10 7

Fig. 8. Time history of the vertical displacement at the tip of the plate.

Fig. 9. Geometric model of the structure.

Fig. 11. Structural mesh.

mensionless parameters because the flow analysis solver FFB treats

parameters as dimensionless quantities. As the length is nondi- A PC cluster is also used for this analysis. It consists of four PCs,
mensionalized at the beginning of the explanation about the ge- each of which has a 3.4 GHz quad-core Intel i7 2600 processor and
ometric models, one second is considered as 56.875 to make a 16 GB of DDR3 memory.
flapping period 1.25. After the simulation, we can dimensionalize The initial velocity and pressure in the fluid domain were both
reversely with giving units to the dimensionless parameters and set to 0. In this paper, we perform four iterations of the FSI anal-
values of the analysis results for a particular purpose. ysis with different design variables, i.e., the Young’s modulus of

Fig. 10. Fluid mesh.

Please cite this article as: T. Yamada et al., Parallel partitioned coupling analysis system for large-scale incompressible viscous fluid–
structure interaction problems, Computers and Fluids (2016), http://dx.doi.org/10.1016/j.compfluid.2016.03.030
JID: CAF
ARTICLE IN PRESS [m5G;April 18, 2016;19:59]

8 T. Yamada et al. / Computers and Fluids 000 (2016) 1–10

Fig. 12. Snapshot of the pressure and streamlines in the fluid domain.

Fig. 15. Relationship between the flexibility of the wing and the lift coefficient.

where Ft is the fluid force on the wing surface at time t, Fti is

the fluid force at time t on each axis (i = x, y, z), Faivg is the time-
averaged fluid force on each axis (i = x, y, z), t is the standard time
step for evaluation, t is the time step, and T is the flapping pe-
riod. Lift and drag forces are defined as
Fig. 13. Definition of the passive feathering angle.
L = Fy (20)

the elastic part of the wing (0.0078 GPa and 0.0117 GPa), and the
maximum angle of flapping (30° and 45°). In all cases, the Young’s
D = F x, (21)
modulus of the rigid part of the wing was 2.88 × 106 GPa, the
flapping frequency and time step were set to 45.5 Hz and 2.20 × where L is the lift force, and D is the drag force. The time-averaged
10−5 s, respectively. The convergence criterion tol = 10−3 was used fluid force is defined as
in the fixed-point iteration. An average of three iterations was re-  
quired for each time step. Favg = Davg , Lavg , Fazvg . (22)
The fluid force on the wing surface at each time step and the Also, the time-averaged lift coefficient is defined as
time-averaged fluid force on each axis are calculated as
  CL =
2Lavg
, (23)
Ft = Ftx , Fty , Ftz (18) ρF U 2S
where CL is the time-averaged lift coefficient, U is the characteristic
t  +(n−1) t 
Fti T velocity that is the average velocity at the endpoint of the leading
t =t 
Faivg = n= , (19) edge, and S is the wing surface area.
n t

Fig. 14. Time history of the flapping and passive feathering angles.

Please cite this article as: T. Yamada et al., Parallel partitioned coupling analysis system for large-scale incompressible viscous fluid–
structure interaction problems, Computers and Fluids (2016), http://dx.doi.org/10.1016/j.compfluid.2016.03.030
JID: CAF
ARTICLE IN PRESS [m5G;April 18, 2016;19:59]

T. Yamada et al. / Computers and Fluids 000 (2016) 1–10 9

Various values, including the lift and drag forces, were calcu- [3] Mittal S, Tezduyar TE. Parallel finite element simulation of 3d incompressible
lated to evaluate the flight performance. The visualized result, as flows: fluid-structure interactions. Int Numer Methods Fluids 1995;21:933–53.
[4] Kalro V, Tezduyar TE. Parallel 3d computational method for fluid-struc-
shown in Fig. 12, helped us to understand the mechanism that en- ture interactions in parachute systems. Comput Methods Appl Mech Eng
hances the lift. Fig. 12 shows that leading edge vortices that devel- 20 0 0;190:321–32.
oped and contributed to increasing the lift when the translational [5] Stein K, Benney R, Kalro V, Tezduyar TE, Leonard J, Accorsi M. Parachute flu-
id-structure interactions: 3-d computation. Comput Methods Appl Mech Eng
velocity of the wing was high. The lines and blue contours indicate 20 0 0;190:373–86.
streamlines and low-pressure (threshold: −30) areas, respectively. [6] Tezduyar TE, Osawa Y. Fluid-structure interactions of a parachute crossing the
Fig. 13 shows the definition of the passive feathering angle be- far wake of an aircraft. Comput Methods Appl Mech Eng 2001;191:717–26.
[7] Ohayon R. Reduced symmetric models for modal analysis of internal struc-
tween two lines: a line connecting the upper and lower sides of
tural-acoustic and hydroelastic-sloshing systems. Comput Methods Appl Mech
the elastic part of the wing and a line normal to the leading edge. Eng 20 01;190:30 09–19.
The passive feathering angle indicates the degree of deformation [8] Wall WA, Genkinger S, Ramm E. A strong coupling partitioned approach for
fluid-structure interaction with free surfaces. Comput Fluids 2002;36:169–83.
of the wing.
[9] Stein K, Tezduyar T, Benney R. Mesh moving techniques for fluid-structure in-
Because the elastic part of the wing has a low Young’s mod- teractions with large displacements. J Appl Mech 2003;70(1):58–63.
ulus (0.0078 GPa), it deformed more than that with high Young’s [10] Bazilevs Y, Calo VM, Zhang Y, Hughes TJR. Isogeometric fluid-structure in-
moduli (0.0117 GPa), as shown in Fig. 14. The deformation influ- teraction analysis with applications to arterial blood flow. Comput Mech
2006;38:1885–95.
enced a phase difference developed between the flapping and pas- [11] Farhat C. CFD-based nonlinear computational aeroelasticity. In: Stein E, de
sive feathering angles, and we colored each phase difference in Borst R, Hughes TRJ, editors. Encyclopedia of computational mechanics, 3. NY:
Fig. 14. Using the time-averaged lift force, we calculated the time- Wiley; 2004.Chapter 13
[12] Torii R, Oshima M, Kobayashi T, Takagi K, Tezduyar TE. Computer model-
averaged lift coefficient. Depending on the maximum flapping an- ing of cardiovascular fluid-structure interactions with the deforming-spatial–
gle, the flexibility of the wing affected the lift coefficient in dif- domain/stabilized space-time formulation. Comput Methods Appl Mech Eng
ferent ways, as shown in Fig. 15. Significant deformation does not 2006;195:1885–95.
[13] Torii R, Oshima M, Kobayashi T, Takagi K, Tezduyar TE. Fluid-structure inter-
always enhance the lift coefficient, which has a maximum limit de- action modeling of aneurysmal conditions with high and normal blood pres-
pending on the maximum flapping angle, even if the elastic part sures. Comput Mech 2006;38:482–90.
of the wing is more flexible. Using the developed FSI analysis sys- [14] Tezduyar TE, Sathe S. Modeling of fluid-structure interactions with the
space-time finite elements: Solution techniques. Int J Numer Methods Fluids
tem, we confirmed the maximum limit needed to design flapping
2007;54:855–900.
MAVs, and will be able to accumulate knowledge on the quanti- [15] Tezduyar TE, Sathe S, Cragin T, Nanna B, Conklin BS, Pausewang J, et al. Mod-
tative relations between various design parameters and the flight elling of fluid-structure interactions with the space-time finite elements: arte-
rial fluid mechanics. Int J Numer Methods Fluids 2007;54:901–22.
performance of a flapping wing.
[16] Löhner R, Cebral JR, Yang C, Baum JD, Mestreau EL, Soto O. Extending the range
and applicability of the loose coupling approach for FSI simulations. In: Bun-
gartz HJ, Schafer M, editors. Fluid-structure interaction. Lecture notes in com-
7. Conclusions putational science and engineering, 53. Berlin: Springer; 2006. p. 82–100.
[17] Sawada T, Hisada T. Fluid-structure interaction analysis of the two-dimensional
A parallel analysis system for solving large-scale FSI problems flag-in-wind problem by an interface-tracking ALE finite element method.
Comput Fluids 2007;36:136–46.
between incompressible viscous fluids and structures was devel-
[18] Tezduyar TE, Sathe S, Pausewang J, Schwaab M, Christopher J, Crabtree J. Inter-
oped in this study. We adopted the iterative partitioned coupling face projection techniques for fluid-structure interaction modeling with mov-
method to integrate two independent parallel flow and struc- ing-mesh methods. Comput Mech 2008;43:39–49.
[19] Tezduyar TE, Sathe S, Schwaab M, Pausewang J, Christopher J, Crabtree J.
ture solvers (FFB and ADVENTURE_Solid, respectively). ADVEN-
Fluid-structure interaction modeling of ringsail parachutes. Comput Mech
TURE_Coupler was used to efficiently handle the data between the 2008;43:133–42.
flow and structure solvers in a parallel computing environment. [20] Küttler U, Wall WA. Fixed-point fluid-structure interaction solvers with dy-
We employed the Broyden’s method to update the interface accel- namic relaxation. Comput Mech 2008;43(1):61–72.
[21] Dettmer WG, Peric D. On the coupling between fluid flow and mesh motion in
erations to obtain robust and rapid convergence. To verify the ac- the modelling of fluid-structure interaction. Comput Mech 2008;43(1):81–90.
curacy and fundamental performance of the developed system, we [22] Isaksen JG, Bazilevs Y, Kvamsdal T, Zhang Y, Kaspersen JH, Waterloo K,
used it to solve the vortex-induced oscillation of a flexible plate et al. Determination of wall tension in cerebral artery aneurysms by numer-
ical simulation. Stroke 2008;39:3172–8.
in the wake of a square column, which is a FSI benchmark prob- [23] Bazilevs Y, Gohean JR, Hughes TJR, Moser RD, Zhang Y. Patient-specific isogeo-
lem. The obtained results agreed quantitatively well with those metric fluid-structure interaction analysis of thoracic aortic blood flow due to
obtained by other researchers. Finally, we applied the system to implantation of the jarvik 20 0 0 left ventricular assist device. Comput Methods
Appl Mech Eng 2009;198:3534–50.
the analysis of the flapping motion of an elastic rectangular plate [24] Bazilevs Y, Hsu MC, Kiendl J, Wüchner R, Bletzinger KU. 3d simulation of wind
and successfully solved it. The accuracy and parallel performance turbine rotors at full scale. part II: fluid-structure interaction modeling with
of the flow analysis solver (FFB) and the structure analysis solver composite blades. Int J Numer Methods Fluids 2011;65:236–53.
[25] Takizawa K, Tezduyar TE. Multiscale space-time fluid-structure interaction
(ADVENTURE_Solid) have already been verified in the latest par-
techniques. Comput Mech 2011;48:247–67.
allel computers, such as the Earth Simulator and the K computer. [26] Hou G, Wang J, Layton A. Numerical methods for fluid-structure interaction -
It has also been verified that ADVENTURE_Coupler is able to solve a review. Commun Comput Phys 2012;12(2):337–77.
[27] Nakata T, Liu H. A fluid-structure interaction model of insect flight with flexi-
acoustic and incompressible viscous FSI problems in parallel en-
ble wings. J Comput Phys 2012;231:1822–47.
vironments. Thus, the system can be applied to large-scale three- [28] Takizawa K, Bazilevs Y, Tezduyar TE. Space-time and ALE-VMS techniques for
dimensional FSI problems. patient-specific cardiovascular fluid-structure interaction modeling. Arch Com-
In the next phase of our research, the parallel performance put Methods Eng 2012;19:171–225.
[29] Bazilevs Y, Takizawa K, Tezduyar TE. Computational fluid-structure interaction:
and robustness of the system will be further improved, and the methods and applications. Chichester, West Sussex, UK: Wiley; 2013.
system will be applied to more problems in engineering, science, [30] Takizawa K, Montes D, Fritze M, McIntyre S, Boben J, Tezduyar TE. Methods
medicine, and everyday life. for FSI modeling of spacecraft parachute dynamics and cover separation. Math
Models Methods Appl Sci 2013;23(3):307–38.
[31] Takizawa K, Tezduyar TE, Kostov N. Sequentially-coupled space-time FSI anal-
References ysis of bio-inspired flapping-wing aerodynamics of an MAV. Comput Mech
2014;54(2):213–33.
[1] Mittal S, Tezduyar TE. A finite element study of incompressible flows [32] Ishihara D, Horie T, Niho T. An experimental and three-dimensional computa-
past oscillating cylinders and aerofoils. Int J Numer Methods Fluids tional study on the aerodynamic contribution to the passive pitching motion
1992;15:1073–118. of flapping wings. Bioinspir Biomim 2014;9:046009.
[2] Tezduyar TE, Aliabadi SK, Behr M, Mittal S. Massively parallel finite element [33] Takizawa K, Tezduyar TE, Buscher A. Space-time computational analysis
simulation of compressible and incompressible flows. Comput Methods Appl of MAV apping-wing aerodynamics with wing clapping. Comput Mech
Mech Eng 1994;119:157–77. 2015;55:1131–41.

Please cite this article as: T. Yamada et al., Parallel partitioned coupling analysis system for large-scale incompressible viscous fluid–
structure interaction problems, Computers and Fluids (2016), http://dx.doi.org/10.1016/j.compfluid.2016.03.030
JID: CAF
ARTICLE IN PRESS [m5G;April 18, 2016;19:59]

10 T. Yamada et al. / Computers and Fluids 000 (2016) 1–10

[34] Zhang Q, Hisada T. Numerical methods for FSI problems with strong de- [56] Gerbeau JF, Vidrascu M. A quasi-newton algorithm based on a reduced model
pendence between fluid and structure. Comput Methods Appl Mech Eng for fluid-structure interaction problems in blood flows. Math Model Numer
2001;190(48):6341–57. Anal 2003;37(4):631–47.
[35] Heil M. An efficient solver for the fully-coupled solution of large-displace- [57] Michler C, van Brummelen EH, de Borst R. An interface newton-krylov
ment fluid-structure interaction problems. Comput Methods Appl Mech Eng solver for fluid-structure interaction. Int J Numer Methods Fluids
2004;193:1–23. 2004;47(10-11):1189–95.
[36] Ishihara D, Yoshimura S. A monolithic approach for interaction of incompress- [58] Deparis S. Numerical analysis of axisymmetric flows and methods for flu-
ible viscous fluid and an elastic body based on fluid pressure poisson equation. id-structure interaction arising in blood flow simulation. Ecole Polytechnique
Int J Numer Methods Eng 2005;64:167–203. Federale de Lausanne; 2004. Ph.d. thesis.
[37] Tezduyar TE, Sathe S, Keedy R, Stein K. Space-time finite element techniques [59] Fernandez MA, Moubachir M. A newton method using exact jacobians for solv-
for computation of fluid-structure interactions. Comput Methods in Appl Mech ing fluid-structure coupling. Comput Struct 2005;83(2-3):127–42.
Eng 20 06;195:20 02–27. [60] Matthies HG, Niekamp R, Steindorf J. Algorithms for strong coupling proce-
[38] Tezduyar TE, Sathe S, Stein K. Solution techniques for the fully discretized dures. Comput Methods Appl Mech Eng 2006;195:2028–49.
equations in computation of fluid-structure interactions with the space-time [61] Yoshimura S, Okamoto N, Yamada T. Stability efficiency of an iterative parti-
formulations. Comput Methods Appl Mech Eng 2006;195(41-43):5743–53. tioned coupling algorithm for fluid-structure interaction problems. Trans Japan
[39] Dettmer W, Peric D. A computational framework for fluid-structure interac- Soc Mech Eng B 2006;72(716):17–24.
tion: finite element formulation and applications. Comput Methods Appl Mech [62] Forster C, Wall WA, Ramm E. Artificial added mass instabilities in sequential
Eng 2006;195(41-43):5754–79. staggered coupling of nonlinear structures and incompressible viscous flows.
[40] Badia S, Quaini A, Quarteroni A. Splitting methods based on algebraic factor- Comput Methods Appl Mech Eng 2007;196(7):1278–93.
ization for fluid-structure interaction. SIAM J Sci Comput 2008;30:1778–805. [63] Jiang YY, Yoshimura S, Imai R, Katsura H, Yoshida T, Kato C. Quantitative eval-
[41] Idelsohn SR, Pin FD, Rossi R, nate EO. Fluid-structure interaction problems with uation of flow-induced structural vibration and noise in turbomachinery by
strong added-mass effect. Int J Numer Methods Eng 2009;80:1261–94. full-scale weakly coupled simulation. J Fluids Struct 2007;23(4):531–44.
[42] Ishihara D, Horie T. A projection method for the interaction of an incompress- [64] Küttler U, Wall WA. Fixed-point fluid-structure interaction solvers with dy-
ible fluid and a structure using new algebraic splitting. Comput Model Eng Sci namic relaxation. Comput Mech 2008;43:61–72.
2014;101:421–40. [65] Küttler U, Wall WA. Vector extrapolation for strong coupling fluid-structure
[43] Manguoglu M, Sameh A, Tezduyar TE, Sathe S. A nested iterative scheme interaction solvers. J Appl Mech 2009;76:1–7.
for computation of incompressible flows in long domains. Comput Mech [66] Degroote J, Bathe KJ, Vierendeels J. Performance of a new partitioned pro-
2008;43(1):73–80. cedure versus a monolithic procedure in fluid-structure interaction. Comput
[44] Manguoglu M, Takizawa K, Sameh AH, Tezduyar TE. Solution of linear systems Struct 2009;87(11-12):793–801.
in arterial fluid mechanics computations with boundary layer mesh refine- [67] Kato C, Yamade Y, Wang H, Guo Y, Miyazawa M, Takaishi T, et al. Numerical
ment. Comput Mech 2010;46(1):83–9. prediction of sound generated from flows with a low mach number. Comput
[45] Manguoglu M, Takizawa K, Sameh AH, Tezduyar TE. Nested and parallel sparse Fluids 2007;36:53–68.
algorithms for arterial fluid mechanics computations with boundary layer [68] Research and Development of Innovative Simulation Software, http://www.ciss.
mesh refinement. Int J Numer Methods Fluids 2011;65:135–49. u-tokyo.ac.jp/riss/.
[46] Minami S, Kawai H, Yoshimura S. A monolithic approach based on the bal- [69] Yoshimura S, Shioya R, Noguchi H, Miyamura T. Advanced general-purpose
ancing domain decomposition method for acoustic fluid-structure interaction. computational mechanics system for large-scale analysis and design. J Comput
J Appl Mech 2011;79(1):010906. Appl Math 2002;49:279–96.
[47] Mandel J. An iterative substructuring method for coupled fluid-solid acoustic [70] ADVENTURE Project, http://adventure.sys.t.u-tokyo.ac.jp/.
problems. J Comput Phys 2002;177(1):95–116. [71] Kumahata K, Inoue S, Minami K. Kernel performance improvement for the
[48] Yamada T, Yoshimura S. Line search partitioned approach for fluid-structure FEM-based fluid analysis code on the k computer. Procedia Comput Sci
interaction analysis of flapping wing. Comput Model Eng Sci 2008;24(1):51–60. 2013;18:2496–9.
[49] Minami S, Yoshimura S. Performance evaluataion of nonlinear algorithms with [72] Charles G, Broyden A. A class of methods for solving nonlinear simultaneous
line search for partitioned coupling technique for fluid-structure interactions. equations. Math Comput 1965;19:577–93.
Int J Numer Methods Fluids 2010;64(10-12):1129–47. [73] Kelley CT. Solving nonlinear equations with Newtons method. Philadelphia, PA:
[50] Kataoka S, Minami S, Kawai H, Yamada T, Yoshimura S. A parallel iterative par- SIAM; 2003.
titioned coupling analysis system for large-scale acoustic fluid-structure inter- [74] Shioya R, Yagawa G. Parallel finite elements on a massively parallel computer
actions. Comput Mech 2014;53(6):1299–310. with domain decomposition. Comput Syst Eng 1994;4:495–503.
[51] Felippa CA, Park KC. Staggered transient analysis procedures for cou- [75] Mandel J. Balancing domain decomposition. Commun Numer Methods Eng
pled mechanical systems: formulation. Comput Methods Appl Mech Eng 1993;9:233–41.
1980;24:61–111. [76] Ogino M, Shioya R, Kawai H, Yoshimura S. Seismic response analysis of full
[52] Tallec PL, Mouro J. Fluid structure interaction with large structural displace- scale nuclear vessel model with ADVENTURE system on the earth simulator. J
ments. Comput Methods Appl Mech Eng 2001;190(24-25):3039–67. Earth Simul 2005;2:41–54.
[53] Park KC, Felippa CA, Ohayon R. Partitioned formulation of internal fluid-struc- [77] Murotani K, Sugimoto S, Kawai H, Yoshimura S. Hierarchical domain decom-
ture interaction problems by localized lagrange multipliers. Comput Methods position with parallel mesh refinement for billions-of-DOF scale finite element
Appl Mech Eng 2001;190(24-25):2989–3007. analyses. Int J Comput Methods 2014;11(4):1350061.
[54] Mok DP. Partitionierte lösungsansätze in der strukturdynamik und der flu- [78] Ishihara D, Yamashita Y, Horie T, Yoshida S, Niho T. Passive maintenance of
id-struktur-interaktion. Institut fur Bautstatik, Universitat Stuttgart; 2001. Ph.d. high angle of attack and its lift generation during flapping translation in crane
thesis. fly wing. J Exp Biol 2009;212:3882–91.
[55] Mok DP, Wall WA. Partitioned analysis schemes for the transient interaction
of incompressible flows and nonlinear flexible structures. In: Wall WA, Blet-
zinger K-U, Schweizerhof K, editors. Trends in computational structural me-
chanics. Barcelona: CIMNE (International Center for Numerical Methods in En-
gineering); 2001. p. 689–98.

Please cite this article as: T. Yamada et al., Parallel partitioned coupling analysis system for large-scale incompressible viscous fluid–
structure interaction problems, Computers and Fluids (2016), http://dx.doi.org/10.1016/j.compfluid.2016.03.030