Composite Structures 184 (2018) 269–278

Contents lists available at ScienceDirect

Composite Structures
journal homepage: www.elsevier.com/locate/compstruct

Dune-composites – A new framework for high-performance finite element MARK

modelling of laminates
⁎
Anne Reinarza, , Tim Dodwellb, Tim Fletcherc, Linus Seelingerd, Richard Butlerc, Robert Scheichla
a
Department of Mathematical Sciences, University of Bath, UK
b
College of Engineering, Mathematics and Physical Sciences, University of Exeter, UK
c
Department of Mechanical Engineering, University of Bath, UK
d
Institute for Scientific Computing, University of Heidelberg, Germany

A B S T R A C T

Finite element (FE) analysis has the potential to offset much of the expensive experimental testing currently
required to certify aerospace laminates. However, large numbers of degrees of freedom are necessary to model
entire aircraft components whilst accurately resolving micro-scale defects. The new module dune-composites,
implemented within DUNE by the authors, provides a tool to efficiently solve large-scale problems using novel
iterative solvers. The key innovation is a preconditioner that guarantees a constant number of iterations re-
gardless of the problem size. Its robustness has been shown rigorously in Spillane et al. (2014) for isotropic
problems. For anisotropic problems in composites it is verified numerically for the first time in this paper. The
parallel implementation in DUNE scales almost optimally over thousands of cores. To demonstrate this, we
present an original numerical study, varying the shape of a localised wrinkle and the effect this has on the
strength of a curved laminate. This requires a high-fidelity mesh containing at least four layers of quadratic
elements across each ply and interface layer, underlining the need for dune-composites, which can achieve run
times of just over 2 min on 2048 cores for realistic composites problems with 173 million degrees of freedom.

1. Introduction conservative ‘knock-down’ factors are applied to strength limits. These

factors are currently obtained by expensive testing at all scales in the
Traditionally, aerospace structures have been largely manufactured Test Pyramid, with large numbers of small coupon tests and fewer
by subtractive processes, such as machining, which is typically applied larger-scale structural tests. Large-scale tests, are extremely expensive
to metals. Composite materials, which constitute over 50% of recent and take place at a stage when it is difficult to make changes and im-
aircraft construction, are created at the same time as the structure itself; prove designs, while small-scale coupons do not represent the perfor-
typically by an additive layup process such as automated fibre place- mance of the material at the structural scale where, for example, defects
ment (AFP). This co-assembly of material and structure is achieved by can be introduced as a result of complex forming processes.
sequential deposition of a number of fibrous layers which are either Furthermore, the boundary conditions and loading in a simple element
pre-impregnated or post-infused with resin, involving hundreds of ad- are very different from the performance of the material within the
ditive operations during the production of a large structural part. Most structure.
of these processes take place before the resin is cured, when the in- One loading of particular interest is corner unfolding, in which
fluence of temperature and pressure on the deposition, forming and though thickness tensile loading acts to separate the layers as the corner
curing of parts with complex geometry is not well understood. For this radius is increased under bending loads. These stresses act in the weak
reason production is vulnerable to defects arising from small process resin-dominated direction of the laminate and can therefore lead to a
variations and achieving high-rate automation of the process is limited. limiting design case. The presence of defects such as mis-aligned fibres
The ability to manufacture composite material-structural systems in the form of out-of-arc wrinkles can have a significant importance.
with repeatable properties at a reasonable rate and competitive cost, Such a reduction in strength is referred to in manufacturing as a knock-
with the quality required for certification, is a major challenge in the down factor. A large number of composite parts contain such corner
global aerospace industry. A number of defect types can arise, and use radii to provide perpendicular surfaces for attachment to other parts.
of non-destructive evaluation is limited in detecting these, hence The opening of these corners can be a critical limitation to the strength

⁎
Corresponding author.
E-mail address: a.k.reinarz@bath.ac.uk (A. Reinarz).

http://dx.doi.org/10.1016/j.compstruct.2017.09.104
Received 13 July 2017; Received in revised form 22 September 2017; Accepted 29 September 2017
Available online 30 September 2017
0263-8223/ © 2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/BY/4.0/).
A. Reinarz et al. Composite Structures 184 (2018) 269–278

of such radii, particularly since manufacturing defects are prone to The failure of curved laminates subjected to corner unfolding has
occur in such regions of high curvature. been shown to be highly unstable with instantaneous propagation fol-
Mukhopadhyay et al. have modelled the effect of defects on com- lowing initiation [14]. Therefore we focus on capturing the initiation of
pressive [23] and tensile [24] strength in flat laminates using 8-node, failure accurately by using a high-fidelity 3D mesh, with 4 or more
solid elements and zero-thickness, 8-node, cohesive elements between elements through the thickness of each ply and interface layer. Such an
plies. Damage modelling accounted for nonlinear shear in plies, trans- approach has been shown to give accurate results for simple flat cou-
verse matrix cracking, mixed mode delamination, tensile fibre fracture pons [13] and 2D models of a few plies [16]. However, this level of
and fibre kinking. The minimum size of the FE mesh was one ply fidelity is typically dropped when modelling larger, more complex
thickness (0.25 mm) in the vicinity of wrinkles and towards laminate parts, such as 3D curved laminates, due to computational limitations.
edges. The compressive strength predictions were in agreement to Here, we do not use any of these more complex models in order to
within 10% of experimental results and were able to pick up a mode reduce the computational requirements.
switch from fibre failure to delamination when defect misalignment More complex modelling approaches have been proposed for mod-
was above ∼9°. In the tensile case, it was noted that wrinkles act as elling of defects in the literature, these include the use of composite
local, through-thickness shear stress concentrators. For multidirectional shell elements [24,23], interface/cohesive elements [18,21] and
laminates, the influence of the defect was exacerbated by edge effects. higher-order continuum models [22]. In particular, cohesive elements
It is well-known that the singularity associated with the free edge of are commonly used to capture propagation [18], but propagation is
laminated test specimens causes a stress concentration, especially when considered less important than initiation for the problem. The for-
a laminate contains plies with fibres in varying orientations. The mulation of such elements, whilst more complex, is possible within
strength of full-sized components may therefore be misrepresented by dune-composites. The solution strategy would require Newton iterations
narrow test specimens, which are frequently cut from wide laminate and path-following methods (available in the DUNE library). However,
sections as witness specimens, to establish manufacturing knock-down even for these non-linear models of failure, computational cost is still
factors. We have been shown previously that applying a 3 mm layer of dominated by the speed of solving a linearised system of equations for
tough resin to the curved edges of corner test specimens reduces the composite materials. In this paper our focus is therefore on developing
stress concentration and increases strength [14]. and implementing an efficient solution strategy for these linearised
There is a wide range of defects that can form in composite lami- equations arising from such massive composite simulation.
nates and a detailed taxonomy of these is presented in [25]. In this
paper we evaluate wrinkle defects, which are more likely to form in 2.1. Dune-composites: High performance FE modelling of large-scale
curved laminates as a result of consolidation onto a male tool [10]. composite structures
They are also likely to form in curved laminates with tapered sections,
which causes double curvature and makes AFP deposition challenging. DUNE (Distributed and Unified Numerics Environment) is an open
These wrinkles are important to the performance of curved laminates. source modular toolbox for solving partial differential equations (PDEs)
In this paper we present a new FE analysis tool dune-composites for with grid-based methods, such as the finite element method (FEM)
efficient, high-fidelity modeling of laminated composite parts. We [4–6]. Written using modern C++ programming techniques, the core
show, using a simple test case that the results of dune-composites match modules of DUNE have been developed by mathematicians and com-
up well with the commercial software package ABAQUS in all six puter scientists to allow users to implement and use state-of-the-art
stresses. For large scale problems, dune-composites crucially relies on mathematical methods across large high performance parallel com-
robust iterative solvers for the resulting FE systems. To this end, we puting architectures. It is a generic package that provides a user with
introduce the preconditioner GenEO [27,28,20], which we have im- the key ingredients for solving any FEM problem, e.g. grid generation,
plemented within DUNE. This preconditioner has previously been different types of finite elements, quadrature rules and a choice of off-
mathematically proven to be robust for isotropic FE systems.+++ We the shelf solvers.
show that these results extend to anisotropic problems and that the Within this platform, we have developed a new module, dune-
solver is suitable for solving large composites problems. Further, we composites, which solves the linear elasticity equations with general,
demonstrate its parallel efficiency and its ability to scale to thousands of anisotropic stiffness tensor, applicable for modelling composite struc-
compute cores, allowing the solution of the large problems with defects tures. This module provides an interfaces to handle composite appli-
mentioned above. We test this module by modeling the unfolding of a cations which includes stacking sequences, complex part geometries
curved laminate part containing manufacturing defects, for which a and complex boundary conditions such as multi-point constraints, or
micron scale mesh is needed to accurately compute stresses. We show periodic boundary conditions. Further, we have implemented a new 20-
that dune-composites is able to accurately predict damage initiation and node 3D serendipity element (with full integration) within dune-pdelab,
that it does so at a fraction of the computational cost required by which is not prone to shear locking and allows comparison with
ABAQUS. ABAQUS’s C3D20R element. This element has degrees of freedom at the
8 nodes of the element as well as on each of the 12 edges.
2. Modelling approach The main advantage of implementing our simulation tool within
DUNE is that it allows us to exploit developments in state of the art
In this section, we introduce the new high performance finite ele- solvers such as Algebraic Multigrid (AMG), see [3], or to implement
ment module dune-composites, and demonstrate its capability of effi- new ones. In particular, as part of our new developments, we im-
ciently and robustly tackling large-scale simulations of composite plemented a novel, robust preconditioner, called GenEO [28,27],
structures. In the example simulations presented, the composite within DUNE. Finally, we will show in Section 2.4 that DUNE allows for
strength of pristine and defected corner radii are accurately predicted. highly parallelised efficiency on hundreds of computer cores. This al-
The analysis assumes standard anisotropic 3D linear elasticity and the lows for the modelling of meshes fine enough to resolve defects or the
failure is assessed using a quadratic damage onset criterion for the in- modelling of wide parts with sufficient accuracy.
itiation of delamination in [8]. This allows the results to be bench-
marked against existing numerical results and experiments, given in 2.2. Verification of DUNE
Fletcher et al. [14]. This study shows that ultimate failure is unstable,
following quickly after initiation, and so linear analysis is justified. For In this section we show that DUNE produces stress results that are
this case, the numerical results of this linear model show good agree- comparable to those produced by standard FE libraries such as
ment with experimental data. ABAQUS. For our comparison we examine a corner unfolding test in

270
A. Reinarz et al. Composite Structures 184 (2018) 269–278

Fig. 1. Sketch of the edge-treated corner-unfolding spe-

cimen (left) and a cross section of the configuration (right).

DUNE and ABAQUS and compare the solutions as well as the run times. edge, 2.156 mm away from the edge, which is in the laminate
We initially use a small computationally inexpensive problem for 0.156 mm (6 elements) away from the resin laminate edge. Since we
comparisons with ABAQUS and move to a larger problem to demon- use identical meshes and identical elements, the numbers of degrees of
strate the parallel efficiency of DUNE. freedom are also the same. However, there are some differences in the
Fig. 1 shows the setup of our problem. For this small test we use a models, such as the quadrature order, use of single precision in
width W = 15 mm and L = 3 mm long limbs. The inner radius R of the ABAQUS and the stress reconstruction, which could account for the
curved section is 6.6 mm. The overall thickness T is 2.98 mm, where the different results. Nevertheless, the results show reasonably good
ply layers are 0.23 mm thick and the resin interfaces are 0.02 mm thick. agreement for all six different stresses.
We then apply a resin treatment of 2 mm to the free edges of the la-
minate. This type of edge treatment has been shown in [14] to be ad- 2.3. Efficiency of dune-composites: Robust iterative solvers
vantageous both for reducing conservatism in the design of aircraft
structures, as well as to make FE analyses more reliable. The mechan- The computational gains of dune-composites demonstrated in this
ical properties for the ply material and for the resin interfaces of the section are a direct consequence of the use of fast and robust iterative
curved laminate are given in Table 1. The fibre angles are given by the solvers for the resulting large systems of linear equations
following 12-ply stacking sequence
Ku = f , (1)
[± 45/90/0/ ∓ 45/ ∓ 45/0/90/± 45].
where K is the global stiffness matrix, f is the load vector arising from
In ABAQUS, the corner unfolding is modelled using a beam-type the applied boundary conditions and u is the solution vector containing
multi-point constraint (MPC) at the end of one limb. In DUNE, a similar the displacement values at each node within the finite element mesh.
effect is achieved by adding a thin layer of very stiff material at the end There are also some significant gains in the setup time due to a more
of the same limb. In both cases, a 96.8 Nmm/mm moment is applied to efficient, problem-adapted data management.
that limb, while fixing all degrees of freedom on the other limb, as Solvers for systems of linear algebraic equations can broadly be
shown in Fig. 1. We denote the stresses using the following directional classified into direct and iterative ones. Direct solvers, based on matrix
notation: arc length s (around the curve); radius r through-thickness factorisation, are more universally applicable and more robust to ill-
and width l across the laminate. These are shown by the coordinate conditioning [12]. Thus, they are typically the default in commercial FE
system in Figure 1a. packages. However, the cost of the factorisation and the memory re-
DUNE and ABAQUS both use a model with second-order serendipity quirements can quickly become infeasibly high for large problems.
elements (element C3D20R in ABAQUS, [1]). There is a small difference Moreover, it is difficult to achieve good parallel efficiencies on large
in that ABAQUS uses reduced integration while DUNE uses full in- multicore computer architectures. Since they do not require any fac-
tegration. Also, the stresses are not recovered in an identical way from torisations, iterative methods for (1) have the potential to scale opti-
the displacements in the two codes. We use a tensor-product grid with mally, both with respect to problem size and with respect to the number
56 elements in the l and s directions and 6 elements in each of the 12 of processors in a parallel implementation, but that crucially depends
fibrous and the 11 interface layers in the r direction. In total, this mesh on the ’conditioning’ of the problem.
contains 432,768 elements. To ensure the effects at the free edge and at In size, we refer to the total number of degrees of freedom N in the
the material discontinuities are sufficiently resolved, the FE mesh is underlying finite element (FE) solution u . Below we will see examples,
graded towards both edges in the l direction, as well as towards the for which N can be very large, up to 60 million. It is important to note
interfaces in each of the resin and fibrous layers in the r direction. The that K is sparse, that is, the number of nonzero entries in K is propor-
bias ratio gives the ratio between width of the smallest and largest tional to N, which is significantly less than the possible N 2 entries in a
element of the mesh. We choose a bias ratio of 400 between the centre full matrix. Iterative methods only require the storage of the original,
and the edges of the structure in the l direction and a bias ratio of 10 sparse stiffness matrix, while the storage of the factors in direct
between the centre and the interfaces in each layer in the r direction. methods is closer to the N 2 entries in a full matrix. The computational
In Fig. 2, all six stresses computed with ABAQUS and DUNE are cost of the best direct methods for FE systems arising from three-di-
compared at the apex of the curve (i.e., at an angle of 45°) near the resin mensional structural calculations, such as those used in ABAQUS, still
grows at least with order N1.5, as we will see below. Iterative solvers
Table 1
that only require multiplication with the sparse matrix K have the po-
Mechanical properties for the curved 12-ply laminate. tential to scale linearly with the problem size N.
The conditioning is a property of the global stiffness matrix K. We
Orthotropic fibrous layer Isotropic interface layer say that K is ill-conditioned, if the nodal displacements u are highly
E11 162 GPa E 10 GPa
sensitive to rounding errors and to small changes in f . Generally, si-
mulations of composite structures are ill-conditioned because of the
E22, E33 10 GPa ν 0.35 strong heterogeneity and the often complex, non-grid aligned aniso-
G12, G13 5.2 GPa tropy of the material. The condition number of K, which is defined as
G23 3.5 GPa Resin edge material
the ratio of the largest and the smallest eigenvalue of K, grows roughly
ν12, ν13 0.35 E 8.5 GPa
ν23 0.5 ν 0.35 linearly with the size of the largest jump in the entries of the stiffness
tensor from one finite element to an adjacent one. But the conditioning

271
A. Reinarz et al. Composite Structures 184 (2018) 269–278

σl σs Fig. 2. Stresses in MPa as a function of radius r from outer

radius at the apex of the curve, at 2.156 mm from the edge
DUNE 100 of the resin-edge-treated laminate (DUNE, solid blue;
20 ABAQUS ABAQUS, dotted red). The background colours indicate the
50 stacking sequence: +45 = red, −45 = blue, 90 = green,
0 = yellow. (For interpretation of the references to colour
10 in this figure legend, the reader is referred to the web ver-
0
sion of this article.)

−50
0
−100
−10 DUNE
−150
ABAQUS

0 0.5 1 1.5 2 2.5 mm 0 0.5 1 1.5 2 2.5 mm

σr τls
20
4

10
3

0
2

−10
1
DUNE DUNE
ABAQUS −20 ABAQUS
0
0 0.5 1 1.5 2 2.5 mm 0 0.5 1 1.5 2 2.5 mm
τlr τsr

4 DUNE
4 ABAQUS

2
2

0 0

−2
−2
DUNE
−4
ABAQUS

0 0.5 1 1.5 2 2.5 mm 0 0.5 1 1.5 2 2.5 mm

also worsens systematically, as the problem size N increases. This PKu = P f . (2)
growth is typically of order N 2/3 in 3D problems. This growth affects the
accuracy of the solution in direct solvers, but it has no effect on the The better the approximation P ≈ K−1, the closer the system matrix
computational cost. in (2) is to the identity and thus to a well-conditioned problem with
By contrast, for iterative methods, such as Gauss–Seidel, conjugate condition number close to 1. However, as outlined above, computing
gradients or GMRES methods, the conditioning of K has a direct impact the inverse of K (e.g. via factorisation) is too expensive. Ideally the cost
on the number of iterations and thus on the cost. The basic idea of of applying P and the memory requirements should again only grow
iterative methods is to approximate the solution iteratively, starting linearly with N. A good choice of P is problem dependent, and must be
from some initial guess, by sequentially reducing the residual carefully designed and tuned, to obtain fast solvers which scale well for
large problems over multiple cores. However, some well-understood
r = Ku− f common design choices for FE discretisations of elliptic partial differ-
ential equations exist. To achieve independence (or at most logarithmic
at each iteration [17]. A ’good’ or fast iterative solver reduces this error dependence) of the number of iterations on the problem size N, it is
quickly, in a few iterations. For well-conditioned problems, standard paramount to apply multilevel preconditioners, either in the form of
black-box iterative solvers (as offered by ABAQUS) work well. But the multigrid [7,30] or multilevel domain decomposition methods [29]. In
number of iterations typically grows with the square-root of the con- addition, the memory demands of such iterative solvers are small
dition number of K. For the poorly conditioned problems of interest in (proportional to N) and the work can be easily distributed over multiple
this paper, standard iterative methods will either converge very slowly cores.
or completely fail to do so. Care has to be taken though to achieve robustness with respect to
A standard approach to improve the conditioning of such problems heterogeneities and anisotropies in the material properties. Since black-
is to apply a preconditioner P to (1), typically a cheap approximation of box multilevel approaches do not scale indefinitely and are not robust,
K−1, and then to iteratively solve more tailor-made approaches are needed. The flexibility to prescribe or

272
A. Reinarz et al. Composite Structures 184 (2018) 269–278

problems this has been proved rigorously in [28]. The robustness is due
to its good approximation properties for problems with highly hetero-
geneous material parameters. This is in fact of independent interest
[2,11]. Details of our specific implementation of GenEO in DUNE are
given in [26].

2.4. Performance tests and parallel efficiency

First, we compare the performance of various preconditioners for

the iterative solver in dune-composites when simulating the problem
specified in Section 2.2, as well as benchmarking them against the di-
Fig. 3. Decomposition of a mesh into several subdomains (Ω), showing an overlap of one rect solver in ABAQUS. The iterative solvers in ABAQUS only converge
element in each subdomain.
for homogeneous problems or for very small numbers of degrees of
freedom. The measures for comparing the solvers are their overall
implement such tailor-made preconditioners is not available in com- computational cost and their parallel efficiency. All the experiments in
mercial software, and therefore the development of codes like dune- Fig. 5 were carried out on the computer cts04 which consists of four 8-
composites is pivotal in solving large-scale problems in composite core Intel Xeon E5-4627v2 Ivybridge processors, each running at
structures. 1.2 GHz and giving a total of 32 available cores.
In dune-composites we use the conjugate gradient (CG) iterative On the left in Fig. 5, we compare the iterative solver with sequential
method for (2) and we test two types of preconditioner: the generic AMG preconditioner in DUNE (with tolerance 10−4 ), an iterative solver
Algebraic Multi Grid (AMG) preconditioner [3], as well as a new two- with an additive Schwarz preconditioner (with and without GenEO
level overlapping domain decomposition preconditioner (GenEO) with coarse space) and the sparse direct solver in ABAQUS for increasing
a multiscale coarse space based on local, generalised eigensolves in problem sizes. When the additive Schwarz preconditioner is used, the
each of the subdomains [28]. For their parallelisation, both methods domain is divided into 32 subdomains with an overlap of two elements.
partition the physical domain and the FE mesh into subdomains of All subdomains are solved on the same core. We see that the iterative
roughly equal size and distribute them across multiple cores (see solver in DUNE scales linearly with the problem size N (the number of
Fig. 3). The key ingredient in both preconditioners is a coarse approx- degrees of freedom) while the cost of the direct solver in ABAQUS
imation of u with significantly fewer degrees of freedom that is cheaper grows significantly faster. Asymptotically, the growth is more than
to obtain then u . This coarse approximation is then interpolated back to quadratic in N. Moreover, the direct solver runs out of memory on one
the original FE mesh. core for the biggest problem tested in DUNE (N ≈ 7.5 million). We
In Algebraic Multigrid, the problem is coarsened repeatedly by ag- specifically use the block version of AMG in our tests in dune-composites
gregating degrees of freedom in the stiffness matrix K, creating a and measure strength of connection between two blocks for the ag-
hierarchy of coarse approximations of decreasing size. The aggregation gregation procedure in the Frobenius norm. As the smoother, we use
process is purely algebraic and based on the fact that the displacement two iterations of symmetric successive overrelaxation (SSOR). The
ui and uj at two neighbouring nodes x i and x j in the FE mesh will be number of iterations remains almost constant and moderate in this test.
similar if the nodes are “strongly connected”, i.e. ui ≈ uj if |Ki,j | is large However, AMG has not originally been designed for serendipity ele-
(relative to |Kii | and |Kjj |). The degrees of freedom at two (or more) ments, and in its current form it does not seem to be robust for this
nodes that are strongly connected are then aggregated to a single de- element, especially in parallel.
gree freedom to obtain a cheaper approximation. This process can be The second comparison, on the right in Fig. 5, shows the parallel
applied recursively until a sufficiently coarse approximation has been scaling of various solvers. Here, we plot the strong scaling, that is, the
obtained that can be efficiently solved using a direct method. On the reduction in computational cost due to an increase in the number of
intermediate levels and on the original mesh we only apply a cheap cores for a fixed problem size. We see that up to 32 cores both the
relaxation method such as Gauss–Seidel. The success of the AMG pre- parallel direct solver in ABAQUS (filled circle), as well as the iterative
conditioner largely depends on how well this aggregation process works solver with domain decomposition preconditioner in DUNE (diamond
(for details see [3,30]). points and hollow circle) show almost optimal parallel scaling, that is,
The GenEO preconditioner is of overlapping Schwarz-type [29] and the CPU time is halved every time the number of cores doubles. We
uses only two levels. The domain is partitioned into overlapping sub- tested two versions of the domain decomposition preconditioner. The
domains of roughly equal size (see Fig. 3). In each of these subdomains, red line corresponds to a one-level overlapping Schwarz preconditioner
the original FE problem subject to zero-displacement boundary condi- with an overlap of two layers of finite elements. The subdomain pro-
tions on the artificial subdomain boundaries is solved either exactly blems are solved with UMFPACK [9], a sparse direct solver, similar to
using a direct method or approximately using, for example the AMG the one in ABAQUS. Again, we use 32 subdomains, and in this case we
preconditioner. To capture the global behaviour of the solution, a partition them evenly onto the available cores. The blue line corre-
coarse problem is introduced. For homogeneous structures, standard sponds to the new GenEO preconditioner, a two-level overlapping
finite elements on a coarser mesh are sufficient [29]. For composite Schwarz preconditioner where, in addition to the subdomain solves, we
structures, especially in the presence of defects, this approach is too also build and solve a problem-adapted coarse problem (see Section
inaccurate. Instead, the GenEO preconditioner computes a few of the 2.3). This coarse problem is again solved using UMFPACK.
smallest energy (eigen) modes of the structure on each of the sub- The addition of the domain decomposition preconditioner was es-
domains. This includes the rigid body modes, but also some more exotic sential for a good parallel efficiency. For some reasons which we cannot
modes that arise due to the particular properties of the composite. Some fully explain, the parallel AMG preconditioner, which is available as a
of these low energy modes can be seen in Fig. 4. Finally, these local default within DUNE did not scale well in parallel. The addition of the
modes are combined to generate a coarse global approximation using a GenEO coarse space dramatically reduces the number of iterations from
so-called partition of unity approach [29]. The coarse problem is again about 400 for the one-level overlapping Schwarz preconditioner (dia-
solved using a direct method or an iterative method. For details see the mond points) to about 40 for GenEO (hollow circle). Both those num-
original paper [28] and the related works [15,2]. The GenEO coarse bers remain roughly constant across this range of cores. In absolute
space has been shown to lead to a fully robust two-level Schwarz pre- terms, the lower number of iterations for GenEO translates into an al-
conditioner which scales well over multiple cores [28,20]. For isotropic most threefold reduction in computational time. Moreover, beyond 32

273
A. Reinarz et al. Composite Structures 184 (2018) 269–278

Fig. 4. Schematic of low energy eigenmodes in a reference

block consisting of a layer of resin (white) between two
layers of plies (green: 90°, blue: −45°, red: 45°); from left to
right, a reference block, a zero energy mode (rotation), a
low energy mode between two 90° plies and a low-energy
twisting mode caused by 45° and −45° plies. (For inter-
pretation of the references to colour in this figure legend,
the reader is referred to the web version of this article.)

Fig. 5. Left: Timing comparison in terms of degrees of

freedom between the sparse direct solver in ABAQUS and
the iterative CG solver (with AMG preconditioner) in DUNE.
Right: Timing comparison in terms of numbers of cores
between the sparse direct solver in ABAQUS and the itera-
tive CG solver (with one- and two-level domain decom-
position preconditioner) in DUNE.

cores, only the GenEO preconditioner continues to scale. The numbers results for two different setups. For the first, larger problem we have
5
of iterations for the one-level Schwarz preconditioner start to increase used 56 elements along the radius, 8 Ncores elements across the width and
with the number of subdomains. 4 elements through thickness in each of the resin and ply layers, re-
Finally, we note that even for the relatively small test we have spectively. This setup ensures that the amount of work handled by each
chosen here (432,768 elements) the iterative solver in DUNE pre- core remains constant as Ncores varies. For the second, smaller problem
conditioned with GenEO is around 5–10 times faster than the direct we have halved the number of elements through the thickness, using
solver in ABAQUS. Moreover, the restriction to shared-memory paral- only 2 elements in each resin and ply layer. In Fig. 6, we see that after a
lelism and the memory requirements of the direct solver in ABAQUS, slight initial growth the scaling of the iterative solver in DUNE with
combined with the O (N 2) growth in computational cost, make it es- GenEO preconditioner is indeed almost optimal to at least 2048 cores,
sentially impossible to carry out a corresponding scaling test with allowing us to increase the size of the tests at a nearly constant run time
ABAQUS beyond 32 cores. and thus, to solve a problem with 173 million degrees of freedom in just
By contrast, in Fig. 6, we now look at the parallel scaling of the over 2 min.
iterative solver with GenEO preconditioner in dune-composites on
thousands of cores solving problems with up to 173 million degrees of
3. Defect analysis and the influence of boundaries
freedom. We use again UMFPACK for all subdomain and coarse solves
in GenEO. For this second scaling test, we move to a larger scale pro-
In manufacturing, small localised defects in the form of misaligned
blem. The model run in this section has 39 plies and a variable width.
fibrous layers can occur, these defects can have a large effect on the
For details on the setup of this problem see Section 3. We also move to a
strength of the materials. In this section we investigate the effect that
larger computer. All the experiments in Fig. 6 were carried out on the
varying the maximum slope and amplitude of a localised wrinkle with
University of Bath HPC cluster Balena. This consists of 192 nodes each
only one oscillation has on the strength of the curved laminate. We
with two 8-core Intel Xeon E5-2650v2 Ivybridge processors, each run-
show that we need a very high-fidelity mesh to be able to compute
ning at 2.6 GHz and giving a total of 3072 available cores.
localised stresses and this problem provides an appropriate application
We carry out a weak scaling experiment, that is, we increase the
for dune-composites. In order to determine the shape and size of the
problem size proportionally to the number of cores used. For an itera-
wrinkle we used an X-ray CT scan of a typical curved laminate with a
tive solver that scales optimally both with respect to problem size and
wrinkle defect, shown in Fig. 7 and described in more detail below.
with the number of cores, the computational time should remain con-
For the defect analysis we examine the influence of width-wise
stant in this experiment. To scale the problem size as the number of
boundary conditions on the results. The mechanical properties for the
cores Ncores grows, we increase the width of the laminate. We present
ply material and the resin interfaces are the same as those for the

Fig. 6. Scaling of dune-composites using the GenEO pre-

conditioner on up to 2048 cores. Solid lines are used for the
problem with 8.4 · 104 degrees of freedom per core; dashed
lines for 4.2·10 4 degrees of freedom per core. For the larger
problem, the problem size varies from about 1.3 million
degrees of freedom on 16 cores to about 173 million degrees
of freedom on 2048 cores.

274
A. Reinarz et al. Composite Structures 184 (2018) 269–278

Fig. 7. Image of a defect. Left: A scan of a wrinkle with the

sech2 (·) superimposed. Center: A radial cut through the
center of the laminate. Right: Decay of the wrinkle across
the width.

1 1
previous smaller test, see Table 1. The geometry of this 39 ply problem Further, b1 = 5 and b3 = 2 . As shown in Fig. 9, the sech2(·) function
is chosen to give a similar ratio of width to radius as the previous test. never reaches zero, however at ± bi it is already negligibly small
Here we use a width W of 52 mm and 10 mm long limbs L. The inner (< 10−2 ). The coordinate s ranges from 0 to π ,r ranges from R to R + T
1
radius R of the curved section is 22 mm. The overall thickness T is and l ranges from 0 to W, which implies that a value of b3 = 2 leads to a
9.93 mm, where the ply layers are 0.24 mm thick and the resin inter- wrinkle spanning half of the width of the part.
faces are 0.015 mm thick. The fibre angles of plies are given by the The wrinkled mesh on the cube is finally mapped to the curved
following stacking sequence section via the mapping

x =̂ g1 (s ,̂ l ,̂ r )̂ = L + r sin ( s )̂
π
[[ ∓ 45/90/0]2 /[ ∓ 45]2 /90/ ∓ 45/90/0/ ∓ 45/0/ ± 45/0/90/ ± 45/90/[± 45]2 ̂ 2
/[0/90/± 45]2 ].
y =̂ g2 (s ,̂ l ,̂ r )̂ = l ̂
The boundary conditions and loading in a narrow specimen are very z =̂ g3 (s ,̂ l ,̂ r )̂ = r cos ( s )̂ ,
π
̂
different from those of the full structure. For this reason we initially 2 (5)
model using periodic boundary conditions at the free edge. This gives where L is the length of the limb.
an approximation of the effect of the wrinkle in a very wide part. For From the form of the wrinkle we can easily calculate the steepest
the later tests we remove the periodic boundary conditions and add a slope by deriving
3 mm wide layer of resin to the free edges. This reduces the strength of
the singularity at this edge and for larger wrinkles, ensures that failure d 2 s−sdef ⎞
tan−1 ⎛ f (s,l,r )⎞ = tan−1 ⎛⎜− tanh ⎛π f (s,l,r )⎞⎟
⎜ ⎟

occurs at the center of the laminate, near the defect. ⎝ ds ⎠ ⎝ b1 ⎝ πb1 ⎠ ⎠ (6)
Fig. 8 shows the process by which the wrinkle is created. We start
with a mesh on a flat plate, then we add the wrinkle to the flat plate and This allows us to classify wrinkles according to the steepest angle
finally transform to the curved geometry. Let (s,l,r ) be coordinates in produced by the defect. In order to ensure that the edge effect does not
the unperturbed, flat geometry. Note, that in the final curved geometry, affect the results we give results with periodic boundary conditions. In
s corresponds to an arc length, r is the radius through the part and l is Fig. 7 we give a sample defect from a CT scan with an overlay showing
the width across the part. the fit to our proposed parameterisation of the wrinkle. We also show
In the flat geometry the wrinkle is given by a perturbation in r, the resulting model with the largest amplitude (and thus slope) of de-
which depends on all three coordinates. The perturbed coordinates fect considered in the following numerical tests.
(s ,̂ r ,̂ l )̂ are given by
3.1. Convergence Analysis
s ̂ = s, l ̂= l and r ̂ = r + f (s,l,r ). (3)
The FE model used for these tests consists of approximately 7.3·106
If we denote by (sdef ,l def ,rdef ) the location where the defect is largest,
degrees of freedom in total. There are 96 elements along the radius, 40
the perturbation is given by
across the width and 4 each in the resin and ply layers. To resolve the
s−sdef ⎞ r −rdef ⎞ l−l defect the model is refined towards the location of the defect, with a
f (s,l,r ) = d sech2 ⎛π
⎜ sech2 ⎛π
⎟ ⎜ sech2 ⎛π def ⎞,
⎟ ⎜ ⎟
bias ratio of 2, i.e. the elements at the center of the model are half as
⎝ πb1 ⎠ ⎝ Rb2 ⎠ ⎝ Wb3 ⎠ (4)
large as those at the edges. For the edge treated case there is also re-
where d gives the amplitude of the defect and bi,i = 1,2,3, are para- finement towards the resin edge with a bias ratio of 400 to resolve the
meters giving the “extent” of the defect in the three directions. stress concentrations that occur near the edge.
For these tests we have chosen b2 so that the wrinkle decays more To ensure that the FE modelling is sufficiently accurate we include a
quickly towards the inner radius, more precisely convergence analysis. In Fig. 9 (right) we plot the relative error in the
1 peak failure criterion given by Eq. (7) below. As an approximation to
⎧ 2 ifr −rdef < 0 the exact solution we used the next refinement level of the FE model
b2 = .
⎨ 1 ifr −rdef > 0 (136 elements along the radius, 60 across the width and 8 each in the
⎩4
resin and ply layers). For the refinement we have chosen (second to last

Fig. 8. Transformation of the flat geometry into a flat geo-

metry with a defect and finally into the curved geometry
with a defect.

275
A. Reinarz et al. Composite Structures 184 (2018) 269–278

Fig. 9. Left: Decay of the sech2() function for 3 different values of parameter B. Right: Convergence plot for a pristine part and a part with a defect with maximum slope 18°, with labels
giving the number of elements through-thickness per ply and interface layer.

point) the relative error has been reduced to around 1% for both the interlaminar shear and direct stresses with a wrinkle of maximum angle
pristine model and for the model containing a defect. For the case with (slope) of 4° and 18° for which an opening/bending moment of 9.58 kN
a wrinkle it is important to note that in the first three refinement levels mm/mm is applied. The interlaminar shear maximises in the region of
have very similar peak failure criterion, even though the model is not greatest slope, while the interlaminar direct stress maximises in the
yet converged. However, in these configurations the wrinkle is not yet region of peak amplitude. These stresses are highly localised and cause
sufficiently resolved and the failure criterion output is still closer to that a sharp rise in failure index through-thickness, as shown in Fig. 11. The
of the pristine model. pristine part is predicted to fail near the mid-thickness, where direct
The models used in this analysis increase the number of elements interlaminar stress maximises. With a wrinkle of slope 4°, there is a rise
through thickness in each layer as follows: 1, 1, 2, 4, 6, 6, 8. The in failure index near the inner radius, however failure is still predicted
number of elements across the part and around the radius and limbs are to first occur near the mid-thickness (as per the pristine model). As the
increased simultaneously. We can see that (especially for a part with a slope of the wrinkle is increased, so failure index near the inner radius
defect) the number of elements through thickness is very important. becomes higher than that near the mid-thickness. This is also indicated
With fewer than 4 elements we do not get a good approximation of the in Table 2, showing failure at interface 16 for pristine and 4° wrinkle
maximum failure criterion. Further, when fewer elements are used the models, and at interface 4 for greater slopes. For the most severe
stresses drop to zero at the surface of the laminate. wrinkle of slope 18°, the failure index is almost double that of the
Around the curve enough elements need to be used to ensure that pristine model, and hence strength is halved.
the wrinkle is resolved, at least 10 elements should be used along the A summary of all models is shown in Table 2. As seen by the
length of the wrinkle. Even with a mesh grading towards the wrinkle maximum recorded displacement, the overall stiffness of the curved
this still requires a large total number of elements across the full length laminate is not significantly affected by the inclusion of a wrinkle de-
of the part. fect. This is because it is very small in comparison to the size of the
whole laminate. However, it does significantly reduce strength, based
3.2. Defect analysis on initiation of failure. When tested in isolation such specimens have
been found to fail in a highly unstable manner, suggesting once in-
The stresses in the vicinity of a defect are complex, including in- itiation is reached there is instantaneous propagation and catastrophic
terlaminar, shear and direct stresses. Therefore a mixed mode failure failure [14]. Note that the two pristine results in Table 2 compare well
criterion is more suitable than a maximum stress criterion. The strength with predictions acquired by modelling the curved laminates using
of the laminates was assessed using a quadratic damage onset criterion, ABAQUS software [14]. Note also that the edge-treated pristine result
defined as compares well with experimental results [14]. Table 2 shows that the
greatest impact of the wrinkle is to introduce significant interlaminar
+ 2 2 2 shear stresses. Interlaminar direct stress is also increased, by approxi-
⎛ σr ⎞ + ⎛ τrs ⎞ + ⎛ τrl ⎞ = F
⎜ ⎟ ⎜ ⎟ ⎜ ⎟
mately 50% between the pristine and 18° wrinkle models; however, the
⎝ s33 ⎠ ⎝ s13 ⎠ ⎝ s13 ⎠ (7)
interlaminar shear stresses go from being negligible in the pristine
with negative values of σr treated as zero and the following allowables: model, to becoming greater than the interlaminar direct stress in the
18° model.
s33 = 61 MPa s13 = 97 MPa. Note that these results do not take into account thermal pre-stresses,
This quadratic criteria follows from original work by Hashin and which occur as a result of the high-temperature curing process and may
Rotem [19] on matrix failure in composite laminates. It has since been be significant. However, the purpose of this paper is to illustrate the
widely used to successfully predict the onset of delamination, e.g. [8]. importance of mesh refinement for rapidly varying through-thickness
Curved laminates subjected to corner unfolding are observed to fail stresses caused by wrinkling defects. Furthermore, the geometric
by delamination of the plies [14]. Since this indicates failure occurs at parameters considered are assumed to be constant but will vary sig-
the interface between plies, we apply the failure criterion only in the nificantly in real applications. For instance the resin rich ply inter
resin-rich interface zones and failure initiates when F = 1. Generally, thicknesses are likely to vary considerably in real laminates and parti-
the failure criterion is applicable in the local coordinates of the mate- cularly in realistic structural components, and this could significantly
rial, however since the interface zone is isotropic, a transformation is influence the interlaminar stresses.
not required.
Having established the wrinkle defect model, we now investigate
the effect of varying the wrinkle severity. Fig. 10 shows the effect on the

276
A. Reinarz et al. Composite Structures 184 (2018) 269–278

Fig. 10. Stresses τrl and σl for defect angle 4° (left), followed by stresses τrl and σl for defect angle 18° (right).

analysis: running thousands of simulations with defects in various lo-

cations and sizes, studying the influence on the integrity of a compo-
nent. The high number of simulations, combined with the requirement
for high fidelity in each simulation, results in a very demanding pro-
blem computationally. The use of a bespoke FE solver, such as the one
developed here in DUNE, is essential for meeting these demands.
Current industry standard FE tools, e.g. ABAQUS, are not able to
deal with these problem sizes, largely due to the direct linear equation
solvers that are employed and the restricted parallel scalability.
Iterative solvers within ABAQUS only converge in very simple cases.
For this reason, as part of our project, a new iterative solver had to be
implemented within DUNE. The heart of this solver is a robust pre-
conditioner (GenEO) that leads to an almost optimal scaling of the
computational time both with respect to the number of degrees of
freedom and the number of compute cores. This has been demonstrated
for problems with up to 173 million degrees of freedom, achieving run
Fig. 11. Through-thickness quadratic damage onset criterion F at the center of each resin
interface (with linear interpolation between resin interfaces). Values are calculated at times of just over 2 min on 2048 cores.
mid-width along the angle of steepest slope (2.1° from the apex of the curve), and at a The chosen example problem evaluates the through-thickness
distance r from the outer radius. stresses caused by unfolding a curved laminate. This serves to illustrate
the importance of using high-fidelity meshes and provides insight into
4. Conclusions the influence of small manufacturing induced defects.

In this paper, a new high-performance FE analysis tool for compo-

site structures is presented and shown to accurately model the beha- Acknowledgements
viour of complex laminate structures with manufacturing defects in a
fraction of the computational time required by commercial software This work was supported by an EPSRC Maths for Manufacturing
packages, such as ABAQUS. grant (EP/K031368/1). Richard Butler holds a Royal Academy of
The stress field around a wrinkle defect in composite materials is Engineering-GKN Aerospace Research Chair in Composites. This re-
highly localised, requiring a high fidelity mesh to adequately capture it. search made use of the Balena High Performance Computing Service at
Such defects that arise in composite materials during manufacture are the University of Bath.
by nature difficult to predict. There is a case for performing stochastic

277
A. Reinarz et al. Composite Structures 184 (2018) 269–278

Table 2
Effect of the defect angle on peak stresses, on the failure criterion F in interfaces and on M the predicted moment of initiation of failure.

Defect slope (deg.) max. disp. (mm) location interface (from outer radius) max. failure crit. F σl (MPa) σs (MPa) σr (MPa) τsl (MPa) τrs (MPa) τrl (MPa) M(kN)

FEA (normalised): Periodic BC

0 3.99 Mid- width 23 1.00 33.76 34.90 61.00 0.19 0.02 −0.23 9.581
4 3.98 Mid- width 23 1.07 37.26 40.72 65.59 0.17 0.95 0.47 8.90
8 3.98 Mid- width 35 1.30 51.67 113.2 65.69 −8.75 6.50 71.05 7.34
12 3.98 Mid- width 35 1.65 45.41 117.4 79.94 −12.80 9.97 97.59 5.78
18 3.98 Mid- width 35 1.94 37.31 131.6 92.99 −17.37 14.91 116.4 4.92

FEA (normalised): Resin edge treated

0 5.16 Edge 23 1.16 80.06 116.25 11.13 6.75 28.13 −107.5 8.262
18 5.16 Mid- width 35 1.99 37.68 144.15 96.67 −16.72 116.80 0.57 4.79

1
ABAQUS result was 9.51 kN [14]
2
ABAQUS result was 8.25 kN, test average 8.65 kN[14]

References [16] Gan KW, Allegri G, Hallett SR. A simplified layered beam approach for predicting
ply drop delamination in thick composite laminates. Mater Design
2016;108:570–80.
[1] Arnold DN, Awanou G. The serendipity family of finite elements. Found Comput [17] Hackbusch W. Iterative solution of large sparse systems of equations. NY: Springer;
Math 2011;11(3):337–44. 1994.
[2] Babuska I, Lipton R. Optimal local approximation spaces for generalized finite [18] Harper PW, Hallett SR. Cohesive zone length in numerical simulations of composite
element methods with application to multiscale problems. Multiscale Model Simul delamination. Eng Fract Mech 2008;75(16):4774–92.
2011;9(1):373–406. [19] Hashin Z, Rotem A. A fatigue failure criterion for fiber reinforced materials. J
[3] Bastian P, Blatt M. On the generic parallelisation of iterative solvers for the finite Compos Mater 1973;7(4):448–64.
element method. Int J Comput Sci Engin 2008;4:56–69. [20] Jolivet P, Hecht F, Nataf F, Prud’homme C. Scalable domain decomposition pre-
[4] Bastian P, Blatt M, Dedner A, Engwer C, Klöfkorn R, Ohlberger M, Sander O. A conditioners for heterogeneous elliptic problems. In: Proc. Internat. Conf. High
generic grid interface for parallel and adaptive scientific computing. Part I: Abstract Performance Computing, Networking, Storage and Analysis (SC ’13), pages
framework Comput 2008;82:103–19. 80:1–80:11, New York, NY, USA, 2013. ACM.
[5] Bastian P, Blatt M, Dedner A, Engwer C, Klöfkorn R, Ohlberger M, et al. A generic [21] Li X, Hallett SR, Wisnom MR. Predicting the effect of through-thickness compressive
grid interface for parallel and adaptive scientific computing. Part II: Implementation stress on delamination using interface elements. Compos Part A 2008;39(2):218–30.
and tests in DUNE. Computing 2008;82:121–38. [22] Liu D, Fleck NA, Sutcliffe MPF. Compressive strength of fibre composites with
[6] Bastian P, Heimann F, Marnach S. Generic implementation of finite element random fibre waviness. J Mech Phys Solids 2004;52(7):1481–505.
methods in the distributed and unified numerics environment (DUNE). Kybernetika [23] Mukhopadhyay S, Jones MI, Hallett SR. Compressive failure of laminates containing
2010;46:294–315. an embedded wrinkle: experimental and numerical study. Compos Part A
[7] Briggs WL, Henson VE, McCormick SF. A Multigrid Tutorial. 2nd Ed. Philadelphia, 2015;73:132–42.
PA, USA: SIAM; 2000. [24] Mukhopadhyay S, Jones MI, Hallett SR. Tensile failure of laminates containing an
[8] Camanho PP, Davila CG, de Moura MF. Numerical simulation of mixed-mode pro- embedded wrinkle; numerical and experimental study. Compos Part A
gressive delamination in composite materials. J Compos Mater 2003;37:1415–38. 2015;77:219–28.
[9] Davis T. Algorithm 832: UMFPACK V4.3 – an unsymmetric-pattern multifrontal [25] Potter K. Understanding the origins of defects and variability in composites man-
method. ACM Trans Math Software 2004;30(2):196–9. ufacture. In: Proc. 17th Internat. Conf. Composite Materials, 2009. Edinburgh,
[10] Dodwell T, Butler R, Hunt G. Out-of-plane ply wrinkling defect during consolidation United Kingdom, July 2009.
over an external radius. Compos Sci Technol 2014;105:151–9. [26] Seelinger L. Robust domain decomposition methods for heterogeneous problems.
[11] Dodwell T, Sandhu A, Scheichl R. Customized coarse models for highly hetero- Master’s thesis, University of Heidelberg, 2017.
geneous materials. In E. Papamichos, P. Papanastasiou, E. Pasternak, and A. Dyskin, [27] Spillane N, Dolean V, Hauret P, Nataf F, Pechstein C, Scheichl R. A robust two-level
editors, Bifurcation and Degradation of Geomaterials with Engineering Applications domain decomposition preconditioner for systems of PDEs. CR Acad Sci
(IWBDG 2017), pages 577–584. Springer, 2017. 2011;349:1255–9.
[12] Duff IS, Erisman AM, Reid JK. Direct Methods for Sparse Matrices. Oxford, UK: [28] Spillane N, Dolean V, Hauret P, Nataf F, Pechstein C, Scheichl R. Abstract robust
Clarendon Press; 1986. coarse spaces for systems of PDEs via generalized eigenproblems in the overlaps.
[13] Esquej R, Castejon L, Lizaranzu M, Carrera M, Miravete A, Miralbes R. A new finite Numer Math 2014;126(4):741–70.
element approach applied to the free edge effect on composite materials. Compos [29] Toselli A, Widlund O. Domain decomposition methods - algorithms and theory.
Struct 2013;98:121–9. Springer Series in Computational Mathematics volume 34. New York: Springer;
[14] Fletcher TA, Kim T, Dodwell TJ, Butler R, Scheichl R, Newley R. Resin treatment of 2004.
free edges to aid certification of through thickness laminate strength. Compos Struct [30] Vassilevski PS. Multilevel block factorization preconditioners: matrix-based analysis
2016;146:26–33. and algorithms for solving finite element equations. New York: Springer; 2008.
[15] Galvis J, Efendiev Y. Domain decomposition preconditioners for multiscale flows in
high-contrast media. Multiscale Model Simul 2010;8(4):1461–83.

278