Preparing scientific applications for the Exascale era
Programming Models
Description
Choosing the right programming model(s) is an important decision that can strongly affect the development, the maintenance, the readability, the portability and the optimization of a code. Programming model challenges are tackled in the Work Package 2 of EoCoE.
Here, the term programming models mainly refers to as programming, optimization and parallelism languages, frameworks or libraries.It could be rephrased High Performance Computing (HPC) challenges. Our goal consists in helping, advising and guiding scientist code developers non-expert in HPC to refactor and optimize their applications to reach the best performance and scalability on massively-parallel super-computers.
Targeted super-computers can be composed of CPUs with accelerators such as GPUs. We currently test our improvements on Tier-0 European machines but our work is not limited to present technologies. Experts in HPC are preparing applications to run on future pre-exascale machines with disruptive technologies such as ARM-based processors and accelerators. We aim at prepare selected codes in the domain of energy to the Exascale ecosystem.
To achieve this goal, the programming model technical challenge has experts in performance evaluation. Having a systematic and continuous evaluation of the code performance enables to guide optimization developments all along the project and to monitor improvements. It helps to determine the best options to solve performance bottlenecks.
Performance analysis and monitoring
Alya
is a high performance computational mechanics code that solves complex coupled multi-physics problems, mostly coming from the engineering realm. Alya will be deeply refactored in order to be able to address heterogeneous computing nodes. A very strong effort on linear algebra is planned for both the fluid and the solid problems handled in the two tasks foreseen in the wind scientific challenge.
Read More
Meso-NH
is an open-source non-hydrostatic mesoscale atmospheric model developed by the Centre National de Recherches Météorologiques (CNRM – Météo France/CNRS) and the Laboratoire d’Aérologie (LA – UPS/CNRS) [1]. This LES model has recently been extended to wind farm flow simulations by IFP Energies Nouvelles, using an Actuator Line Model (ALM) of the wind turbines [2]. It enables to simulate the interactions between wind turbines and surrounding atmospheric flows from meso- to micro-scales. A realistic atmosphere can be simulated with this model, including roughness effects, thermal effects, complex terrains, or even complex micro-physical phenomena.
Read More
waLBerla-wind
is an exascale-enabled tool allowing for the simulation of neutral atmospheric boundary layers and wind turbines based on the Lattice-Boltzmann method and the waLBerla framework. In EoCoE-3, it will be employed as an alternative solution for modelling wind turbines and wind farms, efficiently using heterogeneous computing resources. Thanks to the code generation techniques, vectorization via SIMD intrinsics and reduced floating-point precisions, waLBerla-wind can perform real-time simulations of the interactions between wind farm clusters.
Read MoreWhen talking about hybrid parallelism on both CPU and GPU, it is in fact rare that the two architectures work together asynchronously. In most codes, the CPU acts as a handler: it handles initialization, I/O, communications and some tasks that are better suited to this architecture. The GPU handles computationally intensive tasks. However, when one is running, the other is often on standby.
What could formerly be done by the CPU is simply shifted to the GPU. This means that part of the allocated computing power is systematically lost. Today, task-based methods make it possible to use heterogeneous architectures asynchronously. Nevertheless these methods are still a subject of research and are not integrated into the most common software frameworks. Alya uses a load-balancing method to distribute some of its calculations equitably between the CPU and the GPU. The calculations are then performed at the same time. This allows to get the most power out of the computation nodes.
The Alya code has been adapted to work not only on CPUs but also on GPUs for Computational Fluid Dynamics problems, particularly Large Eddy Simulation cases. For such problems, a semi-implicit approach is used where the momentum equation is solved explicitly while the continuity equation is solved implicitly. Considering that most of Alya can run either in CPUs or GPUs, Alya’s developpers have decided to develop a co-execution approach that makes better use of current pre-exascale supercomputers, which typically blend GPUs and CPUs.
A Fast and scalable geometric mesh partitioning based on Space-Filling Curve (SFC) has been key to enable the co-execution with a correct load balance between the GPUs and CPUs. At the beginning of the simulation, the SFC partitioning is called several times iteratively until an optimum partitioning of the mesh is obtained.
In the rst iteration, each MPI task (be it CPU or GPU) receives a specic portion of the mesh according to some initial weights. With this partition, it calculates a couple of time steps. Based on the computational time taken by each MPI task, it adapts the weights and repartitions again. After a couple of iterations, each processor receives the correct amount of work so that they all take nearly the same time. GPUs obviously receive a more signicant chunk of the mesh than CPUs. In this way, the work done by the CPUs is spared in comparison to a pure GPU calculation.
PDI
ParFlow
GPU porting
A first example is the development done in the ParFlow to port the code on GPU. The developers adopted a strategy of abstracting memory allocation and loop management by designing an in-house DSL. This was done for several reasons. Firstly, it allows to hide from scientific developers the complexity of the code related to parallelism and HPC. Physicists can implement algorithms without worrying about which platform they will run on. Conversely, this allows HPC developers to optimize parallelism and support for new architectures seamlessly. This goes in the direction of a software engineering oriented towards the cutting of expertise and portability. In addition, developers may test the Kokkos programming model to do this work with a C++ formalism. The method used here should serve as an example for porting many applications.
Videos with research results using Alya simulations on wind flows can be found here
EoCoE team introduces order of magnitude improvement in wall modelling implementation for Finite Elements
Large-eddy simulation (LES) has emerged as a valuable tool for the simulation of turbulent flows at high Reynolds numbers and complex geometries, typically encountered in practical engineering applications. In this approach, the larger dynamically important eddies are directly resolved, whereas the smaller ones are assumed to have a universal character and are therefore modeled.
Nearly all real-world problems include a boundary that acts as a wall where the velocity is zero. In the case of Atmospheric Boundary Layer (ABL) flows that are of interest for Wind Energy, the ground is the wall. For wall-bounded flows, the computational cost becomes prohibitively expensive since the dynamically important eddies in the near-wall region become smaller as the Reynolds number increases, and thus, extreme mesh refinement is required to resolve them. Modeling the inner part of the boundary layer with a semiempirical wall model, instead of resolving it, is therefore a key ingredient for high Reynolds numbers flows to be solvable with LES. Even with the most powerful Exascale Supercomputers that will be available in the upcoming years, wall resolved LES simulations of Atmospheric Boundary Layer flows will not be feasible.
In a recent publication entitled “Wall-modeled large-eddy simulation in a finite element framework” we have improved the wall modelling implementation that has traditionally been used by the Computational Fluid Dynamics Finite Element community. When comparing with the traditional approach the error in the prediction of the mean velocity for high Reynolds number flows has been reduced by more than an order of magnitude. For Atmospheric Boundary Layer flows that have a particularly high Reynolds number this has been a key enabler. The traditional approach used by the Finite Element community would predict an average velocity that is approximately twice the correct value, rendering the results of little or no real use. With the improvements we have implemented the error is significantly reduced and Finite Elements become a very competitive solution for ABL flows. For y/H >0.06, that covers the region of interest of wind energy applications, an excellent prediction is obtained.
EoCoE team improves wind turbine simulation using Lattice-Boltzmann method
In order to better understand the flow physics in a wind farm, we need to compute the detailed flow characteristics like wind velocity, turbulent eddies, etc. in a reasonable time and with enough precision. At this stage only high-fidelity Navier-Stokes based Large-Eddy-Simulation (LES) solvers or even meteorological LES flow solvers are able to accurately reproduce such flows, and have been used rather extensively in the wind energy community. Such simulations enable the understanding of some flow phenomena and are very helpful in the improvement of analytical models. However, due to their high computational cost, only canonical cases are considered (single to few turbines), and full wind farm simulations have only been considered either for demonstration purposes or employing mesh resolutions that are not sufficient to accurately represent the flow physics.
To address this issue, FAU University together with IFPEN are currently implementing actuator-line based wind turbine models in the waLBerla LBM flow solver, with particular attention devoted to the collision operators. Whereas the Lattice-Boltzmann method is well-suited for lower Reynolds number regimes, standard collision operators suffer from instabilities at the high Reynolds numbers observed in wind energy applications. To obtain a stable and meaningful simulation, several investigations are conducted, typically using the Single and/or Multiple Relaxation Time collision operators together with different subgrid-scale models, such as e.g. the Smagorinsky and the WALE models. As this approach failed to stabilize the simulation sufficiently, more advanced collision operators, namely entropic, regularised, and cumulant-based are studied. For the wind turbine application, the cumulant Lattice-Boltzmann methods performed best in terms of stability. They have already been validated considering single or tandem wind turbines, and are being ported to Graphical Processing Units (GPU). The following figures illustrate the obtained results using CPU and GPU resources for the NewMexico wind Turbine. For this compressible simulation, Smagorinsky model is used with D3Q27 stencil and SRT collision operator. The simulation on one GPU (NVIDIA P100) substantially and expectedly outperforms the simulation on one CPU core (Intel Skylake G-6140) without any accuracy loss. Eight full nodes, i.e. 144 CPU cores, are required to obtain the same time-to-solution for both CPU and GPU systems.
Scientific publications related to the Programming Models.
Publications
All EoCoE-I and EoCoE-II publications are available here (openAIRE). click the button next
About Company
Innovating for a Sustainable Future
Rays of Progress & Solar Success serves as a powerful beacon, embodying the transformative journey toward a future powered by the boundless Energy of the sun. It's a vision where each
Partner Awaits
Harness the power of water with our AquaForce.
Global Experts
Wind turbines Powering a greener tomorrow.
About Company
Innovating for a Sustainable Future
Rays of Progress & Solar Success serves as a powerful beacon, embodying the transformative journey toward a future powered by the boundless Energy of the sun. It's a vision where each
Partner Awaits
Harness the power of water with our AquaForce.
Global Experts
Wind turbines Powering a greener tomorrow.
Project Success
Clients Reviews
Team Members
Winning Awards
OUR EXPERIENCE
Solar Passion Unveiled Our Story
Business, or everyday living, imagine a world where solar success is not just an aspiration but a reality driven by the Rays of the sun.
Hybrid Energy
Solar Panel
Wind Turbines
Expert Member
Solar Design Engineer
Steve Rhodes
Senior Manager
Lisa Thompson
Senior Engineer
David Martinez
Senior Technician
25,000 Customer
Satisfaction services
consultation
High Quality Solar Energy Solutions
Solar Powering Your Sustainable Scene is a vibrant and compelling phrase that conveys the idea of integrating solar energy into various aspects of life to create a sustainable
- The Environmental Benefits of Solar Energy
- Maximizing Your Renewable Investment
Work in Progress
This section is currently being updated. Please check back soon for more information.
