Managing data at scale
I/O & Data Flow
Four different I/O & Data Flow releated technical challenges are targeted within EoCoE-II:
Improvement of I/O accessibility: Different I/O libraries support a variety of different configuration options. Depending on the situation these options must be continually updated, or a complete new library must be adapted. We want to introduce a generic interface with the help of the Portable Data Interface (PDI), which decouples the I/O API and the application to allow easier switching between different I/O subsystems. These interface should either serve standard I/O operation but should also be useful in context of ensemble or in-situ visualisation data movement.
I/O performance: The data writing and reading time can consume a significant part of the overall application runtime and should be minimized. For this we want to leverage the optimization options of different I/O libraries in use as well by adapting intermediate storage elements such as flash storage devices.
Resiliency: Running an application on a large scale increases the chance of hard- or software problems if more and more computing elements are involved in the calculation. Additional I/O techniques can be used to reduce the effort needed to restart a broken run or even avoid an overall crash, by storing intermediate snapshots to the storage elements. In particular we want to focus on resiliency for ensemble calculations.
Data size reduction: Running an application on a larger scale often implies an increasing data size, which can become unmanageable and consume too much resources. Within this task we want to reduce the overall data size without losing necessary information via in-situ and in-transit processing, moving post-processing elements directly into the frame of the running application.
Tools
Alya
is a high performance computational mechanics code that solves complex coupled multi-physics problems, mostly coming from the engineering realm. Alya will be deeply refactored in order to be able to address heterogeneous computing nodes. A very strong effort on linear algebra is planned for both the fluid and the solid problems handled in the two tasks foreseen in the wind scientific challenge.
Read More
Meso-NH
is an open-source non-hydrostatic mesoscale atmospheric model developed by the Centre National de Recherches Météorologiques (CNRM – Météo France/CNRS) and the Laboratoire d’Aérologie (LA – UPS/CNRS) [1]. This LES model has recently been extended to wind farm flow simulations by IFP Energies Nouvelles, using an Actuator Line Model (ALM) of the wind turbines [2]. It enables to simulate the interactions between wind turbines and surrounding atmospheric flows from meso- to micro-scales. A realistic atmosphere can be simulated with this model, including roughness effects, thermal effects, complex terrains, or even complex micro-physical phenomena.
Read More
waLBerla-wind
is an exascale-enabled tool allowing for the simulation of neutral atmospheric boundary layers and wind turbines based on the Lattice-Boltzmann method and the waLBerla framework. In EoCoE-3, it will be employed as an alternative solution for modelling wind turbines and wind farms, efficiently using heterogeneous computing resources. Thanks to the code generation techniques, vectorization via SIMD intrinsics and reduced floating-point precisions, waLBerla-wind can perform real-time simulations of the interactions between wind farm clusters.
Read More
IME
The Infinite Memory Engine (IME), as designed by partner DDN, one of the main storage technologies used successfully in HPC setups today. IME is a layered hybrid approach, where the capacity file system is extended by a layer of low-latency devices to cope with response time requirements. DDN IME originates from HPC applications and architectures, where I/O has been carefully co- designed over many years, yielding to an eco-system where the hardware is enhanced with complex middleware to meet application requirements. Applications have two possible data path to access to the IME client, either the Fuse interface (expose as a traditional mount point) or directly using the IME Native interface. The former offering compliance and ease of access and the later the extreme level of performance. Once of the technical milestone of EoCoE II is to interface the SIONlib middleware directly with the IME Native Interface.
See MoreFrom data flow to resilience
Results
Data Interface
Along the project, the PDI team has worked on the improvement of the PDI Data Interface in many ways. In addition to the improvement of PDI itself, the team has worked in close collaboration with simulation code developers to integrate and leverage the library in their applications: Alya, ESIAS, Gysela, Parflow, SHEMAT-Suite, …
A new much-improved website has been deployed (https://pdi.dev/), with access to hands-on tutorial, user and reference documentation as well as developer documentation. Packages are now offered to ease installation on Debian, Fedora, Ubuntu, and Spack in addition to the still available source distribution. The library is tested with about thousand of distinct tests on more that 20 architectures to ensure the production-grade quality.
PDI team is proud to have generated more than 30 releases of the library along EoCoE-2, each improving on the previous one.
Improved API
PDI integration in simulation codes has been made easier by improving its type system to match the requirements of its users. Support for new types makes it easy to integrate and inter-operate between C, C++, Fortran or python codes. A type annotation system has been developed that makes it possible to expose data to PDI without having to specify its layout. A dedicated compiler generates PDI data layout information from C code directly.
Additional back-ends
New plugins have been developed to extend the range of actions possible with the data exposed to PDI. A Decl’NetCDF back-end has been developed with support for both the sequential and parallel versions of NetCDF library. The plugin YAML configuration has been designed to be as close as possible to that of Decl’HDF5 hence making the switch from one to the other easy. This plugin leverages PDI new type annotation system developed in the scope of EoCoE to make integration seamless.
A plugin dedicated to the FlowVR framework is now part of PDI distribution. A plugin dedicated to the Melissa framework developed in the project is also developed. This plugin is developed along the Melissa framework itself and a final stable release is expected at the end of EoCoE-2.
Existing back-ends have also been updated and improved with new performance options available in Decl’HDF5. The Decl’SION and FTI plugins are also kept up-to-date to ensure long term support of our users.
In situ data manipulation capabilities
In addition to input and output to disk, PDI has seen huge improvement to its support for in situ data manipulation. PDI now offers direct support for data manipulation from YAML directly in the application process. In addition to its dedicated language, this can be done in python, C, C++ or Fortran.
In addition to the FlowVR and Melissa plugins mentioned previously, two plugins specifically targeted at in situ data analysis support have been developed and tested.
A plugin dedicated to SENSEI usage has been developed. This plugin has been used to interface the Alya simulation code with ADIOS-2 and SENSEI for in situ visualisation at scale[1], see below for more details.
Another plugin has been developed to target Dask. This plugin has been split in a Dask in situ analytics extension: Deisa and an associated PDI plugin. This approach has been used to execute heavy analysis in situ along the execution of the Gysela code with close to no impact on the simulation code or sequential python analytics[2].
[1] Christian Witzler, J. Miguel Zavala-Aké, Karol Sierocinski, and Herbert Owen. Including in situ visualization and analysis in pdi. In Heike Jagode, Hartwig Anzt, Hatem Ltaief, and Piotr Luszczek, editors, High Performance Computing, pages 508–512, Cham, 2021. Springer International Publishing.
[2] Amal Gueroudji, Julien Bigot, and Bruno Raffin. Deisa: dask-enabled in situ analytics. In HiPC 2021 – 28th International Conference on High Performance Computing, Data, and Analytics, pages 1–10, virtual, India, December 2021. IEEE.
Checkpointing in Ensemble Data Assimilation
As part of the EoCoE2 effort on improving I/O for a range of different use cases, we have designed and developed an effortless framework protection leveraging checkpoint/restart. The checkpoints fulfill the purpose of resiliency and hold the climate model results in form of shared HDF5 checkpoint files. The files contain both the background and analysis states of the data assimilation ensemble runs. We use dedicated threads and MPI processes to overlap the checkpoint creation with the framework execution. We demonstrate that the overhead is effectively hidden behind the framework’s normal operation (zero-overhead checkpoint). Our implementation manages further to recover from failures with none or few recomputation (zero-waste recovery). In addition, we derive a model that predicts the average cost of failures during continuous operation.
[3] HiPC 2021: Keller, Kai, Adrian Cristal Kestelman, and Leonardo Bautista-Gomez. “Towards Zero-Waste Recovery and Zero-Overhead Checkpointing in Ensemble Data Assimilation.” 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC). IEEE, 2021.
[4] HiPC 2020: Keller, Kai, Konstantinos Parasyris, and Leonardo Bautista-Gomez. “Design and Study of Elastic Recovery in HPC Applications.” 2020 IEEE 27th International Conference on High Performance Computing, Data, and Analytics (HiPC). IEEE, 2020.
In situ visualization
To enable simulations to integrate in-situ visualization we are using multiple dependencies, that are used in concert while hiding the complexity from the simulation. As the entry layer from the simulation we are using PDI, this allows a unified entry for in-situ and normal IO operations. As a next step pdi2sensei is used, to facilitate an easy setup of the in-situ pipeline, that is visible in the figure below. There you can see that the normal setup uses a divided setup, one part for the simulation and the other one for the data processing. This needs a data transport, which uses ADIOS2 as a transport layer, and allows the special in-situ case that is called in-transit, therefore data is transported to different nodes, to decrease the effect the visualization has on the running simulation. As a middle layer that inputs data and extracts it from ADIOS2 we are using SENSEI, which allows to link to multiple different in-situ frameworks and change the targets with small configuration changes. Therefore, this keeps the flexibility to choose a different data transport later on. In case of the visualization nodes we need this flexibility as well, to allow users a choice for the framework they feel most comfortable with for the actual data visualization. The approach used here is to use Catalyst, which is the in-situ library provided and linked to ParaView. This allows to run predefined visualization scripts and connect it to an interactive GUI to see live results of your data and try out new visualizations while getting new data delivered and update the actual visualization. The described approach was used for two of the EoCoE-II codes, namely in ALYA and ParFlow.
Flash storage evaluation
Besides pure software-based approaches to optimize the I/O capabilities of a certain application, we also analysed the impact of new types of storage devices Infinite Memory Engine (IME) technology by DDN to act as an intermediate cache device between the application and the underlying hardware.
As a testbed, we used the newly released High-Performance Storage Tier (HPST) system at JSC, which is built on top of IME technology and allowed us to be one of the first users to utilize and evaluate this new system of the centre.
To allow us to have a direct comparison we utilize an older version of the Gysela fusion application, which also offered the capabilities to use either a standard HDF5 task-local backend or a SIONlib based backed. Using SIONlib allowed us to also implement the IME native API underneath to evaluate also the utilization of this particular software backend in contrast to the regular FUSE mount based POSIX approach [5].
A weak scaling test, performed on JUWELS in a direct comparison between the HPST and the regular Spectrum Scale based parallel file system highlighted the clear benefit of having an intermediate cache layer. Utilizing the native API underneath of SIONlib slightly improved the bandwidth on smaller scales, where the individual client performance limits the overall bandwidth, while on larger scales the bandwidth is limited by the overall system and the server architecture.
Videos with research results using Alya simulations on wind flows can be found here
EoCoE team introduces order of magnitude improvement in wall modelling implementation for Finite Elements
Large-eddy simulation (LES) has emerged as a valuable tool for the simulation of turbulent flows at high Reynolds numbers and complex geometries, typically encountered in practical engineering applications. In this approach, the larger dynamically important eddies are directly resolved, whereas the smaller ones are assumed to have a universal character and are therefore modeled.
Nearly all real-world problems include a boundary that acts as a wall where the velocity is zero. In the case of Atmospheric Boundary Layer (ABL) flows that are of interest for Wind Energy, the ground is the wall. For wall-bounded flows, the computational cost becomes prohibitively expensive since the dynamically important eddies in the near-wall region become smaller as the Reynolds number increases, and thus, extreme mesh refinement is required to resolve them. Modeling the inner part of the boundary layer with a semiempirical wall model, instead of resolving it, is therefore a key ingredient for high Reynolds numbers flows to be solvable with LES. Even with the most powerful Exascale Supercomputers that will be available in the upcoming years, wall resolved LES simulations of Atmospheric Boundary Layer flows will not be feasible.
EoCoE team improves wind turbine simulation using Lattice-Boltzmann method
In order to better understand the flow physics in a wind farm, we need to compute the detailed flow characteristics like wind velocity, turbulent eddies, etc. in a reasonable time and with enough precision. At this stage only high-fidelity Navier-Stokes based Large-Eddy-Simulation (LES) solvers or even meteorological LES flow solvers are able to accurately reproduce such flows, and have been used rather extensively in the wind energy community. Such simulations enable the understanding of some flow phenomena and are very helpful in the improvement of analytical models. However, due to their high computational cost, only canonical cases are considered (single to few turbines), and full wind farm simulations have only been considered either for demonstration purposes or employing mesh resolutions that are not sufficient to accurately represent the flow physics.
To address this issue, FAU University together with IFPEN are currently implementing actuator-line based wind turbine models in the waLBerla LBM flow solver, with particular attention devoted to the collision operators. Whereas the Lattice-Boltzmann method is well-suited for lower Reynolds number regimes, standard collision operators suffer from instabilities at the high Reynolds numbers observed in wind energy applications. To obtain a stable and meaningful simulation, several investigations are conducted, typically using the Single and/or Multiple Relaxation Time collision operators together with different subgrid-scale models, such as e.g. the Smagorinsky and the WALE models. As this approach failed to stabilize the simulation sufficiently, more advanced collision operators, namely entropic, regularised, and cumulant-based are studied. For the wind turbine application, the cumulant Lattice-Boltzmann methods performed best in terms of stability. They have already been validated considering single or tandem wind turbines, and are being ported to Graphical Processing Units (GPU). The following figures illustrate the obtained results using CPU and GPU resources for the NewMexico wind Turbine. For this compressible simulation, Smagorinsky model is used with D3Q27 stencil and SRT collision operator. The simulation on one GPU (NVIDIA P100) substantially and expectedly outperforms the simulation on one CPU core (Intel Skylake G-6140) without any accuracy loss. Eight full nodes, i.e. 144 CPU cores, are required to obtain the same time-to-solution for both CPU and GPU systems.
Research on I/O and data management
Publications
All EoCoE-I and EoCoE-II publications are available here (openAIRE). click the button next
About Company
Innovating for a Sustainable Future
Rays of Progress & Solar Success serves as a powerful beacon, embodying the transformative journey toward a future powered by the boundless Energy of the sun. It's a vision where each
Partner Awaits
Harness the power of water with our AquaForce.
Global Experts
Wind turbines Powering a greener tomorrow.
About Company
Innovating for a Sustainable Future
Rays of Progress & Solar Success serves as a powerful beacon, embodying the transformative journey toward a future powered by the boundless Energy of the sun. It's a vision where each
Partner Awaits
Harness the power of water with our AquaForce.
Global Experts
Wind turbines Powering a greener tomorrow.
Project Success
Clients Reviews
Team Members
Winning Awards
OUR EXPERIENCE
Solar Passion Unveiled Our Story
Business, or everyday living, imagine a world where solar success is not just an aspiration but a reality driven by the Rays of the sun.
Hybrid Energy
Solar Panel
Wind Turbines
Expert Member
Solar Design Engineer
Steve Rhodes
Senior Manager
Lisa Thompson
Senior Engineer
David Martinez
Senior Technician
25,000 Customer
Satisfaction services
consultation
High Quality Solar Energy Solutions
Solar Powering Your Sustainable Scene is a vibrant and compelling phrase that conveys the idea of integrating solar energy into various aspects of life to create a sustainable
- The Environmental Benefits of Solar Energy
- Maximizing Your Renewable Investment
Work in Progress
This section is currently being updated. Please check back soon for more information.
