|
|
|
|
The Energy-oriented Centre of Excellence (EoCoE-II)
The EoCoE project, funded by the European Commission, applies cutting-edge computational methods in its mission to accelerate the transition to the production, storage and management of clean, decarbonized energy. This world-class consortium of 18 complementary partners from 7 countries, amounting to a unique network of expertise in energy science, scientific computing and HPC, intends to play its part in the reinforcement of a pan-European high performance computing infrastructure, benefitting researchers, key commercial players and SMEs alike.
|
EoCoE is now entering a second phase, EoCoE-II, and it intends to draw from the successful proof-of-principle phase of EoCoE-I, where a large set of diverse computer applications in the energy domain achieved significant efficiency gains thanks to a multidisciplinary approach with experts in applied mathematics and supercomputing.
|
|
The EoCoE consortium, which includes three leading European supercomputing centres, is channeling its efforts into five scientific Exascale challenges in the low-carbon energy sectors of Meteorology, Materials, Water, Wind and Fusion for energy. This multidisciplinary effort will harness innovations in computer science and mathematical algorithms within a tightly integrated co-design approach, and will ultimately demonstrate how exascale supercomputers can support the transition towards a low-carbon economy.
|
|
|
|
|
EoCoE events and activities
The EoCoE consortium held its second face-to-face meeting last September. Graciously hosted by our Belgian partners from the Université Libre de Bruxelles, this meeting was a great opportunity to ascertain the project’s overall progress and strengthen the bond between our respective institution and research teams.
|
Last October, the Erlangen University hosted a weeklong workshop on code optimization, gathering over fifteen participants to alternate between lectures and hands-on sessions. All lectures were recorded and will be available soon on the EoCoE website.
|
And since we are mentioning the EoCoE website, have you checked out the People@EoCoE page? This is where you will find an (almost) exhaustive list of our members!
|
In the upcoming months, EoCoE will keep ramping up its activities. Not only will we be present at SC19, but we will also participate to several events across Europe, including the 15th SIMAI Congress (Parma) and the European Sustainable Energy Week (Brussels) in June 2020.
|
|
The consortium will also hold its third face-to face meeting in the spring of 2020, quite probably on the premises of the Barcelona Supercomputing Centre, and will organize a series of webinars on several EoCoE-related codes and scientific domains.
|
|
|
|
|
|
|
EoCoE presence at SC19
Given SuperComputing’s central part in the animation of the HPC community, the EoCoE-II consortium will maintain a substantial presence at SC 19. You can meet our members and partners on the following institutional booths:
|
- Commissariat à l'énergie atomique et aux énergies alternatives (CEA)
- Forschungszentrum Jülich (FZJ)
- Institut national de recherche en informatique et en automatique (INRIA)
- Istituto Nazionale di Fisica Nucleare (INFN)
- Poznan Supercomputing and Networking Center (PSNC)
- Barcelona Supercomputing Center (BSC)
|
On top of that, Bruno Raffin, our lead INRIA member, will be the paper co-chair of the ISAV workshop on In Situ Infrastructures for Enabling Extreme-scale Analysis and Visualization; and Leonardo Bautista-Gomez, senior researcher at BSC, will chair the SC2020 Tutorials.
|
|
The EoCoE team as a whole is looking forward to meeting you in Denver!
|
|
|
|
Wind for Energy
Wind power is the renewable source with the most successful deployment over the past decade (2007-2016), growing from 93 GW to 490 GW of world-installed capacity during this time (source: “Renewable capacity statistics 2017”, International Renewable Energy Agency). A key role in this spectacular advance has been played by the increased understanding of the turbulent flow in the wind farms, thanks to ever-growing computational power and improved algorithms. In EoCoE-II, the ambition is to be able to simulate a full wind farm over complex terrain with up to 100 wind turbines. Ultimately this will require increases in the size of the problems to be solved and the amount of computational resources of at least two orders of magnitude. The main goal of the wind scientific challenge is to bring the flagship Alya code to Exascale in order to tackle this Large Eddy Simulation (LES) flow problem. Furthermore, the rotating wind turbine blades will be resolved to account for their effect on the flow via a full rotor model. Blade deformation may also be included via the use of sliding meshes. These combined tasks all pose daunting technical challenges for the kernel solvers and meshing algorithms in Alya, for which the EoCoE-II consortium is already well equipped.
|
Codes
|
waLberla (http://www.walberla.net) is an exascale-enabled multi-physics software framework. In EoCoE-2 its Lattice Boltzmann module will be employed as alternative algorithm for modeling wind turbines and wind farms. WaLberla has shown scalability up to 2 million parallel threads and can be used for simulations with up to a trillion (10^12) mesh cells that are partitioned as a forest of octrees.
|
waLBerla features advanced load balancing and performance tuning through advanced code generation and automatic code transformation technology
|
Results
Videos with research results using Alya simulations on wind flows can be found here.
|
|
|
|
Fig. 1 - Large eddy Simulation of the flow over the Bolund Hill (credit Guillermo Marin, BSC)
|
|
|
The wind scientific challenge has two main tasks. The first one is related to the CFD simulation of flow over a wind farm, commonly known as microscale simulations in the wind community. This kind of simulations is performed by energy companies interested in developing a wind farm to assess the wind resource and help identify the best locations for the wind turbines. BSC has been collaborating with Iberdrola, one of the most prominent investors in wind energy, for more than five years. We have adapted our code, Alya so that it can be used directly by the wind experts in Iberdrola. The industry typically relies on RANS simulation for wind resources assessment which is robust and does not require substantial computational resources. However, it is well known that RANS has limitations for the simulation of separated flows that commonly occurs in complex terrain. Our first objective within EoCoE-II is to further develop the LES implantation in Alya so that it can be used for wind resource assessment over complex terrain. LES is more accurate than RANS but also significantly more computationally demanding. It is the logical tool to be used with the advent of Exascale Supercomputers. Figure 1 shows a volumetric rendering of the velocity obtained with a LES simulation using Alya over the Bolund cliff, one of the most well-known benchmarks for flow over complex terrain.
|
|
Our second objective is to perform LES simulations of the rotating wind turbine blades commonly known, as full rotor simulations. The simulation of the rotation of the blades is possible thanks to a sliding mesh approach that is being developed within EoCoE-II. The blade deformation will be taken into account by using Alya’s solids module and its Fluid-Structure Interaction capabilities. Figure 2 shows the Q-vorticity isosurfaces for the flow behind the rotating blades of an NREL-VI wind turbine. Despite such simulation could also have some interest for wind energy companies, we expect that the main interest for this technology should come from wind turbine manufacturers. EoCoE-II has obtained a support letter from Vestas, and we expect that we can provide the industry with a useful technology by the end of the project.
|
|
|
|
Fig. 2 - Full rotor simulation for NREL-VI blades using the sliding mesh approach in Alya (credit Herbert Owen, BSC)
|
|
|
Materials for Energy
Advanced materials can contribute to the reduction in cost, increase in performance and extension of lifetime of the low-carbon energy technologies such as batteries, supercapacitors and solar cells. Thus, there is an urgent need for multi-functional and sustainable materials designed to provide a specific function in the final product. HPC can speed-up the entire process needed to identify new materials and to optimize them for the final use (Materials Roadmap). In particular, the design of advanced materials needs to consider atomic-scale chemistry and how it affects the physical properties at larger scales till the device.
|
The Energy Materials objective in EoCoE-II will focus on three specific flagship applications in energy storage and production respectively: Metalwalls-QMCPack (supercapacitor modelling), PVnegF (high efficiency silicon solar cells) and KMC/DMC (organic/perovskite photovoltaics).
|
Harvesting electricity from salinity and temperature gradient (Metalwalls-QMCPack): The objective of this part will be to ascertain the best electrode/electrolyte combination which optimizes the electricity production in electrochemical systems. Due to the large size of the simulated systems (thousands of electrode atoms, tens of thousands of electrolyte atoms) it is not possible to use electronic DFT for such calculations, therefore the goal is to develop new force fields for classical molecular simulations. In EoCoE-II this will be tackled by performing a series of Quantum Monte Carlo (QMC) reference calculations in order to benchmark the various functionals needed. This important step will be largely beyond the state-of-the-art since it will be necessary to treat large supercells for having realistic interfaces, and thus will ultimately require the use of exascale computers.
|
Optimizing silicon solar cells (PVnegf): The most efficient silicon solar cells belong to the silicon heterojunction (SHJ) and Tunnel-Oxide Passivated Contact (TOPCon) technologies, with efficiencies of 26.7% and 26.1%, respectively. In both cases, amorphous-crystalline heterointerfaces play a crucial role in the photovoltaic operation of the devices, but the microscopic mechanisms of transport and recombination mechanisms at the interface are still poorly understood. The atomistic resolution of the interface structure, the non-local character of the interaction between light, vibrations and charge carriers and the spectral resolution of physical quantities turn this approach in the most insightful and predictive of the existing photovoltaic device simulation formalisms. However, the resulting numerical problem represents a true exascale computing challenge, which in EoCoE-II will be addressed by the application code PVnegf. PVnegf solves the steady-state non-equilibrium Green’s function-Poisson equations for charge carriers interacting with photons and phonons to provide photocarrier dynamics (generation, transport and recombination) of nanostructured regions and at complex interfaces in advanced high-efficiency solar cell devices.
|
Optimizing Organic and Perovskite solar cells (Bath-KMC/DMC): Organic solar cells have blends of organic (carbon based) semiconductors as their active layer. Molecular engineering creates organic semiconductors that absorb different parts of the visible spectrum. Organic materials are flexible and by using flexible electrodes and substrates, fully flexible cells have been manufactured. Cells are cheap to make but are only 15% efficient and have stability concerns. Perovskite solar cells have a perovskite structured compound, commonly a hybrid organic-inorganic lead or tin halide based material, as the light-harvesting active layer. Cell efficiencies are currently 23% and are also cheap to make so have become commercially attractive, with start-up companies promising modules on the market in the next few years. However, perovskite cells are notorious for poor stability and current-voltage hysteresis. For both cell types, device models need extending to systems of 1000+ atoms to understand degradation and find materials and architectures that improve stability. The chosen approach is the Kinetic Monte Carlo (KMC) and Device Monte Carlo (DMC) for simulation of respectively organic and perovskite solar cells. Both codes solve the Boltzmann transport equation to predict current-voltage characteristics, and exhibit a major bottleneck due to inefficient Poisson-Boltzmann and/or Poisson solvers. In EoCoE-II this problem will be overcome by development of multigrid solvers.
|
Codes
QMCPack is an open-source production level many-body ab initio Quantum Monte Carlo code for computing the electronic structure of atoms, molecules, and solids. Variational Monte Carlo (VMC), diffusion Monte Carlo (DMC) and several other advanced QMC algorithms are implemented. Metalwalls is a classical molecular dynamics code aiming at simulating electrochemical cells. It treats electrode as metallic systems held at constant potential and includes polarization effects for the liquids.
|
PVnegf describes photon-carrier dynamics (generation, transport and recombination) in nanostructured regions and at complex interfaces using a non-ballistic approach based on Green’s functions formalism. The code is used to execute advanced simulation of high-efficiency solar cell devices for different input biases.
|
|
Kinetic Monte Carlo (KMC) simulates charge and energy transport in organic solar cells. Device Monte Carlo (DMC) simulates charge transport in perovskite cells. The DMC code includes the effects of mobile ion motion that affects most perovskites.
|
|
|
Water for Energy
In hydropower and geothermal applications, modeling of shallow subsurface flow is of major importance in order to accurately simulate and predict the exchange of groundwater with streams under low-flow conditions, and the transport of energy. The major challenge is the representation of topographically driven groundwater convergence and streamflow generation, and of the geological heterogeneity across a number of space scales ranging from centimeters to thousands of kilometers in case of continental river systems. Constructing hydrologic and geothermal models at this resolution over large spatial scales for scientific and operational applications constitutes a game changer, easily reaching up to 10^12 degrees of freedom, where simulations must additionally assimilate observations to mitigate uncertainties in model data.
|
In EoCoE-II, the integration of hyper-resolved simulation of hydrological fluxes, routing along the river network, and management of storage reservoirs will be performed with a modernized version of ParFlow with adaptive mesh refinement (AMR) capability. The added values of these simulations will be shown by feeding ParFlow gridded runoff time series into the operational hydropower model that will be specifically developed over the Italian Alpine region.
|
Previous work on geothermal reservoir characterization showed the successful application of optimal experimental design (OED) within the simulation code SHEMAT-Suite in order to identify optimal drilling locations for assessing uncertain reservoir parameters within a numerical reservoir model. However, the high computational cost has to date limited this approach to a numerical model with significantly reduced number of unknowns.
|
Collaboration with experts in the EoCoE-II consortium will enable us to create a realistic geothermal reservoir model with vastly improved spatial resolution. Combining optimal experimental design for positioning boreholes with state-of-the art HPC techniques will improve the exploration and exploitation of geothermal reservoir systems, as it enables a sophisticated quantification of uncertainties in the subsurface.
|
Codes
ParFlow (https://parflow.org/) is a physics-based 3D parallel hydrologic model, which simulates surface and 3D subsurface flow, based on Richard's and kinematic wave equations on a finite difference, finite volume grid with a preconditioned, implicit Newton-Krylov solver for the nonlinear PDEs. ParFlow will not be completely rewritten in EoCoE-II, but some code upscaling is planned as well as the activation of the Adaptive Mesh Refinement of the computing grids with the ad hoc numerical scheme and corresponding solver. It is also planned to have a strong effort on IO that will be coordinated with the effort on ensemble runs.
|
SHEMAT-Suite is a code for simulating single- or multi-phase heat and mass transport in porous media. It solves coupled problems comprising heat transfer, fluid flow, and species transport. SHEMAT-Suite can be applied to a range of hydrothermal or hydrogeological problems, be it forward or inverse problems.
|
ExaTerr is a new development that aims at building a common software platform for both SHEMAT-Suite and ParFlow. This platform will be based on Kokkos, a software technology strongly pushed by the US DoE which holds in his heart performance portability. The idea pursued here is to have a single code base that could be executed on both CPU- and GPU-based platforms.
|
Results
|
Continental-scale high-resolution land surface data assimilation system
The land surface data assimilation system CLM-PDAF (consisting of the Community Land Model (CLM 3.5) (Oleson et al., 2004) and the Parallel Data Assimilation Framework (PDAF)(Kurtz et al., 2016) was used to update the soil moisture estimates from the land surface model utilizing the coarse resolution satellite soil moisture data. A key capability of CLM-PDAF is the support for data assimilation that combines land surface processes with satellite and in-situ observations for the estimation of optimal land surface states. The data assimilation structure in CLM-PDAF allows to directly ingest remotely sensed observations of land surface conditions to produce accurate, spatially and temporally consistent fields of land surface states, with reduced associated error. The CLM-PDAF uses an Ensemble Kalman Filter (EnKF) algorithm (Burgers et al., 1998) to generate assimilated or reanalysis products. To effectively simulate the background-error covariances, a large enough ensemble size needs to be maintained in the data assimilation process, which linearly increases the computational resource requirements. The CLM-PDAF is designed for high-performance computing infrastructures and can efficiently cope with the high computational burden of ensemble-based data assimilation.
|
|
Hence, we implemented the CLM-PDAF over Europe to provide downscaled estimates of the soil moisture with complete spatiotemporal coverage by combining historical satellite SM observations with a high resolution LSM using data assimilation techniques. Using the CLM-PDAF, the satellite based soil moisture dataset ESA CCI (the European Space Agency Climate Change Initiative (Wagner et al., 2012) was assimilated into CLM using the EnKF producing a high-resolution European SSM reanalysis (called ESSMRA hereafter) dataset. This product overcomes the shortcomings of sparse spatial and temporal datasets and provides a better estimate of SM than obtained only by modeling or by sparse observations alone. The 3 km ESSMRA is generated by first implementing the regional land surface model setup coupled with the data assimilation framework as shown in letter (a) of the figure below. In the second step the ESA CCI satellite-based data is assimilated into the CLM-PDAF setup to generate the daily 3km ESSMRA product over Europe for the 2000 - 2015 time period, letter (b) of the figure below.
|
|
|
|
(a) Schematic of CLM-PDAF workflow adopted to generate high resolution ESSMRA product.
|
|
(b) Average soil moisture (mm3mm-3) for 2000 -- 2015 time period.
|
|
|
Experimental Design for Geothermal Modeling
|
Drilling boreholes during exploration and development of geothermal reservoirs not only involves high cost, but also bears significant risks of failure. In geothermal reservoir engineering, techniques of optimal experimental design (OED) can improve the decision making process. For instance, during exploration and production of geothermal fields, OED can be used to place additional slim holes, thus decreasing drilling costs and risk. Previous publications explained the formulation and implementation of this mathematical optimization problem and demonstrated its feasibility for finding borehole locations in two- and three-dimensional reservoir models that minimize the uncertainty of estimating hydraulic permeability of a model unit from temperature measurements (Seidler et al. 2014; Seidler et al. 2016). Subsequently, minimizing the uncertainty of the parameter estimation results in a more reliable parametrization of the reservoir simulation, improving the overall process in geothermal reservoir engineering.
|
|
|
OED is a mathematical optimization method. The general approach is to find optimal experimental conditions for constraining model parameters. In other words, OED gives answers to the question: How do I have to design an experiment in order to collect data from which I can predict model parameters with least uncertainty? This is a question of the sensitivity of the model with respect to the unknown parameters. This sensitivity is mathematically described by the Fisher Matrix. It contains the first order derivative of the model output towards the parameters. For evaluating the information contained in the Fisher matrix, OED criteria are formulated. One is the D-optimal design criterion which is based on the determinant of the Fisher Matrix (other criteria are based on the trace or eigenvalue). The minimum of the D-optimal criterion contains the maximum sensitivity.
|
|
Various OED techniques are implemented in the Environment for Combining Optimization and Simulation Software (EFCOSS) (Seidler et al. 2014). This software framework links mathematical optimization software with SHEMAT-Suite, our geothermal simulation code for fluid flow and heat transport through porous media, for addressing problems arising from geothermal modeling.
|
|
Within EoCoE-II we will extend this OED approach with further functionalities for geothermal reservoir modeling and aim at increasing its performance in order to apply it to reservoir scale production runs. As a first step, synthetic test models have been set up for studying certain aspects, such as sensitivity of the OED result to a priori data.
|
|
|
Synthetic OED study: 2D geothermal reservoir model above salt diaper
Here we show first results of an OED study for a 2-dimensional synthetic model of a geothermal reservoir above a salt diaper (yellow unit), which is cross-cut by two high permeable faults (blue and red units) (modified after Rath et al. 2006). Each model unit is characterized by constant thermal and hydraulic properties (e.g., permeability, porosity, thermal conductivity). High heat conductivity of salt results in higher temperatures at shallower depth. This makes the sedimentary units above and close to salt diapers interesting for direct use of geothermal energy. In addition, the permeable faults in our model provide pathways for advective heat transport, resulting in a heat transport towards the surface through the western fault (unit 11, blue). There is a borehole at location x=7475 m with a certain depth z=1475 m providing a temperature log Tlog0. The problem is to find the location for an additional borehole of the same depth for measuring another temperature log Tlog1. These temperature data shall be used for estimating the fault permeability with least uncertainty (i.e., optimally). For this OED problem, the numerical simulation provides two possible optimal horizontal ranges for placing the additional borehole (Figure 1, top) : (i) from 1025 m to 1175 m and (ii) from 8225 m to 9975 m.
|
|
We will further use this model in the course of the project for simulating various OED problems and studying the impact of factors such as a priori assumptions, location of a priori data logs and their data quality.
|
|
|
|
Numerical forward model of a synthetic geothermal reservoir above a salt diaper computed with SHEMAT-Suite Top: Reservoir structure in terms of geological model units, unit 9 in yellow is the salt and units 10 and 11 in red and blue are permeable faults. The remaining units are various sedimentary layers. X0 marks the location of the existing borehole and the black crosses depict the OED result in terms of normalized and binned D-optimality. Red arrows display two optimal ranges for an additional borehole. Middle: Darcy flow in terms of hydraulic reference head and darcy velocity (arrows) for the true reservoir properties. Bottom: Steady state temperature distribution for the true reservoir properties. Image credits: Johanna Bruckmann, M.Sc., E.ON Energy Research Center, RWTH Aachen University.
|
|
|
|
|
  
|
|
|
|
|
|
