In-situ data processing with DASK: Public Webinar
27th March 2025, 10 am
In-situ data processing with DASK
Webinar organized by CASTIEL2
Abstract
In situ data processing refers to the online processing of data produced by large-scale parallel simulations to reduce the amount of data saved to disk and avoid the I/O bottleneck that affects large supercomputers.
There exist various solutions based on the classical MPI model that can be very efficient. However, today data analytics tools are dominated by Python packages that offer highly productive environments familiar to many users (NumPy, Pandas, scikit-learn, PyTorch, JAX).
While these packages often support limited parallelization, frameworks like Dask and Ray have proposed seamless parallelizations to enable processing large datasets based on distributed task-based programming.
The task programming paradigm reduces parallelization effort as users express potential parallelism as a graph of functions where functions without data dependencies can be executed in parallel.
At execution, a scheduler handles task scheduling on available cluster nodes, considering dependencies and data locality.
During this talk we will show how we can combine MPI-based solvers with Dask-based in situ data processing.
We will present work done at INRIA and Maison de La Simulation for leveraging Dask to enable in situ data processing from data produced by MPI codes.
We will discuss different technical points, including:
- data and control transfer between MPI and Dask;
- efficient data selection to minimize unnecessary transfers;
- metadata system for describing data structure and encoding;
- adaptation between simulation and analytics data structures;
- performance at scale.
This work is part of the EoCoe-III projet that will deploy at large scale in situ analytics for the Gysela and Parflow solvers.
EoCoE is a European Horizon 2020 funded project of Centre of Excellence in computing applications. It is designed to enhance numerical simulation efficiency in the international context of the High Performance Computing (HPC) challenges. It focusses its application scope towards low carbon energy domains.
To measure solar energy's profound impact, we must contemplate the “carbon footprint” — a metric that quantifies the total greenhouse gas emissions caused by an individual, organization, event, or product. Traditional electricity production contributes heavily to a nation's footprint, with fossil fuels being the predominant energy source. By harnessing solar energy, we offset these emissions and aid in the progressive decline of our collective carbon footprint.
Switching to solar is more than energy change it's a commitment to a future with reduced carbon emissions
- Roberto Liam
Our Role in Solar Advancement
To measure solar energy's profound impact, we must contemplate the “carbon footprint” — a metric that quantifies the total greenhouse gas emissions caused by an individual, organization, event, or product. Traditional electricity production contributes heavily to a nation's footprint, with fossil fuels being the predominant energy source. By harnessing solar energy, we offset these emissions and aid in the progressive decline of our collective carbon footprint.
In the battle against climate change, the shift towards renewable energy sources plays a pivotal role. Among these, solar energy emerges as a beacon of hope and innovation. Positioned as both a clean and inexhaustible source of power, solar energy has shown significant promise in reducing global carbon emissions. Let’s explore how the adoption of solar power is making strides in the journey towards a more sustainable future.
Active participation from both consumers and industries in the solar movement is imperative. It includes transitioning to solar power where feasible, supporting policies that encourage
