New instrument technologies are enabling a new generation of in-situ and in-operando experiments, with extremely fine spatial and temporal resolution, that allows researchers to observe as physics, chemistry and biology are happening. These new methodologies go hand in hand with an exponential growth in data volumes and rates – petabyte scale data collections and terabyte/sec. At the same time, scientists are pushing for a paradigm shift. As they can now observe processes in intricate details, they want to analyze, interpret and control those processes. Given the multitude of voluminous, heterogenous data streams involved in every single experiment, novel real-time, data-driven analysis and decision support approaches are needed to realize their vision. This talk will discuss state-of-the-art streaming analysis for experimental facilities, its challenges and early successes. It will present work currently carried out at Brookhaven National Laboratory and identify areas for collaboration.
Kerstin Kleese van Dam is the director of the Computational Science Initiative (CSI) at the Department of Energy’s Brookhaven National Laboratory (BNL), in Long Island, NY. With 100 petabytes, BNL hosts the second largest scientific data archive in the US and the fourth largest in the world, it processes annually in access of 400 petabytes of scientific results.