Left to right: Debbie Bard, Superfacility Project lead, and Dula Parkinson, ALS staff scientist and an early proponent of the superfacility concept
Revolutionizing how we do science can mean different things, from entirely new approaches to developing new science instruments. It can also mean finding innovative ways to serve the scientific community.
The concept of the superfacility developed at the Lab is one such innovation. Now implemented with 50 research teams using the National Energy Research Scientific Computing Center (NERSC) facility, and with more projects in the pipeline, the superfacility model is changing the user experience and accelerating discovery.
In the simplest terms, a superfacility connects experiments and instruments with computational, data, and networking facilities. Data produced by light sources, microscopes, telescopes, and other devices can stream in real time to large computing facilities where it can be analyzed, archived, curated, and combined with simulation data. This superfacility model allows for discoveries across data sets, institutions, and domains and makes data from one-of-a-kind facilities and experiments broadly accessible.
It also makes data-intensive science easier for scientists to access. Dula Parkinson, a staff scientist and deputy for operations for photon science at the Advanced Light Source (ALS) user facility and an early proponent of the superfacility model, said, “It’s really about looking at things from the user’s perspective. Each user facility has its own access mechanism and way of doing things. Getting access to the ALS can already be challenging, and the challenge is multiplied when the user also needs access to NERSC or other supercomputing facilities. When fully implemented, the superfacility model could allow the user to submit one proposal, and to have one access point to what they need. It makes users’ lives so much easier and better.”
Debbie Bard, group lead of the Data Science Engagement Group at the Lab’s National Energy Research Scientific Computing Center (NERSC) agreed, “As instruments become more powerful and experiments involve more and more data, scientists increasingly need large-scale supercomputers and data analysis resources that they don’t have locally. Supercomputers and storage facilities like NERSC are essential to many science endeavors. The superfacility model makes it easier for scientists to access these facilities.”
Said Dula, “It’s no longer just about requesting computer resources. There’s so much cool stuff we can do leveraging the data at NERSC. With the data already gathered at NERSC over many years and eventually at other national lab supercomputing facilities, the scientific community can do much more robust data analysis, as well as simulations and AI model training. For example, you can imagine scientists having access to data and simulations even before they begin their experiments, and using this information to inform their experiments. And the data collected at supercomputing facilities can be used to train generative AI for science. It’s an exciting prospect.”
An Inauspicious Start
The superfacility concept has come a long way at the Lab. On November 8, the NERSC Superfacility team was recognized with the Lab’s Director’s Science Award “for developing and demonstrating the revolutionary superfacility concept, coupling high-performance computing, networking, services and tools with a range of experimental and observational science facilities spanning multiple science domains.” But the idea had an uncertain start.
Dula notes that the first time he heard the term “superfacility” was in a 2015 presentation made by Craig Tull, a longtime Lab physicist and computer scientist who retired in 2020. Craig was part of an LDRD project, “Spot Suite – Towards an End-to-end Solution for Light Source Data” with Computing Sciences, NERSC, ALS, ESnet, and the Materials Science Division. The idea was to provide an end-to-end solution for data access, management, and analysis for light source data. The project sparked plans to expand the use of the concept, but not all stakeholders warmed to the idea.
Said Dula, “The idea wasn’t very popular at first. It took some time and encouragement to stick.”
NERSC began to adopt the model around 2015. At first, a handful of individual teams worked with NERSC and Computing Sciences researchers on separate projects. The NERSC team saw that more and more teams would want to use this model. They needed a model that would scale to multiple user groups and be scalable for NERSC staff to support.
Debbie noted, “Once it became clear that the amount of data that scientists would be managing would continue to increase, it became part of NERSC’s mission.”
In 2019, the Computing Sciences Area strategic plan envisioned the Superfacility as one of three pillars critical to the future of the area and NERSC initiated a three-year project for a coordinated superfacility model, called the Superfacility Project, combining the requirements for multiple teams, including a software framework and infrastructure. The goal of the Superfacility Project was to coordinate superfacility work taking place at the Lab and provide a coherent set of guidelines for superfacility work going forward; ultimately, of the eight science teams selected for in-depth engagement with superfacility, five of them successfully implemented autonomous data pipelines and the workflows associated with them. The project was completed in early 2022 and offered new lessons for the team.
Debbie, who spearheaded the project, said, “We found that for that model to be successful, we needed to do more than physically hook up the hardware and make sure the network connection is there. We needed a whole ecosystem of software, infrastructure, and community. Because a superfacility is about multiple experiments connecting with a computing facility, building a community of practice around that model is as important as the physical connections.”
Debbie continued, “I’m thrilled at how far we’ve come. We started with a handful of science teams using the API (the application programming interface that links the experiments with NERSC). Today, just a year and a half later, the program has fully blossomed. Thanks to the infrastructure that we’ve built, we have 50 teams using the API. And all this infrastructure we built as part of the Superfacility project is useful for all 9000 of our users, not just the experimental science teams. There is so much potential for new modes of scientific discovery, the possibilities are very exciting.”
Influencing User Facility Models beyond Berkeley Lab
The superfacility concept didn’t stop at the Lab’s borders. Berkeley Lab scientists and engineers started making presentations about their experience to DOE, which had been thinking about the need for more connected supercomputing facilities, and DOE saw a way for national lab user facilities to build upon Berkeley Lab’s concept. In particular, the idea significantly influenced the thinking of Benjamin Brown, director of the DOE Advanced Scientific Computing Research (ASCR)’s Facilities Division, about an Integrated Research Infrastructure, or IRI, architecting a broader vision that could be applied across the DOE complex. This July, the DOE’s Office of Scientific and Technical Information (OSTI) published a report describing this vision titled “Integrated Research Infrastructure Architecture Blueprint Activity,” co-authored by Berkeley Lab and other national labs.
As noted in the DOE’s report, the Integrated Research Infrastructure (IRI) is designed “to empower researchers to meld DOE’s world-class research tools, infrastructure, and user facilities seamlessly and securely in novel ways to radically accelerate discovery and innovation…experimental facilities could be connected with high performance computing resources for near real-time analysis, and resources should be provided for merging enormous and diverse data for AI/ML techniques and analysis.”
The DOE also began to make related awards; in October, 2023, it announced the selection of the High Performance Data Facility (HPDF) hub, which will create a new scientific user facility specializing in advanced infrastructure for data-intensive science. The project will be a partnership between Thomas Jefferson National Accelerator Facility (JLab) and Berkeley Lab. JLab will be the HPDF Hub Director and house the lead infrastructure; the Berkeley Lab team, led by Lavanya Ramakrishnan, Division Deputy in the Scientific Data Division. The integrated team will provide management and operational support for HPDF, focusing on integration of the HPDF Hub infrastructure with other ASCR facilities including NERSC and ESnet, coordination of the national federated architecture operations of HPDF, and development of strategic directions for the HPDF and IRI efforts in coordination with DOE ASCR and other stakeholders.
The High Performance Data Facility will serve as the foundation for advancing DOE’s Integrated Research Infrastructure (IRI) program, seamlessly melding DOE’s unique data resources, experimental user facilities, and advanced computing resources to accelerate the pace of discovery.
Said Debbie, “I’m looking forward to the day that integrated research infrastructure model is just how science is done for data-intensive research, a one-stop shop for scientists and one that also facilitates new discoveries in ways that were not possible before.”
For more information about the NERSC Superfacility:
- CS Strategic Plan: https://cs.lbl.gov/home/about/strategy/
- NERSC Superfacility info page: https://www.nersc.gov/research-and-development/superfacility/
- Superfacility case studies: https://docs.nersc.gov/science-partners/superfacility-case-studies/
- Story, “Working towards Seamless Analysis of Experimental Data,” https://www.nersc.gov/news-publications/nersc-news/science-news/2023/berkeley-lab/
