Since their development in the 1930s, particle accelerators have become essential tools of science, with applications in industry, medicine, and security. They are highly complex machines that demand advanced computer simulations for their conception, development, and operation. Research is being conducted to develop more compact, and cheaper, particle accelerators to broaden their adoption and to enable applications that are currently out of reach but that have the potential for high societal impact. These new accelerator technologies, however, are stretching the capabilities of state of the art computational power due to their complexity.
One such technology are plasma-based accelerators, one of the holy grails of scientific research tools, which have the potential to enable the next generation of high-energy physics and make more accessible medical applications like ultra-high-dose radiotherapy to treat cancer.
Many institutions are working on the science behind this new type of accelerator, including the Berkeley Lab Laser Accelerator (BELLA) Center. These new accelerators, known as Laser-Plasma Accelerators (LPAs), shoot an intense laser pulse into a plasma, scattering negatively charged particles, which set up intense electric fields that can push electrons up to speeds a thousand times faster than traditional accelerators.
But advancing particle accelerator—including LPA—technologies is no easy feat. Computer simulations of the particle beams are essential to significantly speed up the design and development of these technologies, but are also challenging due to the difference in spatial and time scales involved. The particle beams are very small (for example, microns in LPAs) compared to the accelerating structure (meter-scale for a single plasma stage in LPAs, with up to thousands of stages needed for a collider) through which the beam travels, so an incredibly large number of calculations is required. In addition, in LPAs, the interactions of the electromagnetic fields with the plasma particles are highly complex, requiring sophisticated algorithms.
All this means that the computer simulations are very time-consuming, even with the most powerful supercomputers.
Early Days
The Lab, which recognized the importance of large-scale accelerator simulations to the future of the field and the size of the challenge, has over the years provided critical support through the Laboratory Directed Research & Development (LDRD) program. Not just once or twice, but four times.
The first time that LDRD funded Lab projects for high-performance computing of particle accelerators, including LPAs, was in 2002, when Senior Scientist Phillip Colella, and Jean-Luc Vay, then a physicist in the Heavy Ion Fusion program, proposed a new way of running a simulation that would reduce the time required for these simulations.
The “Adaptive Mesh Refinement” (AMR) technique, first developed by Marsha Berger, Joseph Oliger, and Phillip in the 1980s and applied to the field of fluid dynamics, allowed them to “zoom in” on the specific regions of a problem that are most important to its solution rather than having to perform the entire calculation at the finest resolution. The LDRD program funded Phillip, Jean-Luc, and their collaborators’ proposal.
The strategy worked. Three years later, the project had adapted the AMR technique to the type of “Particle-In-Cell” (PIC) simulations used for particle accelerators and sped up some of the computer simulations by an order of magnitude. While it was a significant step forward, the computer simulations needed to be even faster.
A Breakthrough Idea
In 2007, Jean-Luc had a serendipitous idea. According to Einstein’s Theory of Special Relativity, speed affects mass, time, and space. And because of the relativistic speeds involved in the movement of beams in particle accelerators, the theory should be considered when modeling these accelerators from first principles. With the appropriate frame of reference, the beams would be lengthened while the structures through which the beams traveled would be shortened. This would result in fewer computations and faster calculations—a very important insight for accelerator computer models. Despite Einstein’s theory having been well studied for decades, the effect on the computational cost of simulations of relativistic objects had never been explored before, and thus never been applied in accelerator calculations. Jean-Luc and his team were eager to apply this new idea to the simulations and proposed another LDRD project.
The LDRD program again stepped in to support the research project, titled “Lorentz Compaction of Scales for Ultra-efficient Simulation of Advanced Accelerators (and other Systems)”. Jean-Luc and his colleagues were able to apply, for the first time, the Theory of Special Relativity to speed up a class of simulations for accelerators.
The results were impressive, achieving, for LPAs, first-principle BELLA calculations tens of thousands of times faster than comparable conventional methods. Jean-Luc and his colleagues, Cameron Geddes at the Lab, Estelle Cormier-Michel, then of the Tech-X Corporation in Denver, and David Grote of Lawrence Livermore National Laboratory, published their findings in the March 2011 issue of the journal Physics of Plasma Letters.
Rendering from a 3D simulation using the code Warp of a plasma accelerator in a “boosted frame,” in which the observer moves at near light speed. The laser pulse is represented in blue and red; the wakefields are colored pale blue and yellow. In this frame, the plasma (yellow box) has contracted and the wave fronts are fewer and farther apart, resulting in far fewer calculations and faster results.
Credit: J.-L. Vay
As Physics World reported: “Vay tried his method on several examples of relativistic interactions of relevance to particle accelerators including a free-electron laser, a laser-plasma accelerator, and a particle beam interacting with an electron cloud. For the particle beam example, the simulation in a typical reference frame took over a week to run, whereas in Vay’s relativistic, “ideal” reference frame it took just half an hour.”
Bingo.
LDRD Tips for Researchers
Jean-Luc offers the following advice to other would-be LDRD applicants: “Go for it. But also do your homework and be ready to explain why your idea should be funded and why it should be now,” he said. “If this is your first time applying for LDRD funding, talk to others. If you are applying with a multi-disciplinary project in mind, you need to advocate for your idea across the relevant divisions.”
LDRD Keeps it Going
As the methods developed in this framework began to be applied, another pathway emerged to speed up a different part of accelerator calculations: the use of local Fast Fourier Transform (FFT)-based spectral solvers for Maxwell’s Equations.
The computer simulations for free-electron lasers and plasma based accelerators required solving Maxwell’s equations for electromagnetic waves. At the time, although it was more accurate and enabled better stability for plasma accelerators modeling, the FFT method was not commonly used to solve the equations because deploying it efficiently on massively parallel supercomputers was difficult. To remedy this, Jean-Luc thought: “Why not use multiple local FFTs running simultaneously to solve Maxwell’s equations, exploiting the localized nature of the electromagnetic waves?”
LDRD support stepped in once more in 2011, funding the project, which was titled “High-accuracy Scalable Solvers for Modeling of Future Ultrafast Photon Sources,” also known as “Spectral.” This strategy proved fruitful as well. The team created an impressive tool that significantly improved the accuracy and stability for a given computational cost.
Making it All Come Together: PAS and Exascale WarpX
With all the strategies introduced over the years, the improvements in speed for the particle accelerators simulations were adding up to be significant. In 2014, Jean Luc and his team made one more proposal: to combine all the improvements. The title of the project was “High-Performance Advanced Particle Accelerator Simulator,” or “PAS,” for short. “PAS synthesized all these techniques, resulting in computing speeds that were orders of magnitude faster,” said Jean-Luc.
The project was so successful that the DOE took notice, and one year into the PAS LDRD funding timeline, the DOE’s Office of Science along with the National Nuclear Security Administration decided to fund the Exascale WarpX project, one of the 25 applications selected to participate in the DOE Exascale Computing Project (ECP).
WarpX is a collaboration of multiple research institutions, including Berkeley Lab, Lawrence Livermore National Laboratory, and SLAC National Accelerator Laboratory, and researchers from French Alternative Energies and Atomic Energy Commission (CEA), Deutsches Elektronen Synchrotron (DESY), CERN, and private companies. It is built around the ECP AMReX library, which allowed for the scalable implementation of AMR on multiple computer architectures, and was developed by a team of researchers led by John Bell and Ann Almgren at Berkeley Lab’s Applied Mathematics and Computational Research Division.
Today WarpX is arguably one of the crown jewels in the Department of Energy’s ECP and its computing initiatives. It has won accolades, including in 2022, the prestigious Gordon Bell Prize, which recognizes outstanding achievement in high performance computing. The prize was awarded to the 16-member WarpX team, which includes the Lab’s Axel Huebl, Kevin Gott, Remi Lehe, Andrew Myers, Weiqun Zhang, Jean-Luc Vay and collaborators at other institutions
The Impact of LDRD Support
Today, modeling of complex accelerator systems based on first principles, including LPAs, is within arm’s reach of the scientific community. Jean-Luc’s team is ideally positioned to further develop the technology to the level where it can be considered for designs of future particle colliders to explore the basic structure of the universe. LDRD has been instrumental in this journey, supporting bold, creative ideas with a potentially consequential impact on science that goes beyond LPAs, spawning an ecosystem around WarpX, which is developed and used in research labs, academia, and industry, to tackle a variety of topics, such as all types of particle accelerators, plasma science and fusion devices, and microelectronics.
Jean-Luc, now a senior physicist in the Physical Science Area’s Accelerator Technology and Applied Physics (ATAP) Division and head of the Accelerator Modeling Program, notes that LDRD support has been transformative, not just for the work, but also for him personally. “LDRD funded, at critical points, new ideas to speed up the development of the next generation of accelerators, and provided the stepping stones to secure larger funding,” he said. “And it changed my career. It gave me the opportunity to work on ideas that I had outside of my main line of work at the time. I went from working on these new ideas at night and on weekends, to making them my day job.”
To learn more about plasma accelerator technology:
Berkeley Lab Laser Accelerator (BELLA) Center
“Plasma Particle Accelerators Could Find New Physics,” Scientific American, July 1, 2021
