$5M NSF grant powers AI innovations in nationwide workflow management

Over the last few years, massive sensor networks and the increasing interconnectivity of cyberinfrastructure (CI) systems have created an unprecedented amount of data for researchers to utilize.

While the enormous volume of information enables breakthroughs in complex problems like modeling black holes, tracking space debris, predicting hurricanes, and advancing cancer research, not all researchers have access to the enormous compute power required to process it—or the expertise to craft efficient workflows that spread the data and processing tasks across distributed networks.

"Researchers face tremendous complexity in managing their increasingly sophisticated workflows across the computing continuum," said Michela Taufer, the Dongarra Professor in the University of Tennessee's Min H. Kao Department of Electrical Engineering and Computer Science (EECS). "There is a critical need for intelligent workflow management systems that adapt well to evolving user needs and dynamic resource conditions."

Taufer and EECS Assistant Professor Sai Swaminathan believe that artificial intelligence (AI) is the key to helping every researcher turn their workflows into those intelligent, self-optimizing systems.

"Recent advances like large language models (LLMs) that enable interactive assistance, or neural networks that detect workflow anomalies, enable us to make tools that are both more technically advanced and easier for scientists to use," Swaminathan explained.

Taufer and Swaminathan are part of a team of computer science experts who plan to bring AI-enabled workflows to researchers across the nation by integrating AI into Pegasus, a widely used workflow management network hosted by the National Science Foundation (NSF).

The team, headed by University of Southern California (USC) Research Professor Ewa Deelman, was awarded $5 million from the NSF this summer to develop a new, intelligent version of the workflow management tool—known as PegasusAI—over the next five years.

"PegasusAI will enhance automation and reliability for data-intensive scientific workflows, transforming how researchers engage with and benefit from advanced computing systems," Deelman said.

The project will also contribute foundational technologies to the broader NSF CI ecosystem, enabling researchers across the United States to more easily and efficiently utilize NSF-supported computing platforms and services.

"Advanced computing and AI should be available to everyone, not just experts with big teams and lots of resources," said Swaminathan. "By using human-centered AI to simplify and speed up scientific work, we can help solve complex problems faster—from public health to climate to space."

Multidisciplinary Tools Need Multidisciplinary Input

The development of PegasusAI is funded by NSF's Cyberinfrastructure for Sustained Scientific Innovation (CSSI) program, which supports the development of software frameworks for scientific research that are sustainable and extensible (able to integrate new capabilities as they develop).

"We are tackling complex, real-world problems that require coordinated expertise across multiple layers, from AI model development to user-centered design and evaluation," said Taufer. "No single institution could meet all of these demands in isolation."

That is why Deelman, who also serves as research director of the USC Information Sciences Institute, brought together Taufer; Swaminathan; University of Massachusetts Amherst Professor Michael Zink; and Anirban Mandal, director of the Network Research and Infrastructure group at the University of North Carolina, Chapel Hill.

"This group includes experts in workflow systems, human-computer interaction, AI, performance modeling, real-time analytics, distributed computing, and CI," Swaminathan said. "Together, we will be able to design intelligent, human-centered interfaces that help researchers from any discipline compose, adapt, and steer complex workflows."

A Responsive System that Works with Users

The current version of Pegasus helps researchers across the country access greater computational power, storage, and time than is available at their home institutions. PegasusAI will compound that utility by giving users unprecedented insight into—and control over—the automated workflow management process.

PegasusAI will incorporate graph neural networks, LLMs, and autoencoders to identify and mitigate workflow errors. Users will be able to interrogate AI-driven system decisions, explore alternatives, and adjust workflow parameters in context. Over time, PegasusAI will learn from how users respond, becoming more tailored and intuitive with every use.

"We're moving away from AI as a 'black box,'" Swaminathan said. "PegasusAI will maintain detailed data capturing not just what the system did, but why it did it, and under what conditions."

To ensure that the advanced system is available to as many researchers as possible, the PegasusAI team is also working on interface guides that will adapt the level of support based on users' comfort with composing, launching, and monitoring workflows.

"This project harnesses recent advances in AI to build distributed systems that are not only intelligent, but (…) balance powerful functionality with ease of use," said Deelman.

Just as PegasusAI will respond to user inputs, the researchers will incorporate users' feedback on the tool's development over the next five years.

"PegasusAI is, at its core, a community-driven project," Taufer said. "We are working to create an integrated workflow tool that promotes reliability, transparency, and smooth recovery from setbacks. Engaging users from diverse scientific domains is how we will ensure that the platform evolves sustainably with their needs—and remains open and accessible as the user base continues to grow."

Provided by University of Tennessee at Knoxville