ARPA-E making progress toward achieving mission, says new assessment

The Advanced Research Projects Agency-Energy (ARPA-E) is making progress toward achieving its statutory mission and goals, says a new congressionally mandated report by the National Academies of Sciences, Engineering, and Medicine. ARPA-E has funded research that no other funding source was supporting at the time, and the results of some of these projects have received follow-on funding from private and other public sources for various technologies, the report says.

The agency cannot reasonably be expected to have completely fulfilled its goals given so few years of operation and the size of its budget, the report says. The development of new energy technologies usually requires decades of effort, and decades also must usually elapse before there is evidence that innovations are truly transformative.

The report offers recommendations to guide ARPA-E in building upon its early success, urging the agency to continue its flexible management approach and risk-taking culture, to develop a framework for measuring its impacts, and to streamline its reporting processes.

"ARPA-E has made significant contributions to energy R&D that likely would not take place absent the agency's activities," said Pradeep Khosla, chair of the committee that wrote the report and chancellor of the University of California, San Diego. "We hope our report's recommendations will benefit the agency and the nation as ARPA-E continues to evolve and improve its operations."

Congress created ARPA-E within the U.S. Department of Energy with the mission "to overcome the long-term and high-risk technological barriers in the development of energy technologies." When Congress authorized ARPA-E in the 2007 America COMPETES Act, it requested an early assessment following 6 years of operation to examine the agency's progress toward achieving its statutory mission and goals. The Academies' new report documents the results of that assessment.

While the full market impacts of ARPA-E-funded technologies will not be seen for years, some intermediate outcomes are evident now, the report says. One quarter of the supported project teams or technologies have received follow-on funding for continued work. Roughly half of supported projects have published the results of their research in peer-reviewed journals, and about 13 percent have obtained patents. All of these are positive indicators for technologies on a trajectory toward commercialized products.

ARPA-E is in many cases successfully enhancing the economic and energy security of the United States by funding transformational activities, white space (technology areas that are novel or underexplored and unlikely to be addressed by the private sector or by other federal research programs), and feasibility studies to open up new technological directions and evaluate the technical merit of potential directions, the report says.

Importantly at this early stage, the committee found no signs that ARPA-E is failing to deliver on its mission and goals, on a path to failing, or in need of reform. In fact, attempts to reform the agency - such as applying pressure for ARPA-E to show short-term successes rather than focusing on its long-term mission and goals - would pose a significant risk of harming its efforts and chances of achieving its mission and goals, the report says.

The report also recommends that the ARPA-E director and program directors develop and implement a framework for measuring and assessing the agency's impact in achieving its mission and goals as soon as practicable, providing the agency with greater ability to demonstrate its value and impact. While the agency has in place an extensive data gathering and record keeping system at the project level, its system for collecting, tracking, and reporting publicly available high-level innovation metrics—publications, funding from other sources, and disclosures and patents over time - is less extensive. The agency could link data from its robust internal database of project-level metrics to program-level goals, including indicators of commercial and noncommercial outcomes over the short and long terms; connect those goals to standard, observable innovation metrics; and then translate those metrics into progress toward achieving the agency-level mission and goals.

ARPA-E's program directors have been empowered to take risks in selecting projects, in line with the agency's mission, and to actively manage ongoing projects, which includes altering project milestones, budgets, and timelines. ARPA-E should strive to preserve this management approach, the report says. In addition, one of ARPA-E's strengths is its focus on funding high-risk, potentially transformative technologies and overlooked "off-roadmap" opportunities pursued by neither private firms nor other funding agencies, including other programs and offices within DOE. ARPA-E leadership and the secretary of energy should actively work to sustain this culture.

ARPA-E is a positive agent of change in DOE and the federal government, and its best practices are being adopted in some DOE offices, the report says. For example, the Office of Energy Efficiency and Renewable Energy has incorporated several elements of ARPA-E's approach into management of its own programs, including use of a workshop to define a program, use of concept papers to screen funding applicants, and early termination of underperforming projects. The secretary of energy should ensure that other offices and programs within DOE continue to explore and adopt elements of ARPA-E's practices that can improve the department's operations.

In addition, the report recommends that ARPA-E reconceptualize its "tech-to-market" program, which aims to provide researchers with business and other non-technical support and advice to help ensure that projects develop with market needs in mind. The agency considers this program to be an ongoing experiment, and the challenge of developing such a program may be greater than originally thought. The roughly three-year time frame of an ARPA-E project is too short to expect a technology to move from concept to market. ARPA-E should reconfigure the program to account for the wide variation in support needed across programs and performers with respect to prospective funding, commercialization, and deployment pathways. For example, ARPA-E should consider making full tech-to-market plans optional, but requiring performers to describe potential product applications if they can prove technological feasibility.

The report also recommends that the agency streamline its reporting requirements. The "high touch" nature of ARPA-E, where program directors review progress of those performing ARPA-E projects quarterly, is a hallmark of the agency. However, since the quarterly written reporting required can be challenging, ARPA-E should consider changing the process so that it consists only of quarterly presentations to the program directors and their teams, with only fourth-quarter or annual reports providing full written details.

Provided by National Academies of Sciences, Engineering, and Medicine