As part of RFF’s Regulatory Performance Initiative, led by RFF Senior Fellow Richard Morgenstern, a number of experts analyzed the actual performance of environmentally oriented regulations throughout the federal government to determine if the realized costs and benefits were consistent with projections. Morgenstern sat down with Resources to discuss how such analysis might benefit the development of future regulations.
RESOURCES: You just finished this in-depth project on retrospective performance analysis of environmental regulations. What is the value of such an undertaking?
RICHARD MORGENSTERN: Regulatory analysis is currently done before regulations are issued—at a time when much of the analysis is based on unverifiable assumptions. Retrospective analysis, on the other hand, provides the opportunity to look back and see how rules actually performed, what the goals were, whether they were fully achieved or not, and at what cost. The broad point is to learn from past experiences and improve future rulemaking.
Retrospective analysis is difficult to do because the available data are generally limited. There are not routine ways of collecting information after the fact and, frankly, the incentives are not particularly well structured to do that type of analysis.
RESOURCES: Despite these challenges, you and your colleagues were able to examine the actual costs and benefits of a number of regulations.
MORGENSTERN: That’s right. The team, which includes experts from inside and outside RFF, completed nine studies that look at regulations from the US Environmental Protection Agency (EPA), Department of Energy (DOE), Department of the Interior, and Food and Drug Administration. The selection of rules is quite diverse, which creates both challenges and opportunities in drawing conclusions. Because some of the studies involved multiple rules, we have a total of 34 ex ante–ex post benefit or cost comparisons in the group.
RESOURCES: Are the rules providing the benefits or environmental improvements that were forecast originally, and are they doing so at the original cost estimates?
MORGENSTERN: Although it’s difficult to generalize from our small sample, one can interpret our results to suggest somewhat of a tendency to overstate both the costs and benefits in the regulatory impact analyses that take place before the rules are issued. Of course, there is no iron law of overstatement, as we have plenty of examples of both underestimates and accurate ex ante analyses. Now, the next question is, are the benefits overstated more than the costs or vice versa? Unfortunately, we really can’t answer that in this study.
RESOURCES: There are some specific issues that can affect the outcomes and costs of regulations. Trading programs, for example, would seem to present opportunities for cost savings, taking the dynamism of the marketplace into account. What did you find with the trading programs that you examined?
MORGENSTERN: We were very fortunate to have a truly outstanding study done of Phase Two of the Acid Rain Trading Program. This is not a new program, but that’s part of the reason it was chosen—because it already has a track record.
No one disputes the massive net benefits from this program. However, our team asked a somewhat different question: had the rule been issued as a uniform performance standard rather than one that allowed utilities to trade with each other, what would the results have been? And the findings are quite interesting. On the one hand, the cost savings from trading were considerable. On the other, they were not as great as had been forecasted in the rulemaking. Going forward, the case study authors suggest the textbook case of the least-cost economic solution should be treated as the lower bound of costs rather than the best-case scenario. Such an approach might provide a more realistic estimate of the costs of a new regulation.
The team also looked at the benefits of trading: How did the trading component itself change the overall benefits? They found that because the program moved certain emissions from areas of relatively low population density in the Midwest to higher-density areas in the East, we ended up with slightly lower overall benefits compared to what would have occurred under a uniform performance standard.
I want to emphasize that this does not undermine the overwhelming success of the acid rain program in terms of achieving enormous benefits at relatively low costs. But there are some important nuances. On the benefits side, the team argues for greater consideration of the potential distributional consequences before a trading-based rule is put in place. In fact, modified designs could have been introduced—for example, zoned trading—to prevent this particular problem from occurring.
RESOURCES: One issue that is often discussed is the costs and benefits of regulations requiring new environmental technologies. Did you examine any such cases, and did these technologies deliver what they had promised?
MORGENSTERN: Two of our case studies feature issues of technology development. One looked at the minimum efficiency performance standards in the DOE regulations for household appliances, and the other looked at the development of new technology under EPA’s renewable fuel regulation. And the results of the two studies are opposite of one another.
DOE tended to overestimate the costs of these new technologies. Had its analysis been more accurate, it might have ended up with more stringent standards. EPA, on the other hand, tended to underestimate the costs of renewable fuels. Had it been more accurate, that might have resulted in a less stringent standard than was ultimately adopted. Ironically, EPA has had to soften its standard on more than one occasion since that initial promulgation.
RESOURCES: It seems that technology and the path it takes exemplify the tremendous uncertainties that regulators have to face when they’re crafting a rule, but are there other examples?
MORGENSTERN: Uncertainty is prevalent in everyday life, and it’s no less prevalent in the regulatory world. For the renewable fuels rule, for example, EPA seemed to have not taken full account of the uncertainty of future energy prices. It assumed that the prices in place at the time the rule was promulgated would continue to rise at healthy levels from that day forward. Failing to recognize the long historical record of volatility in energy prices meant that it didn’t understand that the economics of renewable fuels could be adversely affected by a major drop, for example, in world oil prices.
One recommendation from this study is for the agencies to have more explicit considerations of uncertainty. We refer to them as “stress tests,” borrowing the popular term from the banking sector, and suggest that stress tests be applied to rules that are particularly vulnerable to such macro-level fluctuations.
RESOURCES: Several experts at RFF are currently conducting a major analysis of EPA’s Clean Power Plan, which could include trading, depending on what form of regulation the states adopt. Technology is going to be a big part, as are distributional aspects and uncertainties. What insights do you have for this work, based on the analysis you’ve just completed?
MORGENSTERN: Evaluating the Clean Power Plan while it’s under way would clearly have a great deal of value. Looking at the differences across states and planned requirements deserves careful attention, ex post as well as ex ante. There are going to be major data issues associated with evaluating this rule—they’re going to need to collect information on energy production and costs, greenhouse gas emissions, and the ancillary emissions effects because the co-benefits are an important part of the overall value of the rule as predicted by EPA. Also, coordination with the Energy Information Administration and the states is going to be essential to understanding what’s really going on with the Clean Power Plan.
RESOURCES: It seems like this type of analysis is critical to improving the efficiency of regulations, as well as maximizing benefits and reducing costs. What do you think needs to happen to encourage more retrospective analysis of regulations in the future?
MORGENSTERN: A major barrier to this work is the difficulty of obtaining useful data for evaluation purposes. One thing that the agencies could do for new regulations is to build a plan of retrospective analysis into the regulation at the time it’s promulgated rather than having researchers piece it together after the fact. Obviously, you can’t do it for all rules—especially in times of tight budgets. You should be selective. But the idea of picking out some high-priority rules and having the agencies design retrospective studies up front is an appealing idea. I think the Office of Management and Budget has a role to play in developing guidance for the agencies on how they should be measuring the outcomes of rules. Our study contains some guidance based on our case studies—including many examples of what to do and what not to do.
Find the case studies from RFF’s Regulatory Performance Initiative and more at www.rff.org/regulatoryperformance.
A version of this article appeared in print in the Winter 2016 issue of Resources magazine.
