The New World of Big-Data-Informed Performance Management
How much will the increasing capacity of states to gather and manipulate large quantities of data help improve the use of performance measurement to make decisions? The possibilities are exciting and just begin with: an increased capacity to disaggregate performance measures, which helps attract public attention; better validation of performance measures and the capacity to make more use of information about the value-added aspects of programs.
About the Authors
CSG Senior Fellows Katherine Barrett and Richard Greene are experts on state government who work with Governing magazine, the Pew Charitable Trusts, the Volcker Alliance and others. As CSG senior fellows, Barrett and Greene serve as advisers on state government policy and programming and assist in identifying emerging trends affecting states.
Performance-informed management has been around for many decades. There was a rebirth of interest around 25 years ago, but this was only the beginning of public sector excitement about its possibilities. States that had never considered making an effort to replace the private sector profit-and-loss statement with their own measures of success were just beginning to explore the value of such efforts and the process necessary to make first steps. It wasn’t until 2007 that all 50 states had bought into the notion that performance management—and its primary tool, measurement of progress within programs—was a worthwhile effort statewide.1
Just a couple of decades ago, in fact, the possibility of measuring the results of government programs was often spoken of in the future tense. The big topic of the day was Total Quality Management, an effort underway in about 36 states to help ideas percolate from the bottom, carefully determine what citizens really want and measure the progress made.2 But though interest was high, for many states Total Quality Management eventually wound up on a list of forgotten management initiatives.
What’s more, for those states that were beginning to issue performance measurement reports back in the 1990s, there was a huge gap between what governments wanted to measure and what they could measure. Performance measurement documents were riddled with NAs—for “Not Available”—when it came to actually showing data that supported the measurements. There were plenty of state officials who were still dubious about the whole process, arguing that performance measurement would steal far too much time and resources away from actually providing direct services to the public. As one state budget officer told the authors of this essay in 1992, “I think that the people who want to see more performance measurement are dying off—one zealot at a time.”3
Back then, the focus of the effort, even for many enthusiasts, was on “performance reporting in a static fashion,” according to Donald Moynihan, a professor at the La Follette School of Public Affairs at the University of Wisconsin. You made it available to interested parties, “and your job was done.”4
Of course, that kind of thinking changed as time passed. Just publishing reports, particularly ones with lots of missing data, did not produce a strong impact. As Moynihan recalls, the more people thought about performance measurement, the more they came to the conclusion that it had to be utilized in a fashion that would actually permit it to affect decision making.5
As years passed, pressure began to accumulate to move from measuring simple outputs—like the total number of people served in a drug rehabilitation program—to outcomes, like the percentage of those people who were still off drugs after three or five years. By 2008, the vast majority of the states had agencies that were involved in just this kind of work. As the Pew Charitable Trusts’ Government Performance Project stated in early 2008, “A growing number of (governors) are now personally involved in improving the way information is used to manage their states.”6
As published in Governing magazine, the report offered some specific examples. During his term, Ohio Gov. Ted Strickland began the “Turn Around Ohio” plan that includes flexible performance agreements with his agency heads. Similarly, Maryland’s former Gov. Martin O’Malley built StateStat, a comprehensive means for making decisions based on data, similar to his CitiStat effort in Baltimore. He described it as a system “that actually sets goals and has the guts to measure progress towards achieving those goals.”7 Though the researchers for the Government Performance Project couldn’t have known it at the time, the emphasis on heightened use of so-called “big data” has become the new Holy Grail in many states’ efforts to measure—and manage—performance. The term “big data” means different things to different people. But for the purposes of this essay, the authors will use the term simply to mean the technologically infused capacity of states to gather and analyze huge amounts of concrete measurable information—across agency silos, wherever possible—to help guide their actions. “Big data has reduced the barriers to useful performance management,” said Dustin Brown, deputy assistant director for personnel and performance management at the U.S. Office of Management and Budget.8
The arrival of big data on the public sector stage has generated a kind of electricity in the air that is similar to that felt a couple of decades ago about performance measurement generally. As John Kamensky, senior fellow for the IBM Center for the Business of Government said, “The biggest thing in the whole performance measurement world is the introduction of data analytics. The use of analytics with performance measures can actually change the performance on the ground.”9 Better data analysis will cause the states to think about how they were going to get to their objectives. “I’m just thrilled when governments use any kind of data and start to play with it,” said Jeff Tryens, a performance measurement authority whose experience includes stints in South Australia, Oregon and New York City.10
Paul Epstein, principal with New York-based consulting firm Epstein and Fass, emphasizes that he hasn’t seen examples where this marriage has been fully consummated. Before big data can help improve performance measurement, “a government organization needs a functioning performance management system that agency managers are comfortable working with in a performance improvement mode, not in a defensive mode where they try to get away with conservative targets,” he said.11
The degree to which big data will change the performance management world won’t be known for some time. But there’s room for optimism. For one thing, public—and state employee—familiarity
with the potential for data use has grown enormously. “They know they can Google anything,” said Philip Kase, performance management chief at the Oregon Department of Transportation.12 Nathalie Molliet-Ribet, senior associate director of Virginia’s Joint Legislative Audit and Review Commission agreed. “The expectations have gotten really high that folks should have access to everything we know about what’s happening,” she said.13
Benefits of Big-data-informed Performance Measurement
There are many ways in which performance measurement can benefit from the use of big data, not the least of which is the credibility that validated information brings to any kind of work. Legislators are far more inclined to pay attention to the information that they find on their desks if it’s backed with careful analysis of great quantities of data—even if, realistically, they don’t actually look at the data itself.
There also has been a growing emphasis in legislative circles on considering policies that have roots in actual evidence; successful experiences from other states or other agencies within their state. This underlines the importance of standardized data, which enables states and the federal government to compare and analyze the success of programs.
“Many states have started to embrace the technological changes that would make it possible to electronically track performance. It comes down to common data formats,” said Hudson Hollister, executive director of the Data Coalition. “If the information is searchable across all of the different agencies and offices of the states, then it can be used to automatically track whatever the subject matter is. But if the information is not searchable and there is not a common format, then complicated translations and analytics are necessary.”14
Another area for which growing integration of data and performance measurement is important is in the use of social impact bonds, sometimes called social benefit bonds. These bonds are agreements between investors and the public sector to invest in programs that will result eventually in better outcomes and savings. The savings are then to be used to pay back the investors with interest. While the use of such instruments isn’t common yet, they are being considered by many states, and it’s clear that their success will come only if the public has faith that the savings will be added up using sophisticated data-based techniques coupled with measures of performance.
Of course, one of the most obvious performance management uses for big data comes with the ability it provides to compare entities across state government and invest tax dollars wisely. This is the kind of thing that simply is not possible if statewide systems don’t exist to gather and validate parallel information from counties, cities, towns and so on. Similarly, big data can help provide states with the ability to work with and aid local governments in comparing themselves to others and uncover the cities or counties that can serve as models.
There are other benefits to be found at the juncture of big data and performance measurement. A few of these:
- It offers new abilities to communicate with citizens—particularly given the potential to disaggregate information—so that it is meaningful to individuals, who can see the impact of programs on their own lives. Data analysis can shift the measures of government performance away from a technocratic world and turn them into information that the average taxpayer can understand and use.
- Simple performance outcome measures can take users off track, but combined with databased evidence, analysts more easily can discern whether the measures are providing good information or if the results are deceiving. For example, positive results can be obtained when performance measures focus on an unrepresentative group of service users. But data analytics can help to reveal whether that’s been the case.
- Data analysis also can help to move from a straightforward look at comparative outcomes to a sometimes more sophisticated approach in which value added is measured. Take the comparison of two universities as an example. One may start with the cream of the crop in its student body, and show terrific educational outcomes. The other may begin with less well prepared first-year students. Simple outcomes would make the first school look better. But with the use of big data, the value added to the two student bodies can be compared and the second school may come out on top.
Why Technology Makes a Difference
Not so many years ago, when data was transferred manually from spreadsheet to spreadsheet and generating the data in the first place was extremely time consuming, states had every reason to push back at the talk of generating more and better information. But with new technology, “we’re going to have a lot more state detailed information, rather than just high level,” said Molliet-Ribet. “There were probably ways to do this with an asset database or a spreadsheet,” she said. “But I think, obviously, technology has gotten better, faster and cheaper, and is helping make these things happen.”15
Generating performance information also has become easier because citizens themselves can now feed into the process important pieces of information. In the past, gathering data from millions of individuals was a gargantuan task. Now with most Americans plugged into car-based GPS systems, cell phones, kiosks in selective locations and so on, they can provide information nearly effortlessly, which can be transmitted to a central location, aggregated and then disaggregated in dozens of different fashions. Not only does this ease the accumulation of data, it also means that it can be reported in real time. What’s more, according to the 2013 report “Collaboration Across Boundaries,” published by the IBM Center for the Business of Government, individuals can help to build a data-based performance management system in four important ways.16
- “As explorer, citizens can identify, discover, and define emerging and existing problems in public services. For example, the New York-based Datakind initiative involves citizen volunteers using their data analysis skills to mine public data in health, education, environment and more areas to identify important civic issues and problems.
- “As ideator, citizens can conceptualize novel solutions to well-defined problems in public services. For example, initiatives such as Challenge. gov and OpenIDEO employ online contests and competitions to solicit innovative ideas to solve important civic problems.
- “As designer, citizens can design and/or develop implementable solutions to well-defined problems in public services. For example, as part of initiatives such as NYC Big Apps and Apps for California, citizens have designed mobile apps to address specific issues such as public parking availability, public transport delays, and more.
- “As diffuser, citizens can directly support or facilitate the adoption and diffusion of public service innovations and solutions among well defined target populations. For example, physicians interacting with peers in dedicated online communities have assisted government agencies in diffusing health technology innovations.”
A Helping Hand
The federal government has been leading the way for states to make progress in the use of data for performance measurement. The 2014 Data Act, for example, is shifting federal agencies to use a common data format to report their spending and to make interagency comparisons far more clear. The act is still in its beginning stages of implementation and is supposed to be fully implemented in another year. While it is focused on spending data, it may lead to more standardized performance reporting, said Kamensky. “Ultimately, after you figure out how to report financial data then you can come up with a model of how you would be able to report performance data.”17
“I don’t know of a state program that is as ambitious as the data act,” said Hollister. “But I always like to point to Ohio, which has already published all of its spending transactions on one platform.”18
The new law will largely affect states in terms of grant-related reporting, though that is still in a pilot phase. In the future, it also will likely lead the way to more standardized data generally. “I thought states and universities would find the Data Act highly intrusive,” said Kamensky. But as it turns out, “I haven’t heard a lot of screaming,” he added. Kamensky theorizes that is because the states have discovered that technology already in place has made reporting less onerous than it would have been not so many years ago. That’s good news for advocates of standardization within the states themselves.
There are other examples of the push on the part of the federal government to focus increasingly on the use of data, including the reporting that was required in the American Recovery and Investment Act of 2009 and the aggressive federal push for electronic health records. While the health record initiative has great potential for improving patients’ experience when they enter the medical system, it also represents a good example of how difficult this kind of work can be.
Robert Wachter, author of a recently published book, The Digital Doctor, cites a number of obstacles that have been encountered along the way. For example, the technology that is geared to ease the transference of data is often designed by technology specialists who don’t have a solid understanding of how the data eventually will be used. Even when medical data is more easily obtained, the users often are not trained effectively and there are many examples of routine errors in simple data entry.19
Wachter points to these challenges as cautionary notes, not as evidence that they won’t be overcome as time goes on. He quotes from the book Smarter than You Think by Clive Thompson, “The past turns out to be oddly reassuring, because a pattern emerges. Each time we’re faced with bewildering new thinking tools, we panic—then quickly set about deducing how they can be used to help us work, meditate and create.”20
In all cases, the federal government can only get things rolling in the states, like pushing a hoop with a stick. The same is even true with edicts from the highest levels of state governments. “Federal and state policies can establish a supportive framework,” wrote Patrick Lester, director of the Social Innovation Research Center. “But actual improvements must take place on the front lines.”21
The Challenges to Come …
An examination of the trend of combining big data with performance measurement doesn't’t only reveal the potential benefits; a number of challenges were pointed out by the men and women who are deep in the trenches of states that are trying to make this connection.22
For example, often the data that is collected is the easiest to collect and not necessarily the most useful. Performance measures may be selected for much the same reason. The information that is needed to measure and manage performance needs to be selected with consideration of the impact the measurements will have. Similar thought needs to go to what data is collected.
Additionally, it’s critical that all the participants involved in this process understand the utility of the data they are collecting and analyzing. If the use of the data is not well understood, the individuals collecting the data may not pay necessary attention to ensuring its accuracy.
Efforts to optimize the utility and accuracy of performance measures from agency to agency “requires some sort of centralization,” said Mike Lawson, former head of performance for the International City/County Management Association (ICMA) and currently a consultant in the field.23 But centralization can be difficult to achieve, particularly when agencies relish their autonomy and are disinclined to play together nicely. As a result, leadership from the top can be critical to ensure a focus on data and how it’s going to be used.
Indiana provides a good example of leadership from the very top. Gov. Mike Pence signed an executive order in 2014 creating a “management and performance hub,” in the state. “Hoosiers can benefit from a comprehensive and coordinated effort by state agencies to share data and improve and strengthen services,” he wrote in the order. “Centralized data sharing, correlation and analysis capability will enable the state to achieve efficiencies in the administration of state programs.”24
The list of other challenges to the successful use of data for performance measurement—and other purposes, as well—goes on to include lack of resources, legislators who want data and measures but are disinclined to use them to make decisions, inaccurate data that can lead to false conclusions, data overload, misinterpretations of the actual meaning of the data and the context for the measures, and unclear definitions. “When you’ve got all this data, people can very quickly not understand what they’re looking for and that can lead to confusion and even distrust,” said Kase.25
But, while these challenges may all play a part in slowing states down in their efforts to utilize performance measures and big data to their fullest potential, they needn’t prevent them from making the effort. The benefits are so clear—and have sufficient potential for transforming the way government does its business—that the resultant accomplishments will be worth the risks and costs along the way.
1 The Pew Charitable Trusts, Government Performance Project research.
2 Financial World magazine, May 11, 1993.
3 Authors’ interview, 1995.
4 Authors’ Interview, Jan. 21, 2016.
6 Katherine Barrett and Richard Greene, “Measuring Performance: The State Management Report Card for 2008,” Governing, March 2008.
8 Authors’ interview, Feb. 3, 2016.
9 Authors’ interview, Jan. 12, 2016.
10 Authors’ interview, Jan. 20, 2016.
11 “Thoughts on ‘Big Data’ and Government Performance Management,” a brief prepared for this essay by Paul Epstein, Jan. 13, 2016.
12 Authors’ interview, Feb. 2, 2016.
13 Authors’ interview, Feb. 1, 2016.
14 Authors’ interview, Feb. 10, 2016.
15 Authors’ interview, Feb. 1, 2016.
16 Satish Nambisan and Priya Nambisan, “Engaging Citizens in Co-Creation in Public Services: Lessons Learned and Best Practices,” IBM Center for the Business of Government, 2013.
17 Authors’ interview, Jan. 12, 2016.
18 Authors’ interview, Feb. 10, 2016.
19 Robert Wachter, The Digital Doctor: Hope, Hype and Harm at the Dawn of Medicine’s Computer Age, McGraw-Hill Education, 2015.
21 Patrick Lester, “Building Performance Systems in Child Welfare,” Social Innovation Research Center, Feb. 8, 2016.
22 Katherine Barrett and Richard Greene, “The Causes, Costs and Consequences of Bad Government Data,” Governing, June 24, 2015.
23 Authors’ interview, Feb. 2, 2016.
24 State of Indiana Executive Department. Establishing the Governor’s Management and Performance Hub, Executive Order: 14-06, posted by Legislative Services Agency, May 14, 2014.
25 Authors’ interview, Feb. 2, 2016.
|The New World of Big-Data-Informed Performance Management||74.6 KB|