A State Official's Guide to Science-Based Decision-Making
Policymakers are bombarded by information in today’s ever-connected, fast-paced world. Advances in communication platforms, like social media, and information technology have brought a sea change in the public’s ability to access data at unimaginable depths and speeds. This interconnectedness can also pose challenges for state officials trying to solve already difficult issues by adding another layer of complexity to the public policymaking process. As information becomes more available and immediate, policymakers often must make decisions without the technical background necessary to fully vet all the considerations involved.
The Council of State Governments developed A State Official’s Guide to Science-based Decision-making to provide strategic guidance that can cut through the jargon and spin that can accompany technical issues. The guide includes recommendations and helpful tools for policymakers, regardless of background, to confidently assess the assumptions, conclusions and results found in state public hearing witness testimony and scientific studies. The aim of the guide is not to suggest what to think; rather, the impetus is to provide a roadmap of how to approach an issue so state thought leaders can make the most informed decision possible.
Assessing the Methods
Data can be derived from many sources, including case studies, observational studies, controlled studies and risk assessments. This process analyzes how the data collection was performed or how information was gathered, which underpins a study or set of results. Many studies make calculated assumptions and can involve intricate variables. How those variables are controlled can have a tremendous impact on the final results, which can form the basis of an economic impact analysis, policy decision or regulatory action, etc.
Questions to Ask and Warning Signs:
»»Did you gather this information yourself?
»»What specific data or studies form the basis for your scientific conclusions?
»»Have the data or studies been evaluated by other scientists? How extensive was the review?
»»Have important variables been overlooked or ignored?
»»Is the sample size used in the study truly representative?
»»What specific hypotheses or questions did the researchers set out to test? Were steps taken to control other effects?
»»Watch for studies that lack accepted standards or credible references.
Assessing the Results
Once data are produced, a policymaker must then be able to make sense of its outcome and relevancy. One common mistake is a confusion of correlation, which is an apparent connection between variables when they frequently occur together, with causation, in essence, a statement of cause begetting an effect. Further, a general understanding of the statistical significance of the results in a study is paramount. Statistical significance is a technique that calculates the probability that an effect observed in a research study is occurring because of chance.
Questions to Ask and Warning Signs:
»»What are the underlying systematic uncertainties? What degree of uncertainty surrounds the results?
»»Were conclusions based on personal stories/anecdotal evidence?
»»Do the results demonstrate causality, rather than correlation?
»»Are the results statistically significant?
»»Are statistics adequately explained? If not, ask for a practical explanation.
»»To whom or for what do the results apply? Can they be extrapolated to the general population?
»»Do the conclusions logically follow from the scientific results?
»»Be on guard for conclusions making statements of absolute certainty.
Integrating the Knowledge
This segment focuses on critically assessing and formulating the disparate data points and facts learned or gleaned from testimony, study methodologies and results to help inform making the actual decision-making process. This segment will address the role of risk assessments in integrating knowledge and helping policymakers in their decision-making.
Questions to Ask:
»»Have the study results been published? If yes, was it in a journal that requires peer review prior to publication? Have any scientific peer review panels considered the study results?
»»Is there a consensus about the key findings of the studies? What are the areas of agreement/disagreement?
»»Do other scientists share your views? Who doesn’t and why?
»»Have the results been repeated, confirmed or supported by other studies?
»»What are your views on the practical applications of this scientific knowledge/decision?
»»Is the timeliness of study relevant to today’s issues?
»»What are the consequences/implications of action or inaction, including risks? Is there a balanced approach? Why or why not?
Science informs just one part, albeit a critically important one, of the policymaking process; it is not a policy itself. Sometimes there simply is no clear-cut answer science can provide to determine consensus. Policymakers must be able to distinguish advocacy from expertise, understand how scientific data was gathered, be willing to assess the process to gather the data and recognize the relevancy of the data to present day topics. After policymakers learn and assess data and facts, they must then utilize risk assessment to make decisions. Ultimately, the choices made in resolving complex technical issues reside in a leader’s ability to ask straightforward questions about the data and distinguish facts from advocacy—no easy task in today’s hypercharged political environment. This guide strives to help make that process less daunting and facilitate better outcomes that could be applied across a host of policy discussions.