Greg Randolph MD, MPH
President and CEO, Population Health Improvement Partners
Professor of Pediatrics and Public Health, University of North Carolina at Chapel Hill
Measurement is a fundamental aspect of quality improvement (QI). Some fear that measurement for population-level improvement initiatives is exceedingly different than in a typical QI project, or even impossible. However, I have good news:the same principles apply; they just require minor adaptations, mostly in the number of measures needed and the related feasibility issues.
At the beginning of any improvement initiative, it is critical for leaders of the initiative team to set measurable goals that address the question, "What are we trying to accomplish?" Once these goals are established, the team can begin to develop a set of measures that answer another key question: "Are the changes we're making leading to improvement and moving us toward achieving our goals?"
Using a Set of Measures
A cornerstone of improvement science is the concept of a system. All systems tend to be complex and dynamic. It's not surprising, that when dealing with the health of a maternal and child population, complexity and the dynamic nature are even greater. Thus measuring the impact of improvement at the population level must take this complexity into consideration, and more measures (e.g., five to 10 measures) might be needed than in typical QI efforts. There will never be a "silver bullet" measure that can accurately reflect improvement within a complex system, so we need to think of a set of measures, including outcome, process and "balancing" measures.
Outcome measures address how policies and services affect the health, functional status and satisfaction of the population targeted in the improvement initiative (or simply, the experience of the population and its members). Process measures address how and what services and policies are provided (i.e., what programs and public health officials do). Balancing measures address potential unintended consequences to the system as it is changed (i.e., what could go wrong).
Outcome measures are very important for all stakeholders, especially initiative leaders, who want to know the ultimate impact of the initiative; they must be a part of any improvement initiatives' measure set. Unfortunately, outcome measures are often slow to change (e.g., MCH datasets like PRAMS, BRFFS and YRBS are primarily outcomes), so inclusion of one or more process measures is required so the team can understand the effects of the improvement effort quickly and is able to assess whether the changes they are making are resulting in improvement. In addition, at the end of the initiative, the process measures can help demonstrate that the intended changes were indeed implemented. Finally, any time that changes are made to a system, there can be unintended, adverse consequences. We don't want to improve one aspect of a system at the expense of another, as this could lead to less or no overall improvement, or even overall harm. Usually the team can identify several possible things that could go wrong early in the planning stage, such as decreasing client satisfaction with the time spent with Women, Infants and Children (WIC) staff when attempting to increase the efficiency of a WIC site. In that case, we don't want to be more efficient by decreasing the quality of the interactions with staff.
Feasibility Issues – Data Collection and Analysis
It is often best to use existing data sources when possible. In addition to the above MCH data sources, various partners and stakeholders, community or state health assessments, Community Commons and other local, state and national resources can be vital. However, due to the need for both process and balancing measures, teams will often need to collect some of their own data. Feasibility and cost are key issues but can be mitigated by using some of these strategies:
- Sampling strategies are key, so consider using small sample sizes (e.g., 20 to40 observations) collected frequently (e.g., monthly or quarterly) and/or representative convenience samples.
- Simple data collection instruments and methods, like check sheets and very brief surveys, can help minimize costs and effort.
- Leveraging technology can also help (e.g., email surveys or scannable forms).
Finally, due to the dynamic nature of complex systems and the need to track progress , the best way to analyze data for improvement is to report your data graphically in run charts.
In this era of health transformation and a strong focus on value, we should consider economic impact measures (sometimes referred to as "ROI" for "return on investment") as outcomes as well. Measuring economic impact is consistent with the "triple aim" of better quality services, better outcomes and lower costs that is fundamental to the Affordable Care Act and related federal health policy. Similarly, it is very important to address and measure health disparities in population health improvement work. This can often be done by stratifying measures among priority populations in the population that is targeted for improvement.
Measurement Resources for your Population Health Improvement Efforts
- Our "QI Step by Step Guide" has some useful QI measurement tools. Find it here.
- A comprehensive approach to QI measurement is outlined in this article.
- For an overview of using ROI in public health settings, review this free, full-text article.
Population Health Improvement Partners is a national leader in building community and organizational capacity to improve and sustain population health.