Feedback
Ways to measure public health program and policy effectiveness
This is an excerpt from Foundations of Physical Activity and Public Health 3rd Edition With HKPropel Access by Harold W. Kohl III,Tinker D. Murray & Deborah Salvo.
As soon as the program has been defined, thoughts should turn to defining the evaluation. This definition is an interactive process that can result in changes to the program plan. Because the program implementation plan and the evaluation plan inform each other, they should be addressed simultaneously. Asking questions about formative, process, outcome, and cost-effectiveness evaluations helps to clarify aspects of the program that may not have been part of the original idea. The logic model, discussed later, helps to create this interactive experience.
Although the evaluation questions one can ask about a physical activity program may seem endless, they can be conveniently categorized into four categories (see figure 16.1):
- Formative evaluation
- Process evaluation
- Outcome evaluation
- Cost-effectiveness evaluation
Formative Evaluation
Formative evaluation is the first level of a physical activity program evaluation. In formative evaluation, the fundamental questions focus on the needs, utility, and design features of a physical activity promotion program or policy and its individual components. Questions asked during the formative evaluation stage do not focus on the expected outcome (e.g., changes in physical activity behaviors) but rather on the overall design of the program. Formative evaluation questions for the senior balance exercise program discussed earlier could include the following: What is the extent of the problem of falling in the group? How frequently will participants be willing to come to the exercise classes? Will incentives be needed to increase participation? Is there a leader in the target population who can assist with outreach? What kind of equipment will be needed? How should instructors be trained? Clearly, this information can help the program manager or policy decision-maker craft components that will have the best chance of hitting the intended target.
The primary information sources of data during formative evaluation are (1) expert opinion and previous work and (2) the target population of the program. Expert opinion, including that of the people implementing the program, comes from previous experience, similar work in other settings, published and unpublished examples, and other sources. The experiences of people who have done something similar can be invaluable to a program manager looking to build a physical activity promotion program. Often, these experiences are not published and are available only through networking with people who have similar intentions. A quick search of the Internet or published literature indexing services (such as MEDLINE), or both, can begin the journey down the road to gathering expert opinion.
Needs assessments provide the second source of data for formative evaluations. Needs assessments can be formal, standardized surveys or interviews, or more qualitative discussions with key people knowledgeable about effective strategies. However it is done, the general purpose of a needs assessment is to gather information from the target population (or one that is similar). For the municipal bicycle lane example, good needs assessment questions would be, How great is the need? Would people even be interested in using the bicycle lane? If not, why not? If so, what will keep riders using the bicycle lane over time? Will incentives to participate be helpful? How about disincentives to driving in the same area? What type of outreach should be used? In other words, what might work to fill the need?
Formative evaluation should be used throughout the program. Although it is necessary before the program starts to determine the best targets and strategies for implementation, it should not stop once the program begins. A well-designed formative evaluation provides feedback throughout program implementation so that changes and adjustments can be made. For example, interviews with participants and nonparticipants in a balance exercise class for seniors can help program planners understand the characteristics of each group, why some continue to come, and why others dropped out or never came. They can then adjust the program accordingly.
Progress Evaluation
Unlike formative evaluation, process evaluation focuses on program or policy implementation. Of interest is how well the physical activity program or policy is operating and what can be done to improve those operations. Process evaluations help program managers assess the quality of program or policy delivery. They examine delivery strategies that appear to be successful and those that may not have worked as planned. Both are important not only for evaluating the program or policy, but also for informing future efforts.
Process evaluation should include the assessment of delivery alternatives. Program directors and policy decision makers can learn how best to deliver a program or implement a policy by testing their strategy against an alternative. For example, in a walk-to-school program, a pilot test could be designed to determine whether leader-supported walking groups (a parent volunteer leads children to school) result in more children walking to school than individual groups of children and their families. Such a systematic approach makes for a much stronger evaluation. Alternatives can be identified in the formative evaluation stage (discussed earlier), with input from potential program participants or experts. They can then be tested during a process evaluation.
Using the balance exercise class for seniors example, process evaluation questions may include the following: Are all exercise classes being offered as scheduled? Is instructor training and certification being offered consistently according to predetermined protocols? What is the attendance at the classes? How do these data compare to prestated goals? Are there trends in class attendance over the weeks and months that the program is being offered? Are other systems that have been put in place working as intended? Why or why not? What program adaptations seem to affect attendance?
The information gleaned from these process evaluation questions can be useful for monitoring the implementation fidelity of the program or policy. Implementation fidelity refers to the extent to which the program or policy is being implemented as originally planned. For instance, do all senior balance exercise classes follow the preestablished curriculum? Or do all balance exercise classes start on time? Additionally, process evaluation data can be used for understanding the program or policy outcome evaluation data (discussed next). For example, if outcome data for the balance exercise class show one or two sites that have reduced the risk of fall-related injuries among participants by 60 to 80%, the process evaluation data can be used to determine whether differences in implementation exist between the sites that are doing well and those that are not. If class attendance was routinely very high at the sites where falls have become less prevalent—and conversely, attendance was very low or intermittent at sites that have seen no change in the risk of falling among participants—it could be that the dose of physical activity was higher among those participants who are now at lower risk. Clearly, this process is not linear but requires an understanding of evaluation outcomes on all levels.
Outcome Evaluation
Outcome evaluation, sometimes referred to as impact evaluation, focuses on cause and effect. Did the program or policy have the intended effect on the outcome of interest? For the purposes of physical activity promotion programs and policies, the outcome of interest is usually increasing levels of participation in overall physical activity or in a specific physical activity (e.g., bicycling). Questions about the design and implementation of the program or policy have been answered in the formative and process evaluation stages. Now the question is, Has the program or policy increased physical activity behavior? Outcome evaluation seeks to answer this question while also assessing, by process and formative evaluations, how the intended effects were reached (or not). Outcome evaluation metrics for physical activity promotion programs or policies could include minutes per week that the target population engaged in moderate- or vigorous-intensity physical activity, changes in their physical fitness levels, the percentage of the target population meeting physical activity guidelines, or the number of people bicycling for practical purposes at least once per week.
Because cause and effect is of central interest in outcome evaluation, changes in metrics such as these are often used as outcome (impact) measures. This, of course, assumes that a baseline assessment was done prior to the start of the program or implementation of the policy that will allow the calculation of change in the outcome of interest.
Outcomes of physical activity promotion programs and policies can be assessed using any of the measures discussed in chapter 4, with each having its strengths and weaknesses. In the walk-to-school example, the number of pedometer-measured steps that randomly selected students take per day, both before and after program implementation, may be an important outcome of interest.
The underlying reasons for observing an outcome can be as important as measuring the outcome itself. To this end, physical activity program and policy evaluations also focus on hypothesized upstream determinants as well. Upstream determinants are the known or hypothesized factors that cause a certain outcome (or do not cause a certain outcome) to a person or community involved in a program. For example, self-efficacy (i.e., in the context of this example, a person’s belief in her ability to become more physically active and sustain that behavior) is a likely mediator in participation in physical activity. If the balance exercise program for seniors shows an increase in participants’ self-efficacy and no such increase in those who did not participate, then the program evaluator might reasonably conclude that increases in participants’ beliefs that they can do the exercise program are one of the reasons physical activity increased in the group. A thorough outcome evaluation of a physical activity program not only measures the effects of the program or policy on physical activity behavior but also suggests possible upstream determinants of that behavior. This gives program staff a better understanding of the mechanisms through which the program may be working.
Evaluation Questions
Physical activity evaluations come in all shapes and sizes. One characteristic of successful evaluations is that they have very clearly conceptualized evaluation questions. This helps define the what, the how, and the who for the target population as well as for the program staff. Before starting any physical activity evaluation, program planners should work hard to state the questions as clearly as possible.
Cost-Effectiveness Evaluation
A final category of evaluation, and one that is often overlooked in public health evaluations, is related to the economics of the program or policy. The purpose of a cost-effectiveness evaluation is to assess not only the overall costs of delivery but also how these costs compare with those of alternative program delivery options. Also of interest is how the costs of program or policy delivery and implementation compare to costs (real or estimated) of not delivering the program or policy.
In public health, the costs related to health issues (short-term as well as lifetime) must be balanced against program delivery costs. These health costs include the money spent to treat a disease, illness, or medical condition per person as well as indirect costs such as loss in productivity. Although a complete treatment of methods of cost-effectiveness evaluation is beyond the scope of this textbook, the basic message is that this form of evaluation should not be overlooked. In the balance exercise program for seniors example, costs of implementation include staff salaries, physical resources, evaluation costs, and participation time (the cost of doing one thing at the expense of something else). These costs are then measured against any medical care expenditures (savings) that may be due to the balance exercise program (e.g., fewer emergency room visits, bone fractures, joint replacement surgeries).
Ding and colleagues (2016) published a study examining the global economic burden of physical inactivity. Using information on the known effects of physical inactivity increasing the risk of different health outcomes (e.g., cardiovascular disease, diabetes, cancer), as well as information on the cost of these conditions to health systems around the world each year, they estimated that physical inactivity cost health care systems $53.8 billion (international dollars) in 2013. Additionally, they estimated that deaths due to physical inactivity contribute to $13.7 billion in productivity losses each year, with physical inactivity being responsible for 13.4 million years lost due to disability (disability-adjusted life-years: DALYs).
Laine and colleagues (2014) conducted a systematic review of the evidence on the cost-effectiveness of different physical activity strategies. Cost-effectiveness analyses use information on physical activity program costs as well as gains in health outcomes to be expected from increases in physical activity (e.g., lower risks of heart disease, some cancers, and diabetes). The results of the synthesis of this evidence showed that the evidence-based physical activity promotion strategies highlighted in chapters 11 through 14 of this text provided a good value for the money needed to implement them. In their review, Laine and colleagues found that among the most cost-effective strategies to promote physical activity were the expansion of community trails and paths ($0.006 per MET-hour gained [2012 USD]), the use of pedometers as a behavioral strategy to increase walking ($0.014 per MET-hour gained [2012 USD]), and school-based physical activity promotion programs ($0.056 per MET-hour gained [2012 USD]). Although there is a wide range of costs and benefits for evidence-based and effective physical activity programs, for many of them, the health gains associated with increased physical activity far outweighed the costs.
Santos and colleagues (2023) conducted the first global study to provide estimates of the number of new cases and associated public health care costs that would occur from 2020 to 2030 that could be prevented if levels of physical inactivity were reduced or eliminated. The authors found that there would be 499.2 million new cases of preventable noncommunicable diseases by 2030 if the prevalence of physical inactivity does not change, and the direct health care costs will escalate to $520 billion (international dollars).
Of the four types of evaluation covered in this chapter, the cost-effectiveness evaluation often has the most influence on policy makers and leaders. If a physical activity promotion program or policy is shown to actually save money by reducing the burden (and costs) of disease and disability, leaders and policy makers often consider it a wise investment.
How Much Should an Evaluation Cost?
Program planners frequently neglect to budget for an adequate evaluation, particularly at the proposal stage. Even the best, most innovative physical activity promotion program will be unable to demonstrate effectiveness without an adequate evaluation budget. How much money is needed? Although there are no clearly developed guidelines, a good rule of thumb is to dedicate 10% of an overall project budget to evaluation activities (personnel, data collection, analysis, and reporting).
More Excerpts From Foundations of Physical Activity and Public Health 3rd Edition With HKPropel AccessSHOP
Get the latest insights with regular newsletters, plus periodic product information and special insider offers.
JOIN NOW