Evaluating the Intervention: Chapter 12
- Evaluate both implementation and patient clinical outcomes to consider the full impact of a patient-reported outcome measures (PROMs) program
- Consider the pros/cons of experimental and non-experimental designs
- Communicate findings about the intervention back to the health system and use them to improve future iterations of the program
Evaluating an intervention is a crucial aspect of gauging its success and determining areas for improvement. Measuring the PROM intervention’s effect on both implementation outcomes, such as acceptability and appropriateness, as well as on patient clinical outcomes, such as symptom burden, survival, and quality of life, can provide a fuller picture of how well the intervention met its objectives.
Different methods can be used to evaluate the implementation of PROM systems. These include both experimental and non-experimental designs. These methods span more controlled methods (e.g. randomized controlled trials) to more flexible and adaptive methods (e.g. quality improvement studies). Quality improvement studies may be more appropriate during the initial implementation phases while more rigorous designs may be more appropriate to evaluate the impact of established programs. Just as implementations science frameworks can be used to guide the incorporation of PROs into the workflow (see Chapter 8, Incorporating in Clinical Workflow), they can also inform an evaluation strategy for the intervention.
Questions and Considerations
A. What outcomes can be evaluated?
- Implementation outcomes describe how well a program is delivered and received
- Examples include acceptability, appropriateness, feasibility, adoption/uptake, fidelity, reach/penetration, costs, sustainability
Patient clinical outcomes
- Patient clinical outcomes include those related to the patient’s health and health service use
- Examples of clinical outcomes for evaluation include symptom burden, use of emergency services, survival, quality of life, physical function, satisfaction with care
B. What evaluation methods can be used?
- Experimental designs include randomized controlled trials, cluster-randomized trials, and similar approaches
- These designs are considered the ‘gold standard’ in evaluating an intervention as they are typically rigorous and maximize internal validity
- Experimental designs typically test efficacy rather than effectiveness, but some implementation science experimental designs have been adapted to evaluate effectiveness
- Conducting an experimental evaluation requires substantial monetary and professional resources, typically requires grant support, and often needs institutional review and approval including ethical review
- These studies may be more appropriate to evaluate established programs
Feasibility studies, quality improvement designs, quasi-experimental designs, and observational studies
- Includes improvement research, plan-do-study-act cycles, cross-over, case-control, and time series research. These approaches may be better suited to capture effectiveness of an intervention as they are evaluating a hybrid of its implementation and its effect on patient outcomes
- These studies may be more appropriate to evaluate initial implementations but can also be used to continuously improve established programs
- The non-experimental nature of the design increases risk of bias and decreases generalizability
- Both qualitative and quantitative methods can be used to evaluate the system
- These approaches are typically lower cost but may still require some monetary and personnel resources to administer, as well as possibly requiring institutional and ethical approval
Figure 12.1: Example implementation and clinical outcomes
C. What should you do with the results of the intervention evaluation?
- Inform PROM system improvement
- Communicate learnings throughout the organization
- Share learnings with the field. Consider disseminating results of the intervention through conferences, publications, or other communication channels to increase awareness of both the PROM program and efforts to evaluate it
- Consider whether the outcomes meet the goals of the PROM system – and adjust the PROM strategy if they do not
Relevant Primary Resources
The information presented here is an overview of evaluating the intervention. For more detailed information please see the following sources: