Social Impact Program Evaluation

Abstract

Inadequate public and grant funding often inhibits social service agencies from covering all of their program participant and agency needs. Compounding this problem, funders now have greater expectations that agencies demonstrate a return on investment, document their success, and demonstrate benefits to the community. This has led agencies to seek or develop more effective tools to evaluate program impact. In this study, I drew from the literature a proposed approach of social impact measurement that was applied as a four-step blueprint for social impact program evaluation, also referred to as “blueprint” later in the study, in order to demonstrate greater effectiveness of program goals and outcomes, justify current and future funding needs, and ultimately validate social impact. My first and second studies explored 1) the usefulness of a new approach for identifying and quantifying the impact of social service programs and 2) the pros and cons of involving stakeholders throughout the social impact measurement process, and whether stakeholder involvement substantially changed the process and results. My third study further explored the social impact measurement process with additional consideration emphasizing funders and policy advocates as stakeholders and investigated their impressions and influence on the measurement process. This social impact measurement approach consists of four steps: applying the Theory of Change to assist with appropriately identifying the program’s long-term goals; initiating a Logic Model to improve the connection between inputs, activities, outputs, outcomes and impact with the long-term goals; designing an Impact Measurement analysis to connect the specific outcomes with the intrinsic value each outcome creates for the well-being of the program participants; and the involvement of stakeholders throughout the impact-oriented program evaluation process to utilize published and program-specific evaluation studies to calculate the Social Return on Investment. To learn more about the potential and pitfalls of the proposed four-step model of social impact measurement, I partially applied this model to one program at my organization, the Philadelphia Corporation for Aging, for my first study. Following the application of the proposed blueprint at my organization, challenges emerged related to the lack of stakeholder involvement, which led to the inclusion of stakeholders in the blueprint application for my second study. Practitioner literature suggests that involving stakeholders in every step of impact-oriented program evaluation will strengthen the quality of the Impact Measurement. Utilizing action research, I sought to understand this contention by comparing levels of stakeholder involvement across programs and different organizations to explore the ways in which stakeholder involvement might help or hinder Impact Measurement. The results suggest that exclusively relying on homogeneous workgroups to provide deeper insights regarding program impact to the community, or solely relying on external stakeholders that lack the micro-level nuances of the program processes and procedures, does not lead to well-rounded social impact measurement. However, combining internal and external stakeholders in a workgroup seems to provide the best and most consistent path to effective social impact measurement by diversifying perspectives and leading to the development and application of metrics that are both reflective of community impact and practical to track. My third study focused on gaining additional insights regarding the usefulness of engaging funders and policy advocates in the design and implementation of the social impact measurement model and the usefulness of a new approach for identifying and quantifying the impact of social service programs. For this stage of my study, semi-structured interviews were completed with leadership from State Units on Aging throughout the nation with differing demographic compositions. Additional interviewees included leaders from the Aging Network with a broad perspective of the Area Agency on Aging (AAA) Network including an executive from a key trade association, a former cabinet level staff member within a former Governor’s administration, and key leaders from managed long-term service and support system payors in different states. To better understand the influence of funders as stakeholders, the Resource Dependence Theory, highlighting the delicate balance of power between funders and recipient organizations (Hung, C., and Berrett, J., 2021), and strategies used in the acquisition of resources, was further explored (Klein, L. L. and Diniz Pereira, B. A., 2016). While several challenges were presented regarding why impact measurement strategies have not progressed further, progress was noted when leadership identified successful strategies utilized by other industries or government agencies, or when they sought individuals with specific skillsets to strengthen their internal processes. Many of these strategies were possible due to the support of leadership and the addition of requisite state resources. However, the additional resources would often avoid duplicating resources that funders were initiating, and strategies would be considered if they did not significantly challenge funders or jeopardize funding. My study contributions include combining existing theoretical concepts into a proposed four-step blueprint for social impact program evaluation that focused on impact and outcomes for study one, which was further refined with the addition of stakeholders in study two, and study three continued to explore the role that stakeholders, including influential ones like funders, played in developing and applying social impact measurement tools. Additionally, study three further contributes to Resource Dependence Theory research by further assessing the balance of power between organizations and the leveraging of stakeholder partnerships to acquire resources from the external environment to minimize the dependence on funders. Strengthening the social impact measurement alignment between funders and funding recipients has the capacity to improve the consistency in the Impact Measurement processes. Utilizing the strategies from this study provides a pathway to align policies and procedures between funders and funding recipients; encourage the transition from measuring outputs to outcomes; lead to collaboration that increases organizational resources from external sources, which aids the balance of power between funding and funding recipient and minimizes dependence; and provides a potential for a replicable tool that can be used nation-wide and across different types of social service agencies.