By: Harrison Angoff
At ECS, many clients run programs focused on improving participants’ mental health and well-being. This includes ongoing programming conducted across communities and individual programs implemented in various settings, including schools, community centers, and youth-serving agencies. Evaluating these mental health programs and efforts requires careful consideration to multiple factors, three of which I will discuss in more depth below: 1) program aim, including short vs. long-term outcomes; 2) assessment of group differences; and 3) culturally appropriate and tailored data collection techniques and measurement selection.
Program Aim – Short vs. Long-term Outcomes
At ECS, we assist our clients with the development of a logic model to organize and articulate one’s program and organizational aims more effectively. In short, a logic model is a tool that provides a visual representation of how one’s program, effort, or initiative is supposed to work. This begins with a specification of the problem/goal that is meant to be addressed, risk factors for this problem, and local conditions that influence the persistence of the problem. When considering the problem/goal to be addressed for mental health programming, identifying the short and long-term outcomes is increasingly important.
Short-term outcomes are the immediate targets of change (e.g., help-seeking behavior; knowledge; confidence). To identify your intended short-term outcomes, it can be helpful to ask yourself, “What does it mean for my program to work” or, “What do I hope participants gain from my program?” Long-term outcomes are the mental health problems or indicators to be prevented or reduced (e.g., suicide attempts, depressive symptoms, school absenteeism). It can be helpful to think of short-term outcomes as the mechanisms in which long-term outcomes work through. While identifying your long-term outcomes assists with identifying your aims and overall program strategy, measuring short-term outcomes related to mental health indicators is particularly important as it allows you to understand specific program components that effectively reach your target audience. This is especially important in mental health programming because symptom reduction often takes considerable time. As such, by only measuring symptoms or one’s mental health status before and after your program or initiative (typically a long-term outcome), your evaluation may not fully capture the impact it had. Instead, by specifying short-term outcomes, you can more effectively measure the behaviors that can lead to mental health improvement.
Assessment of Group Differences
After determining the specific outcomes to be addressed, it is also important to consider how you analyze the data for your evaluation. For outcomes in which mental health variables are utilized, assessing group differences is a culturally conscious and ethical practice. Because individuals from minority groups (e.g., LGBTQ+ individuals, racial/ethnic minorities, individuals with disabilities) experience higher levels of mental health concerns and tend to have less access to culturally competent resources to assist with alleviating their concerns, assessing whether your program has a similar impact across all participants is necessary. As a function of continuous quality improvement (CQI), findings from the assessment of group differences could then be integrated back into the planning and implementation of your ongoing initiatives. This can occur through further refinement of the specific program implemented or through more targeted strategies.
Measurement Selection & Data Collection
When identifying measures and data collection techniques related to mental health program evaluation, it is helpful to consider both etic (culture-universal) and emic (culture-specific) distinctions in how mental health is communicated and experienced (Hwang et al., 2008). Indeed, approaching your participants in a culturally conscious manner during data collection is important for both ethical considerations and scientific implications regarding the validity and representativeness of your data. This includes factors such as linguistic barriers and recognizing how the past experiences of your sample may impact their comfort in discussing and answering questions surrounding mental health.
Regarding linguistic difficulties, Rodríguez-Lainz et al. (2016) recommend three practices to reduce language barriers during data collection: 1) translation of data collection instruments into the main languages spoken by those completing your survey; 2) ensuring cultural validity of data collection instruments; and 3) use trained interviewers and/or third-party interpreters who are fluent in the languages spoken by individuals being sampled. Additionally, in responding to participants’ comfort towards mental health, it is important to consider the overall climate surrounding mental health in the community you are working in and your positionality as an “insider” vs. “outsider”. A couple questions you can ask yourself as the programmer or evaluator are, “How may participants view me as I enter their space?” or “How has my intended sample been treated historically by researchers or those who are seen as experts?” These questions are important in relation to mental health programming and evaluation as historical experiences within the mental health system (e.g., racism and discrimination, overdiagnosis, stigma) have been found to increase mistrust towards mental health services, practitioners, and researchers – limiting engagement in mental health data collection and research among marginalized groups (Suite et al., 2007; Woodall et al., 2010). Thus, understanding your positionality and the historical experiences of your participants can assist with your decision-making as you seek to maximize data collected and improve participants’ experiences and comfort throughout your evaluation.
These are just a few considerations regarding data collection and evaluation for mental health programming. Feel free to contact Harrison for additional technical assistance and consultation regarding your mental health programming, implementation, or evaluation.
Harrison Angoff is an Evaluator at Epiphany Community Services and a fifth-year doctoral candidate in Clinical Psychology at Bowling Green State University. He specializes in mixed-methods program evaluation related to school-based mental health, mental health service delivery, and substance use.
References:
Hwang, W-C., Myers, H. F., Abe-Kim, J., & Ting, J. Y. (2008). A conceptual paradigm for understanding culture’s impact on mental health: The cultural influences on mental health (CIMH) model. Clinical Psychology Review, 28, 211-227. https://doi.org/10.1016/j.cpr.2007.05.001
Rodríguez-Lainz, A., McDonald, M., Penman-Aguilar, A., & Barrett, D. H. (2016). Getting data right – and righteous tot improve Hispanic or Latino health. Journal of Healthcare Science and the Humanities, 6(3), 60-83.
Suite, D. H., La Bril, R., Primm, A., & Harrison-Ross, P. (2007). Beyond misdiagnosis, misunderstanding and mistrust: Relevance of the historical perspective in the medical and mental health treatment of people of color. Journal of the National Medical Association, 99(8), 879-885.
Woodall, A., Morgan, C., Sloan, C., & Howard, L. (2010). Barriers to participation in mental health research: Are there specific gender, ethnicity, and age related barriers? BMC Psychiatry, 10(103). https://doi.org/10.1186/1471-244X-10-103