Introduction: Community engagement in research is widely accepted as best practice, despite gaps in existing frameworks to evaluate its process, context, and impact on research. The Screening in High Schools to Identify, Evaluate, and Lower Depression (SHIELD) study evaluated the use of a school-based major depressive disorder screening tool in the identification of symptoms and treatment initiation among adolescents, and was developed, implemented, and disseminated in partnership with a Stakeholder Advisory Board (SAB). We summarize outcomes of the evaluation strategy applied through our partnership with the SAB and explore gaps in the available engagement evaluation tools for mixed stakeholder populations including youth.
Methods: SHIELD study SAB members (n = 13; adolescents, parents, mental health and primary care providers, and professionals from education and mental health organizations) advised on study design, implementation, and dissemination over a three-year period. Both SAB members and study team members (i.e., clinician researchers, project managers) were invited to quantitatively and qualitatively evaluate stakeholder engagement after each project year. At the conclusion of the study, SAB members and study team members were asked to evaluate the application of engagement principles in overall stakeholder engagement across the study period, using portions of the Research Engagement Survey Tool (REST).
Results: SAB members and study team members responded similarly when evaluating engagement process (i.e., valued on team, voice represented); means ranged from 3.9 to 4.8 out of 5 points across all three project years. Reported engagement within study-specific engagement activities (i.e., meetings, study newsletter) varied from year to year, with some discrepancy between SAB member and study team evaluations. Using REST, SAB members reported the alignment of their experience with key engagement principles the same or higher than study team members. Qualitative feedback at the conclusion of the study generally matched quantitative measures; adolescent SAB members, however, reported disengagement from stakeholder activities that was not accurately or effectively captured in evaluation strategies employed across the study period.
Conclusions: Challenges exist in effectively engaging stakeholders and evaluating their engagement, particularly among heterogenous groups that include youth. Evaluation gaps should be addressed through the development of validated instruments that quantify the process, context, and impact of stakeholder engagement on study outcomes. Consideration should be given to collecting parallel feedback from stakeholders and study team members to fully understand the application and execution of engagement strategy.
Keywords: Adolescents and young adults; Community-engaged research; Engagement evaluation; Program evaluation; Stakeholders.
We conducted a study (Screening in High Schools to Identify, Evaluate, and Lower Depression) to understand if an adolescent major depressive disorder screening tool delivered in the school setting aided in the identification of symptoms and treatment. We planned and conducted this study with the guidance of a stakeholders, including adolescents.At the end of each study year, we sent an evaluation survey to stakeholders to understand their experience, such as how appropriately the study included stakeholders and their perspectives. We also surveyed the team leading the study to understand their perspectives about stakeholder involvement. In general, both stakeholders and the study team reported feeling positive about stakeholder involvement; However, some stakeholders felt less involved as the study moved forward, and for some activities stakeholders and study team did not agree on how much the stakeholders were involved in study activities. Additionally, adolescent stakeholders reported low involvement in the study when completing the final evaluation, which, unfortunately, was not captured in the evaluations conducted in earlier study years.By evaluating the experiences of stakeholders, along with gathering perspectives of the study team, we were able to understand how well we involved stakeholders. However, additional questions remain unanswered, such as how best to involve adolescents as stakeholders, and how involving stakeholders impacted the results of our study. Evaluation tools to best understand these impacts are needed across the field of community-engaged research to answer these questions for future studies.
© 2023. The Author(s).