SEARCHRCTpediatrics, family medicine

RCT·pediatrics, family medicine, public health, biomedical research, research methods·PMC10375039

A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care

Implementation Research and Practice · 9 authors, 6 centres

AI SUMMARY

FIDELITY 100%

This summary was generated by AI from a single paper. It has not been reviewed by a clinician and is not clinical advice. Verify against the source before acting on it.

AGENT 01 · STRUCTURED SUMMARY

TRACE TO SOURCE

“

TL;DR

This study tested two machine learning models (TF-IDF and BERT) to automatically rate the fidelity and parent engagement of the Family Check-Up 4 Health program from session transcripts. The models' ratings were significantly correlated with expert observer ratings and predicted parent engagement outcomes, showing promise for scalable implementation monitoring.

Full summary

1,161 CHARS

The study evaluated two natural language processing (NLP) methods, TF-IDF and BERT, to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program. Using 197 session transcripts from 113 families in a type 2 hybrid trial, models were trained and evaluated. For concurrent validity, both NLP models' ratings were significantly associated with expert observer ratings on the COACH fidelity measure, showing improvement over baseline with average mean squared error improvements of 24%. For predictive validity, the NLP ratings significantly predicted multiple parent engagement indicators, including in-session engagement, session attendance, home practice competence, and motivation to change, with an approximate average improvement of 18% over baseline. Limitations include the relatively small sample size, the use of human-created transcripts (not direct audio), and a sample primarily composed of Latino families, which may limit generalizability. The results support the potential of NLP for low-burden, timely implementation monitoring, though further research on feasibility, acceptability, and effectiveness is needed before widespread use.

STUDY TYPE

RCT

SPECIALTY

pediatrics

SUMMARISED BY

AI pipeline

FIDELITY CHECK

100% · A

ORIGINAL PAPER

PMC10375039

A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care

Implementation Research and Practice

READ THE FULL PAPER

ABSTRACT

Background Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner. Methods In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes. Results Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83–1.02 to 0.62–0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81–27.3 to 0.62–19.50, resulting in an approximate average improvement of 18%. Conclusions These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented. Trial registration NCT03013309 ClinicalTrials.gov.

CITATION EXPORT

Cady Berkel, Dillon C. Knox, Nikolaos Flemotomos et al.. A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care. Implementation Research and Practice. (2023). https://doi.org/10.1177/26334895231187906

AUTHORS

9 · 6 CENTRES

Cady Berkel, Dillon C. Knox, Nikolaos Flemotomos, Victor R. Martinez, David C. Atkins, Shrikanth S. Narayanan, Lizeth Alonso Rodriguez, Carlos G. Gallo, Justin D. Smith

1College of Health Solutions, 7864Arizona State University, Phoenix, AZ, USA
2Ming Hsieh Department of Electrical Engineering, USC Viterbi School of Engineering, REACH Institute, 7864Arizona State University, Tempe, AZ, USA
3Signal Analysis and Interpretation Laboratory, 5116University of Southern California, Los Angeles, CA, USA
4Department of Psychiatry and Behavioral Sciences, 7284University of Washington, Seattle, WA, USA
5Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago, IL, USA
6Department of Population Health Sciences, Spencer Fox Eccles School of Medicine, University of Utah, Salt Lake City, UT, USA