Cognitive inference as the main predictor of AI reliability in automated behavioral coding of parent–child interactions

Amorocho, José - Balsa, Ana - Giraldo-Huertas, Juan José - Bloomfield, Juanita - Patrone, Paula - Cid, Alejandro

Resumen:

Observational coding of parent–child interactions is a gold standard in developmental science but remains unscalable. Multimodal generative AI could help, yet its reliability and failure modes are not well characterized. We benchmarked a multi-agent pipeline (GABRIEL) against a multi-rater expert consensus when scoring 22 PICCOLO items on 156 ten-minute free-play interactions from Uruguay. Agreement was summarized with Percent Agreement (PA) and Cohen’s κ, and disagreement with a unit-free normalized mean squared error (nMSE = MSE/Var(Yitem)). A priori item classes indexed cognitive inference (Low/Medium/High). Final calibration yielded modest agreement (PA = 50.7%, κ = .216). Disagreement was chiefly structured by inference (Kruskal–Wallis H = 308.70, p<.001), a pattern that persisted in late iterations. The model also overused the middle category (1) and underused “2.” No systematic differences in nMSE emerged by sex, age quartile, or maternal education. We conclude that generative AI is promising for scalable detection of concrete, low-inference behaviors, whereas high-inference judgments still require expert adjudication. A human-in-the-loop, co-intelligence workflow aligns current strengths with ethical oversight and supports equitable deployment at scale.

Detalles Bibliográficos
2026
Generative AI
Parent-Child Interaction
Observational Coding
Inter-Rater Reliability
PICCOLO
Multimodal AI
Inglés
Universidad de Montevideo
REDUM
https://hdl.handle.net/20.500.12806/2795
Acceso abierto
Attribution-NonCommercial-NoDerivatives 4.0 Internacional