Pitfalls in using ML to predict cognitive function performance

Scientific Reports, vol. 15, pp. 37747

Abstract

Machine learning analyses are widely used for predicting cognitive abilities, yet there are pitfalls that need to be considered during their implementation and interpretation of the results. Hence, the present study aimed at drawing attention to the risks of erroneous conclusions incurred by confounding variables illustrated by a case example predicting executive function (EF) performance by prosodic features. Healthy participants (n = 231) performed speech tasks and EF tests. From 264 prosodic features, we predicted EF performance using 66 variables, controlling for confounding effects of age, sex, and education. A reasonable prediction performance was apparently achieved for EF variables of the Trail Making Test. However, in-depth analyses revealed indications of confound leakage, leading to inflated prediction accuracies, due to a strong relationship between confounds and targets. These findings highlight the need to control confounding variables in ML pipelines and caution against potential pitfalls in ML predictions.

Authors 6

1. Gianna Kuhles corresponding

   Forschungszentrum Jülich · Heinrich Heine University Düsseldorf

   Affiliation as printed

   Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany. g.kuhles@fz-juelich.de

   Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany. g.kuhles@fz-juelich.de

   Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany

   Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany

2. Stefan Heim Aachen Department of Psychiatry, Psychotherapy and Psychosomatics Medical Faculty Institute for Midwifery Science

   RWTH Aachen University · Forschungszentrum Jülich

   Affiliation as printed

   Department of Psychiatry, Psychotherapy and Psychosomatics, Medical Faculty, RWTH Aachen University, Aachen, Germany

   Institute for Midwifery Science, Medical Faculty, RWTH Aachen University, Aachen, Germany

   Institute of Neuroscience and Medicine, Structural and Functional Organization of the Brain (INM-1), Research Centre Jülich, Jülich, Germany

3. Simon Bodo Eickhoff

   Forschungszentrum Jülich · Heinrich Heine University Düsseldorf

   Affiliation as printed

   Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany

   Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany

4. Kaustubh R. Patil

   Forschungszentrum Jülich · Heinrich Heine University Düsseldorf

   Affiliation as printed

   Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany

   Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany

5. Julia A. Camilleri

   Forschungszentrum Jülich · Heinrich Heine University Düsseldorf

   Affiliation as printed

   Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany

   Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany

6. Susanne Weis

   Forschungszentrum Jülich · Heinrich Heine University Düsseldorf

   Affiliation as printed

   Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany

   Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany