Automating Data Science

Cognitive technologies, pp. 269–282

Abstract

Abstract It has been observed that, in data science, a great part of the effort usually goes into various preparatory steps that precede model-building. The aim of this chapter is to focus on some of these steps. A comprehensive description of a given task to be resolved is usually supplied by the domain expert. Techniques exist that can process natural language description to obtain task descriptors (e.g., keywords), determine the task type, the domain, and the goals. This in turn can be used to search for the required domain-specific knowledge appropriate for the given task. In some situations, the data required may not be available and a plan needs to be elaborated regarding how to get it. Although not much research has been done in this area so far, we expect that progress will be made in the future. In contrast to this, the area of preprocessing and transformation has been explored by various researchers. Methods exist for selection of instances and/or elimination of outliers, discretization and other kinds of transformations. This area is sometimes referred to asdata wrangling. These transformations can be learned by exploiting existing machine learning techniques (e.g., learning by demonstration). The final part of this chapter discusses decisions regarding the appropriate level of detail (granularity) to be used in a given task. Although it is foreseeable that further progress could be made in this area, more work is needed to determine how to do this effectively.

Authors 4

1. Pavel Brazdil corresponding

   Universidade do Porto

   Affiliation as printed

   Laboratory of Artificial Intelligence and Decision Support, University of Porto, Porto, Portugal

2. Jan N. van Rijn Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden University, Leiden, The Netherlands

3. Carlos Soares

   Universidade do Porto

   Affiliation as printed

   Porto Business School, Porto, Portugal

4. Joaquin Vanschoren

   Eindhoven University of Technology

   Affiliation as printed

   Department of Mathematics and Computer Science, Technische Universiteit Eindhoven, Eindhoven, The Netherlands