Less is not more: We need rich datasets to explore

Future Generation Computer Systems, vol. 142, pp. 117–130

Abstract

Traditional datacenter analysis is based on high-level, coarse-grained metrics. This obscures our vision of datacenter behavior, as we do not observe the full picture nor subtleties that might make up these high-level, coarse metrics. There is room for operational improvement based on fine-grained temporal and spatial, low-level metric data. We leverage in this work one of the (rare) public datasets providing fine-grained information on datacenter operations, with over 60 billion measurements captured in 15-second intervals. We show evidence that fine-grained information reveals new operational aspects, that the different metrics cannot be derived from one another (and thus need to be captured), and that many low-level metrics, gathered frequently are key to understanding datacenter operations. We propose a holistic analysis for datacenter operations, providing statistical characterization of node and workload aspects. Our analysis reveals both generic and machine learning-specific aspects, summarized in over 30 observations, providing deep insight into this dataset and the originating cluster. We give actionable insights, surprising findings, and exemplify how our observations support performance-engineering tasks such as workload prediction and long-term datacenter design.

Authors 7

1. Laurens Versluis corresponding

   Vrije Universiteit Amsterdam

   Affiliation as printed

   VU Amsterdam, The Netherlands

2. Mehmet Berk Cetin

   Vrije Universiteit Amsterdam

   Affiliation as printed

   VU Amsterdam, The Netherlands

3. Caspar Greeven
   Affiliation as printed

   Surf Amsterdam, The Netherlands

4. Damian Podareanu
   Affiliation as printed

   Surf Amsterdam, The Netherlands

5. Valeriu Codreanu
   Affiliation as printed

   Surf Amsterdam, The Netherlands

6. Alexandru Uta Aachen

   Leiden University

   Affiliation as printed

   Leiden University, The Netherlands

7. Alexandru Iosup

   Vrije Universiteit Amsterdam

   Affiliation as printed

   VU Amsterdam, The Netherlands