SortedAP: Rethinking evaluation metrics for instance segmentation

IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 3925–3931

Abstract

Designing metrics for evaluating instance segmentation revolves around comprehensively considering object detection and segmentation accuracy. However, other important properties, such as sensitivity, continuity, and equality, are overlooked in the current study. In this paper, we reveal that most existing metrics have a limited resolution of segmentation quality. They are only conditionally sensitive to the change of masks or false predictions. For certain metrics, the score can change drastically in a narrow range which could provide a misleading indication of the quality gap between results. Therefore, we propose a new metric called sortedAP, which strictly decreases with both object- and pixel-level imperfections and has an uninterrupted penalization scale over the entire domain. We provide the evaluation toolkit and experiment code at https://www.github.com/looooongChen/sortedAP.

Authors 4

1. Long Chen Aachen Institute of Imaging & Computer Vision

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Institute of Imaging & Computer Vision,Germany

2. Yuli Wu Aachen Institute of Imaging & Computer Vision

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Institute of Imaging & Computer Vision,Germany

3. Johannes Stegmaier Aachen Institute of Imaging & Computer Vision

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Institute of Imaging & Computer Vision,Germany

4. Dorit Merhof

   University of Regensburg

   Affiliation as printed

   University of Regensburg,Faculty of Informatics and Data Science,Germany

   Faculty of Informatics and Data Science, University of Regensburg, Germany