Intra- and interobserver variability in response evaluation metrics used for localized soft tissue sarcoma

ESMO rare cancers., vol. 7, pp. 100517

Abstract

Background Reliable imaging-based response assessment in localized soft tissue sarcoma (STS) is essential for guiding treatment and interpreting clinical trial outcomes. We quantified intra- and interobserver variability and evaluated its impact on classification using Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 and magnetic resonance (MR)-adapted Choi criteria across different measurement tools. Materials and methods Nineteen pre- and postradiotherapy magnetic resonance imaging pairs were evaluated by 10 radiologists using visual estimation, digital rulers, and segmentation; signal intensity was normalized using circular or freehand muscle regions of interest. Intraobserver error was assessed through mirrored lesions. Interobserver error analysis was split into systematic and random components. Clinical impact of measurement variability was evaluated by analyzing response category threshold crossings and agreement in RECIST 1.1 and MR-adapted Choi classifications under realistic clinical scenarios, including when baseline and follow-up scans were assessed by the same or different readers. Results Segmentation-based diameters showed the lowest variability and highest agreement. RECIST 1.1 classifications were consistent across readers when using segmentation (Fleiss' k = 0.92-1.00), but ruler-based measurements showed lower agreement (Fleiss' k = 0.40-0.46). MR-adapted Choi criteria were more variable, particularly for signal intensity measurements, which led to false progression/response calls in up to 16% of identical scan comparisons. Agreement when comparing real-world baseline follow-up scenarios was also lower (Fleiss' k = 0.31-0.53). Conclusions Substantial intra- and interobserver variability exists in STS response assessment, especially with intensity-based criteria. Segmentation-derived diameters improve reproducibility and agreement. Refinement and automation of intensity-based methods are needed to ensure reliable response evaluation in clinical trials.

Authors 8

1. I. van der Loo

   The Netherlands Cancer Institute · Maastricht University · Deventer Ziekenhuis

   Affiliation as printed

   Department of Radiology, The Netherlands Cancer Institute–Antoni van Leeuwenhoek Hospital, Amsterdam, the Netherlands

   GROW–Research Institute for Oncology & Reproduction, Maastricht University, Maastricht, the Netherlands

   Research Office, Deventer Ziekenhuis, Deventer, the Netherlands

2. L.G.P.H. Vroomen

   The Netherlands Cancer Institute

   Affiliation as printed

   Department of Radiology, The Netherlands Cancer Institute–Antoni van Leeuwenhoek Hospital, Amsterdam, the Netherlands

3. N. Gennaro

   Northwestern University · The Netherlands Cancer Institute

   Affiliation as printed

   Department of Radiology, Northwestern University Feinberg School of Medicine, Chicago, USA

   Department of Radiology, The Netherlands Cancer Institute–Antoni van Leeuwenhoek Hospital, Amsterdam, the Netherlands

4. Olga Hamming‐Vrieze

   The Netherlands Cancer Institute

   Affiliation as printed

   Department of Radiation Oncology, Netherlands Cancer Institute–Antoni van Leeuwenhoek Hospital, Amsterdam, the Netherlands

5. R.G.H. Beets-Tan

   University of Southern Denmark · The Netherlands Cancer Institute · Maastricht University

   Affiliation as printed

   Faculty of Health Sciences, University of Southern Denmark, Odense M, Denmark

   GROW–Research Institute for Oncology & Reproduction, Maastricht University, Maastricht, the Netherlands

   The Netherlands Cancer Institute–Antoni van Leeuwenhoek, Amsterdam, the Netherlands

6. Rick L. Haas Aachen Department of Radiation Oncology

   Leiden University Medical Center · The Netherlands Cancer Institute

   Affiliation as printed

   Department of Radiation Oncology, Leiden University Medical Center, Leiden, the Netherlands

   Department of Radiation Oncology, Netherlands Cancer Institute–Antoni van Leeuwenhoek Hospital, Amsterdam, the Netherlands

7. A. L. T. Imholz

   Deventer Ziekenhuis

   Affiliation as printed

   Department of Oncology, Deventer Ziekenhuis, Deventer, the Netherlands

8. Stefano Trebeschi corresponding

   The Netherlands Cancer Institute · Maastricht University

   Affiliation as printed

   Department of Radiology, The Netherlands Cancer Institute–Antoni van Leeuwenhoek Hospital, Amsterdam, the Netherlands

   GROW–Research Institute for Oncology & Reproduction, Maastricht University, Maastricht, the Netherlands