OP0039 How calculating consensus change scores can go wrong: lessons from multi-reader imaging assessments in axial spondyloarthritis

Annals of the Rheumatic Diseases, vol. 84, pp. 35–36

Abstract

Background: In research, imaging findings in various diseases are usually assessed by multiple readers. For analytical convenience, individual reader scores are typically combined into one aggregate score to determine the presence of lesions. In longitudinal studies with multiple reader assessments over time, presence of lesions is then modified into change of lesions. Different strategies to arrive at aggregate change scores are at hand, but these may yield different results. Objectives: To establish a methodological framework for aggregating change scores derived from imaging assessments, using the example of syndesmophyte progression in patients with axial spondyloarthritis (axSpA). Methods: Data from the Sensitive Imaging in Ankylosing Spondylitis (SIAS) cohort were used. Patients with axSpA with established structural spinal damage were followed over time with the aim of evaluating damage progression. Syndesmophytes were assessed on vertebral corners (hereafter ‘site level') on conventional radiography (CR) and low-dose computed tomography (ldCT) of the spine at baseline and 2 years. CR were scored by three trained central readers, and the aggregate status consensus score was defined as ≥2 out of 3 reader agreement. For ldCTs, data from two readers were available and the status consensus score was only considered as ‘present' when both readers agreed on the presence of a syndesmophyte (otherwise ‘not present'). Images were scored per modality, and all readers were blinded for time order and patient information. We assessed new syndesmophytes (change score) after 2 years using data from reading assessments of CR and ldCT modalities, applying 2 different strategies. Strategy 1 determines first the change score per reader, and then the consensus change score at the site level, while in strategy 2 the consensus change score is derived from consensus status scores (agreement on the presence of a syndesmophyte at each timepoint) (detailed description in Box 1 ). The means and standard deviations (SD) of the different aggregate change scores (total number of new syndesmophytes at the site level) were compared. For analyses at the patient level, the total number of new syndesmophytes were calculated as the average of all readers. Results: Complete data from 51 patients were included. Figure 1A provides three clarifying scoring examples of assessment of a new syndesmophyte ("yes/no") per site level. While both strategies do not yield different consensus change scores in examples 1 and 2, they do in example 3. Here, reader B identified a syndesmophyte at baseline and 2 years on CR (no change over time), while reader C identified a syndesmophyte solely at 2 years. When consensus change score is derived from change scores of individual readers (strategy 1), because only reader C detected a new syndesmophyte, there is no new syndesmophyte. In contrast, when consensus status scores are determined first (strategy 2), indicating no syndesmophyte at baseline but one at 2 years, the consensus change score suggests the development of a new syndesmophyte. Notably, when using strategy 2, we describe the change score as reflecting agreement on the presence of a new syndesmophyte at the site level after 2 years. However, only one out of three readers detected an actual change during follow-up. Thus, the agreement in strategy 2 reflects a derived consensus based on status scores rather than direct agreement on progression. Figure 1B illustrates how extrapolating separate site level scores by both strategies into patient-sum scores may result in importantly different mean change scores. Although strategy 2 is more sensitive than strategy 1, identifying 1.3 times more new syndesmophytes on CR (mean [SD]: 0.86 [1.00] vs 0.69 [0.95]) and 3.4 times more on ldCT (6.98 [7.31] vs 2.04 [3.88]), this comes at the cost of higher variability (larger SD). Only strategy 1 represents a "true consensus" among readers, however this mean number of new syndesmophytes requires agreement at the site level, which is strict and compromises sensitivity to change at the patient level. For this purpose, the total number of new syndesmophytes can also be calculated as the average of the total number of new syndesmophytes among all readers, reflecting overall progression at the patient level (mean [SD] total number of new syndesmophytes: 0.92 [0.96] on CR; 6.36 [6.62] on ldCT). Conclusion: Consensus change scores must be based on individual readers' change scores rather than on change based on consensus status scores. Strategy 1 approaches true change best, while strategy 2 artificially inflates change but ignores part of the noise. For descriptive purposes at the patient level, the average total number of new syndesmophytes among readers (‘mean sum score'), which has highest sensitivity to change, but does not require absolute site level-score-agreement, could best be reported. While this specific example focuses on syndesmophyte progression in axSpA, this framework is broadly applicable to other dichotomous imaging findings across rheumatic and non-rheumatic diseases. Advanced statistical analysis beyond descriptive purposes alone should preferably be performed on individual reader scores, rather than on manipulated consensus scores, using multilevel models that will properly account for correlated data and variability in reader assessment. REFERENCES: NIL . Acknowledgements: The authors would like to acknowledge Fundação para a Ciência e a Tecnologia (FCT) for supporting Ana Bento da Silva's PhD (grant 2024.01498.BD). Disclosure of Interests: Ana Bento da Silva: None declared, Sofia Ramiro has received speaker fees from Eli Lilly, Novartis and UCB, consultancy fees from AbbVie, Eli Lilly, Galapagos/Alfasigma, Janssen, MSD, Pfizer, UCB and Sanofi, and grants from AbbVie, Galapagos/Alfasigma, MSD, Novartis, Pfizer and UCB, Floris van Gaalen has received consultancy fees from Novartis, MSD, AbbVie Bristol Myers Squibb and Eli Lilly, and research grants from Stichting vrienden van Sole Mio, Stichting ASAS, Novartis and UCB, Robert Landewé has received speaker fees from Abbie, UCB, Novartis, Galapagos and Jansen pharma, and consultancy fees from AbbVie, UCB, Galapagos and Jansen, Miranda van Lunteren: None declared, Liese de Bruin: None declared, Gizem Ayan: None declared, Xenofon Baraliakos has received fees from Abbvie, Alphasigma, Amgen, BMS, Cesas, Celltrion, Galapagos, Janssen, Lilly, Moonlake, Novartis, Pfizer, Roche, Sandoz, Springer, Stada, Takeda, UCB and Zuellig, speaker fees from Abbvie, Alphasigma, Amgen, BMS, Cesas, Celltrion, Galapagos, Janssen, Lilly, Moonlake, Novartis, Pfizer, Roche, Sandoz, Springer, Stada, Takeda, UCB and Zuellig, consultancy fees from Abbvie, Alphasigma, Amgen, BMS, Cesas, Celltrion, Galapagos, Janssen, Lilly, Moonlake, Novartis, Pfizer, Roche, Sandoz, Springer, Stada, Takeda, UCB and Zuellig, and grants from Abbvie, Celltrion, Janssen, Moonlake and Novartis, Monique Reijnierse: None declared, Désirée van der Heijde has received consultancy fees from AbbVie, Alfasigma, ArgenX, BMS, Elly-Lilly, Grey-Wolf Therapeutics, Janssen, Novartis, Pfizer, Takeda and UCB Pharma, Manouk de Hooge has received consultancy fees from UCB Pharma. © The Authors 2025. This abstract is an open access article published in Annals of Rheumatic Diseases under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Neither EULAR nor the publisher make any representation as to the accuracy of the content. The authors are solely responsible for the content in their abstract including accuracy of the facts, statements, results, conclusion, citing resources etc.

Authors 9

1. Sofía Ramiro Aachen

   Leiden University Medical Center

   Affiliation as printed

   Leiden University Medical Center, Rheumatology, Leiden, Netherlands

2. Floris A van Gaalen Aachen Department of Rheumatology, Leiden

   Leiden University Medical Center

   Affiliation as printed

   Leiden University Medical Center, Department of Rheumatology, Leiden, Netherlands

3. Robert Landewé

   Zuyderland Medisch Centrum · Amsterdam University Medical Centers

   Affiliation as printed

   Amsterdam University Medical Center, Rheumatology & Clinical immunology, Amsterdam, Netherlands

   Zuyderland Medical Center, Rheumatology, Heerlen, Netherlands

4. M. van Lunteren Aachen

   Leiden University Medical Center

   Affiliation as printed

   Leiden University Medical Center, Rheumatology, Leiden, Netherlands

5. Gizem Ayan Aachen

   Leiden University Medical Center · Ankara Eğitim ve Araştırma Hastanesi

   Affiliation as printed

   Ankara Research and Training Hospital, Ankara, Türkiye

   Leiden University Medical Center, Rheumatology, Leiden, Netherlands

6. Xenofon Baraliakos

   Rheumazentrum Ruhrgebiet · Ruhr University Bochum

   Affiliation as printed

   Rheumazentrum Ruhrgebiet Herne, Ruhr-University Bochum, Herne, Germany

7. M. Reijnierse Aachen

   Leiden University Medical Center

   Affiliation as printed

   Leiden University Medical Center, Rheumatology, Leiden, Netherlands

8. Désirée M F M van der Heijde Aachen

   Leiden University Medical Center

   Affiliation as printed

   Leiden University Medical Center, Rheumatology, Leiden, Netherlands

9. Manouk de Hooge

   Ghent University Hospital

   Affiliation as printed

   Ghent University Hospital, Rheumatology, Ghent, Belgium