Histopathology-based protein multiplex generation using deep learning
Nature Machine Intelligence, vol. 7, pp. 1292–1307
Abstract
Multiplexed protein imaging offers valuable insights into interactions between tumours and their surrounding tumour microenvironment, but its widespread use is limited by cost, time and tissue availability. Here we present HistoPlexer, a deep learning framework that generates spatially resolved protein multiplexes directly from standard haematoxylin and eosin (H&E) histopathology images. HistoPlexer jointly predicts multiple tumour and immune markers using a conditional generative adversarial architecture with custom loss functions designed to ensure pixel- and embedding-level similarity while mitigating slice-to-slice variations. A comprehensive evaluation of metastatic melanoma samples demonstrates that HistoPlexer-generated protein maps closely resemble real maps, as validated by expert assessment. They preserve crucial biological relationships by capturing spatial co-localization patterns among proteins. The spatial distribution of immune infiltration from HistoPlexer-generated protein multiplex enables stratification of tumours into immune subtypes. In an independent cohort, integration of HistoPlexer-derived features into predictive models enhances performance in survival prediction and immune subtype classification compared to models using H&E features alone. To assess broader applicability, we benchmarked HistoPlexer on publicly available pixel-aligned datasets from different cancer types. In all settings, HistoPlexer consistently outperformed baseline methods, demonstrating robustness across diverse tissue types and imaging conditions. By enabling whole-slide protein multiplex generation from routine H&E images, HistoPlexer offers a cost- and time-efficient approach to tumour microenvironment characterization with strong potential to advance precision oncology.
Authors 92
-
ETH Zurich · University of Basel · University Hospital Zurich · SIB Swiss Institute of Bioinformatics · Institute for Machine Learning
Affiliation as printed
Computational and Translational Pathology Group, Department of Biomedical Engineering, University of Basel, Basel, Switzerland
Department of Computer Science, ETH Zurich, Zurich, Switzerland
Swiss Institute of Bioinformatics, Zurich, Switzerland
Biomedical Informatics, University Hospital Zurich, Zurich, Switzerland
Department of Computer Science, Institute of Machine Learning, ETH Zurich, Zurich, Switzerland
-
ETH Zurich · SIB Swiss Institute of Bioinformatics · Computer Vision Laboratory
Affiliation as printed
AI Center, ETH Zurich, Zurich, Switzerland
Computer Vision Laboratory, Department of Information Technology and Electrical Engineering, ETH Zurich, Zurich, Switzerland
Department of Computer Science, ETH Zurich, Zurich, Switzerland
Swiss Institute of Bioinformatics, Zurich, Switzerland
-
ETH Zurich · University Hospital Zurich · SIB Swiss Institute of Bioinformatics · Institute for Machine Learning
Affiliation as printed
Department of Computer Science, ETH Zurich, Zurich, Switzerland
Swiss Institute of Bioinformatics, Zurich, Switzerland
Biomedical Informatics, University Hospital Zurich, Zurich, Switzerland
Department of Computer Science, Institute of Machine Learning, ETH Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Computer Science, ETH Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
-
University of Zurich · University Hospital Zurich
Affiliation as printed
Department of Pathology and Molecular Pathology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
-
University of Zurich · University Hospital Zurich
Affiliation as printed
Department of Pathology and Molecular Pathology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
-
ETH Zurich · Institute for Molecular Systems Biology
Affiliation as printed
Institute of Molecular Systems Biology, Department of Biology, ETH Zurich, Zurich, Switzerland
-
ETH Zurich · University Hospital Zurich · SIB Swiss Institute of Bioinformatics
Affiliation as printed
Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
AI Center, ETH Zurich, Zurich, Switzerland
Department of Computer Science, ETH Zurich, Zurich, Switzerland
Swiss Institute of Bioinformatics, Zurich, Switzerland
-
ETH Zurich · SIB Swiss Institute of Bioinformatics · NEXUS Personalized Health Technologies
Affiliation as printed
NEXUS Personalized Health Technologies, ETH Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
Affiliation as printed
Department of Health Sciences and Technology, ETH Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Health Sciences and Technology, ETH Zurich, Zurich, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
-
Affiliation as printed
F. Hoffmann-La Roche Ltd, Basel, Switzerland
-
Affiliation as printed
Roche Pharmaceutical Research and Early Development, Roche Innovation Center Zurich, Schlieren, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
-
Affiliation as printed
VP Medicine, University of Zurich, Zurich, Switzerland
-
ETH Zurich · SIB Swiss Institute of Bioinformatics
Affiliation as printed
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
Affiliation as printed
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland
-
Affiliation as printed
Clinical Trials Center, University Hospital Zurich, Zurich, Switzerland
-
ETH Zurich · SIB Swiss Institute of Bioinformatics · NEXUS Personalized Health Technologies
Affiliation as printed
NEXUS Personalized Health Technologies, ETH Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
ETH Zurich · Institute for Machine Learning
Affiliation as printed
Department of Computer Science, Institute of Machine Learning, ETH Zurich, Zurich, Switzerland
-
ETH Zurich · SIB Swiss Institute of Bioinformatics · NEXUS Personalized Health Technologies
Affiliation as printed
NEXUS Personalized Health Technologies, ETH Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
-
Affiliation as printed
Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
-
ETH Zurich · SIB Swiss Institute of Bioinformatics · NEXUS Personalized Health Technologies
Affiliation as printed
NEXUS Personalized Health Technologies, ETH Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Biomedicine, University Hospital Basel and University of Basel, Basel, Switzerland
-
Affiliation as printed
F. Hoffmann-La Roche Ltd, Basel, Switzerland
-
Affiliation as printed
Institute of Molecular Life Sciences, University of Zurich, Zurich, Switzerland
-
ETH Zurich · Institute for Machine Learning
Affiliation as printed
Department of Computer Science, Institute of Machine Learning, ETH Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland
-
Affiliation as printed
Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
-
Affiliation as printed
Roche Pharmaceutical Research and Early Development, Roche Innovation Center Basel, Basel, Switzerland
-
Affiliation as printed
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland
-
Affiliation as printed
Institute of Molecular Life Sciences, University of Zurich, Zurich, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Biomedicine, University Hospital Basel and University of Basel, Basel, Switzerland
-
Affiliation as printed
Roche Diagnostics GmbH, Penzberg, Germany
-
Affiliation as printed
Department of Health Sciences and Technology, ETH Zurich, Zurich, Switzerland
-
ETH Zurich · SIB Swiss Institute of Bioinformatics · NEXUS Personalized Health Technologies
Affiliation as printed
NEXUS Personalized Health Technologies, ETH Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
Affiliation as printed
Institute of Molecular Life Sciences, University of Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Chemistry and Applied Biosciences, ETH Zurich, Zurich, Switzerland
-
Affiliation as printed
Medical University Clinic, Cantonal Hospital Baselland, Liestal, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Biomedicine, University Hospital Basel and University of Basel, Basel, Switzerland
Gynecological Cancer Center, University Hospital Basel, Basel, Switzerland
-
Affiliation as printed
Institute of Molecular Life Sciences, University of Zurich, Zurich, Switzerland
-
Affiliation as printed
Roche Pharmaceutical Research and Early Development, Roche Innovation Center Zurich, Schlieren, Switzerland
-
ETH Zurich · Institute for Machine Learning
Affiliation as printed
Department of Computer Science, Institute of Machine Learning, ETH Zurich, Zurich, Switzerland
Max Planck ETH Center for Learning Systems, Stuttgart, Germany
-
Affiliation as printed
Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Biomedicine, University Hospital Basel and University of Basel, Basel, Switzerland
-
Affiliation as printed
F. Hoffmann-La Roche Ltd, Basel, Switzerland
-
Affiliation as printed
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland
-
ETH Zurich · University Hospital Zurich · SIB Swiss Institute of Bioinformatics · Institute for Machine Learning
Affiliation as printed
Biomedical Informatics, University Hospital Zurich, Zurich, Switzerland
Department of Computer Science, Institute of Machine Learning, ETH Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
-
ETH Zurich · University Hospital Zurich · SIB Swiss Institute of Bioinformatics · Institute for Machine Learning
Affiliation as printed
Biomedical Informatics, University Hospital Zurich, Zurich, Switzerland
Department of Computer Science, Institute of Machine Learning, ETH Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
University Hospital Zurich · SIB Swiss Institute of Bioinformatics
Affiliation as printed
Department of Pathology and Molecular Pathology, University Hospital Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
Viktor Hendrik Koelzer corresponding
University of Zurich · University of Basel · University Hospital of Basel · University Hospital Zurich
Affiliation as printed
Computational and Translational Pathology Group, Department of Biomedical Engineering, University of Basel, Basel, Switzerland
Department of Pathology and Molecular Pathology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
-
Affiliation as printed
University Hospital Basel, Basel, Switzerland
-
ETH Zurich · SIB Swiss Institute of Bioinformatics
Affiliation as printed
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
Affiliation as printed
Department of Information and Communication Technology, University Hospital Basel, Basel, Switzerland
-
Affiliation as printed
Brustzentrum, University Hospital Basel, Basel, Switzerland
-
RWTH Aachen University · Universitätsklinikum Aachen · ETH Zurich · University Hospital Cologne · Institute for Machine Learning
Affiliation as printed
Cancer Research Center Cologne-Essen, University Hospital Cologne, Cologne, Germany
Center for Integrated Oncology Aachen (CIO-A), Aachen, Germany
Department of Computer Science, Institute of Machine Learning, ETH Zurich, Zurich, Switzerland
Joint Research Center Computational Biomedicine, University Hospital RWTH Aachen, Aachen, Germany
-
Affiliation as printed
Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Biomedicine, University Hospital Basel and University of Basel, Basel, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Biomedicine, University Hospital Basel and University of Basel, Basel, Switzerland
-
Affiliation as printed
F. Hoffmann-La Roche Ltd, Basel, Switzerland
-
Affiliation as printed
Roche Diagnostics GmbH, Penzberg, Germany
-
Affiliation as printed
Department of Medical Oncology and Hematology, University Hospital Zurich, Zurich, Switzerland
-
ETH Zurich · Institute for Machine Learning
Affiliation as printed
Department of Computer Science, Institute of Machine Learning, ETH Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Health Sciences and Technology, ETH Zurich, Zurich, Switzerland
-
ETH Zurich · Institute for Molecular Systems Biology
Affiliation as printed
Institute of Molecular Systems Biology, Department of Biology, ETH Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Gynecology, University Hospital Zurich, Zurich, Switzerland
-
University of Zurich · University Hospital Zurich
Affiliation as printed
Department of Medical Oncology and Hematology, University Hospital Zurich, Zurich, Switzerland
Faculty of Medicine, University of Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Health Sciences and Technology, ETH Zurich, Zurich, Switzerland
-
Affiliation as printed
Department of Pathology and Molecular Pathology, University Hospital Zurich, Zurich, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
-
Affiliation as printed
Services and Support for Science IT, University of Zurich, Zurich, Switzerland
-
University of Basel · University Hospital of Basel · University of Bern
Affiliation as printed
Department of BioMedical Research, University of Bern, Bern, Switzerland
Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
-
Affiliation as printed
Department of Pathology and Molecular Pathology, University Hospital Zurich, Zurich, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Biomedicine, University Hospital Basel and University of Basel, Basel, Switzerland
-
Affiliation as printed
Department of Biology, ETH Zurich, Zurich, Switzerland
-
Affiliation as printed
Institute of Molecular Life Sciences, University of Zurich, Zurich, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Biomedicine, University Hospital Basel and University of Basel, Basel, Switzerland
Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
-
ETH Zurich · SIB Swiss Institute of Bioinformatics · NEXUS Personalized Health Technologies
Affiliation as printed
NEXUS Personalized Health Technologies, ETH Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
ETH Zurich · University Hospital Zurich · SIB Swiss Institute of Bioinformatics · Institute for Machine Learning
Affiliation as printed
Biomedical Informatics, University Hospital Zurich, Zurich, Switzerland
Department of Computer Science, Institute of Machine Learning, ETH Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Biomedicine, University Hospital Basel and University of Basel, Basel, Switzerland
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Biomedicine, University Hospital Basel and University of Basel, Basel, Switzerland
-
Affiliation as printed
Roche Pharmaceutical Research and Early Development, Roche Innovation Center Basel, Basel, Switzerland
-
ETH Zurich · SIB Swiss Institute of Bioinformatics · NEXUS Personalized Health Technologies
Affiliation as printed
NEXUS Personalized Health Technologies, ETH Zurich, Zurich, Switzerland
SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland
-
Gunnar Rätsch corresponding
ETH Zurich · University Hospital Zurich · SIB Swiss Institute of Bioinformatics
Affiliation as printed
AI Center, ETH Zurich, Zurich, Switzerland
Department of Computer Science, ETH Zurich, Zurich, Switzerland
Medical Informatics Unit, University Hospital Zurich, Zurich, Switzerland
Swiss Institute of Bioinformatics, Zurich, Switzerland
-
Affiliation as printed
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland
Cited by 24 stored of 24
No patents citing this paper on Lens.org (checked 2026-10-06).