Vishwesh Ramanathan

2025

1. 

   ModalTune: Fine-Tuning Slide-Level Foundation Models with Multi-Modal Information for Multi-task Learning in Digital Pathology

   Vishwesh Ramanathan^*, Tony Xu^*, Pushpak Pati, and 3 more authors

   International Conference on Computer Vision (ICCV), 2025

   Abs DOI Code

   Prediction tasks in digital pathology are challenging due to the massive size of whole-slide images (WSIs) and the weak nature of training signals. Advances in computing, data availability, and self-supervised learning (SSL) have paved the way for slide-level foundation models (SLFMs) that can improve prediction tasks in low-data regimes. However, working with these models is challenging, with issues such as catastrophic forgetting during fine-tuning and underutilization of shared information between tasks and modalities. To overcome these two challenges, we propose ModalTune, a novel fine-tuning framework which introduces the Modal Adapter to integrate new modalities without modifying SLFM weights. Additionally, we use large-language models (LLMs) to encode labels as text, capturing semantic relationships and enhancing generalization across multiple tasks and cancer types in a single training recipe. ModalTune achieves state-of-the-art (SOTA) results against both uni-modal and multi-modal models across four cancer types, jointly improving survival and cancer subtype prediction while remaining competitive in pan-cancer settings. Additionally, we show ModalTune is highly generalizable to two out-of-distribution (OOD) datasets. To our knowledge, this is the first unified fine-tuning framework for multi-modal, multi-task, and pan-cancer modeling in digital pathology.

2024

1. 

   Ensemble of Prior-guided Expert Graph Models for Survival Prediction in Digital Pathology

   Vishwesh Ramanathan^*, Pushpak Pati^*, Matthew McNeil, and 1 more author

   In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2024

   Abs DOI Code

   Survival prediction in pathology is a dynamic research field focused on identifying predictive biomarkers to enhance cancer survival models, providing valuable guidance for clinicians in treatment decisions. Graph-based methods, especially Graph Neural Networks (GNNs) leveraging rich interactions among different biological entities, have recently successfully predicted survival. However, the inherent heterogeneity among the entities within tissue slides significantly challenges the learning of GNNs. GNNs, operating with the homophily assumption, diffuse the intricate interactions among heterogeneous tissue entities in a tissue microenvironment. Further, the convoluted downstream task relevant information is not effectively exploited by graph-based methods when working with large slide-graphs. We propose a novel prior-guided, edge-attributed tissue-graph construction to address these challenges, followed by an ensemble of expert graph-attention survival models. Our method exploits diverse prognostic factors within numerous targeted tissue subgraphs of heterogeneous large slide-graphs. Our method achieves state-of-the-art results on four cancer types, improving overall survival prediction by 4.33% compared to the competing methods

2. 

   Detecting Noisy Labels with Repeated Cross-Validations

   Jianan Chen^*, Vishwesh Ramanathan^*, Tony Xu, and 1 more author

   In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2024

   Abs DOI Code

   Machine learning models experience deteriorated performance when trained in the presence of noisy labels. This is particularly problematic for medical tasks, such as survival prediction, which typically face high label noise complexity with few clear-cut solutions. Inspired by the large fluctuations across folds in the cross-validation performance of survival analyses, we design Monte-Carlo experiments to show that such fluctuation could be caused by label noise. We propose two novel and straightforward label noise detection algorithms that effectively identify noisy examples by pinpointing the samples that more frequently contribute to inferior cross-validation results. We first introduce Repeated Cross-Validation (ReCoV), a parameter-free label noise detection algorithm that is robust to model choice. We further develop fastReCoV, a less robust but more tractable and efficient variant of ReCoV suitable for deep learning applications. Through extensive experiments, we show that ReCoV and fastReCoV achieve state-of-the-art label noise detection performance in a wide range of modalities, models and tasks, including survival analysis, which has yet to be addressed in the literature

2023

1. 

   Ink removal in whole slide images using hallucinated data

   Vishwesh Ramanathan, Wenchao Han, Dina Bassiouny, and 2 more authors

   In Medical Imaging 2023: Digital and Computational Pathology, 2023

   Abs DOI PDF Code

   Pathologists regularly use ink markings on histopathology slides to highlight specific areas of interest or orientation, making it an integral part of the workflow. Unfortunately, digitization of these ink-annotated slides hinders any computer-aided analyses, particularly deep learning algorithms, which require clean data free from artifacts. We propose a methodology that can identify and remove the ink markings for the purpose of computational analyses. We propose a two-stage network with a binary classifier for ink filtering and Pix2Pix for ink removal. We trained our network by artificially generating pseudo ink markings using only clean slides, requiring no manual annotation or curation of data. Furthermore, we demonstrate our algorithm’s efficacy over an independent dataset of H&E stained breast carcinoma slides scanned before and after the removal of pen markings. Our quantitative analysis shows promising results, achieving 98.7% accuracy for the binary classifier. For Pix2Pix, we observed a 65.6% increase in structure similarity index, a 21.3% increase in peak signal-to-noise ratio, and a 30% increase in visual information fidelity. As only clean slides are required for training, the pipeline can be adapted to multiple colors of ink markings or new domains, making it easy to deploy over different sets of histopathology slides

2. 

   Self Supervised Multi-view Graph Representation Learning in Digital Pathology

   Vishwesh Ramanathan, and Anne L Martel

   In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI-W) - Graphs in Biomedical Image Analysis, 2023

   Best Paper Award Abs DOI Code

   • Best Paper Award at the MICCAI Workshop on Graphs in Biomedical Image Analysis 2023, held in conjunction with MICCAI 2023 in Vancouver, Canada

   Graph Neural Networks (GNNs) hold great promise for solving many challenges in digital pathology by leveraging the rich relationships between cells and tissues in histology images. However, the shortage of annotated data in digital pathology presents a significant challenge for training GNNs. To address this, self-supervision can be used to enable models to learn from data by capturing rich structures and relationships without requiring annotations. Inspired by pathologists who take multiple views of a histology slide under a microscope for exhaustive analysis, we propose a novel methodology for graph representation learning using self-supervision. Our methodology leverages multiple graph views constructed from a given histology image to capture diverse information. We maximize mutual information across nodes and graph representations of different graph views, resulting in a comprehensive graph representation. We showcase the efficacy of our methodology on the BRACS dataset where our algorithm generates superior representations compared to other self-supervised graph representation learning algorithms and comes close to pathologists and supervised learning algorithms