Bridging Vision and Language in Medical Imaging: Towards Robust Automated Report Generation from PET/CT Scans

Abstract

Vision-Language Models (VLMs) have achieved significant success in multimodal reasoning in general domains, yet their application to medical imaging remains limited, especially in specialized data domains such as PET/CT. In this study, we introduce Vietnamese Positron Emission Tomography - Vision-Language Model (ViPET-VLM), a novel pipeline specifically designed for medical report generation and visual question answering tasks on PET/CT data. ViPET-VLM integrates a fusion module to combine morphological information from CT with functional signals from PET, thereby forming a richer multimodal representation. To enhance clinical reliability, we propose a regularization mechanism with specialized loss functions that not only ensure diagnostic accuracy but also guide the model's attention to critical regions of interest in the images. ViPET-VLM was evaluated on a comprehensive, expert-validated PET/CT dataset and demonstrated marked improvements over current state-of-the-art methods in both report generation and medical question answering. The model shows potential for enhancing the accuracy and clinical applicability of VLMs for medical imaging.