Automated Medical Report Generation and Visual Question Answering

The rapid growth of medical imaging data has far outpaced the availability of trained radiologists, significantly increasing their workload. To alleviate this burden, reduce diagnostic errors, and streamline clinical workflows, the need for automated medical diagnostic report generation has become more urgent than ever. However, this task is particularly challenging, as it requires the ability to capture and describe clinically significant fine-grained visual differences in highly similar medical images. Additionally, critical disease-related keywords can easily be overshadowed by the prevalence of similar phrases describing common image content. Moreover, generating comprehensive reports that detail both normal and pathological findings within images adds to the complexity.

In this presentation, I will showcase our latest research on automated medical diagnostic report generation and medical visual question answering, highlighting how we have tackled these challenges. Our work has transitioned from traditional encoder-decoder models to cutting-edge approaches utilizing large language models (LLMs). I will also discuss the current limitations of these methods and propose potential future directions.

Specifically, I will present two methods we developed before the advent of pretrained LLMs, which enhance fine-grained recognition for medical report generation from different angles. The first is a self-boosting framework designed to learn highly correlated image and text features, enabling the model to narrate even finer visual changes in the generated reports. The second method is inspired by the 'multi-expert joint diagnosis' scenario and introduces multiple learnable 'expert' tokens into the transformer architecture, with each expert focusing on distinct image regions. These complementary perspectives are then aggregated to produce a final, more accurate report. In addition to report generation, I will also present our efforts in improving medical visual question answering (VQA).

Following this, I will introduce our recent work on integrating LLMs for medical report generation. I will outline two frameworks we developed: the first employs a frozen LLM for report generation, training only a lightweight visual alignment module to achieve state-of-the-art performance. The second framework goes a step further by integrating a knowledge graph to unlock disease-related knowledge within the LLM, thereby enhancing the clinical relevance of the generated reports. Additionally, I will share our latest investigation into GPT-4V's multimodal capabilities in chest X-ray analysis and discuss the limitations of current evaluation metrics for radiology report generation. To address these limitations, I will introduce our recently developed MRScore framework, which guides LLMs in radiology report evaluation to ensure alignment with human expert analysis.