Stanford and Google Researchers Develop a Foundation Model for Chest X-Ray Interpretation

Chexagent Is an Instruction-Tuned Foundation Model That Can Analyze and Summarize Chest X-Rays Using Natural Language Instructions.

Title of the paper: CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation

Authors: Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, Emily B. Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Akshay S. Chaudhari, and Curtis Langlotz.

What problem this paper solves?

Chest X-rays (CXRs) are the most frequently performed imaging test in clinical practice. However, interpreting CXRs is challenging and requires expert knowledge and experience. Moreover, there is a lack of large-scale vision-language datasets and models in the medical image domain, as well as a lack of evaluation frameworks for benchmarking the abilities of foundation models (FMs) on CXR interpretation. This paper addresses these problems by introducing CheXinstruct, a large-scale instruction-tuning dataset for CXR interpretation, CheXagent, an instruction-tuned FM for CXR analysis and summarization, and CheXbench, a novel benchmark for evaluating FMs on CXR interpretation tasks.

What approach this paper utilizes?

The paper uses an instruction-tuning approach, which is a form of self-supervised learning that leverages natural language instructions as weak supervision signals for training FMs. The paper first curates CheXinstruct, a dataset of 28 publicly-available CXR datasets, with over 1.2 million CXR images and 1.4 million natural language instructions. The paper then presents CheXagent, an FM that consists of a clinical large language model (LLM) for parsing radiology reports, a vision encoder for representing CXR images, and a network to bridge the vision and language modalities. CheXagent is trained on CheXinstruct using an instruction-tuning objective, which aims to generate CXR summaries that match the given instructions. The paper also introduces CheXbench, a benchmark that consists of 8 clinically-relevant CXR interpretation tasks, such as disease detection, localization, severity assessment, and differential diagnosis.

What are the impacts of this approach on AI research?

The paper makes several contributions to the AI research community, such as:

• • It provides a large-scale vision-language dataset and a novel instruction-tuning objective for CXR interpretation, which can facilitate the development and evaluation of FMs in the medical image domain.
  • It demonstrates the effectiveness and versatility of instruction-tuning for CXR analysis and summarization, which can enable FMs to perform various tasks using natural language instructions.
  • It proposes a novel FM architecture that combines a clinical LLM, a vision encoder, and a vision-language network, which can capture the complexities and nuances of medical data and language.
  • It introduces a comprehensive benchmark for CXR interpretation, which can systematically evaluate the abilities and limitations of FMs on different aspects of CXR interpretation.
  • It shows that CheXagent outperforms previously-developed general- and medical-domain FMs on CheXbench tasks, and receives positive feedback from expert radiologists.

Summary of the research

The paper presents CheXagent, an instruction-tuned FM for CXR interpretation, which can analyze and summarize CXRs using natural language instructions. The paper also introduces CheXinstruct, a large-scale instruction-tuning dataset for CXR interpretation, and CheXbench, a novel benchmark for evaluating FMs on CXR interpretation tasks. The paper demonstrates that CheXagent can achieve high-quality and high-diversity results, while being fast and efficient. The paper also highlights the potential and challenges of using FMs for CXR interpretation, and calls for more research and regulation on the responsible use of FMs in the medical image domain.

Also Read: Chinese Researchers Develop a Novel AI Model to Predict Abnormal Brain Connections in Alzheimer’s Disease

Also Read: The Rise of InstantID: A New AI Image Generation Method That Could Revolutionize Deepfakes

Also Read: Hugging Face and Google Cloud Join Forces to Boost Open AI Development