Eddie-Transformer: Enriched Disease Embedding Transformer For X-Ray Report Generation

Automatic medical report generation is an emerging field that aims to generate medical reports based on medical images. The report writing process can be tedious for senior radiologists and challenging for junior ones. It is therefore of great importance to expedite the process. In this work, we propose an EnricheD DIsease Embedding based Transformer (Eddie-Transformer) model, which jointly performs disease detection and medical report generation. This is done by decoupling the latent visual features into semantic disease embeddings and disease states via our state-aware mechanism. It is followed by entangling the learned diseases and their states, thus enabling explicit and precise disease representations. Finally, the enriched disease representations go through the Transformer model to generate high-quality medical reports. Our approach demonstrates promising results on the widely-used Open-I benchmark and a COVID-19 dataset.