Re-Ranking Strategies In Cross-Modality Microscopy Retrieval

For many cancer diagnoses tissue samples stained by hematoxylin and eosin are inspected in a brightfield (BF) microscope. It is becoming increasingly common to additionally inspect second harmonic generation (SHG) images alongside their BF counterparts as such multimodal image pairs carry complimentary information about the tissue. To match BF and SHG images captured in different microscopes, Breznik et al. (2022) recently proposed a method for image retrieval to match unaligned multimodal image pairs of BF and SHG: it creates a bag-of-words (BoW) based on SURF features and image representations called CoMIRs; and finally a re-ranking step to refine the retrieval among the best-ranking matches. Here, we evaluate three different re-ranking strategies (one relying on global features, two relying on local features) for cross-modality image retrieval of SHG and BF images and evaluate them on a publicly available dataset.