Toward perceptually driven image retrieval in mammography: A pilot observer study to assess visual similarity of masses

Development of a fully automated system retrieving visually similar images is a task that could be helpful as the basis of a computer-assisted diagnostic (CADx) tool in mammography. Our study aims at a better understanding of the concept of visual similarity as it pertains to mammographic masses. Such understanding is a necessary step for building effective perceptually-driven image retrieval systems. In our study we deconstruct the concept of visual mass similarity into three components: similarity of size, similarity of shape, and similarity of margin. We present the results of a pilot observer study to determine the importance of each component when human observers assess the overall similarity of two masses. Seven observers of various expertise participated in the study: 1 highly experienced mammographer, 1 expert in visual perception, 3 CAD researchers, and 2 novices. Each observer assessed the similarity between 100 pairs of mammographic regions of interest (ROIs) depicting benign and malignant masses. Visual similarity was assessed in four categories (shape, size, margin, overall) using a web-based interface and a 10-point rating scale. Preliminary analysis of the results suggests the following. First, there is a moderate agreement between observers in similarity assessment for all mentioned categories. Second, all components substantially affect the overall similarity rating, with mass margin having the highest significance and mass size having the lowest significance relatively to the other factors. These findings varied somewhat based on the observer's expertise. Third, some low-level morphological features extracted from the masses can be used to mimic the overall visual similarity ratings and its specific components.