The increasing number of medical images of various imaging modalities is challenging the accuracy and efficiency of radiologists. In order to retrieve the images from medical databases, radiologists will confine their search to the image modality. In this paper, we present an improved image feature to represent medical images for image modality classification. The proposed image descriptor is an ensemble descriptor that combines the Harris Corner encoded by the SIFT algorithm fused with Local Binary Pattern. Furthermore, we propose an ensemble classifier with surrogate splits to be used in medical image modality classification in order to improve the performance. It is shown that the proposed ensemble classifier with surrogate splits and ensemble descriptor encoded with bag-of-visual-words representation outperforms other conventional approaches applied in medical image modality classification. © 2015 IEEE.