Medical images are increasing at an alarming rate. This increasing number of images affects the interpreting capacity of radiologists. In order to reduce the burden of radiologists, automatic categorization of medical images based on modality is the need of the hour. Because image modality is an important and fundamental image characteristic. The important factor in the automatic medical image categorization based on modality are the features used for categorization purpose, because nice treatment on these subtleties can lead to good results. Many descriptors have been proposed in the literature for medical image categorization. It is unclear which descriptor encodes the content information efficiently. The descriptors that are calculated from these medical images should be descriptive, distinctive and robust to various transformations. The stability of these descriptors are evaluated under various transformations and are then analyzed for their discriminatory ability for the task of classification. In this study the criteria of transformations, repeatability, matching and classification accuracy on the basis of precision recall is used to evaluate the performance of these descriptors. The experimental results illustrates that among global descriptors local features patches histogram and among local descriptors SIFT encodes the content information quite efficiently. © 2016 Penerbit UTM Press. All rights reserved.