Modality corresponding to medical images is a vital filter in medical image retrieval systems, as radiologists or physicians are interested in only one of radiology images e.g CT scan, MRI, X-ray. Various handcrafted feature schemes have been proposed for medical image modality classification. On the other hand not enough attempts have been made for deep learned feature extraction. A comparative evaluation of both handcrafted and deep learned features for medical image modality classification is presented in this paper. The experiments are performed on IMAGECLEF 2012 data. After carrying out the experiments it is shown that the handcrafted features outperforms the deep learned features and shows the potential of handcrafted feature extraction models in the medical image field. © 2016 IEEE.