In this paper, we present a motion and oriented gradient based approach for behavior representation in a sparse crowd scene. We present a method that builds upon the previous ideas such as local space-time features and space-time pyramid. The method is aimed at exploiting the activity coherently and effectively by extracting low-level features at spatial-temporal interest point's neighborhood; a histogram of optical flow and a histogram of the oriented gradient. Relying on the measurable attributes of objects description and motion characteristics, specific behavior such as crossing, walking, merging and splitting can be detected more accurately. We present a new method for crowd behavior classification based on space-time features. An experimental evaluation is conducted on publicly available crowd analytic datasets. The result indicates that radial basis function support vector machine shows a good accuracy, precision and recall in classifying human behavior when compared to a nearest neighbor classifier. © 2016 IEEE.