In this paper, we propose a vehicle detection and tracking algorithm. The detection is done using the median filtering and blob extraction. Median filtering is used for background extraction which is later subtracted from the motion frames for object detection. Morphological operators are employed for blob extraction. Hence, object detection is achieved using median filtering and morphological closing operation. Kalman filtering is used for object tracking which uses location of blobs. One of the advantages of this system is that each vehicle in the frame is classified into different color boxes. We present preliminary research results that will finally lead to the identification of the tracked vehicle.