The utilization of video surveillance systems is becoming common and is expected to more widespread as societies become more complex and the population continues to grow. The implementation of these video surveillance systems has provided valuable information and assistance in monitoring large areas. These systems typically use human operators to determine human behavior and to manually track people or objects of interest over an array of cameras. With the application of computers with video surveillance, real time surveillance of large public areas, people and their activities has been made possible for monitoring and security. The main contribution of this paper is to show the effect of color space on tracking methods in video surveillance. Results from evaluations on different tracking methods have indicated that YCbCr and HSV color spaces have better tracking ability compared to grayscale and RGB color spaces. In addition, the results from evaluations have also indicated that data from selected layers in some color spaces can be used for the purpose of tracking namely the Cb and Cr layers from the YCbCr color space and the H layer from the HSV color space.