Video segmentation is one of video image processing application that deployed by video surveillance system. The high computation power must be provided to support real time performance. This paper presents the implementation of VLSI based hardware accelerator design for real time video segmentation system. The algorithm of Sobel edge detection operator is used to develop this hardware accelerator. The NTSC standard definition video is digitized at 720�480 with a video rate of 30 frames per second. To develop hardware accelerator datapath architecture the management of memory access is deployed and architecture based pipeline are made with the potential improvements in acceleration to the read data pixel from memory. In addition, a finite state machine is used to ensure the hardware accelerator controls the sequence of derivative computation, the write and read operations. The hardware accelerator design is implemented on Altera Stratix III DSP development board and enables application of co-processor without requiring new application specific digital signal processor. The implementation result shows a field programmable gate arrays (FPGAs) acting as coprocessor platforms for user defined co-processor, with real time performance at a frame rate of 30 fps with a resolution of 720 � 480. The parallel and pipeline technique are utilized in memory access, resulting more than 70 memory bandwidth reduction. © 2011 IEEE.