Real-time video processing is the basic requirement for applications such as video surveillance, traffic management and medical imaging. The high computation power is a requirement to support this operation. This requirement could be fulfilled by utilizing the hardware accelerator architecture for computation part. This paper presents the development of edge detection hardware accelerator architecture for real time video processing systems. The algorithm of Sobel edge detection operator is used to develop this hardware accelerator. The NTSC standard definition video is digitized at 720�480 with a video rate of 30 frames per second. To develop hardware accelerator datapath architecture the management of memory access is deployed and architecture based pipeline are made with the potential of improvements in acceleration to read the data pixel from memory. In addition, a finite state machine is used to ensure the hardware accelerator controls the sequence of derivative computation, write and read operations. Initial simulation shows that the hardware accelerator architecture manages to achieve approximately 75 memory bandwidth reduction compare to previous work 5.