This paper presents an optimal design and implementation of hardware system units for tracking moving objects on FPGA. Designs outline dataflow between different frequency domains using pipelining and FIFOs as buffers. The system implements a large amount of operations in order to apply a tracking algorithm called Adaptive Hybrid Difference. Two core units were built to implement the algorithm on FPGA, the adaptive threshold unit and the binary image builder unit. By implementing the proposed design with pipelining and parallelism, the number of clock cycles required fo calculate the adaptive threshold over 30 frames has reduced from 23 � cycles to 4.3 � cycles. Additionally, optimal logic elements utilization has been achieved through the design. © 2011 IEEE.