The H.264/AVC video compression standard incorporates new features to enhance video coding. It supports a variety of block sizes and shapes for motion estimation (ME), and gives more flexibility than previous standards. In this paper, we propose a method of determining the block mode to be used for a particular video sequence to give the optimum performance for the encoder. To reduce the number of computations involved, the block mode decision phase is done on the first few frames only and fixed for subsequent frames. The decision is based on motion vectors (MVs) found from previous frames. From our results, the block computation time can be reduced as much as 47.72 for a video sequence with low motion activity. On average, most of the test video sequences suffer degradation in PSNR value of less than 0.5dB. © 2006 IEEE.