The ultimate goal of computer vision research is to understand a scene semantically from an image or a video. Real-time object detection received significant attention over the past few years. Many challenges remain, especially in the focus of extraction of localized object information for scene representation. In order to have an accurate, intelligent and fast real-time object detection, the implementation of accurate localized information in the machine is inevitable. This research will focus on developing the object localization extractor that can extract the localized object information from the scene for further scene prediction and inference. In particular, (i) our localized extractor can encode significantly high-level features information; (ii) this rich localized information will be used for scene representation and understanding. © 2018 IEEE.