A stark increase in the vision-based applications in recent years has made the interpretation of imagery data a challenging problem at scale. For accurate visual scene comprehension, the need for accurate segmentation modules is witnessed to rise exponentially. This raises demands analysis of granular pixel information captured in different scenarios (direct/low lightning, reflections, fog/rain). This paper presents two non-trivial issues of existing semantic segmentation techniques which proves to be impeding factors for accurate scene understanding. Firstly, a review of existing semantic segmentation techniques for analysis of image data leading to the identification of a problem is presented. Second, the possible factors triggering the problem are identified which eventually marks the failure of optimal semantic segmentation results. Then, empirical evidence of the presence of the problem through erroneous results produced by testing existing solutions is presented. Finally, a hybrid framework is proposed by visually presenting different integrated modules to tackle the identified problem. The proposed framework has relevance to wide range of applications including facial recognition in online learning setups where image analysis of dynamic images is expected. © 2021 IEEE.