The nanofluids are considered as an effective medium for thermal transport in various applications due to improved properties. Limited experimental studies are accomplished on the convective heat transfer of nanofluids. Natural convective heat transfer behavior in nanofluids is experimentally studied in a vertical rectangular enclosure (aspect ratio = 4) with one heating and one cooling wall. The fluid under investigation is a novel and highly stable functionalized alumina-thermal oil-based nanofluid. The investigations are carried out for different concentrations of nanofluids ranging from 0 to 3 wt. The effectiveness of the natural convection heat transfer process is mainly dependent on the properties of cooling media. The measured thermophysical properties of nanofluids are used for the estimation of heat transfer characteristics with a Prandtl number range of 228�592. The heat transfer coefficient and Nusselt number are obtained at different nanoparticle concentrations. A heat flux is applied on the hot wall with a range of 1593.75�3150 W/m2. An improvement in the cooling performance of nanofluids is observed. The high nanoparticle concentrations of nanofluids exhibit higher heat transfer coefficient as compared to the pure thermal oil. A correlation is developed for the Nusselt number in terms of Rayleigh number (4.43 � 105�2.59 � 106), nanoparticle concentration and effective thermophysical properties. © 2017 Elsevier Ltd