Oily wastewater from the oil and gas industry negatively affects the environment. Oily wastewater typically exists in the form of an oil-in-water emulsion. Conventional methods to treat oily wastewater have low separation efficiency and long separation time and use large equipment. Therefore, a simple but effective method must be developed to separate oil-in-water emulsions with high separation efficiency and short separation times. Magnetite-reduced graphene oxide (M-RGO) nanocomposites were used as a demulsifier in this work. Magnetite nanoparticles (Fe3O4) were coated on reduced graphene oxide (rGO) nanosheets via an in situ chemical synthesis method. The synthesized M-RGO nanocomposites are environmentally friendly and can be recovered after demulsification by an external magnetic field. M-RGO characterization was performed using X-ray diffraction, Fourier transform infrared spectroscopy, X-ray photoelectron spectroscopy, field emission scanning microscopy, Raman spectroscopy, and vibrating sample magnetometry. Demulsification performance was evaluated in terms of M-RGO dosage, effects of pH, and brine concentration. The demulsification capability of M-RGO was determined based on the residual oil content of the emulsion, which was measured with a UV-vis spectrometer. The response surface method was used to determine the optimum conditions of the input variables. The optimum demulsification efficiency achieved at pH 4 and M-RGO dosage of 29 g L-1 was approximately 96. This finding demonstrates that M-RGO nanocomposites are potential magnetic demulsifiers for oily wastewater that contains oil-in-water emulsions. Also, the recyclability of this nanocomposite has been tested and the results shown that it is a good recyclable demulsifier. © 2019 The Royal Society of Chemistry.