It is challenging to predict the mechanical properties of modified asphalt binders because of their complex non-linear viscoelastic behavior. This study evaluates and compares the feasibility of using the response surface methodology (RSM) and machine learning (ML) methods to predict the shear strain, accumulated shear strain, non-recoverable creep compliance (Jnr), and percentage of recovery (R) of the base binder, nanosilica (NS)-modified, waste denim fiber (WDF)-modified, and NS/WDF composite asphalt binders. The study conducts an extensive investigation using ML algorithms to accurately predict the multiple stress creep recovery (MSCR) rutting parameters for the base and modified asphalt binders. The RSM statistical analysis revealed that the NS and WDF significantly affect the shear strain, accumulated shear strain, Jnr, and R at different levels of shear stress within the 95 confidence interval. Besides, the RSM-based predictive models have correlation coefficients (R2) greater than 0.8 for all responses, indicating an adequate consistency between the predicted MSCR parameters by the developed models and the parameters from the experimental work. Analysis of the ML models shows that the Extreme Gradient Boosting regression (XGB regression) is among the most accurate models for predicting the shear strain and accumulated strain. Of the evaluated ML models, Decision Tree Regression (DTR) shows the best performance in predicting Jnr and R, with the highest R2 of 0.99 and smallest root mean square error (RMSE) of <1, which indicates its ability to represent the experimental MSCR parameters accurately. Evaluation of the XGB regression and DTR performance shows that the developed ML models outperform the RSM in predicting the MSCR rutting parameters. © 2022 Elsevier Ltd