Wellbore trajectory design is a nonlinear and constrained mathematical optimization problem that is used to build a cost-efficient, safe, and easily reachable trajectory. True measured depth (TMD), torque, and strain energy are used as objective functions to evaluate the wellbore trajectory design in this work. The minimum values of these objective functions enable a trajectory to be drilled with minimum drilling cost and maximum safety. A lot of modifications to the original metaheuristic methods were made during previous applications, which mostly improve the exploration capability of original algorithms keeping exploitation capability unaddressed. To address this issue a new hybridization of cellular automata (CA) technique with grey wolf optimization (GWO) and particle swarm optimization (PSO) algorithms is proposed in this paper which solves these three optimization objectives of drilling through 17 tuning variables. The improvements of the original PSO Algorithm are proposed by updating its exploitation phase with the incorporation of the GWO algorithm and the exploration phase by using a cellular automaton. During the optimization, the operational constraints of a wellbore such as true vertical depth and casings along with the bounds of tuning variables were utilized. Better performances were observed in cases of Pareto optimal front, search capabilities, and diversity of solutions by comparing the proposed method with other standard methods like MOCPSO, MOGWO, and MOPSO. Several parametric tests (IGD, SP, MS) were done to investigate the effect of proposed hybridization. The mean value of IGD was 0.0208 by the proposed method, which is 46.8 better than MOCPSO, 49.78 than MOPSO, and 60.80 better than the MOGWO. The proposed optimization method also had the minimum spacing metric and maximum spread. © 2020 Elsevier B.V.